Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Stuff The Internet Says On Scalability For October 31st, 2014

Hey, it's HighScalability time:


A CT scanner without its clothes on. Sexy.
  • 255Tbps: all of the internet’s traffic on a single fiber; 864 million: daily Facebook users
  • Quotable Quotes:
    • @chr1sa: "No dominant platform-level software has emerged in the last 10 years in closed-source, proprietary form”
    • @joegravett: Homophobes boycotting Apple because of Tim Cook's brave announcement are going to lose it when they hear about Turing.
    • @CloudOfCaroline: #RICON MySQL revolutionized Scale-out. Why? Because it couldn't Scale-up. Turned a flaw into a benefit - @martenmickos
    • chris dixon: We asked for flying cars and all we got was the entire planet communicating instantly via pocket supercomputers
    • @nitsanw: "In the majority of cases, performance will be programmer bound" - Barker's Law
    • @postwait: @coda @antirez the only thing here worth repeating: we should instead be working with entire distributions (instead of mean or q(0.99))
    • Steve Johnson: inventions didn't come about in a flash of light — the veritable Eureka! moment — but were rather the result of years' worth of innovations happening across vast networks of creative minds.

  • On how Google is its own VC. cromwellian: The ads division is mostly firewalled off from the daily concerns of people developing products at Google. They supply cash to the treasury, people think up cool ideas and try to implement them. It works just like startups, where you don't always know what your business model is going to be. Gmail started as a 20% project, not as a grand plan to create an ad channel. Lots of projects and products at Google have no business model, no revenue model, the company does throw money at projects and "figure it out later" how it'll make money. People like their apps more than the web. Mobile ads are making a lot of money.  

  • Hey mobile, what's for dinner? "The world," says chef Benedict Evans, who has prepared for your pleasure a fine gourmet tasting menu: Presentation: mobile is eating the world. Smart phones are now as powerful as Thor and Hercules combined. Soon everyone will have a smart phone. And when tech is fully adopted, it disappears. 

  • How much bigger is Amazon’s cloud vs. Microsoft and Google?: Amazon’s cloud revenue at more than $4.7 billion this year. TBR pegs Microsoft’s public cloud IaaS revenue at $156 million and Google’s at $66 million. If those estimates are correct than Amazon’s cloud revenue is 30 times bigger than Microsoft’s.

  • Great discussion on the Accidental Tech Podcast (at about 25 minutes in) on how the the time of open APIs has ended. People who made Twitter clients weren't competing with Twitter, they were helping Twitter become who they are today. For Apple, developers add value to their hardware and since Apple makes money off the hardware this is good for Apple, because without apps Apple hardware is way less valuable. With their new developer focus Twitter and developer interests are still not aligned as Twitter is still worried about clients competing with them. Twitter doesn't want to become an infrastructure company because there's no money in it. In the not so distant past services were expected to have an open API, in essence services were acting as free infrastructure, just hoping they would become popular enough that those dependent on the service could be monetized. New services these days generally don't have full open APIs because it's hard to justify as a business case. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Azure: Announcing New Real-time Data Streaming and Data Factory Services

ScottGu's Blog - Scott Guthrie - Fri, 10/31/2014 - 07:39

The last three weeks have been busy ones for Azure.  Two weeks ago we announced a partnership with Docker to enable great container-based development experiences on Linux, Windows Server and Microsoft Azure.

Last week we held our Cloud Day event and announced our new G-Series of Virtual Machines as well as Premium Storage offering.  The G-Series VMs provide the largest VM sizes available in the public cloud today (nearly 2x more memory than the largest AWS offering, and 4x more memory than the largest Google offering).  The new Premium Storage offering (which will work with both our D-series and G-series of VMs) will support up to 32TB of storage per VM, >50,000 IOPS of disk IO per VM, and enable sub-1ms read latency.  Combined they provide an enormous amount of power that enables you to run even bigger and better solutions in the cloud.

Earlier this week, we officially opened our new Azure Australia regions – which are our 18th and 19th Azure regions open for business around the world.  Then at TechEd Europe we announced another round of new features – including the launch of the new Azure MarketPlace, a bunch of great network improvements, our new Batch computing service, general availability of our Azure Automation service and more.

Today, I’m excited to blog about even more new services we have released this week in the Azure Data space.  These include:

  • Event Hubs: is a scalable service for ingesting and storing data from websites, client apps, and IoT sensors.
  • Stream Analytics: is a cost-effective event processing engine that helps uncover real-time insights from event streams.
  • Data Factory: enables better information production by orchestrating and managing diverse data and data movement.

Azure Event Hub is now available in general availability, and the new Azure Stream Analytics and Data Factory services are now in public preview. Event Hubs: Log Millions of events per second in near real time

The Azure Event Hub service is a highly scalable telemetry ingestion service that can log millions of events per second in near real time.  You can use the Event Hub service to collect data/events from any IoT device, from any app (web, mobile, or a backend service), or via feeds like social networks.  We are using it internally within Microsoft to monitor some of our largest online systems.

Once you collect events with Event Hub you can then analyze the data using any real-time analytics system (like Apache Storm or our new Azure Stream Analytics service) and store/transform it into any data storage system (including HDInsight and Hadoop based solutions).

Event Hub is delivered as a managed service on Azure (meaning we run, scale and patch it for you and provide an enterprise SLA).  It delivers:

  • Ability to log millions of events per second in near real time
  • Elastic scaling support with the ability to scale-up/down with no interruption
  • Support for multiple protocols including support for HTTP and AMQP based events
  • Flexible authorization and throttling device policies
  • Time-based event buffering with event order preservation

The pricing model for Event Hubs is very flexible – for just $11/month you can provision a basic Event Hub with guaranteed performance capacity to capture 1 MB/sec of events sent to your Event Hub.  You can then provision as many additional capacity units as you need if your event traffic goes higher. 

Getting Started with Capturing Events

You can create a new Event Hub using the Azure Portal or via the command-line.  Choose New->App Service->Service Bus->Event Hub in the portal to do so:

image

Once created, events can be sent to an Event Hub with either a strongly-typed API (e.g. .NET or Java client library) or by just sending a raw HTTP or AMQP message to the service.  Below is a simple example of how easy it is to log an IoT event to an Event Hub using just a standard HTTP post request.  Notice the Authorization header in the HTTP post – you can use this to optionally enable flexible authentication/authorization for your devices:

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1<?xml:namespace prefix = "o" />

Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=tYu8qdH563Pc96Lky0SFs5PhbGnljF7mLYQwCZmk9M0%3d&se=1403736877&skn=RootManageSharedAccessKey

ContentType: application/atom+xml;type=entry;charset=utf-8

Host: your-namespace.servicebus.windows.net

Content-Length: 42

Expect: 100-continue

 

{ "DeviceId":"dev-01", "Temperature":"37.0" }

Your Event Hub can collect up to millions of messages per second like this, each storing whatever data schema you want within them, and the Event Hubs service will store them in-order for you to later read/consume.

Downstream Event Processing

Once you collect events, you no doubt want to do something with them.  Event Hubs includes an intelligent processing agent that allows for automatic partition management and load distribution across readers.  You can implement any logic you want within readers, and the data sent to the readers is delivered in the order it was sent to the Event Hub.

In addition to supporting the ability for you to write custom Event Readers, we also have two easy ways to work with pre-built stream processing systems: including our new Azure Stream Analytics Service and Apache Storm.  Our new Azure Stream Analytics service supports doing stream processing directly from Event Hubs, and Microsoft has created an Event Hubs Storm Spout for use with Apache Storm clusters.

The below diagram helps express some of the many rich ways you can use Event Hubs to collect and then hand-off events/data for processing:

image

Event Hubs provides a super flexible and cost effective building-block that you can use to collect and process any events or data you can stream to the cloud.  It is very cost effective, and provides the scalability you need to meet any needs.

Learning More about Event Hubs

For more information about Azure Event Hubs, please review the following resources:

Stream Analytics: Distributed Stream Processing Service for Azure

I’m excited to announce the preview our new Azure Stream Analytics service – a fully managed real-time distributed stream computation service that provides low latency, scalable processing of streaming data in the cloud with an enterprise grade SLA. The new Azure Stream Analytics service easily scales from small projects with just a few KB/sec of throughput to a gigabyte/sec or more of streamed data messages/events.  

Our Stream Analytics pricing model enable you to run low throughput streaming workloads continuously at low cost, and enables you to only have to scale up as your business needs increase.  We do this while maintaining built in guarantees of event delivery, and state management for fast recovery which enables mission critical business continuity.

Dramatically Simpler Developer Experience for Stream Processing Data

Stream Analytics supports a SQL-like language that dramatically lowers the bar of the developer expertise required to create a scalable stream processing solution. A developer can simply write a few lines of SQL to do common operations including basic filtering, temporal analysis operations, joining multiple live streams of data with other static data sources, and detecting stream patterns (or lack thereof).

This dramatically reduces the complexity and time it takes to develop, maintain and apply time-sensitive computations on real-time streams of data. Most other streaming solutions available today require you to write complex custom code, but with Azure Stream Analytics you can write simple, declarative and familiar SQL.

Fully Managed Service that is Easy to Setup

With Stream Analytics you can dramatically accelerate how quickly you can derive valuable real time insights and analytics on data from devices, sensors, infrastructure, or applications. With a few clicks in the Azure Portal, you can create a streaming pipeline, configure its inputs and outputs, and provide SQL-like queries to describe the desired stream transformations/analysis you wish to do on the data. Once running, you are able to monitor the scale/speed of your overall streaming pipeline and make adjustments to achieve the desired throughput and latency.

You can create a new Stream Analytics Job in the Azure Portal, by choosing New->Data Services->Stream Analytics:

image

Setup Streaming Data Input

Once created, your first step will be to add a Streaming Data Input.  This allows you to indicate where the data you want to perform stream processing on is coming from.  From within the portal you can choose Inputs->Add An Input to launch a wizard that enables you to specify this:

image

We can use the Azure Event Hub Service to deliver us a stream of data to perform processing on. If you already have an Event Hub created, you can choose it from a list populated in the wizard above.  You will also be asked to specify the format that is being used to serialize incoming event in the Event Hub (e.g. JSON, CSV or Avro formats).

Setup Output Location

The next step to developing our Stream Analytics job is to add a Streaming Output Location.  This will configure where we want the output results of our stream processing pipeline to go.  We can choose to easily output the results to Blob Storage, another Event Hub, or a SQL Database:

image

Note that being able to use another Event Hub as a target provides a powerful way to connect multiple streams into an overall pipeline with multiple steps.

Write Streaming Queries

Now that we have our input and output sources configured, we can now write SQL queries to transform, aggregate and/or correlate the incoming input (or set of inputs in the event of multiple input sources) and output them to our output target.  We can do this within the portal by selecting the QUERY tab at the top.

image

There are a number of interesting queries you can write to processing the incoming stream of data.  For example, in the previous Event Hub section of this blog post I showed how you can use an HTTP POST command to submit JSON based temperature data from an IoT device to an Event Hub with data in JSON format like so:

{ "DeviceId":"dev-01", "Temperature":"37.0" }

When multiple devices are streaming events simultaneously into our Event Hub like this, it would feed into our Stream Analytics job as a stream of continuous data events that look like the sequence below:

Wouldn’t it be interesting to be able to analyze this data using a time-window perspective instead?  For example, it would be useful to calculate in real-time what the average temperature of each device was in the last 5 seconds of multiple readings.

With the Stream Analytics Service we can now dynamically calculate this over our incoming live stream of data just by writing a SQL query like so:

SELECT DateAdd(second,-5,System.TimeStamp) as WinStartTime, system.TimeStamp as WinEndTime, DeviceId, Avg(Temperature) as AvgTemperature, Count(*) as EventCount 
    FROM input
    GROUP BY TumblingWindow(second, 5), DeviceId

Running this query in our Stream Analytics job will aggregate/transform our incoming stream of data events and output data like below into the output source we configured for our job (e,g, a blog storage file or a SQL Database):

The great thing about this approach is that the data is being aggregated/transformed in real time as events are being streamed to us, and it scales to handle literally gigabytes of data event streamed per second.

Scaling your Stream Analytics Job

Once defined, you can easily monitor the activity of your Stream Analytics Jobs in the Azure Portal:

image

You can use the SCALE tab to dynamically increase or decrease scale capacity for your stream processing – allowing you to pay only for the compute capacity you need, and enabling you to handle jobs with gigabytes/sec of streamed data. 

Learning More about Stream Analytics Service

For more information about Stream Analytics, please review the following resources:

Data Factory: Fully managed service to build and manage information production pipelines

Organizations are increasingly looking to fully leverage all of the data available to their business.  As they do so, the data processing landscape is becoming more diverse than ever before – data is being processed across geographic locations, on-premises and cloud, across a wide variety of data types and sources (SQL, NoSQL, Hadoop, etc), and the volume of data needing to be processed is increasing exponentially. Developers today are often left writing large amounts of custom logic to deliver an information production system that can manage and co-ordinate all of this data and processing work.

To help make this process simpler, I’m excited to announce the preview of our new Azure Data Factory service – a fully managed service that makes it easy to compose data storage, processing, and data movement services into streamlined, scalable & reliable data production pipelines. Once a pipeline is deployed, Data Factory enables easy monitoring and management of it, greatly reducing operational costs. 

Easy to Get Started

The Azure Data Factory is a fully managed service. Getting started with Data Factory is simple. With a few clicks in the Azure preview portal, or via our command line operations, a developer can create a new data factory and link it to data and processing resources.  From the new Azure Marketplace in the Azure Preview Portal, choose Data + Analytics –> Data Factory to create a new instance in Azure:

image

Orchestrating Information Production Pipelines across multiple data sources

Data Factory makes it easy to coordinate and manage data sources from a variety of locations – including ones both in the cloud and on-premises.  Support for working with data on-premises inside SQL Server, as well as Azure Blob, Tables, HDInsight Hadoop systems and SQL Databases is included in this week’s preview release. 

Access to on-premises data is supported through a data management gateway that allows for easy configuration and management of secure connections to your on-premises SQL Servers.  Data Factory balances the scale & agility provided by the cloud, Hadoop and non-relational platforms, with the management & monitoring that enterprise systems require to enable information production in a hybrid environment.

Custom Data Processing Activities using Hive, Pig and C#

This week’s preview enables data processing using Hive, Pig and custom C# code activities.  Data Factory activities can be used to clean data, anonymize/mask critical data fields, and transform the data in a wide variety of complex ways.

The Hive and Pig activities can be run on an HDInsight cluster you create, or alternatively you can allow Data Factory to fully manage the Hadoop cluster lifecycle on your behalf.  Simply author your activities, combine them into a pipeline, set an execution schedule and you’re done – no manual Hadoop cluster setup or management required. 

Built-in Information Production Monitoring and Dashboarding

Data Factory also offers an up-to-the moment monitoring dashboard, which means you can deploy your data pipelines and immediately begin to view them as part of your monitoring dashboard.  Once you have created and deployed pipelines to your Data Factory you can quickly assess end-to-end data pipeline health, pinpoint issues, and take corrective action as needed.

Within the Azure Preview Portal, you get a visual layout of all of your pipelines and data inputs and outputs. You can see all the relationships and dependencies of your data pipelines across all of your sources so you always know where data is coming from and where it is going at a glance. We also provide you with a historical accounting of job execution, data production status, and system health in a single monitoring dashboard:

image

Learning More about Stream Analytics Service

For more information about Data Factory, please review the following resources:

Other Great Data Improvements

Today’s releases make it even easier for customers to stream, process and manage the movement of data in the cloud.  Over the last few months we’ve released a bunch of other great data updates as well that make Azure a great platform to perform any data needs.  Since August: 

We released a major update of our SQL Database service, which is a relational database as a service offering.  The new SQL DB editions (Basic/Standard/Premium ) support a 99.99% SLA, larger database sizes, dedicated performance guarantees, point-in-time recovery, new auditing features, and the ability to easily setup active geo-DR support. 

We released a preview of our new DocumentDB service, which is a fully-managed, highly-scalable, NoSQL Document Database service that supports saving and querying JSON based data.  It enables you to linearly scale your document store and scale to any application size.  Microsoft MSN portal recently was rewritten to use it – and stores more than 20TB of data within it.

We released our new Redis Cache service, which is a secure/dedicated Redis cache offering, managed as a service by Microsoft.  Redis is a popular open-source solution that enables high-performance data types, and our Redis Cache service enables you to standup an in-memory cache that can make the performance of any application much faster.

We released major updates to our HDInsight Hadoop service, which is a 100% Apache Hadoop-based service in the cloud. We have also added built-in support for using two popular frameworks in the Hadoop ecosystem: Apache HBase and Apache Storm.

We released a preview of our new Search-As-A-Service offering, which provides a managed search offering based on ElasticSearch that you can easily integrate into any Web or Mobile Application.  It enables you to build search experiences over any data your application uses (including data in SQLDB, DocDB, Hadoop and more).

And we have released a preview of our Machine Learning service, which provides a powerful cloud-based predictive analytics service.  It is designed for both new and experienced data scientists, includes 100s of algorithms from both the open source world and Microsoft Research, and supports writing ML solutions using the popular R open-source language.

You’ll continue to see major data improvements in the months ahead – we have an exciting roadmap of improvements ahead.

Summary

Today’s Microsoft Azure release enables some great new data scenarios, and makes building applications that work with data in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu omni

Categories: Architecture, Programming

Start Your Digital Vision by Reenvisioning Your Customer Experience

You probably hear a lot about the Mega-Trends (Cloud, Mobile, Social, and Big Data), or the Nexus of Forces (the convergence of social, mobility, cloud and data insights patterns that drive new business scenarios), or the Mega-Trend of Mega-Trends (Internet of Things).

And you are probably hearing a lot about digital transformation and maybe even about the rise of the CDO (Chief Digital Officer.)

All of this digital transformation is about creating business change, driving business outcomes, and driving better business results.

But how do you create your digital vision and strategy?   And, where do you start?

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

3 Perspectives of Digital Vision

When it comes to creating your digital vision, you can focus on reenvisioning the customer experience, the operational processes, or your business model.

Via Leading Digital:

“Where should you focus your digital vision? Digital visions usually take one of three perspectives: reenvisioning the customer experience, reenvisioning operational processes, or combining the previous two approaches to reenvision business models.  The approach you take should reflect your organization’s capabilities, your customer’s needs, and the nature of competition in your industry.”

Start with Your Customer Experience

One of the best places to start is with your customer experience.  After all, a business exists to create a customer.  And the success of the business is how well it creates value and serves the needs of the customer.

Via Leading Digital:

“Many organizations start by reenvisioning the way they interact with customers.  They want to make themselves easier to work with, and they want to be smarter in how they sell to (and serve) customers.  Companies start from different places when reenvisioning the customer experience.”

Transform the Relationship

You can use the waves of technologies (Cloud, Mobile, Social, Data Insights, and Internet of Things), to transform how you interact with your customers and how they experience your people, your products, and your services.

Via Leading Digital:

“Some companies aim to transform their relationships with their customers.  Adam Bortman, chief digital officer of Starbucks, shared this vision: 'Digital has to help more partners and help the company be the way we can ... tell our story, build our brand, and have a relationship with our customers.' Burberry's CEO Angela Ahredts focused on multichannel coherence. 'We had a vision, and the vision was to be the first company who was fully digital end-to-end ... A customer will have total access to Burberry across any device, anywhere.'  Mare Menesquen, managing director of strategic marketing at cosmetics gitan L'Oreal, said, 'The digital world multiples the way our brands can create an emotion-filled relationship with their customers.’”

Serve Your Customers in Smarter Ways

You can use technology to personalize the experience for your customers, and create better interactions along the customer experience journey.

Via Leading Digital:

“Other companies envision how they can be smarter in serving (and selling to) their customers through analytics.  Caesars started with a vision of using real-time customer information to deliver a personalized experience to each customer.  The company was able to increase customer satisfaction and profits per customer using traditional technologies.  Then, as new technologies arose, it extended the vision to include a mobile, location-based concierge in the palm of every customer's hand.”

Learn from Customer Behavior

One of the most powerful things you can now do with the combination of Cloud, Mobile, Social, Big Data and Internet of Things is gain better customer insights.  For example, you can learn from the wealth of social media insights, or you can learn through better integration and analytics of your existing customer data.

Via Leading Digital:

“Another approach is to envision how digital tools might help the company to learn from customer behavior.  Commonwealth Bank of Australia sees new technologies as a key way of integrating customer inputs in its co-creation efforts.  According to CIO Ian Narev, 'We are progressively applying new technology to enable customers to play a greater part in product design.  That helps us create more intuitive products and services, readily understandable to our customers and more tailored to their individual needs.”

Change Customers’ Lives

If you focus on high-value activities, you can create breakthroughs in the daily lives of your customers.

Via Leading Digital:

“Finally, some companies are extending their visions beyond influencing customer experience to actually changing customers' lives.  For instance, Novartis CEO Joseph Jimenez wrote of this potential: ‘The technologies we use in our daily lives, such as smart phones and tablet devices, could make a real difference in helping patients to manage their own health.  We are exploring ways to use these tools to improve compliance rates and enable health-care professionals to monitor patient progress remotely.’”

If you want to change the world, one of the best places to start is right from wherever you are.

With a Cloud and a dream, what can you do to change the world?

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

The Future of Jobs

Management Innovation is at the Top of the Innovation Stack

McKinsey on Unleashing the Value of Big Data

Categories: Architecture, Programming

How to Dockerize your Dropwizard Application

Xebia Blog - Wed, 10/29/2014 - 10:47

If you want to deploy your Dropwizard Application on a Docker server, you can Dockerize your Dropwizard Application. Since a Dropwizard Application is already packaged as an executable Java ARchive file, creating a Docker image for such an application should be easy.

 

In this blog, you will learn how to Dockerize a Dropwizard Application using 4 easy steps.

Before you start

  • You are going to use the Dropwizard-example application, which can be found at the Dropwizard GitHub repository.
  • Additionally you need Docker. I used Boot2Docker to run the Dockerized Dropwizard Application on my laptop. If you use boot2Docker, you may need this Boot2Docker workaround to access your Dockerized Dropwizard application.
  • This blog does not describe how to create Dropwizard applications. The Dropwizard getting started guide provides an excellent starting point if you like to know more about building your own Dropwizard applications.

 

Step 1: create a Dockerfile

You can start with creating a Dockerfile. Docker can automatically build images by reading the instructions described in this file. Your Dockerfile could look like this:

FROM dockerfile/java:openjdk-7-jdk

ADD dropwizard-example-1.0.0.jar /data/dropwizard-example-1.0.0.jar

ADD example.keystore /data/example.keystore

ADD example.yml /data/example.yml

RUN java -jar dropwizard-example-1.0.0.jar db migrate /data/example.yml

CMD java -jar dropwizard-example-1.0.0.jar server /data/example.yml

EXPOSE 8080

 

The Dropwizard Application needs a Java Runtime, so you can start from an base image already available at Docker Hub, for example: dockerfile/java:openjdk-7-jdk.

You must add the Dropwizard Application files to the image, using the ADD instruction in your Dockerfile.

Next, simply specify the commands of your Dropwizard Application, which you want to execute during image build and container runtime. In the example above, the db migrate command is executed when the Docker image is build and the server command is executed when you issue a Docker run command to create a running container.

Finally, the EXPOSE instruction tells Docker that your container will listen on the specified port(s) at runtime.

 

Step 2: build the Docker image

Place the Dockerfile and your application files in a directory and execute the Docker build command to build an Docker image.

docker@boot2docker:~$ docker build -t dropwizard/dropwizard-example ~/dropwizard/

 

In the console output you should be able to that the Dropwizard Application db migrate command is executed. If everything is ok, the last line reported informs you that the image is successfully build.

Successfully built dd547483b57b

 

Step 3: run the Docker image

Use the Docker run command to create a container based on the image you have created. If you need to find your image id use the Docker images command to list your images. It should take around 3 seconds to start the Dockerized Dropwizard example application.

Docker run –p 8080:8080 dd547483b57b

Notice that I included the –p option to include a network port binding, which maps 8080 inside the container to port 8080 on the Docker host.  You can verify whether your container is running using the docker ps command.

docker@boot2docker:~$ docker ps

CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                    NAMES

3b6fb75adad6        dropwizard/dropwizard-example:latest   "/bin/sh -c 'java -j   3 minutes ago       Up 3 minutes        0.0.0.0:8080->8080/tcp   high_turing

 

  1. Test the application

Now the application is ready for use. You can access the application using your Docker host ip address and the forward port 8080. For example, use the Google Advanced Rest Client App to register “John Doe”.

GoogleRestClient

Sponsored Post: Apple, TokuMX, Hypertable, VSCO, Gannett, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Site Reliability Engineer. As a member of the Apple Pay SRE team, you’re expected to not just find the issues, but to write code and fix them. You’ll be involved in all phases and layers of the application, and you’ll have a direct impact on the experience of millions of customers. Please apply here.
    • Software Engineering Manager. In this role, you will be communicating extensively with business teams across different organizations, development teams, support teams, infrastructure teams and management. You will also be responsible for working with cross-functional teams to delivery large initiatives. Please apply here
    • Sr. Software Developer. We are looking for a solid senior-level Java/C programmer who will be working on security software development. This software will provide the data protection, integrity, and service authentication services for iOS devices. Please apply here.
    • DevOps Software Engineer - Apple Pay, iOS Systems.  The iOS Systems team is looking for an outstanding DevOps software engineer to help make our rapidly growing platform manageable, scalable, and reliable using state of the art technologies and cutting edge system automation. Come join the team to strategize, architect, and build infrastructure to help our systems perform and scale. Please apply here

  • VSCO. Do you want to: ship the best digital tools and services for modern creatives at VSCO? Build next-generation operations with Ansible, Consul, Docker, and Vagrant? Autoscale AWS infrastructure to multiple Regions? Unify metrics, monitoring, and scaling? Build self-service tools for engineering teams? Contact me (Zo, zo@vs.co) and let’s talk about working together. vs.co/careers.

  • Gannett Digital is looking for talented Front-end developers with strong Python/Django experience to join their Development & Integrations team. The team focuses on video, user generated content, API integrations and cross-site features for Gannett Digital’s platform that powers sites such as http://www.usatoday.com, http://www.wbir.com or http://www.democratandchronicle.com. Please apply here.

  • Platform Software Engineer, Sprout Social, builds world-class social media management software designed and built for performance, scale, reliability and product agility. We pick the right tool for the job while being pragmatic and scrappy. Services are built in Python and Java using technologies like Cassandra and Hadoop, HBase and Redis, Storm and Finagle. At the moment we’re staring down a rapidly growing 20TB Hadoop cluster and about the same amount stored in MySQL and Cassandra. We have a lot of data and we want people hungry to work at scale. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you. http://www.aerospike.com/aerospike-training/

  • November TokuMX Meetups for Those Interested in MongoDB. Join us in one of the following cities in November to learn more about TokuMX and hear TokuMX use cases. 11/5 - London;11/11 - San Jose; 11/12 - San Francisco. Not able to get to these cities? Check out our website for other upcoming Tokutek events in your area - www.tokutek.com/events.
Cool Products and Services
  • Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at http://www.hypertable.com/. Connect with Hypertable: @hypertable--Blog.

  • FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.

  • Diagnose server issues from a single tab. Scalyr replaces all your monitoring and log management services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. Engineers say it's powerful and easy to use. Customer support teams use it to troubleshoot user issues. CTO's consider it a smart alternative to Splunk, with enterprise-grade functionality, sane pricing, and human support. Trusted by in-the-know companies like Codecademy – learn more!

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Azure: New Marketplace, Network Improvements, New Batch Service, Automation Service, more

ScottGu's Blog - Scott Guthrie - Tue, 10/28/2014 - 15:35

Today we released a major set of updates to Microsoft Azure. Today’s updates include:

  • Marketplace: Announcing Azure Marketplace and partnerships with key technology partners
  • Networking: Network Security Groups, Multi-NIC, Forced Tunneling, Source IP Affinity, and much more
  • Batch Computing: Public Preview of the new Azure Batch Computing Service
  • Automation: General Availability of the Azure Automation Service
  • Anti-malware: General Availability of Microsoft Anti-malware for Virtual Machines and Cloud Services
  • Virtual Machines: General Availability of many more VM extensions – PowerShell DSC, Octopus, VS Release Management

All of these improvements are now available to use immediately (note that some features are still in preview).  Below are more details about them: Marketplace: Announcing Azure Marketplace and partnerships with key technology partners

Last week, at our Cloud Day event in San Francisco, I announced a new Azure Marketplace that helps to better connect Azure customers with partners, ISVs and startups.  With just a couple of clicks, you can now quickly discover, purchase, and deploy any number of solutions directly into Azure.

Exploring the Marketplace

You can explore the Azure Marketplace by clicking the Marketplace title that is pinned by default to the home-screen of the Azure Preview Portal:

image

Clicking the Marketplace tile will enable you to explore a large selection of applications, VM images, and services that you can provision into your Azure subscription:

image

Using the marketplace provides a super easy way to take advantage of a rich ecosystem of applications and services integrated to run great with Azure.  Today’s marketplace release includes multi-VM templates to run Hadoop clusters powered by Cloudera or Hortenworks, Linux VMs powered by Unbuntu, CoreOS, Suse, CentOS, Microsoft SharePoint Server Farms, Cassandra Clusters powered by DataStax, and a wide range of security virtual appliances.

You can click any of the items in the gallery to learn more about them and optionally deploy them.  Doing so will walk you though a simple to follow creation wizard that enables you to optionally configure how/where they will run, as well as display any additional pricing required for the apps/services/VM images that you select.

For example, below is all it takes to stand-up an 8-node DataStax Enterprise cluster:

image

Solutions you purchase through the Marketplace will be automatically billed to your Azure subscription (avoiding the need for you to setup a separate payment method).  Virtual Machine images will support the ability to bring your own license or rent the image license by the hour (which is ideal for proof of concept solutions or cases where you need the solution for only a short period of time).  Both Azure Direct customers as well as customers who pay using an Enterprise Agreement can take advantage of the Azure Marketplace starting today.

You can learn more about the Azure Marketplace as well as browse the items within it here.

Networking: Lots and lots of New Features and Improvements

This week’s Azure update includes a ton of new capabilities to the Azure networking stack.  You can use these new networking capabilities immediately in the North Europe region, and they will be supported worldwide in all regions in November 2014.  The new network capabilities include:

Network Security Groups

You can now create Network Security groups to define access control rules for inbound and outbound traffic to a Virtual machine or a group of virtual machines in a subnet. The security groups and the rules can be managed and updated independent of the life cycle of the VM.

Multi-NIC Support

You can now create and manage multiple virtual network interfaces (NICs) on a VM.  Multi-NIC support is a fundamental requirement for a majority of network virtual appliances that can be deployed in Azure. Having this support now enabled within Azure will enable even richer network virtual appliances to be used.

Forced Tunneling

You can now redirect or “force” all Internet-bound traffic that originates in a cloud application back through an on-premises network via a Site-to-Site VPN tunnel for inspection and auditing. This is a critical security capability for enterprise grade applications.

ExpressRoute Enhancements

You can now share a single ExpressRoute connection across multiple Azure subscriptions. Additionally, a single Virtual Network in Azure can now be linked to more than one ExpressRoute circuit, thereby enabling much richer backup and disaster recovery scenarios.

image

New VPN Gateway Sizes

To cater to the growing hybrid connectivity throughput needs and the number of cross premise sites, we are announcing the availability of a higher performance Azure VPN gateway. This will enable a faster ExpressRoute and Site-to-Site VPN gateways with more tunnels.

Operations and audit logs for VNet Gateways and ExpressRoute

You can now view operations logs for Virtual Network Gateways and ExpressRoute circuits. The Azure portal will now show operations logs and information on all API calls you make and important infrastructure changes made such as scheduled updates to gateways.

Advanced Virtual Network Gateway policies

We now enable the ability for you to control encryption for the tunnel between Virtual Networks. You now have a choice between 3DES, AES128, AES256 and Null encryption, and you can also enable Perfect Forward Secrecy (PFS) for IPsec/IKE gateways.

Source IP Affinity

The Azure Load Balancer now supports a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). You can now load balance traffic based on a 2-tuple (Source-IP, Destination-IP) or 3-tuple (Source-IP, Destination-IP and Protocol) distribution modes.

Nested policies for Traffic Manager

You can now create nested policies for traffic management. This allows tremendous flexibility in creating powerful load-balancing and failover schemes to support the needs of larger, more complex deployments.

Portal Support for Managing Internal Load Balancer, Reserved and Instance IP addresses for Virtual Machines

It is now possible to use the Azure Preview Portal to manage creating and setting up internal load balancers, as well as reserved and instance IP addresses for virtual machines.

Automation: General Availability of Azure Automation Service

I am excited to announce the General Availability of the Azure Automation service. Azure Automation enables the creation, deployment, monitoring, and maintenance of resources in an Azure environment using a highly scalable and reliable workflow engine. The service can be used to orchestrate time-consuming and frequently repeated operational tasks across Azure and third-party systems while decreasing operating expenses.

Azure Automation allows you to build runbooks (PowerShell Workflows) to describe your administration processes, provides a secure global assets store so you don’t need to hardcode sensitive information within your runbooks, and offers scheduling so that runbooks can be triggered automatically.

Runbooks can automate a wide range of scenarios – from simple day to day manual tasks to complex processes that span multiple Azure services and 3rd party systems. Because Automation is built on PowerShell, you can take advantage of the many existing PowerShell modules, or author your own to integrate with third party systems.

Creating and Editing Runbooks

You can create a runbook from scratch, or start by importing an existing template in the runbook gallery:

image

Editing experience for runbooks can also be performed directly in the administration portal:

image

Pricing

Available as a pay-as-you-go service, Automation is billed based on the number of job run time minutes used in a given Azure subscription.  500 minutes of free job runtime credits are also included each month for Azure customers to use at no charge.

Learn More

To learn more about Azure Automation, check out the following resources:

Batch Service: Preview of Azure Batch - new job scheduling service for parallel and HPC apps

I’m excited to announce the public preview of our new Azure Batch Service. This new platform service provides “job scheduling as a service” with auto-scaling of compute resources, making it easy to run large-scale parallel and high performance computing (HPC) work in Azure. You submit jobs, we start the VMs, run your tasks, handle any failures, and then shut things down as work completes.

Azure Batch is the job scheduling engine that we use internally to manage encoding for Azure Media Services, and for testing Azure itself. With this preview, we are excited to expand our SDK with a new application framework from GreenButton, a company Microsoft acquired earlier in the year. The Azure Batch SDK makes it easy to cloud-enable parallel, cluster, and HPC applications by describing jobs with the required resources, data, and one or more compute tasks.

Azure Batch can be used to run large volumes of similar tasks or applications in parallel, programmatically. A command line program or script takes a set of data files as input, processes the data in a series of tasks, and produces a set of output files. Examples of batch workloads that customers are running today in Azure include calculating risk for banks and insurance companies, designing new consumer and industrial products, sequencing genes and developing new drugs, searching for new energy sources, rendering 3D animations, and transcoding video.

Azure Batch makes it easy for these customers to use hundreds, thousands, tens of thousands of cores, or more on demand. With job scheduling as a service, Azure developers can focus on using batch computing in their applications and delivering services without needing to build and manage a work queue, scaling resources up and down efficiently, dispatching tasks, and handling failures.

image

The scale of Azure helps batch computing customers get their work done faster, experiment with different designs, run larger and more precise models, and test a large number of different scenarios without having to invest in and maintain large clusters.

Learn more about Azure Batch and start using it for your applications today. Virtual Machines: General Availability of Microsoft Anti-Malware for VMs and Cloud Services

I’m excited to announce that the Microsoft Anti-malware security extension for Virtual Machines and Cloud Services is now generally available.  We are releasing it as a free capability that you can use at no additional charge.

The Microsoft Anti-malware security extension can be used to help identify and remove viruses, spyware or other malicious software.  It provides real-time protection from the latest threats and also supports on-demand scheduled scanning.  Enabling it is a good security best practice for applications hosted either on-premises or in the cloud.

Enabling the Anti-Malware Extension

You can select and configure the Microsoft Antimalware security extension for virtual machines using the Azure preview portal, Visual Studio or API’s/PowerShell.  Antimalware events are then logged to the customer configured Azure Storage account via Azure Diagnostics and can be piped to HDInsight or a SIEM tool for further analysis. More information is available in the Microsoft Antimalware Whitepaper.

To enable antimalware feature on existing virtual machine, select the EXTENSIONS tile on a Virtual Machine in the Azure Preview Portal, then click ADD in the command bar and select the Microsoft Antimalware extension. Then, click CREATE and customize any settings:

image Virtual Machines: General Availability of even more VM Extensions

In addition to enabling the Microsoft Anti-Malware extension for Virtual Machines, today’s release also includes support for a whole bunch more new VM extensions that you can enable within your Virtual Machines.  These extensions can be added and configured using the same EXTENSIONS tile on Virtual Machine resources within the Azure Preview Portal (the same screen-shot as in the Anti-malware section above).

The new extensions enabled today include:

PowerShell Desired State Configuration

The PowerShell Desired State Configuration Extension can be used to deploy and configure Azure VMs using Desired State Configuration (DSC) technology. DSC enables you to declaratively specify how you want your software environment to be configured. DSC configuration can also be automated using the Azure PowerShell SDK, and you can push configurations to any Azure VM and have them enacted automatically. For more details, please see this desired state configuration blog post.

image 

Octopus

Octopus simplifies the deployment of ASP.NET web applications, Windows Services and other applications by automatically configuring IIS, installing services and making configuration changes. Octopus integration of Azure was one of the top requested features on Azure UserVoice and with this integration we will simplify the deployment and configuration of octopus on the VM.

image

Visual Studio Release Management

Release Management for Visual Studio is a continuous delivery solution that automates the release process through all of your environments from TFS through to production. Visual Studio Release Management is integrated with TFS and you can configure multi-stage release pipelines to automatically deploy and validate your applications on multiple environments. With the new Visual Studio Release Management extension, VMs can be preconfigured with the necessary components for required for Release Management to operate.

image Summary

Today’s Microsoft Azure release enables a ton of great new scenarios, and makes building applications hosted in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu omni

Categories: Architecture, Programming

Sketching API Connections

Coding the Architecture - Simon Brown - Tue, 10/28/2014 - 14:52

Daniel Bryant, Simon and I recently had a discussion about how to represent system communication with external APIs. The requirement for integration with external APIs is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams.


How to Represent an External System?

The first thing we discussed was what symbol to use for a system supplying an API. Traditionally, UML has used the Actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (UML Superstructure Specification, v2.1.2). Therefore a system providing an API may look like this:


Actor


I've found that this symbol tends to confuse those who aren't well versed in UML as most people assume that the Actor symbol always represents a *person* rather than a system. Sometimes this is stereotyped to make it more obvious e.g.


Actor with stereotype


However the symbol is very powerful and tends to overpower the stereotype. Therefore I prefer to use a stereotyped box for an external system supplying an API. Let's compare two context diagrams using Boxes vs Stick Actors.


Actor

In which diagram is it more obvious what are systems or people?

Note that ArchiMate has a specific symbol for Application Service that can be used to represent an API:


Actor (Application Service notation from the Open Group's ArchiMate 2.1 Specification)


An API or the System that Supplies it?

Whatever symbol we choose, what we've done is to show the *system* rather than the actual API. The API is a definition of a service provided by the system in question. How should we provide more details about the API?

There are a number of ways we could do this but my preference is to give details of the API on the connector (line connecting two elements/boxes). In C4 the guidelines for a container diagram includes listing protocol information on the connector and an API can be viewed as the layer above the protocol. For example:


Actor


Multiple APIs per External System

Many API providers supply multiple services/APIs (I'm not referring to different operations within an API but multiple sets of operations in different APIs, which may even use different underlying protocols.) For example a financial marketplace may have APIs that do the following:

  1. Allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over HTTP.
  2. Supply real time, low latency updates of market prices via bespoke messages over UDP.
  3. Allow entry of trades via industry standard FPML over a queuing system.
  4. Supply a bulk, batch download of trades for end-of-day reconciliation via FPML over HTTP.

Two of the services use the same protocol (xml over HTTP) but have very different content and use. One of the APIs is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at EOD).

There are multiple ways of showing this. We could:

  1. Have a single service element, list the APIs on it and have all components linking to it.
  2. Show each service/API as a separate box and connect the components that use the individual service to the relevant box.
  3. Show a single service element with multiple connections. Each connection is labeled and represents an API.
  4. Use a Port and Connector style notation to represent each API from the service provider. Provide a key for the ports.
  5. Use a UML style 'cup and ball' notation to define interfaces and their usage.

Some examples are below:


A Single Service element and simple description


Actor


In the above diagram the containers are stating what they are using but contain no information about how to use the APIs. We don't know if it is a single API (with different operations) or anything about the mechanisms used to transport the data. This isn't very useful for anyone implementing a solution or resolving operational issues.


Single, Service box with descriptive connectors


Actor

In this diagram there is a single, service box with descriptive connectors. The above diagram shows all the information so is much more useful as a diagnostic or implementation tool. However it does look quite crowded.


Services/APIs shown as separate boxes


Actor


Here the external system has its services/APIs shown as separate boxes. This contains all the information but might be mistaken as defining the internal structure of the external system. We want to show the services it provides but we know nothing about the internal structure.


Using Ports to Represent APIs


Actor with


In the above diagram the services/APIs are shown as 'ports' on the external system and the details have been moved into a separate key/table. This is less likely to be mistaken as showing any internal structure of the external service. (Note that I could have also shown outgoing rPorts from the Brokerage System.)


UML Interfaces


Actor

This final diagram is using a UML style interface provider and requirer. This is a clean diagram but requires the user to be aware of what the cup and ball means (although I could have explained this in the key).


Conclusion

Any of these solutions could be appropriate depending on the complexity of the API set you are trying to represent. I'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed BUT remember to use a key to explain any elements you use!

Categories: Architecture

Software architecture vs code (DevDay 2014)

Coding the Architecture - Simon Brown - Mon, 10/27/2014 - 19:10

I had the pleasure of delivering the closing keynote at the DevDay 2014 conference in Krakow, Poland last month. It's a one day event, with a bias towards the .NET platform, and one of my favourite conferences from this year. Beautiful city, fantastic crowd and top-notch hospitality. If you get the chance to attend next year, do it!

If you missed it, you can find videos of the talks to watch online. Here's mine called Software architecture vs code. It covers the conflict between software architecture and code, how we can resolve this, the benefits of doing so, fishing and a call for donations to charity every time you write public class without thinking. Enjoy!

p.s. I've written about some of these same topics on the blog ... for example, Modularity and testability and Software architecture vs code. My Structurizr project is starting to put some of this into practice too.

Categories: Architecture

Microservices in Production - the Good, the Bad, the it Works

This is a guest repost written by Andrew Harmel-Law on his real world experiences with Microservices. The original article can be found here.

It’s reached the point where it’s even a cliche to state “there’s a lot written about Microservices these days.” But despite this, here’s another post on the topic. Why does the internet need another? Please bear with me…

We’re doing Microservices. We’re doing it based on a mash-up of some “Netflix Cloud” (as it seems to becoming known - we just call it “Archaius / Hystrix”), a gloop of Codahale Metrics, a splash of Spring Boot, and a lot of Camel, gluing everything together. We’ve even found time to make a bit of Open Source ourselves - archaius-spring-adapter - and also contribute some stuff back.

Lets be clear; when I say we’re “doing Microservices”, I mean we’ve got some running; today; under load; in our Production environment. And they’re running nicely. We’ve also got a lot more coming down the dev-pipe.

All the time we’ve been crafting these we’ve been doing our homework. We’ve followed the great debate, some contributions of which came from within Capgemini itself, and other less-high-profile contributions from our very own manager. It’s been clear for a while that, while there is a lot of heat and light generated in this debate, there is also a lot of valid inputs that we should be bearing in mind.

Despite this, the Microservices architectural style is still definitely in the honeymoon period, which translates personally into the following: whenever I see a new post on the topic from a Developer I respect my heart sinks a little as I open it and read… Have they discovered the fatal flaw in all of this that everyone else has so far missed? Have they put their finger on the unique aspect that mean 99% of us will never realise the benefits of this new approach and that we’re all off on a wild goose chase? Have they proven that Netflix really are unicorns and that the rest of us are just dreaming?

Despite all this we’re persisting. Despite always questioning every decision we make in this area far more than we normally would, Microservices still feel right to us for a whole host of reasons. In the rest of this post I hope I’ll be able to point out some of the subtleties which might have eluded you as you’ve researched and fiddled, and also, I’ve aimed to highlight some of the old “givens” which might not be “givens” any more.

The Good
Categories: Architecture

How to create Java microservices with Dropwizard

Xebia Blog - Mon, 10/27/2014 - 15:10

On Tuesday October 14th the Amsterdam Middleware Meetup experimented with Dropwizard. The idea was to find out what this technology is about, where it could be useful and what the alternatives are. So below I’ll give you an overview of Dropwizard and compare it to Spring Boot.
The Dropwizard website claims:

Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.

I’ll discuss each of these claims below.

Stable and mature
Dropwizard uses Jetty, Jersey, Jackson and Metrics as its most important frameworks, but also a host of other stuff like Guava, Liquibase and Joda Time. The latest Dropwizard release is version 0.7.1, released on June 20th 2014. It depends on these versions of some core libraries:
Jetty - 9.2.3.v20140905 - May 2014
Jackson - 2.4.1 - June 2014
Jersey - 2.11 - July 2014

The table shows that stable != out-of-date which is fine of course. The versions of core libraries used are recent though. I guess ‘stable’ means libraries with a long history.

Simple
The components of a Dropwizard application are shown below (taken from the tutorial
http://dropwizard.io/getting-started.html):
Dropview components overview

  1. Application (HelloWorldApplication.java): the applications main method, responsible for startup.
  2. Configuration (HelloWorldConfiguration.java) sets configuration for an environment, this is where you may set hostnames for systems the application depends on or set usernames.
  3. Data object (Saying.java).
  4. Resource (HelloWordResource.java): service implementation entry point
  5. Health Check (TemplateHealthCheck.java): runtime tests that show if the application still works.

Light weight
We did some experiments trying to answer the question whether Dropwizard applications are light weight. The table below summarizes some of the sizes of deployments and tools.
Tomcat size 14 mb
Tomcat lib folder size 7 MB
Jetty size 14,6 MB
Jetty in Dropwizard jar: 5,4 MB
Dropwizard tutorial example 10 mb
Dropwizard extended example 20 MB
Dropwizard Hibernate classes in package: 5 MB

A Tomcat or Jetty installation takes about 14 MB, but if you count only the lib folder the size goes down to about 7 MB. The Jetty folder in Dropwizard however is only 5.5 MB. Apparently Dropwizard managed to strip away some code you don’t really need (or is packaged somewhere else, we didn’t look into that).
Building the tutorial results in a 10 MB jar, so if you would run a webapp in its own Tomcat container, switching to Dropwizard saves quite a bit. On the other hand, deployment size isn’t all that important if we’re still talking < 50 MB.
Compared to your default Weblogic install (513 MB, Weblogic-only on OSX) however, savings are humongous (but this is also true when you compare Weblogic to Tomcat or Jetty).

Productivity
We tried to run the build for the tutorial application (dropwizard-example in the dropwizard project on Github). This works fine and takes about 8 seconds using mocks for external connections. One option to explore would be to run tests against a deployed application. What we’re used to is that deploying an application for test takes lots of time and resources, but starting a Dropwizard app is quite cheap. Therefore it would be possible to run an integration test of services at the end of a build. This would be quite hard to do with e.g. Weblogic or Websphere.

Spring boot
Spring boot is interesting, as well as the discussion around the differences between Spring boot and Dropwizard. See https://groups.google.com/forum/#!topic/dropwizard-user/vH1h2PgC8bU

The official Spring boot website says: Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that can you can "just run". We take an opinionated view of the Spring platform and third-party libraries so you can get started with minimum fuss. Most Spring Boot applications need very little Spring configuration.
It’s good to see a platform change according to new insights, but still, I remember Rod Johnson saying some ten years ago that J2EE was bloated and complex and Spring was the answer. Now it seems we need Spring boot to make Spring simple? Or is it just that we don’t need application servers anymore to divide resources among processes?

Dropwizard and Docker
Finally we experimented with running Dropwizard in a Docker container. This can be done with limited effort because Dropwizard applications have such a small number of dependencies. Thomas Kruitbosch will report on this later.

References
Spring boot: http://projects.spring.io/spring-boot/
Dropwizard: http://dropwizard.io/

100 Top Agile Blogs

Luis Goncalves has put together a list called the 100 Top Agile Blogs:

If you don't know Luis, he lives and breathes driving adoption of Agile practices.

Luis is also an Agile Coach, Co-Author, Speaker, and Blogger.  He is also the co-founder of a MeetUp group called High Performing Teams, and he is a certified Scrum Master and Product Owner.

Here is a preview of the list of top 100 Agile Blogs:

image

 

For the rest of the list, check out 100 Top Agile Blogs.

Lists like these are a great way to discover blogs you may not be aware of.  

While there will be a bunch of blogs you already know, chances are, with that many at a glance, there will be at least a few new ones you can add to your reading list.

Categories: Architecture, Programming

Big changes

Gridshore - Sun, 10/26/2014 - 08:27

The first 10 years of my career I worked as a consultant for Capgemini and Accenture. I learned a lot in that time. One of the things I learned was that I wanted something else. I wanted to do something with more impact, more responsibility and together with people that wanted to do challenging projects. Not pure to get up in your career but more because they like doing cool stuff. Therefore I left Accenture to become part of a company called JTeam. It has been over 6 years that this took place.

I started as Chief Architect at JTeam. The goal was to become a leader to the other architects and create a team together with Bram. That time I was lucky that Allard joined me. We share a lot of ideas, which makes it easier to set goals and accomplish them. I got to learn a few very good people at JTeam, to bad that some of them left, but that is life.

After a few years bigger changes took place. Leonard left for Steven and the shift to a company that needs to grow started. We took over two companies (Funk and Neteffect), we now had all disciplines of software development available. From front-end to operations. As the company grew some things had to change. I got more involved in arranging things like internships, tech events, partnerships and human resource management.

We moved into a bigger building and we had better opportunities. One of the opportunities was a search solution created by Shay Banon. Gone was Steven, together with Shay he founded Elasticsearch. We got acquired by Trifork. In this change we lost most of our search expertise because all of our search people joined the elasticsearch initiative. Someone had to pick up search at Trifork and that was me together with Bram.

For over 2 years I invested a lot of time in learning about mainly elasticsearch. I created a number of workshops/trainings and got involved with multiple customers that needed search. I have given trainings to a number of customers to groups varying between 2 and 15 people. In general they were all really pleased with the trainings I have given.

Having so much focus for a while gave me a lot of time to think, I did not need to think about next steps for the company, I just needed to get more knowledgeable about elasticsearch. In that time I started out on a journey to find out what I want. I talked to my management about it and thought about it myself a lot. Then, right before summer holiday I had a diner with two people I know through the Nljug, Hans and Bert. We had a very nice talk and in the end they gave me an opportunity that I really had to have some good thoughts about. It was really interesting, a challenge, not really a technical challenge, but more an experience that is hard to find. During summer holiday I convinced myself this was a very interesting direction and I took the next step.

I had a lunch meeting with my soon to be business partner Sander. After around 30 minutes it already felt good. I really feel the energy of creating something new, I feel inspired again. This is the feeling I have been missing for a while. In September we were told that Bram was leaving Trifork. Since he is the person that got me into JTeam back in the days it felt weird. I understand his reasons to go out and try to start something new. Bram leaving resulted in a vacancy for a CTO and the management team had decided to approach Allard for this role. This was a surprise to me, but a very nice opportunity for Allard and I know he i going to do a good job. At the end of September Sander and myself presented the draft business plan to the board for Luminis. That afternoon hands were shaken. It was than that I made the last call and decided to resign from my Job at Trifork and take this new opportunity at Luminis.

I feel sad about leaving some people behind. I am going to mis the morning talks in the car with Allard about everything related to the company, I am going to mis doing projects with Roberto (We are a hell of team), I am going to mis Byron for his capabilities (You make me feel proud that I guided your first steps within Trifork), I am going to mis chasing customers with Henk (We did a good job the passed year) and I am going to mis Daphne and the after lunch walks . To all of you and all the others at Trifork, it is a small world …

Luminis logo

Together with Sander, and with the help of all the others at Luminis, we are going to start Luminis Amsterdam. This is going to be a challenge for me, but together with Sander I feel we are going to make it happen. I feel confident that the big changes to come will be good changes.

The post Big changes appeared first on Gridshore.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For October 24th, 2014

Hey, it's HighScalability time:


This is an ultrasound powered brain implant! (65nm GP CMOS technology, high speed, low power (100 µW))
  • 70: percentage of the worlds transactions processed using COBOL.  
  • Quotable Quotes:
    • John Siracusa: Apple has shown that it wants to succeed more than it fears being seen as a follower.
    • @Dries: "99% of Warren Buffett's wealth was built after his 50th birthday."
    • @Pinboard: It is insane to run a bookmarking site on AWS at any kind of scale. Unless you are competing with me, in which case it’s a great idea—do it!
    • @dvellante: I sound like a broken record but AWS has the scale to make infrastructure outsourcing marginal costs track SW curve 
    • @BrentO: LOL RT @SQLPerfTips: "guess which problem you are more likely to have - needing joins, or scaling beyond facebook?"
    • @astorrs: Legacy systems? Yes they're still relevant. ~20x the number of transactions as Google searches @IBM #DOES14 
    • @SoberBuildEng: "It was all the Agile guys' fault at the beginning.Y'know, if the toilet overflowed, it was 'What, are those Agile guys in there?!'" #DOES14
    • @cshl1: #DOES14  @netflix "$1.8M revenue / employee" << folks, this is an amazing number
    • Isaac Asimov: Probably more inhibiting than anything else is a feeling of responsibility. The great ideas of the ages have come from people who weren’t paid to have great ideas, but were paid to be teachers or patent clerks or petty officials, or were not paid at all. The great ideas came as side issues.

  • With Fabric can Twitter mend the broken threads of developer trust? A good start would be removing 3rd party client user limit caps. Not sure a kit of many colors will do it.

  • Not only do I wish I had said this, I wish I had even almost thought it. tjradcliffe: I distinguish between two types of puzzles: human-made (which I call puzzles) and everything else (which I call problems.) In those terms, I hate puzzles and love problems. Puzzles are contrived by humans and are generally as much psychology problems as anything else. They basically require you to think like the human who created them, and they have bizarre and arbitrary constraints that are totally unlike the real world, where, as Feyrabend told us, "Anything goes."

  • David Rosenthal with a great look at Facebook's Warm Storage: 9 [BLOB] types have dropped by 2 orders of magnitude within 8 months...the vast majority of the BLOBs generate I/O rates at least 2 orders of magnitude less than recently generated BLOBs...Within a data center it uses erasure coding...Between data centers it uses XOR coding...When fully deployed, this will save 87PB of storage...heterogeneity as a way of avoiding correlated failures.

  • Gene Tene on is it a CPU bound future: I don't think CPU speed is a problem. The CPUs and main RAM channels are still (by far) the highest performing parts of our systems. For example, yes, you can move ~10-20Gbps over various links today (wired or wifi, "disk" (ssd) or network), but a single Xeon chip today can sustain well over 10x that bandwidth in random access to DRAM. A single chip has more than enough CPU bandwidth to stream through that data, too. E.g. a single current Haswell core can move more than that 10-20Gbps in/out of it's cache levels. and even relatively low end chips (e.g. laptops) will have 4 or more of these cores on a single chip these days. < BTW, a great thread if you are interested in latency issues.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

The Future of IT Leaders

I’ll need to elaborate on this at some point, to share what I’ve experienced across lots of businesses large and small, as well as some of the biggest businesses on the planet, as they transform themselves for the digital economy.

Meanwhile, here is an interesting read on CIO Straight Talk magazine.

In their words, "CIO Straight Talk is a series of "straight talking" articles from senior IT executives and leading companies and government and nonprofit organizations."

This first edition is focused on learning, failing and learning in the Second Machine Age, and features two non-practitioner experts on current topics:

“Andrew McAfee, co-author of the New York Times bestseller The Second Machine Age, cofounder of MIT’s Initiative on the Digital Economy and Principal Research Scientist at MIT Sloan School of Management, talks about ‘The CIO’s role in the enterprise of the future.’ Says McAfee: ‘The overall trend is that companies of all stripes will need, proportionately, many fewer people in IT. Those who remain will be very highly valued, very highly skilled, very important… Enterprises are going to need someone to help them navigate the second machine age… I think that if the CIO plays her cards right, this can absolutely be her role in the enterprise.’”

Michelle Gallen, the CEO of Shhmooze, a social networking start-up, talks about failure, not to be confused Failure Lite – ‘I failed. How nice. I learned so much’ – often hailed breezily by management experts as something everyone should experience and every company should encourage. Real failure, according to this serial entrepreneur, isn’t pretty. Says Gallen: ‘I don’t think you learn without failing… In the start-up world, innovation is the ability to take an idea and turn it into an invoice. Lots of larger business organizations also rely on cash flow to keep them alive, and therefore innovation has to be monetized. If you’re Apple or Microsoft, you’ve got a war chest, and you can actually allow failure. A lot of companies can’t actually afford it. It’s quite an expensive hobby, failing.’”

So there you have it -- failure is an expensive hobby and the few IT leaders left in organizations will be very highly valued, very highly skilled, and very important.

There’s more to the story and I’ll share what I’ve learned over the past few years helping companies cross the Cloud chasm and accelerating their digital transformation.

Categories: Architecture, Programming

Software architecture sketching in Iceland

Coding the Architecture - Simon Brown - Thu, 10/23/2014 - 10:46

I'll be in Iceland next month for the Agile Iceland 2014 conference, which I'm really looking forward to as everybody tells me that Iceland is a fantastic country to visit. While in Iceland, I'll also be running my 1-day software architecture sketching workshop on the 6th of November. If you're interested in learning how to communicate the design of your software in a simple yet effective way without using lots of complex UML diagrams, please do join me. Everybody who attends will also get a copy of my Software Architecture for Developers ebook too. :-)

Categories: Architecture

Paper: Actor Model of Computation: Scalable Robust Information Systems

With Reactive Systems becoming the new old hotness, it will help to have a thorough grounding in the Actor Model. Here's a good start. Carl Hewitt in Actor Model of Computation: Scalable Robust Information Systems gives a very thorough and relatively concise explanation of the Actor model.

Here's the abstract.

The Actor model is a mathematical theory that treats "Actors" as the universal primitives of concurrent digital computation. The model has been used both as a framework for a theoretical understanding of concurrency, and as the theoretical basis for several practical implementations of concurrent systems. Unlike previous models of computation, the Actor model was inspired by physical laws. It was also influenced by the programming languages Lisp, Simula 67 and Smalltalk-72, as well as ideas for Petri Nets, capability-based systems and packet switching. The advent of massive concurrency through client-cloud computing and many-core computer architectures has galvanized interest in the Actor model.


Actor technology will see significant application for integrating all kinds of digital information for individuals, groups, and organizations so their information usefully links together. Information integration needs to make use of the following information system principles:
    * Persistence. Information is collected and indexed.
    * Concurrency: Work proceeds interactively and concurrently, overlapping in time.
    * Quasi-commutativity: Information can be used regardless of whether it initiates new work or become relevant to ongoing work.
    * Sponsorship: Sponsors provide resources for computation, i.e., processing, storage, and communications.
    * Pluralism: Information is heterogeneous, overlapping and often inconsistent.
    * Provenance: The provenance of information is carefully tracked and recorded

The Actor Model is intended to provide a foundation for inconsistency robust information integration.

Related Articles
Categories: Architecture

Think a Series of Sprints, Not Marathons

When you drive business change and digital initiatives with Cloud, Mobile, Social, and Big Data (and Internet of Things), successful businesses think a series of sprints, not marathons.

Successful businesses go digital by transforming their customer experiences, their employee experiences, and their back-office experiences through rapid prototyping, building proofs-of-concept, testing pilots, and going to production.  It’s a fast cycle of prototype –> pilot –> POC –> production.

These short cycles create rapid learning loops, build momentum, and help adapt for change.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned in driving digital initiatives and agile transformation.

The Digital World Moves Quickly

Avoid Big Up Front Design.  Whenever there is a big lag time between designing it, developing it, and using it, you’re introducing more risk.  You’re breaking feedback loops.  You’re falling into the pit of analysis paralysis.   Focus on “just enough design” so that you can test what works and what doesn’t, and respond accordingly.

Via Leading Digital:

“The digital world moves quickly.  The rapid pace of technology innovation today does not lend itself to multiyear planning and waterfall development methods common in the ERP era.  Markets change, new technologies become mainstream, an disruptive entrants begin courting your customers.  Your roadmap will need to be nimble enough to recognize these changes, adapt for them, and course-correct.”

Keep a Vision in Mind and Build on Success Along the Way

Hold on to the vision and use that to guide you as you test your ideas and implement them, without getting bogged down.

Via Leading Digital:

“To design an agile transformation, borrow an approach that has become common among today's leading software companies.  Keep people committed to the end goal, but pace your initiatives as short sprints of effort.  Create prototype solutions, and experiment with new technologies or approaches.  Evaluate the results, and incorporate the results into your evolving roadmap.  Adam Brotman, Starbucks CDO, explained the iterative process: 'We didn't have all the answers, but we started thinking about other things we could do ... I think it worked not to go too far, too fast, but to keep a vision in mind and keep building on success along the way.”

Test Ideas, Save Time, Adapt to Changes

Short cycle times help you respond to market change and adapt as you learn what works and what doesn’t.

Via Leading Digital:

“The test-and-learn approach will require some new ways of working in its own right, but it enjoys some distinct advantages.  By marketing ideas quickly before they go to scale, this approach saves time and money.  It's short cycle times also make it more adaptive to external changes.  Finally, it enables your transformation to sustain momentum through small, incremental successes, rather than the big-bang approach of long-term programs.”

When it comes to your digital strategy and driving business transformation, drive your business change the agile way.

You Might Also Like

10 Ways to Make Agile Design More Effective

Building Better Business Cases for Digital Initiatives

Cloud Changes the Game from Deployment to Adoption

How Digital is Changing Physical Experiences

McKinsey on Unleashing the Value of Big Data Analytics

Categories: Architecture, Programming

Comparing solutions

Coding the Architecture - Simon Brown - Wed, 10/22/2014 - 14:38

Here's a little snippet that my class really picked up on yesterday. During the training course, we get people into groups and ask them to design a solution based upon some simple requirements. The deliverable is "one or more diagrams to describe your solution". Aside from answering a few questions about the business domain and the environment, that's pretty much all the guidance that groups get.

As you can probably imagine, the resulting diagrams are all very different. Some are very high-level, others very low-level. Some show static structure, others show runtime and behavioural views. Some show technology, others don't. Without a consistent approach, these differing diagrams make it hard for people to understand the solutions being presented to them. But furthermore, the differing diagrams make it really hard to compare solutions too.

As I've said before, I don't actually teach people to draw pictures. What I do instead is to teach people how to think about, and therefore describe, their software using a simple set of abstractions. This is my C4 model. With these abstractions in place, groups then redraw their diagrams. Despite the notations still differing between the groups, the solutions are much easier to understand. The solutions are much easier to compare too, because of the consistency in the way they are being described. A common set of abstractions is much more important than a common notation.

Categories: Architecture

Facebook Mobile Drops Pull For Push-based Snapshot + Delta Model

We've learned mobile is different. In If You're Programming A Cell Phone Like A Server You're Doing It Wrong we learned programming for a mobile platform is its own specialty. In How Facebook Makes Mobile Work At Scale For All Phones, On All Screens, On All Networks we learned bandwidth on mobile networks is a precious resource. 

Given all that, how do you design a protocol to sync state (think messages, comments, etc.) between mobile nodes and the global state holding servers located in a datacenter?

Facebook recently wrote about their new solution to this problem in Building Mobile-First Infrastructure for Messenger. They were able to reduce bandwidth usage by 40% and reduced by 20% the terror of hitting send on a phone.

That's a big win...that came from a protocol change.

Facebook Messanger went from a traditional notification triggered full state pull:

Categories: Architecture

How to deploy a Docker application into production on Amazon AWS

Xebia Blog - Fri, 10/17/2014 - 17:00

Docker reached production status a few months ago. But having the container technology alone is not enough. You need a complete platform infrastructure before you can deploy your docker application in production. Amazon AWS offers exactly that: a production quality platform that offers capacity provisioning, load balancing, scaling, and application health monitoring for Docker applications.

In this blog, you will learn how to deploy a Docker application to production in five easy steps.

For demonstration purposes, you are going to use the node.js application that was build for CloudFoundry and used to demonstrate Deis in a previous post. A truly useful app of which the sources are available on github.

1. Create a Dockerfile

First thing you need to do is to create a Dockerfile to create an image. This is quite simple: you install the node.js and npm packages, copy the source files and install the javascript modules.

# DOCKER-VERSION 1.0
FROM    ubuntu:latest
#
# Install nodejs npm
#
RUN apt-get update
RUN apt-get install -y nodejs npm
#
# add application sources
#
COPY . /app
RUN cd /app; npm install
#
# Expose the default port
#
EXPOSE  5000
#
# Start command
#
CMD ["nodejs", "/app/web.js"]
2. Test your docker application

Now you can create the Docker image and test it.

$ docker build -t sample-nodejs-cf .
$ docker run -d -p 5000:5000 sample-nodejs-cf

Point your browser at http://localhost:5000, click the 'start' button and Presto!

3. Zip the sources

Now you know that the instance works, you zip the source files. The image will be build on Amazon AWS based on your Dockerfile.

$ zip -r /tmp/sample-nodejs-cf-srcs.zip .
4. Deploy Docker application to Amazon AWS

Now you install and configure the amazon AWS command line interface (CLI) and deploy the docker source files to elastic beanstalk.  You can do this all manually, but here you use the script deploy-to-aws.sh that I created.

$ deploy-to-aws.sh \
         sample-nodejs-cf \
         /tmp/sample-nodejs-cf-srcs.zip \
         demo-env

After about 8-10 minutes your application is running. The output should look like this..

INFO: creating application sample-nodejs-cf
INFO: Creating environment demo-env for sample-nodejs-cf
INFO: Uploading sample-nodejs-cf-srcs.zip for sample-nodejs-cf, version 1412948762.
upload: ./sample-nodejs-cf-srcs.zip to s3://elasticbeanstalk-us-east-1-233211978703/1412948762-sample-nodejs-cf-srcs.zip
INFO: Creating version 1412948762 of application sample-nodejs-cf
INFO: demo-env in status Launching, waiting to get to Ready..
...
INFO: demo-env in status Launching, waiting to get to Ready..
INFO: Updating environment demo-env with version 1412948762 of sample-nodejs-cf
INFO: demo-env in status Updating, waiting to get to Ready..
...
INFO: demo-env in status Updating, waiting to get to Ready..
INFO: Version 1412948762 of sample-nodejs-cf deployed in environment
INFO: current status is Ready, goto http://demo-env-vm2tqi3qk4.elasticbeanstalk.com
5. Test your Docker application on the internet!

Your application is now available on the Internet. Browse to the designated URL and click on start. When you increase the number of instances at Amazon, they will appear in the application. When you deploy a new version of the application, you can observe how new versions of the application  appear without any errors on the client application.

For more information, goto Amazon Elastic Beanstalk adds Docker support. and Dockerizing a Node.js Web App.