Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!


The Ultimate Personal Productivity Platform is You

“Amateurs sit and wait for inspiration, the rest of us just get up and go to work.” ― Stephen King

The ultimate personal productivity platform is you.

Let’s just put that on the table right up front so you know where personal productivity ultimately comes from.  It’s you.

I can’t possibly give you anything that will help you perform better than an organized mind firing on all cylinders combined with self-awareness.

You are the one that ultimately has to envision your future.  You are the one that ultimately has to focus your attention.  You are the one that ultimately needs to choose your goals.  You are the one that ultimately has to find your motivation.  You are the one that ultimately needs to manage your energy.  You are the one that ultimately needs to manage your time.  You are the one that ultimately needs to take action.  You are the one that needs to balance work and life.

That’s a lot for you to do.

So the question isn’t are you capable?  Of course you are.

The real question is, how do you make the most of you?

Agile Results is a personal productivity platform to help you make the most of what you’ve got.

Agile Results is a simple system for getting better results.  It combines proven practices for productivity, time management, and motivation into a simple system you can use to achieve better, faster, easier results for work and life.

Agile Results works by integrating and synthesizing positive psychology, sport psychology, project management skills, and peak performance insights into little behavior changes you can do each day.  It’s also based on more than 10 years of extensive trial and error to help people achieve high performance.

If you don’t know how to get started, start simple:

Ask yourself the following question:  “What are three things I want to achieve today?”

And write those down.   That’s it.

You’re doing Agile Results.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For August 28th, 2015x

Hey, it's HighScalability time:

The oldest known fossil of a flowering plant. 130 million years old. What digital will last so long?
  • 32.6: Ashley Madison password cracks per hour; 1 million: cores in the Human Brain Project's silicon brain; 54,000: tennis balls used at Wimbledon; 4 kB: size of first web page; 1.2 million: million messages per second Apache Samza performance on a single node; 27%: higher conversion for sites loading one second faster; 

  • Quotable Quotes:
    • @adrianco: Apple first read about Mesos on  and for a year have run Siri on the worlds biggest cluster 
    • @Besvinick: Interesting recurring sentiment from recent grads: We lived most of our college lives on Snapchat—now we don't have any "tangible" memories.
    • Robin Hobb: For most moments of our lives, we have forgotten almost all of the world around us, except for what currently claims our interest.
    • @Carnage4Life: I'd like to thank all the Amazon employees who cried at their desks to make this possible
Categories: Architecture

Trying out the Serenity BDD framework; a report

Xebia Blog - Fri, 08/28/2015 - 12:47

“Serenity, that feeling you know you can trust your tests.” Sounds great, but I was thinking of Firefly first when I heard the name ‘Serenity’. In this case, we are talking about a framework you can use to automate your tests.

The selling points of this framework are that it integrates your acceptance tests (BDD) with reporting and acts like living documentation. It can also integrate with JIRA and all that jazz. Hearing this, I wasn’t ‘wowed’ per se. There are many tools out there that can do that. But Serenity isn’t supporting just one approach. Although it is heavily favouring Webdriver/Selenium, you can also use JBehave, JUnit, Cucumber. That is really nice! 

Last weekend, at the Board of Agile Testers, we tried the framework with a couple of people. Our goal was to see if it’s really easy to set up, to see if the reports are useful and how easy it is to implement features. We used Serenity ‘Cucumber-style’ with Selenium/Webdriver (Java) and the Page Object Pattern.


It maybe goes a little too far to say a totally non-technical person could set up the framework, but it was pretty easy. Using your favorite IDE, all you had to do was import a Maven archetype (we used a demo project) and all the Serenity dependencies are downloaded for you. We would recommend using Java 7 at least, Java 6 gave us problems.

Using the tool

The demo project tests ran alright, but we noticed it was quite slow! The reason is probably that Serenity takes a screenshot at every step of your test. You can configure this setting, thankfully.

At the end of each test run, Serenity generates an HTML report. This report looks really good! You get a general overview and can click on each test step to see the screenshots. There is also a link to the requirements and you can see the ‘coverage’ of your tests. I’m guessing they mean functional coverage here, since we’re writing acceptance tests.

Serenity Report

Writing our own tests

After we got a little overview of the tool we started writing our own tests, using a Calculator as the System Under Test. The Serenity specific Page Object stuff comes with the Maven archetype so the IDE could help you implement the tests. We tried to do it in a little TDD cycle. Run the test, let it fail and let the output give you a hint on how to implement the step definitions. Beyond the step definition you had to use your Java skills to implement the actual tests.


The tool is pretty developer oriented, there’s no denying that. The IDE integration is very good in my opinion. With the Community IntelliJ edition you have all the plugins you need to speed up your workflow. The reporting is indeed the most beautiful I had seen, personally. Would we recommend changing your existing framework to Serenity? Unless your test reports are shit: no. There is in fact a small downside to using this framework; for now there are only about 15 people who actively contribute. You are of course allowed to join in, but it is a risk that there are only a small group of people actively improving it. If the support base grows, it will be a powerful framework for your automated tests and BDD cycle. For now, I'm going to play with the framework some more, because after using it for about 3 hours I think we only scratched the surface of its possibilities. 


Announcing Great New SQL Database Capabilities in Azure

ScottGu's Blog - Scott Guthrie - Thu, 08/27/2015 - 17:13

Today we are making available several new SQL Database capabilities in Azure that enable you to build even better cloud applications.  In particular:

  • We are introducing two new pricing tiers for our  Elastic Database Pool capability.  Elastic Database Pools enable you to run multiple, isolated and independent databases on a private pool of resources dedicated to just you and your apps.  This provides a great way for software-as-a-service (SaaS) developers to better isolate their individual customers in an economical way.
  • We are also introducing new higher-end scale options for SQL Databases that enable you to run even larger databases with significantly more compute + storage + networking resources.

Both of these additions are available to start using immediately.  Elastic Database Pools

If you are a SaaS developer with tens, hundreds, or even thousands of databases, an elastic database pool dramatically simplifies the process of creating, maintaining, and managing performance across these databases within a budget that you control. 


A common SaaS application pattern (especially for B2B SaaS apps) is for the SaaS app to use a different database to store data for each customer.  This has the benefit of isolating the data for each customer separately (and enables each customer’s data to be encrypted separately, backed-up separately, etc).  While this pattern is great from an isolation and security perspective, each database can end up having varying and unpredictable resource consumption (CPU/IO/Memory patterns), and because the peaks and valleys for each customer might be difficult to predict, it is hard to know how much resources to provision.  Developers were previously faced with two options: either over-provision database resources based on peak usage--and overpay. Or under-provision to save cost--at the expense of performance and customer satisfaction during peaks.

Microsoft created elastic database pools specifically to help developers solve this problem.  With Elastic Database Pools you can allocate a shared pool of database resources (CPU/IO/Memory), and then create and run multiple isolated databases on top of this pool.  You can set minimum and maximum performance SLA limits of your choosing for each database you add into the pool (ensuring that none of the databases unfairly impacts other databases in your pool).  Our management APIs also make it much easier to script and manage these multiple databases together, as well as optionally execute queries that span across them (useful for a variety operations).  And best of all when you add multiple databases to an Elastic Database Pool, you are able to average out the typical utilization load (because each of your customers tend to have different peaks and valleys) and end up requiring far fewer database resources (and spend less money as a result) than you would if you ran each database separately.

The below chart shows a typical example of what we see when SaaS developers take advantage of the Elastic Pool capability.  Each individual database they have has different peaks and valleys in terms of utilization.  As you combine multiple of these databases into an Elastic Pool the peaks and valleys tend to normalize out (since they often happen at different times) to require much less overall resources that you would need if each database was resourced separately:

databases sharing eDTUs

Because Elastic Database Pools are built using our SQL Database service, you also get to take advantage of all of the underlying database as a service capabilities that are built into it: 99.99% SLA, multiple-high availability replica support built-in with no extra charges, no down-time during patching, geo-replication, point-in-time recovery, TDE encryption of data, row-level security, full-text search, and much more.  The end result is a really nice database platform that provides a lot of flexibility, as well as the ability to save money.

New Basic and Premium Tiers for Elastic Database Pools

Earlier this year at the //Build conference we announced our new Elastic Database Pool support in Azure and entered public preview with the Standard Tier edition of it.  The Standard Tier allows individual databases within the elastic pool to burst up to 100 eDTUs (a DTU represents a combination of Compute + IO + Storage performance) for performance. 

Today we are adding additional Basic and Premium Elastic Database Pools to the preview to enable a wider range of performance and cost options.

  • Basic Elastic Database Pools are great for light-usage SaaS scenarios.  Basic Elastic Database Pools allows individual databases performance bursts up to 5 eDTUs.
  • Premium Elastic Database Pools are designed for databases that require the highest performance per database. Premium Elastic Database Pools allows individual database performance bursts up to 1,000 eDTUs.

Collectively we think these three Elastic Database Pool pricing tier options provide a tremendous amount of flexibility and optionality for SaaS developers to take advantage of, and are designed to enable a wide variety of different scenarios. Easily Migrate Databases Between Pricing Tiers

One of the cool capabilities we support is the ability to easily migrate an individual database between different Elastic Database Pools (including ones with different pricing tiers).  For example, if you were a SaaS developer you could start a customer out with a trial edition of your application – and choose to run the database that backs it within a Basic Elastic Database Pool to run it super cost effectively.  As the customer’s usage grows you could then auto-migrate them to a Standard database pool without customer downtime.  If the customer grows up to require a tremendous amount of resources you could then migrate them to a Premium Database Pool or run their database as a standalone SQL Database with a huge amount of resource capacity.

This provides a tremendous amount of flexibility and capability, and enables you to build even better applications. Managing Elastic Database Pools

One of the the other nice things about Elastic Database Pools is that the service provides the management capabilities to easily manage large collections of databases without you having to worry about the infrastructure that runs it.   

You can create and mange Elastic Database Pools using our Azure Management Portal or via our Command-line tools or REST Management APIs.  With today’s update we are also adding support so that you can use T-SQL to add/remove new databases to/from an elastic pool.  Today’s update also adds T-SQL support for measuring resource utilization of databases within an elastic pool – making it even easier to monitor and track utilization by database.

image Elastic Database Pool Tier Capabilities

During the preview, we have been and will continue to tune a number of parameters that control the density of Elastic Database Pools as we progress through the preview.

In particular, the current limits for the number of databases per pool and the number of pool eDTUs is something we plan to steadily increase as we march towards the general availability release.  Our plan is to provide the highest possible density per pool, largest pool sizes, and the best Elastic Database Pool economics while at the same time keeping our 99.99 availability SLA.

Below are the current performance parameters for each of the Elastic Database Pool Tier options in preview today:


Basic Elastic

Standard Elastic

Premium Elastic

Elastic Database Pool

eDTU range per pool (preview limits)

100-1200 eDTUs

100-1200 eDTUs

125-1500 eDTUs

Storage range per pool

10-120 GB

100-1200 GB

63-750 GB

Maximum database per pool (preview limits)




Estimated monthly pool and add-on  eDTU costs (preview prices)

Starting at $0.2/hr (~$149/pool/mo).

Each additional eDTU $.002/hr (~$1.49/mo)

Starting at $0.3/hr (~$223/pool mo). 

Each additional eDTU $0.003/hr (~$2.23/mo)

Starting at $0.937/hr (`$697/pool/mo).

Each additional eDTU $0.0075/hr (~$5.58/mo)

Storage per eDTU

0.1 GB per eDTU

1 GB per eDTU

.5 GB per eDTU

Elastic Databases

eDTU max per database (preview limits)




Storage max per DB

2 GB

250 GB

500 GB

Per DB cost (preview prices)

$0.0003/hr (~$0.22/mo)

$0.0017/hr (~$1.26/mo)

$0.0084/hr (~$6.25/mo)

We’ll continue to iterate on the above parameters and increase the maximum number of databases per pool as we progress through the preview, and would love your feedback as we do so.

New Higher-Scale SQL Database Performance Tiers

In addition to the enhancements for Elastic Database Pools, we are also today releasing new SQL Database Premium performance tier options for standalone databases. 

Today we are adding a new P4 (500 DTU) and a P11 (1750 DTU) level which provide even higher performance database options for SQL Databases that want to scale-up. The new P11 edition also now supports databases up to 1TB in size.

Developers can now choose from 10 different SQL Database Performance levels.  You can easily scale-up/scale-down as needed at any point without database downtime or interruption.  Each database performance tier supports a 99.99% SLA, multiple-high availability replica support built-in with no extra charges (meaning you don’t need to buy multiple instances to get an SLA – this is built-into each database), no down-time during patching, point-in-time recovery options (restore without needing a backup), TDE encryption of data, row-level security, and full-text search.


Learn More

You can learn more about SQL Databases by visiting the web-site.  Check out the SQL Database product page to learn more about the capabilities SQL Databases provide, as well as read the technical documentation to learn more how to build great applications using it.


Today’s database updates enable developers to build even better cloud applications, and to use data to make them even richer more intelligent.  We are really looking forward to seeing the solutions you build.

Hope this helps,


Categories: Architecture, Programming

7 Strategies for 10x Transformative Change

Peter Thiel, VC, PayPal co-founder, early Facebook investor, and most importantly, the supposed inspiration for Silicon Valley's intriguing Peter Gregory character, argues in his book Zero to One that a successful business needs to make a product that is 10 times better than its closest competitor

The title Zero to One refers to the idea of progress as either horizontal/extensive or vertical/intensive. For a more detailed explanation take a look at Peter Thiel's CS183: Startup - Class 1 Notes Essay.

Horizontal/extensive progress refers to copying things that work. Observe, imitate, and repeat.  The one word summary for the concept is  "globalization.” For more on this PAYPAL MAFIA: Reid Hoffman & Peter Thiel's Master Class in China is an interesting watch.

Vertical/intensive progress means doing something genuinely new, that is going from zero to one, as apposed to going from one to N, which is merely globalization. This is the creative spark. The hero's journey of over coming obstacles on the way to becoming the Master of the Universe you were always meant to be.

We see this pattern with Google a lot. Google often hits scaling challenges long before anyone else and because they have a systematizing culture they produce discrete replicatable technologies that then diffuse out to the rest of the world, often through open source efforts.

Google told us about the Google File System in 2003, MapReduce in 2004, Bigtable in 2006, The Datacenter as a Computer in 2009, Percolator (real-time updates) in 2010, Pregel (graph processing) in 2010, Dremel (interactive analysis) in 2010, Spanner (globally distributed database) in 2012,  Omega (cluster scheduling) in 2013, Borg (cluster manager) in 2015, and Jupiter Rising (advanced networking) in 2015.

Sometime later we've seen the development of open source parallels like HDFS, Hadoop, HBase, Giraph, YARN, Drill, and Mesos. 

So, how can you rise up and meet the 10x challenge?

Murat Demirbas, a computer science and engineering professor at SUNY Buffalo, and awesome writer on all things distributed, came up with some good suggestions in How to go for 10X

Categories: Architecture

Help, my diagram doesn't fit on one page!

Coding the Architecture - Simon Brown - Tue, 08/25/2015 - 17:16

This definitely goes into the category of a frequently asked question because it crops up time and time again, both during and after my software architecture sketching workshop.

I'm following your C4 approach but my software system is much bigger than the example you use in your workshop. How do you deal with the complexity of these larger systems? And what happens when my diagram doesn't fit on one page?

Even with a relatively small software system, it's tempting to try and include the entire story on a single diagram. For example, if you have a web application, it seems logical to create a single component diagram that shows all of the components that make up that web application. Unless your software system really is that small, you're likely to run out of room on the diagram canvas or find it difficult to discover a layout that isn't cluttered by a myriad of overlapping lines. Using a larger diagram canvas can sometimes help, but large diagrams are usually hard to interpret and comprehend because the cognitive load is too high. And if nobody understands the diagram, nobody is going to look at it.

Instead, don't be afraid to split that single complex diagram into a larger number of simpler diagrams, each with a specific focus around a business area, functional area, functional grouping, bounded context, use case, user interaction, feature set, etc. You can see an example of this in One view or many?, where I create one component diagram per web MVC controller rather than having a single diagram showing all components. You can see this in action in the software architecture diagrams for that are hosted on Structurizr. The key is to ensure that each of the separate diagrams tells a different part of the same overall story, at the same level of abstraction.

Categories: Architecture

Ask HighScalability: Choose an Async App Server or Multiple Blocking Servers?

Jonathan Willis, software developer by day and superhero by night, asked an interesting question via Twitter on StackOverflow

tl;dr Many Rails apps or one Vertx/Play! app?

I've been having discussions with other members of my team on the pros and cons of using an async app server such as the Play! Framework (built on Netty) versus spinning up multiple instances of a Rails app server. I know that Netty is asynchronous/non-blocking, meaning during a database query, network request, or something similar an async call will allow the event loop thread to switch from the blocked request to another request ready to be processed/served. This will keep the CPUs busy instead of blocking and waiting.

I'm arguing in favor or using something such as the Play! Framework or, something that is non-blocking... Scalable. My team members, on the other hand, are saying that you can get the same benefit by using multiple instances of a Rails app, which out of the box only comes with one thread and doesn't have true concurrency as do apps on the JVM. They are saying just use enough app instances to match the performance of one Play! application (or however many Play! apps we use), and when a Rails app blocks the OS will switch processes to a different Rails app. In the end, they are saying that the CPUs will be doing the same amount of work and we will get the same performance.

What do you think? The marketplace has seemingly moved, in the form of node.js, Golang, Akka, and even Java, to the async server model. Does that mean it's the only right way?

Here's my attempt at a response:

Categories: Architecture

Every Employee is a Digital Employee

“The questions that we must ask ourselves, and that our historians and our children will ask of us, are these: How will what we create compare with what we inherited? Will we add to our tradition or will we subtract from it? Will we enrich it or will we deplete it?”
― Leon Wieseltier

Digital transformation is all around us.

And we are all digital employees according to Gartner.

In the article, Gartner Says Every Employee Is a Digital Employee, Gartner says that the IT function no longer holds a monopoly on IT.

A Greater Degree of Digital Dexterity

According to Gartner, employees are creating increasing digital dexterity from the devices and apps they use, to participating in sharing economies.

Via Gartner Says Every Employee Is a Digital Employee:

"'Today's employees possess a greater degree of digital dexterity,' said Matt Cain, research vice president at Gartner. 'They operate their own wireless networks at home, attach and manage various devices, and use apps and Web services in almost every facet of their personal lives. They participate in sharing economies for transport, lodging and more.'"

Workers are Streamlining Their Work Life

More employees are using technology to simplify, streamline, and scale their work.

Via Gartner Says Every Employee Is a Digital Employee:

"This results in unprecedented numbers of workers who enjoy using technology and recognize the relevance of digitalization to a wide range of business models. They also routinely apply their own technology and technological knowledge to streamline their work life."

3 Ways to Exploit Digital Dexterity

According to Gartner, there are 3 Ways the IT organization should exploit employees' digital dexterity:

  1. Implement a digital workplace strategy
  2. Embrace shadow IT
  3. Use a bimodal approach
1. Implement a Digital Workplace Strategy

While it’s happening organically, IT can also help shape the digital workplace experience.  Implement a strategy that helps workers use computing resources in a more friction free way and that play better with their pains, needs, and desired outcomes.

Via Gartner Says Every Employee Is a Digital Employee:

“Making computing resources more accessible in ways that match employees' preferences will foster engagement by providing feelings of empowerment and ownership. The digital workplace strategy should therefore complement HR initiatives by addressing and improving factors such as workplace culture, autonomous decision making, work-life balance, recognition of contributions and personal growth opportunities.”

2. Embrace shadow IT

Treat shadow IT as a first class citizen.  IT should partner with the business to help the business realize it’s potential, and to help workers make the most of the available IT resources.

Via Gartner Says Every Employee Is a Digital Employee:

“Rather than try to fight the tide, the IT organization should develop a framework that outlines when it is appropriate for business units and individuals to use their own technology solutions and when IT should take the lead. IT should position itself as a business partner and consultant that does not control all technology decisions in the business.”

3. Use a bimodal approach

Traditional IT is slow.   It’s heavy in governance, standards, and procedures.   It addresses risk by reducing flexibility.   Meanwhile, the world is changing fast.  Business needs to keep up.  Business needs fast IT. 

So what’s the solution?

Bimodal IT.  Bimodal IT separates the fast demands of digital business from the slow/risk-averse methods of traditional IT.

Via Gartner Says Every Employee Is a Digital Employee:

“Bimodal IT separates the risk-averse and ‘slow’ methods of traditional IT from the fast-paced demands of digital business, which is underpinned by the digital workplace. This dual mode of operation is essential to satisfy the ever-increasing demands of digitally savvy business units and employees, while ensuring that critical IT infrastructure and services remain stable and uncompromised.”

Everyone has technology at their fingertips.  Every worker has the chance to re-imagine their work in a Mobile-First, Cloud-First world. 

With infinite compute, infinite capacity, global reach, and real-time insights available to you, how could you evolve your job?

You can evolve your digital work life right under your feet.

You Might Also Like

Empower Every Person on the Planet to Achieve More

Satya Nadella on a Mobile-First, Cloud-First World

We Help Our Customers Transform

Categories: Architecture, Programming

What Life is Like with Agile Results

“Courage doesn't always roar. Sometimes courage is the little voice at the end of the day that says I'll try again tomorrow.” -- Mary Anne Radmacher

Imagine if you could wake up productive, where each day is a fresh start.  As you take in your morning breath, you notice your mind is calm and clear.

You feel strong and well rested.

Before you start your day, you picture in your mind three simple scenes of the day ahead:

In the morning, you see yourself complete a draft you’ve been working on.

In the afternoon, you see yourself land your idea and win over your peers in a key meeting.

In the evening, you see yourself enjoying some quiet time as you sit down and explore your latest adventures in learning.

With an exciting day ahead, and a chance to rise and shine, you feel the day gently pull you forward with anticipation. 

You know you’ll be tested, and you know some things won’t work out as planned.   But you also know that you will learn and improve from every setback.  You know that each challenge you face will be a leadership moment or a learning opportunity.  Your challenges make you stronger.

And you also know that you will be spending as much time in your strengths as you can, and that helps keeps you strong, all day long. 

You motivate yourself from the inside out by focusing on your vision for today and your values.  You value achievement.  You value learning.  You value collaboration.  You value excellence.  You value empowerment.   And you know that throughout the day, you will have every chance to apply your skills to do more, to achieve more, and to be more. 

Each task, or each challenge, is also a chance to learn more.  From yourself, and from everyone all around you.  And this is how you never stop learning.

You may not like some of the tasks before you, but you like the chance to master your craft.  And you enjoy the learning.  And you love how you get better.  With each task on your To-Do list for today, you experiment and explore ways to do things better, faster, and easier.

Like a productive artist, you find ways to add unique value.   You add your personal twist to everything you do.  Your twist comes from your unique experience, seeing what others can’t see from your unique vantage point, and applying your unique strengths.

And that’s how you do more art.  Your art.  And as you do your art, you feel yourself come alive.  You feel your soul sing, as you operate at a higher level.  As you find your flow and realize your potential, your inner-wisdom winks in an approving way.  Like a garden in full bloom on a warm Summer’s day, you are living your arête.

As your work day comes to an end, you pause to reflect on your three achievements, your three wins, for the day.   You appreciate the way you leaned in on the tough stuff.  You surprised yourself in how you handled some of your most frustrating moments.  And you learned a new way to do your most challenging task.  You take note of the favorite parts of your day, and your attitude of gratitude feels you with a sense of accomplishment, and a sense of fulfillment.

Fresh and ready for anything, you head for home.

Try 30 Days of Getting Results.  It’s free. Surprise yourself with what you’re capable of.

Categories: Architecture, Programming

HTTP/2 Server Push

Xebia Blog - Sun, 08/23/2015 - 17:11

The HTTP/2 standard was finalized in May 2015. Most major browsers support it, and Google uses it heavily.

HTTP/2 leaves the basic concepts of Requests, Responses and Headers intact. Changes are mostly at the transport level, improving the performance of parallel requests - with few changes to your application. The go HTTP/2 'gophertiles' demo nicely demonstrates this effect.

A new concept in HTTP/2 is Server Push, which allows the server to speculatively start sending resources to the client. This can potentially speed up initial page load times: the browser doesn't have to parse the HTML page and find out which other resources to load, instead the server can start sending them immediately.

This article will demonstrate how Server Push affects the load time of the 'gophertiles'.

HTTP/2 in a nutshell

The key characteristic of HTTP/2 is that all requests for a server are sent over one TCP connection, and responses can come in parallel over that connection.

Using only one connection reduces overhead caused by TCP and TLS handshakes. Allowing responses to be sent in parallel is an improvement over HTTP/1.1 pipelining, which only allows requests to be served sequentially.

Additionally, because all requests are sent over one connection, there is a Header Compression mechanism that reduces the bandwidth needed for headers that previously would have had to be repeated for each request.

Server Push

Server Push allows the server to preemptively send a 'request promise' and an accompanying response to the client.

The most obvious use case for this technology is sending resources like images, CSS and JavaScript along with the page that includes them. Traditionally, the browser would have to first fetch the HTML, parse it, and then make subsequent requests for other resources. As the server can fairly accurately predict which resources a client will need, with Server Push it does not have to wait for those requests and can begin sending the resources immediately.

Of course sometimes, you really do only want to fetch the HTML and not the accompanying resources. There are 2 ways to accomplish this: the client can specify it does not want to receive any pushed resources at all, or cancel an individual push after receiving the push promise. In the latter case the client cannot prevent the browser from initiating the Push, though, so some bandwidth may have been wasted. This will make deciding whether to use Server Push for resources that might already have been cached by the browser subtle.


To show the effect HTTP/2 Server Push can have, I have extended the gophertiles demo to be able to test behavior with and without Server Push, available hosted on an old raspberry pi.

Both the latency of loading the HTML and the latency of loading each tile is now artificially increased.

When visiting the page without Server Push with an artificial latency of 1000ms, you will notice that loading the HTML takes at least one second, and then loading all images in parallel again takes at least one second - so rendering the complete page takes well above 2 seconds.

With server push enabled, you will see that after the DOM has loaded, the images are almost immediately there, because they have been Push'ed already.

All that glitters, however, is not gold: as you will notice when experimenting (especially at lower artificial latencies), while Server Push fairly reliably reduces the complete load time, it sometimes increases the time until the DOM-content is loaded. While this makes sense (the browser needs to process frames relating to the Server Push'ed resources), this could have an impact on the perceived performance of your page: for example, it could delay running JavaScript code embedded in your site.

HTTP/2 does give you tools to tune this, such as Stream Priorities, but this might need careful tuning and be supported by the http2 library you are choosing.


HTTP/2 is here today, and can provide a considerable improvement in perceived performance - even with few changes in your application.

Server Push potentially allows you to improve your page loading times even further, but requires careful analysis and tuning - otherwise it might even have an adverse effect.

Stuff The Internet Says On Scalability For August 21st, 2015

Hey, it's HighScalability time:

Hunter-Seeker? Nope. This is the beauty of what a Google driverless car sees. Great TED talk.
  • $2.8 billion: projected Instagram ad revenue in 2017; 1 trillion: Azure event hub events per month; 10 million: Stack Overflow questions asked; 1 billion: max volts generated by a lightening strike; 850: apps downloaded every second from the AppStore; 2000: years data can be stored in DNA; 60: # of robots needed to replace 600 humans; 1 million: queries per second with Nginx, Ubuntu, EC2

  • Quotable Quotes:
    • Tales from the Lunar Module Guidance Computer: we landed on the moon with 152 Kbytes of onboard computer memory.
    • @ijuma: Included in JDK 8 update 60 "changes GHASH internals from using byte[] to long, improving performance about 10x
    • @ErrataRob: I love the whining over the Bitcoin XT fork. It's as if anarchists/libertarians don't understand what anarchy/libertarianism means.
    • Network World: the LHC Computing Grid has 132,992 physical CPUs, 553,611 logical CPUs, 300PB of online disk storage and 230PB of nearline (magnetic tape) storage. It's a staggering amount of processing capacity and data storage that relies on having no single point of failure.
    • @petereisentraut: Chef is kind of a distributed monkey-patching festival running as root.
    • @SciencePorn: If you were to remove all of the empty space from the atoms that make up every human on earth, all humans would fit into an apple.
    • SDN for the cloud: Most of the concepts presented in the papers have been put into practice in Microsoft cloud infrastructures. As a result of these improvements, modern Azure services can carry up to 1,400,000 SQL databases. Moreover, a typical Azure event hub sees as high as 1 trillion events per month.

  • On the Alphabet Google reorg...what Horace Dediu has to say on functional vs divisional organizations may provide insight. A functional organization, which is used by the Army and Apple, prevents cross divisional fights for resources and power. Those are the kind of internal politics that kill a company. Why not just sidestep all that?

  • Here's how Pinterest shards MySQL to scale: All data needed to be replicated to a slave machine for backup, with high availability and dumping to S3 for MapReduce...You never want to read/write to a slave in production...Slaves lag, which causes strange bugs; I still recommend startups avoid the fancy new stuff — try really hard to just use MySQL. Trust me. I have the scars to prove it...We created a 64 bit ID that contains the shard ID...To create a new Pin, we gather all the data and create a JSON blob...A mapping table links one object to another...there are three primary ways to add more capacity...more up new ranges...move some shards to new machines...This system is best effort. It does not give you Atomicity, Isolation or Consistency in all cases...We stored the shard configuration table in ZooKeeper...This system has been in production at Pinterest for 3.5 years now and will likely be in there forever. 

  • Nobody expects the quadruple fault! Google loses data as lightning strikes: four successive lightning strikes on the local utilities grid that powers our European datacenter caused a brief loss of power to storage systems...only a very small number of disks remained affected, totalling less than 0.000001% of the space of allocated persistent disks...full recovery is not possible.

  • Flxone upgraded to Go version 1.5 and reduced their 95-percentile garbage collector from 279 milliseconds down to just 10 ms, a 96% decrease in garbage collection pause time. Average request latency dropped by 53%. I wonder now if they can reduce the number of nodes required to meet their SLA? And would the results hold if they wrote their app more natively, that is to generate garbage?

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Announcing Windows Server 2016 Containers Preview

ScottGu's Blog - Scott Guthrie - Wed, 08/19/2015 - 17:01

At DockerCon this year, Mark Russinovich, CTO of Microsoft Azure, demonstrated the first ever application built using code running in both a Windows Server Container and a Linux container connected together. This demo helped demonstrate Microsoft's vision that in partnership with Docker, we can help bring the Windows and Linux ecosystems together by enabling developers to build container-based distributed applications using the tools and platforms of their choice.

Today we are excited to release the first preview of Windows Server Containers as part of our Windows Server 2016 Technical Preview 3 release. We’re also announcing great updates from our close collaboration with Docker, including enabling support for the Windows platform in the Docker Engine and a preview of the Docker Engine for Windows. Our Visual Studio Tools for Docker, which we previewed earlier this year, have also been updated to support Windows Server Containers, providing you a seamless end-to-end experience straight from Visual Studio to develop and deploy code to both Windows Server and Linux containers. Last but not least, we’ve made it easy to get started with Windows Server Containers in Azure via a dedicated virtual machine image. Windows Server Containers

Windows Server Containers create a highly agile Windows Server environment, enabling you to accelerate the DevOps process to efficiently build and deploy modern applications. With today’s preview release, millions of Windows developers will be able to experience the benefits of containers for the first time using the languages of their choice – whether .NET, ASP.NET, PowerShell or Python, Ruby on Rails, Java and many others.

Today’s announcement delivers on the promise we made in partnership with Docker, the fast-growing open platform for distributed applications, to offer container and DevOps benefits to Linux and Windows Server users alike. Windows Server Containers are now part of the Docker open source project, and Microsoft is a founding member of the Open Container Initiative. Windows Server Containers can be deployed and managed either using the Docker client or PowerShell. Getting Started using Visual Studio

The preview of our Visual Studio Tools for Docker, which enables developers to build and publish ASP.NET 5 Web Apps or console applications directly to a Docker container, has been updated to include support for today’s preview of Windows Server Containers. The extension automates creating and configuring your container host in Azure, building a container image which includes your application, and publishing it directly to your container host. You can download and install this extension, and read more about it, at the Visual Studio Gallery here:

Once installed, developers can right-click on their projects within Visual Studio and select “Publish”:


Doing so will display a Publish dialog which will now include the ability to deploy to a Docker Container (on either a Windows Server or Linux machine):


You can choose to deploy to any existing Docker host you already have running:


Or use the dialog to create a new Virtual Machine running either Window Server or Linux with containers enabled.  The below screen-shot shows how easy it is to create a new VM hosted on Azure that runs today’s Windows Server 2016 TP3 preview that supports Containers – you can do all of this (and deploy your apps to it) easily without ever having to leave the Visual Studio IDE:

image Getting Started Using Azure

In June of last year, at the first DockerCon, we enabled a streamlined Azure experience for creating and managing Docker hosts in the cloud. Up until now these hosts have only run on Linux. With the new preview of Windows Server 2016 supporting Windows Server Containers, we have enabled a parallel experience for Windows users.

Directly from the Azure Marketplace, users can now deploy a Windows Server 2016 virtual machine pre-configured with the container feature enabled and Docker Engine installed. Our quick start guide has all of the details including screen shots and a walkthrough video so take a look here


Once your container host is up and running, the quick start guide includes step by step guides for creating and managing containers using both Docker and PowerShell. Getting Started Locally Using Hyper-V

Creating a virtual machine on your local machine using Hyper-V to act as your container host is now really easy. We’ve published some PowerShell scripts to GitHub that automate nearly the whole process so that you can get started experimenting with Windows Server Containers as quickly as possible. The quick start guide has all of the details at

Once your container host is up and running the quick start guide includes step by step guides for creating and managing containers using both Docker and PowerShell.

image Additional Information and Resources

A great list of resources including links to past presentations on containers, blogs and samples can be found in the community section of our documentation. We have also setup a dedicated Windows containers forum where you can provide feedback, ask questions and report bugs. If you want to learn more about the technology behind containers I would highly recommend reading Mark Russinovich’s blog on “Containers: Docker, Windows and Trends” that was published earlier this week. Summary

At the //Build conference earlier this year we talked about our plan to make containers a fundamental part of our application platform, and today’s releases are a set of significant steps in making this a reality.’ The decision we made to embrace Docker and the Docker ecosystem to enable this in both Azure and Windows Server has generated a lot of positive feedback and we are just getting started.

While there is still more work to be done, now users in the Window Server ecosystem can begin experiencing the world of containers. I highly recommend you download the Visual Studio Tools for Docker, create a Windows Container host in Azure or locally, and try out our PowerShell and Docker support. Most importantly, we look forward to hearing feedback on your experience.

Hope this helps,

Scott omni

Categories: Architecture, Programming

The Microsoft Take on Containers and Docker

This is a guest repost by Mark Russinovich, CTO of Microsoft Azure (and novelist!). We all benefit from a vibrant competitive cloud market and Microsoft is part of that mix. Here's a good container overview along with Microsoft's plan of attack. Do you like their story? Is it interesting? Is it compelling?

You can’t have a discussion on cloud computing lately without talking about containers. Organizations across all business segments, from banks and major financial service firms to e-commerce sites, want to understand what containers are, what they mean for applications in the cloud, and how to best use them for their specific development and IT operations scenarios.

From the basics of what containers are and how they work, to the scenarios they’re being most widely used for today, to emerging trends supporting “containerization”, I thought I’d share my perspectives to better help you understand how to best embrace this important cloud computing development to more seamlessly build, test, deploy and manage your cloud applications.

Containers Overview

In abstract terms, all of computing is based upon running some “function” on a set of “physical” resources, like processor, memory, disk, network, etc., to accomplish a task, whether a simple math calculation, like 1+1, or a complex application spanning multiple machines, like Exchange. Over time, as the physical resources became more and more powerful, often the applications did not utilize even a fraction of the resources provided by the physical machine. Thus “virtual” resources were created to simulate underlying physical hardware, enabling multiple applications to run concurrently – each utilizing fractions of the physical resources of the same physical machine.

We commonly refer to these simulation techniques as virtualization. While many people immediately think virtual machines when they hear virtualization, that is only one implementation of virtualization. Virtual memory, a mechanism implemented by all general purpose operating systems (OSs), gives applications the illusion that a computer’s memory is dedicated to them and can even give an application the experience of having access to much more RAM than the computer has available.

Containers are another type of virtualization, also referred to as OS Virtualization. Today’s containers on Linux create the perception of a fully isolated and independent OS to the application. To the running container, the local disk looks like a pristine copy of the OS files, the memory appears only to hold files and data of a freshly-booted OS, and the only thing running is the OS. To accomplish this, the “host” machine that creates a container does some clever things.

The first technique is namespace isolation. Namespaces include all the resources that an application can interact with, including files, network ports and the list of running processes. Namespace isolation enables the host to give each container a virtualized namespace that includes only the resources that it should see. With this restricted view, a container can’t access files not included in its virtualized namespace regardless of their permissions because it simply can’t see them. Nor can it list or interact with applications that are not part of the container, which fools it into believing that it’s the only application running on the system when there may be dozens or hundreds of others.

For efficiency, many of the OS files, directories and running services are shared between containers and projected into each container’s namespace. Only when an application makes changes to its containers, for example by modifying an existing file or creating a new one, does the container get distinct copies from the underlying host OS – but only of those portions changed, using Docker’s “copy-on-write” optimization. This sharing is part of what makes deploying multiple containers on a single host extremely efficient.

Second, the host controls how much of the host’s resources can be used by a container. Governing resources like CPU, RAM and network bandwidth ensure that a container gets the resources it expects and that it doesn’t impact the performance of other containers running on the host. For example, a container can be constrained so that it cannot use more than 10% of the CPU. That means that even if the application within it tries, it can’t access to the other 90%, which the host can assign to other containers or for its own use. Linux implements such governance using a technology called “cgroups.” Resource governance isn’t required in cases where containers placed on the same host are cooperative, allowing for standard OS dynamic resource assignment that adapts to changing demands of application code.

The combination of instant startup that comes from OS virtualization and reliable execution that comes from namespace isolation and resource governance makes containers ideal for application development and testing. During the development process, developers can quickly iterate. Because its environment and resource usage are consistent across systems, a containerized application that works on a developer’s system will work the same way on a different production system. The instant-start and small footprint also benefits cloud scenarios, since applications can scale-out quickly and many more application instances can fit onto a machine than if they were each in a VM, maximizing resource utilization.

Comparing a similar scenario that uses virtual machines with one that uses containers highlights the efficiency gained by the sharing. In the example shown below, the host machine has three VMs. In order to provide the applications in the VMs complete isolation, they each have their own copies of OS files, libraries and application code, along with a full in-memory instance of an OS. Starting a new VM requires booting another instance of the OS, even if the host or existing VMs already have running instances of the same version, and loading the application libraries into memory. Each application VM pays the cost of the OS boot and the in-memory footprint for its own private copies, which also limits the number of application instances (VMs) that can run on the host.

App Instances on Host

The figure below shows the same scenario with containers. Here, containers simply share the host operating system, including the kernel and libraries, so they don’t need to boot an OS, load libraries or pay a private memory cost for those files. The only incremental space they take is any memory and disk space necessary for the application to run in the container. While the application’s environment feels like a dedicated OS, the application deploys just like it would onto a dedicated host. The containerized application starts in seconds and many more instances of the application can fit onto the machine than in the VM case.

Containers on Host

Docker’s Appeal
Categories: Architecture

Release Burn Down Brought to Life

Xebia Blog - Tue, 08/18/2015 - 23:48

Inspired by the blog of Mike Cohn [Coh08] "Improving On Traditional Release Burndown Charts" I created a time lapsed version of it. It also nicely demonstrates that forecasts of "What will be finished?" (at a certain time) get better as the project progresses.

The improved traditional release burn down chart clearly show what (a) is finished (light green), (b) what will very likely be finished (dark green), (c) what will perhaps be finished, and perhaps not (orange), and (d) what almost is guaranteed not to be finished (red).

This knowledge supports product owners in ordering the backlog based on the current knowledge.


The result is obtained doing a Monte Carlo simulation of a toy project, using a fixed product backlog of around 100 backlog items with various sized items. The amount of work realized also varies per projectday based on a simple uniform probability distribution.

Forecasting is done using a 'worst' velocity and a 'best' velocity. Both are determined using only the last 3 velocities, i.e. only the last 3 sprints are considered.

The 2 grey lines represent the height of the orange part of the backlog, i.e. the backlog items that might be or not be finished. This also indicates the uncertainty over time of what actually will be delivered by the team at the given time.


The Making Of...

The movie above has been created using GNU plot [GNU Plot] for drawing the charts, and ffmpeg [ffmpeg] has been used to creat the time lapsed movie from the set of charts.


Over time the difference between the 2 grey lines gets smaller, a clear indication of improving predictability and reduction of risk. Also, the movie shows that the final set of backlog items done is well between the 2 grey lines from the start of the project.

This looks very similar to the 'Cone of Uncertainty'. Besides that the shape of the grey lines only remotely resembles a cone, another difference is that the above simulation merely takes statistical chances into account. The fact that the team gains more knowledge and insight over time, is not considered in the simulation, whereas it is an important factor in the 'Cone of Uncertainty'.


[Coh08] "Improving On Traditional Release Burndown Charts", Mike Cohn, June 2008,

[GNU Plot] Gnu plot version 5.0, "A portable command-line driven graphing utility",

[ffmpeg] "A complete, cross-platform solution to record, convert and stream audio and video",

Sponsored Post: Surge, Redis Labs,, VoltDB, Datadog, MongoDB, SignalFx, InMemory.Net, Couchbase, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • VoltDB's in-memory SQL database combines streaming analytics with transaction processing in a single, horizontal scale-out platform. Customers use VoltDB to build applications that process streaming data the instant it arrives to make immediate, per-event, context-aware decisions. If you want to join our ground-breaking engineering team and make a real impact, apply here.  

  • At Scalyr, we're analyzing multi-gigabyte server logs in a fraction of a second. That requires serious innovation in every part of the technology stack, from frontend to backend. Help us push the envelope on low-latency browser applications, high-speed data processing, and reliable distributed systems. Help extract meaningful data from live servers and present it to users in meaningful ways. At Scalyr, you’ll learn new things, and invent a few of your own. Learn more and apply.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Surge 2015. Want to mingle with some of the leading practitioners in the scalability, performance, and web operations space? Looking for a conference that isn't just about pitching you highly polished success stories, but that actually puts an emphasis on learning from real world experiences, including failures? Surge is the conference for you.

  • Your event could be here. How cool is that?
Cool Products and Services
  • MongoDB Management Made Easy. Gain confidence in your backup strategy. MongoDB Cloud Manager makes protecting your mission critical data easy, without the need for custom backup scripts and storage. Start your 30 day free trial today.

  • In a recent benchmark for NoSQL databases on the AWS cloud, Redis Labs Enterprise Cluster's performance had obliterated Couchbase, Cassandra and Aerospike in this real life, write-intensive use case. Full backstage pass and and all the juicy details are available in this downloadable report.

  • Real-time correlation across your logs, metrics and events. just released its operations data hub into beta and we are already streaming in billions of log, metric and event data points each day. Using our streaming analytics platform, you can get real-time monitoring of your application performance, deep troubleshooting, and even product analytics. We allow you to easily aggregate logs and metrics by micro-service, calculate percentiles and moving window averages, forecast anomalies, and create interactive views for your whole organization. Try it for free, at any scale.

  • In a recent benchmark conducted on Google Compute Engine, Couchbase Server 3.0 outperformed Cassandra by 6x in resource efficiency and price/performance. The benchmark sustained over 1 million writes per second using only one-sixth as many nodes and one-third as many cores as Cassandra, resulting in 83% lower cost than Cassandra. Download Now.

  • Datadog is a monitoring service for scaling cloud infrastructures that bridges together data from servers, databases, apps and other tools. Datadog provides Dev and Ops teams with insights from their cloud environments that keep applications running smoothly. Datadog is available for a 14 day free trial at

  • Turn chaotic logs and metrics into actionable data. Scalyr replaces all your tools for monitoring and analyzing logs and system metrics. Imagine being able to pinpoint and resolve operations issues without juggling multiple tools and tabs. Get visibility into your production systems: log aggregation, server metrics, monitoring, intelligent alerting, dashboards, and more. Trusted by companies like Codecademy and InsideSales. Learn more and get started with an easy 2-minute setup. Or see how Scalyr is different if you're looking for a Splunk alternative or Sumo Logic alternative.

  • SignalFx: just launched an advanced monitoring platform for modern applications that's already processing 10s of billions of data points per day. SignalFx lets you create custom analytics pipelines on metrics data collected from thousands or more sources to create meaningful aggregations--such as percentiles, moving averages and growth rates--within seconds of receiving data. Start a free 30-day trial!

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex goes beyond monitoring and measures the system's work on your MySQL and PostgreSQL servers, providing unparalleled insight and query-level analysis. This unique approach ultimately enables your team to work more effectively, ship more often, and delight more customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here:

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Iterables, Iterators and Generator functions in ES2015

Xebia Blog - Mon, 08/17/2015 - 20:45

ES2015 adds a lot of new features to javascript that make a number of powerful constructs, present in other languages for years, available in the browser (well as soon as support for those features is rolled out of course, but in the meantime we can use these features by using a transpiler such as Babeljs or Traceur).
Some of the more complicated additions are the iterator and iterable protocols and generator functions. In this post I'll explain what they are and what you can use them for.

The Iterable and Iterator protocols

These protocols are analogues to, for example, the Java Interfaces and define the contract that an object must adhere to in order to be considered an iterable or an iterator. So instead of new language features they leverage existing constructs by agreeing upon a convention (as javascript does not have a concept like an interface in other typed languages). Let's have a closer look at these protocols and see how they interact with each other.

The Iterable protocol

This protocol specifies that for an object object to be considered iterable (and usable in, for example, a `for ... of` loop) it has to define a function with the special key "Symbol.iterator" that returns an object that adheres to the iterator protocol. That is basically the only requirement. For example say you have a datastructure you want to iterate over, in ES2015 you would do that as follows:

class DataStructure {
  constructor(data) { = data;

  [Symbol.iterator]() {
    let current = 0
    let data =;
    return  {
      next: function () {
        return {
          value: data[current++],
          done: current > data.length
let ds = new DataStructure(['hello', 'world']);

console.log([...ds]) // ["hello","world"]

The big advantage of using the iterable protocol over using another construct like `for ... in` is that you have more clearly defined iteration semantic (for example: you do not need explicit hasOwnProperty checking when iterating over an array to filter out properties on the array object but not in the array). Another advantage is the when using generator functions you can benefit of lazy evaluation (more on generator functions later).

The iterator protocol

As mentioned before, the only requirement for the iterable protocol is for the object to define a function that returns an iterator. But what defines an iterator?
In order for an object to be considered an iterator it must provided a method named `next` that returns an object with 2 properties:
* `value`: The actual value of the iterable that is being iterated. This is only valid when done is `false`
* `done`: `false` when `value` is an actual value, `true` when the iterator did not produce a new value

Note that when you provided a `value` you can leave out the `done` property and when the `done` property is `true` you can leave out the value property.

The object returned by the function bound to the DataStructure's `Symbol.iterator` property in the previous example does this by returning the entry from the array as the value property and returning `done: false` while there are still entries in the data array.

So by simply implementing both these protocols you can turn any `Class` (or `object` for that matter) into an object you can iterate over.
A number of built-ins in ES2015 already implement these protocols so you can experiment with the protocol right away. You can already iterate over Strings, Arrays, TypedArrays, Maps and Sets.

Generator functions

As shown in the earlier example implementing the iterable and iterator protocols manually can be quite a hassle and is error-prone. That is why a language feature was added to ES2015: generator functions. A generator combines both an iterable and an iterator in a single function definition. A generator function is declared by adding an asterisk (`*`) to the function name and using yield to return values. Big advantage of using this method is that your generator function will return an iterator that, when its `next()` method is invoked will run up to the first yield statement it encounters and will suspend execution until `next()` is called again (after which it will resume and run until the next yield statement). This allows us to write an iteration that is evaluated lazily instead of all at once.

The following example re-implements the iterable and iterator using a generator function producing the same result, but with a more concise syntax.

class DataStructure {
  constructor(data) { = data;

  *[Symbol.iterator] () {
    let data =;
    for (let entry of data) {
      yield entry;
let ds = new DataStructure(['hello', 'world']);

console.log([...ds]) // ["hello","world"]
More complex usages of generators

As mentioned earlier, generator functions allow for lazy evaluation of (possibly) infinite iterations allowing to use constructs known from more functional languages such as taking a limited subset from an infinte sequence:

function* generator() {
  let i = 0;
  while (true) {
    yield i++;

function* take(number, gen) {
  let current = 0;
  for (let result of gen) {
    yield result;
    if (current++ >= number) {
console.log([...take(10, generator())]) // [0,1,2,3,4,5,6,7,8,9]
console.log([...take(10, [1,2,3])]) // [1,2,3]

Delegating generators
Within a generator it is possible to delegate to a second generator making it possible to create recursive iteration structures. The following example demonstrates a simple generator delegating to a sub generator and returning to the main generator.

function* generator() {
  yield 1;
  yield* subGenerator()
  yield 4;

function* subGenerator() {
  yield 2;
  yield 3;

console.log([...generator()]) // [1,2,3,4]

30 Day Sprints for Personal Development: Change Yourself with Skill

"What lies behind us and what lies before us are small matters compared to what lies within us. And when we bring what is within us out into the world, miracles happen." -- Ralph Waldo Emerson

I've written about 30 Day Sprints before, but it's time to talk about them again:

30 Day Sprints help you change yourself with skill.

Once upon a time, I found that when I was learning a new skill, or changing a habit, or trying something new, I wasn't getting over that first humps, or making enough progress to stick with it.

At the same time, I would get distracted by shiny new objects.  Because I like to learn and try new things, I would start something else, and ditch whatever else I was trying to work on, to pursuit my new interest.  So I was hopping from thing to thing, without much to show for it, or getting much better.

I decided to stick with something for 30 days to see if it would make a difference.  It was my personal 30 day challenge.  And it worked.   What I found was that sticking with something past two weeks, got me past those initial hurdles.  Those dips that sit just in front of where breakthroughs happen.

All I did was spend a little effort each day for 30 days.  I would try to learn a new insight or try something small each day.  Each day, it wasn't much.  But over 30 days, it accumulated.  And over 30 days, the little effort added up to a big victory.

Why 30 Day Sprints Work So Well

Eventually, I realized why 30 Day Sprints work so well.  You effectively stack things in your favor.  By investing in something for a month, you can change how you approach things.  It's a very different mindset when you are looking at your overall gain over 30 days versus worrying about whether today or tomorrow gave you immediate return on your time.  By taking a longer term view, you give yourself more room to experiment and learn in the process.

  1. 30 Day Sprints let you chip away at the stone.  Rather than go big bang or whole hog up front, you can chip away at it.  This takes the pressure off of you.  You don't have to make a breakthrough right away.  You just try to make a little progress and focus on the learning.  When you don't feel like you made progress, you at least can learn something about your approach.
  2. 30 Day Sprints get you over the initial learning curve.  When you are taking in new ideas and learning new concepts, it helps to let things sink in.  If you're only trying something for a week or even two weeks, you'd be amazed at how many insights and breakthroughs are waiting just over that horizon.  Those troughs hold the keys to our triumphs.
  3. 30 Day Sprints help you stay focused.  For 30 days, you stick with it.  Sure you want to try new things, but for 30 days, you keep investing in this one thing that you decided was worth it.  Because you do a little every day, it actually gets easier to remember to do it. But the best part is, when something comes up that you want to learn or try, you can add it to your queue for your next 30 Day Sprint.
  4. 30 Day Sprints help you do things better, faster, easier, and deeper.  For 30 days, you can try different ways.  You can add a little twist.  You can find what works and what doesn't.  You can keep testing your abilities and learning your boundaries.  You push the limits of what you're capable of.  Over the course of 30 days, as you kick the tires on things, you'll find short-cuts and new ways to improve. Effectively, you unleash your learning abilities through practice and performance.
  5. 30 Day Sprints help you forge new habits.  Because you focus for a little bit each day, you actually create new habits.  A habit is much easier to put in place when you do it each day.  Eventually, you don't even have to think about it, because it becomes automatic.  Doing something every other day, or every third day, means you have to even remember when to do it.  We're creatures of habit.  Just replace how you already spend a little time each day, on your behalf.

And that is just the tip of the iceberg.

The real power of 30 Day Sprints is that they help you take action.  They help you get rid of all the excuses and all the distractions so you can start to achieve what you’re fully capable of.

Ways to Make 30 Day Sprints Work Better

When I first started using 30 Day Sprints for personal development, the novelty of doing something more than a day or a week or even two weeks, was enough to get tremendous value.  But eventually, as I started to do more 30 Day Sprints, I wanted to get more out of them.

Here is what I learned:

  1. Start 30 Day Sprints at the beginning of each month.  Sure, you can start 30 Day Sprints whenever you want, but I have found it much easier, if the 17th of the month, is day 17 of my 30 Day Sprint.  Also, it's a way to get a fresh start each month.  It's like turning the page.  You get a clean slate.  But what about February?  Well, that's when I do a 28 Day Sprint (and one day more when Leap Year comes.)
  2. Same Time, Same Place.  I've found it much easier and more consistent, when I have a consistent time and place to work on my 30 Day Sprint.  Sure, sometimes my schedule won't allow it.  Sure, some things I'm learning require that I do it from different places.  But when I know, for example, that I will work out 6:30 - 7:00 A.M. each day in my living room, that makes things a whole lot easier.  Then I can focus on what I'm trying to learn or improve, and not spend a lot of time just hoping I can find the time each day.  The other benefit is that I start to find efficiencies because I have a stable time and place, already in place.  Now I can just optimize things.
  3. Focus on the learning.  When it's the final inning and the score is tied, and you have runners on base, and you're up at bat, focus is everything.  Don't focus on the score.  Don't focus on what's at stake.  Focus on the pitch.  And swing your best.  And, hit or miss, when it's all over, focus on what you learned.  Don't dwell on what went wrong.  Focus on how to improve.  Don't focus on what went right.  Focus on how to improve.  Don't get entangled by your mini-defeats, and don't get seduced by your mini-successes.  Focus on the little lessons that you sometimes have to dig deeper for.

Obviously, you have to find what works for you, but I've found these ideas to be especially helpful in getting more out of each 30 Day Sprint.  Especially the part about focusing on the learning.  I can't tell you how many times I got too focused on the results, and ended up missing the learning and the insights. 

If you slow down, you speed up, because you connect the dots at a deeper level, and you take the time to really understand nuances that make the difference.

Getting Started

Keep things simple when you start.  Just start.  Pick something, and make it your 30 Day Sprint. 

In fact, if you want to line your 30 Day Sprint up with the start of the month, then just start your 30 Day Sprint now and use it as a warm-up.  Try stuff.  Learn stuff.  Get surprised.  And then, at the start of next month, just start your 30 Day Sprint again.

If you really don't know how to get started, or want to follow a guided 30 Day Sprint, then try 30 Days of Getting Results.  It's where I share my best lessons learned for personal productivity, time management, and work-life balance.  It's a good baseline, because by mastering your productivity, time management, and work-life balance, you will make all of your future 30 Day Sprints more effective.

Boldly Go Where You Have Not Gone Before

But it's really up to you.  Pick something you've been either frustrated by, inspired by, or scared of, and dive in.

Whether you think of it as a 30 Day Challenge, a 30 Day Improvement Sprint, a Monthly Improvement Sprint, or just a 30 Day Sprint, the big idea is to do something small for 30 days.

If you want to go beyond the basics and learn everything you can about mastering personal productivity, then check out Agile Results, introduced in Getting Results the Agile Way.

Who knows what breakthroughs lie within?

May you surprise yourself profoundly.

Categories: Architecture, Programming

How Autodesk Implemented Scalable Eventing over Mesos

This is a guest post by Olivier Paugam, SW Architect for the Autodesk Cloud. I really like this post because it shows how bits of infrastructure--Mesos, Kafka, RabbitMQ, Akka, Splunk, Librato, EC2--can be combined together to solve real problems. It's truly amazing how much can get done these days by a small team.

I was tasked a few months ago to come up with a central eventing system, something that would allow our various backends to communicate with each other. We are talking about activity streaming backends, rendering, data translation, BIM, identity, log reporting, analytics, etc.  So something really generic with varying load, usage patterns and scaling profile.  And oh, also something that our engineering teams could interface with easily.  Of course every piece of the system should be able to scale on its own.

I obviously didn't have time to write too much code and picked up Kafka as our storage core as it's stable, widely used and works okay (please note I'm not bound to using it and could switch over to something else).  Now I of course could not expose it directly and had to front-end it with some API. Without thinking much I also rejected the idea of having my backend manage the offsets as it places too much constraint on how one deals with failures for instance.

So what did I end up with?

Categories: Architecture

Persistence with Docker containers - Team 1: GlusterFS

Xebia Blog - Mon, 08/17/2015 - 10:05

This is a follow-up blog from KLM innovation day

The goal of Team 1 was to have GlusterFS cluster running in Docker containers and to expose the distributed file system to a container by ‘mounting’ it through a so called data container.

Setting up GlusterFS was not that hard, the installation steps are explained here [installing-glusterfs-a-quick-start-guide].

The Dockerfiles we eventually created and used can be found here [docker-glusterfs]
Note: the Dockerfiles still containe some manual steps because you need to tell GlusterFS about the other node so they can find each other. In an real environment this could be done by for example Consul.

Although setting up GlusterFS cluster was not hard, mounting it on CoreOS proved much more complicated. We wanted to mount the folder through a container using the GlusterFS client but to achieve that the container needs to run in privileged mode or with ‘SYS_ADMIN’ capabilities. This has nothing to do with GlusterFS it self, Docker doesn’t allow mounts without these options. Eventually mounting of the remote folder can be achieved but exposing this mounted folder as Docker volume is not possible. This is an Docker shortcoming, see docker issue here

Our second - not so prefered - method was mounting the folder in CoreOS itself and then using it in a container. The problem here is that CoreOS does not have support for GlusterFS client but does have NFS support. So to make this work we exposed GlusterFS through NFS, the step to enable it can be found here [Using_NFS_with_Gluster].  After enabling NFS on GlusterFS we mounted the exposed folder in CoreOS and used it in a container which worked fine.

Mounting GlusterFS through NFS was not what we wanted, but luckily Docker released their experimental volume plugin support. And our luck did not end there, because it turned out David Calavera had already created a volume plugin for GlusterFS. So to test this out we used the experimental Docker version 1.8 and run the plugin with the necessary settings. This all worked fine but this is where our luck ran out. When using the experimental Docker daemon, in combination with this plugin, we can see in debug mode the plugin connects to the GlusterFS and saying it is mounting the folder. But unfortunately it receives an error, it seems from the server and then unmounts the folder.

The volume plugin above is basically a wrapper around the GlusterFS client. We also found a Go API for GlusterFS. This could be used to create a pure Go implementation of the volume plugin, but unfortunately we ran out of time to actually try this.


Using distributed files system like GlusterFS or CEPH sounds very promising especially combined with the Docker volume plugin which hides the implementation and allows to decouple the container from the storage of the host.


Between the innovation day and this blog post the world evolved and Docker 1.8 came out and with it a CEPH docker volume plugin.

Innovation day at KLM: Persistence with Docker containers

Xebia Blog - Sat, 08/15/2015 - 12:06

On 3th of July KLM and Cargonauts joined forces at KLM headquarters for an innovation day. The goal was to share knowledge and find out how to properly do “Persistence with Docker containers”.

Persistence is data that you want to have available after the reboot, and to make it more complex in some cases you also want to share that data over multiple nodes. Examples of this are an upload folder that is shared or a database. Our innovation day case is focusing on a MySQL database, we want to find out how we can host MySQL data reliable and highly available.

Persistence with Docker containers poses the same dilemmas as when you don’t use them but with more options to chose from. Eventually those options could be summarized to this:

  1. Don’t solve the problem on the Docker platform; Solve it at the application level by using a distributed database like Cassandra.
  2. Don’t solve the problem on the Docker platform; Use a SAAS or some technology that provides a high available storage to your host.
  3. Do fix it on the Docker platform, so you are not tied to a specific solution. This will allows you to deploy everywhere as long as Docker is available.

Since this was an innovation day and we are container enthusiasts, we focused on the last option. To make it more explicit we decided to try and investigate into these two possible solutions for our problem:

  1. Provide the host with a highly available distributed file system with GlusterFS. This will allow us to start a container anywhere and move it freely because the data is available anywhere on the platform.
  2. GlusterFS might not provide the best performance due to its distributed nature. So to have better performance we need to have the data available on the host. For this we investigated Flocker.

Note: We realise that these two solutions only solve part of the problem because what happens if the data gets corrupted? To solve that we still need some kind of a snapshot/backup solution.

Having decided on the approach we split the group into two teams, team 1 focused on GlusterFS and team 2 focused on Flocker. We will post their stories in the coming days. Stay tuned!

The innovation day went exactly as an innovation day should go, with a lot of enthusiasm followed with some disappointments which led to small victories and unanswered questions, but with great new knowledge and a clear vision where to focus your energy next.

We would like to thank KLM for hosting the innovation day!