Robert Scoble is a kind of Brothers Grimm for the digital age. Instead of inspired romantics walking around the country side collecting the folk tales of past ages, he is an inspired technologist documenting the current mythology of startups.

One of the developments Robert is exploring is the rise of the contextual age. Where every bit of information about you is continually being prodded, pulled, and observed, shoveled into a great learning machine, and turned into a fully actionable knowledge graph of context. A digital identity more real to software than your physical body ever was. 

Sinner or saviour, the Age of Context has interesting implications for startups. It raises the entrance bar to dizzying heights. Much of the reason companies are tearing down the Golden Age of the Web, one open protocol at a time, is to create a walled garden of monopolized information.

To operate in this world you will have to somehow create a walled garden of your own. And it will be a damn big garden. So you must scale. Facebook and Google have an insane number of servers. Then you must gather to yourself the people capable of running these servers, creating the applications and cutting edge devices that tease out the data, store the data, learn from the data, and create a market for the data. A serious capital and technological barrier to entry. 

It's popular to say don't worry about scaling until you need to. It's a wonderful problem to have, etc. A curious aspect of these new contextual systems, Robert noticed, is that they need to plan on scale from the start. Not only must the infrastructure scale to gather and process context and turn it into smart data, the cost of each new customer is now huge.

One of the upsides of your typical software service is new customers are incrementally cheap to service, so they are highly profitable. The data is typically dead. It just sits in inexpensive storage and comes alive on demand, only when a user needs it. Economies of scale are your friend.

In the contextual age economies of scale are not your friend...

Hidden in every computer is a hardware backplane for moving signals around. Hidden in every application are ways of moving messages around and giving code CPU time to process them. Unhiding those capabilities and making them first class facilities for the programmer to control is the idea behind AppBackplane.

This goes directly against the trend of hiding everything from the programmer and doing it all automagically. Which is great, until it doesn't work. Then it sucks. And the approach of giving the programmer all the power also sucks, until it's tuned to work together and performance is incredible even under increasing loads. Then it's great.

These are two different curves going in opposite directions. You need to decide for your application which curve you need to be on.

AppBackplane is an example framework supporting the multiple application architectures we talked about in Beyond Threads And Callbacks. It provides a scheduling system that supports continuous and high loads, meets critical path timing requirements, supports fair scheduling amongst priorities; is relatively easy to program; and supports higher degrees of parallelism than can be supported with a pure tasking model.

It's a bit much for simple applications. But if you are looking to go beyond a basic thread per request model and think of an application as a container where a diverse set of components must somehow all share limited resources to accomplish work, then some of the ideas may prove useful.

In case you are still wondering where the name AppBackplane comes from, it's something I made up a while ago as a takeoff of computer backplane:

All I wanted to do was create a number of plugins and examples for Deployit using the different techniques available. While working on examples I was frustrated by having to clean up remainders of previous attempts, so following in the footsteps of greater men than my humble self (most notably professor Knuth who created TeX so he could finish writing a series of books on computer science) I first wrote a script to create junk in the Deployit repository and then get rid of it in one sweeping go.

I’ve placed my code on Github (git clone git://github.com/jvermeir/DeployitBlogs.git).
To try out the examples you’ll need Deployit 3.8.4 (the version I used, but I’m quite sure it’ll work fine with other versions as well). I’ve installed the server and cli in a directory referenced by an environment variable $DEPLOYIT_HOME. I’m using *nix style variable and scripts in my examples. If you insist on using Microsoft you can always download Cygwin or change the examples yourself. After checking out the sources and installing Deployit, $DEPLOYIT_HOME should contain the following folders:
deployit-3.8.4-server
deployit-3.8.4-cli
DeployitBlogs

DeployitBlogs contains a directory named tools that for now holds no more than a couple of scripts, most notably three scripts named utils.py, cleanup.cli and deploy.cli. In src/test you’ll find a dar file I’m using to try out my scripts and plugins (deps.dar was build from another example I hope to explain later, for now it’s not important; all it does is copy some files and print a message to the console).
Utils.py isn’t really a script but a plugin. It is supposed to be placed in $DEPLOYIT_HOME/cli/ext, but that can become rather tedious very quickly if you’re going through lots of versions, so I left the script in the tools directory and created a symbolic link to it so the cli will pick up new versions when it is restarted. Create a link in $DEPLOYIT_HOME/cli/ext:
cd $DEPLOYIT_HOME/deployit-3.8.4-cli/ext
ln -s $DEPLOYIT_HOME/DeployitBlogs/tools/utils.py utils.py

One more piece of setup is necessary: create two test environments that are required by the scripts in examples I hope to discuss later. Open the Deployit client interface and create two environments named ‘localenv’ and ‘AnotherLocalEnv’. They should contain a overthere.SshHost named ‘local’ and ‘AnotherLocalHost’ respectively. I defined both hosts with default settings and a username of ‘deployit’, but the username doesn’t (yet) matter for the examples. It should exist though, so in my case Deployit just connects to localhost as user ‘deployit’ to run the scripts.

The scripts in cleanup.cli and deploy.cli are no more than convenience wrappers that make it easier to test new versions of the plugin. After some command line input checking all they do is call one of the functions defined in utils.py.
cleanup.cli is used as follows:
$DEPLOYIT_HOME/deployit-3.8.4-cli/bin/cli.sh -username admin -password admin -f $DEPLOYIT_HOME/DeployitBlogs/tools/deploy.cli -- $DEPLOYIT_HOME/DeployitBlogs/tools/src/test/deps.dar localenv

The command above starts the cli, connects using default credentials, starts the script named in the -f parameter and passes the two parameters following the double dash to the script.

To avoid even more typing I created a shell script d.sh that does no more than run the command line above. It takes the environment name as a parameter like this:
d.sh localenv

To call the cleanup script use:
c.sh deps

With all these details out of the way we can discuss the utils.py script.
utils.py is read by the cli when it is started. Each method in the script is available from the cli command line, but there are two main entry points:
def deployApp(fileName, environmentName):
def deleteApp(appName):

These are the methods that are called from deploy.cli and cleanup.cli respectively.

deployApp is nothing more than a straightforward deployment of an application from a dar file, like explained in the cli guide (climanual.html) you can find in the server/doc/html directory (open the file and search for ‘Performing deployments’).
Before it deploys an app from a DAR archive, deployApp calls deleteApp to erase all traces of previously imported or installed versions from each environment.
DeleteApp calls undeployApps (see picture below) to find all versions of the app and undeploys them so they can be subsequently deleted by the call to deleteVersions.
deleteVersions works like undeployApps: get a list of stuff to delete, loop and delete each version using repository.delete().

Main methods of utils.py

If you’re interested in the details you can clone the repo at
git clone git://github.com/jvermeir/DeployitBlogs.git
cd DeployitBlogs/
git checkout 94ff4af

The code for this blog is located in the ‘tools’ directory.

Hey, it's HighScalability time: 

• 1 Billion/Month : Active Mostly Mobile YouTube Users
• Quotable Quotes:
  • @NeckbeardHacker: "Wait...he reimplemented swatch and rsync in chef, node and mongo?""Yup." "......Why?""Go easy on him...he's a Ninjipsterstar."
  • @b6n: Scale myth #1: Your service is a unique snowflake.
  • @polotek: If you don't care about bugs, design or scalability, it only takes 2 days to build anything.
  • @NasHope: Even with the rapid scalability of Rackspace, I've still waited for over an hour for Dominos to actually deliver my pizza.
  • George Dyson: Computers may turn out to be less important as an end product of technological evolution and more important as catalysts facilitating evolutionary processes through the incubation and propagation of self-replicating filaments of code.
  • George Dyson: Von Neumann believed that all fields of science, including pure mathematics, derive their sustenance through contact with real problems in the physical world.
• For those of us on the outside looking in, Sebastian Stadil gives a rare view from the cool kids table with an early look at How Google Compute Engine stacks up to Amazon EC2. Is it as wonderful as we all imagined? The differences: AWS has a much richer set of services; GCE is on-demand only, so AWS can be cheaper; GCE faster disk; GCE faster network IO, especially between datacenters; GCE faster boot times, GCE can mount read-only partitions multiple machines; GCE shares images across regions. Sebastian ponders what new architectures Google's feature set will encourage to flourish? Interesting was the idea that because the inter-datacenter network IO is so fast it will be possible to put read slaves in multiple datacenters, replicate in real-time. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Previously  I have discussed the use of the word 'architecting' and whether it is a valid word when describing the thing that architects do.

One of the people who commented on that blog entry informed me that the IEEE have updated the architecture standard, IEEE-1471 which describes the architecture of a software-intensive system to ISO/IEC/IEEE 42010:2011, Systems and software engineering — Architecture description. They have also updated slightly the definition of the word architecting to: 
"The process of conceiving, defining, expressing, documenting, communicating, certifying proper implementation of, maintaining and improving an architecture throughout a system’s life cycle (i.e., "designing")." Interesting that they have added that last bit in brackets "that is designing". I always fall back on the words used by Grady Booch to resolve that other ongoing discussion about whether the word architecture is valid at all in describing what we do "all architecture is design but not all design is architecture".

John McCutchan, after a long career spent working on the Linux kernel and being hired out as a code optimization guru, joined Google's Dart team. A curious hire until you watch Bringing SIMD to the Web via Dart, where John makes a programmer accessible explanation of why he likes Dart: performance, performance, performance.

Dart is an open-source Web programming language developed by Google. The motivation for Dart is twofold: provide a language capable of scaling up to the complex web applications that are becoming the norm. Think Gmail. And provide a single language capable of working on both the client and server. Towards those ends Dart is a complete language, full tool environment, and provides an advanced Web UI framework for building web applications at a high level of abstraction.

Why isn't JavaScript good enough? The fear is for large web apps the nature of JavaScript puts crippling limits on potential performance improvements, which will cause web apps to lose out to mobile apps. Before you scream in protest, Dart is created by the same guys who worked on V8, so all their JavaScript experience has fed into Dart.

John’s pitch is that despite many assertions to the contrary, the web is not fast. For example, when comparing the performance of a 2D physics engine across C and the web, the web version is 20x slower than C. Java is twice as slow as C. So the browser is not fast. Dart and the new libraries it supports can close that gap, allowing for the development of high performance applications in the browser.

You may be wondering which of the gazillion web frameworks to use and Dart may just be on the edge of your radar. If so this is an excellent video to take a look at.

Why might you like Dart? Here’s John’s list:

Who's Hiring?

• Fitbit is hiring a Site Operations Lead to help us on our mission to make the world a healthier place! Fitbit's wearable fitness devices are worn by people across the world, each syncing with the web site, wirelessly and automatically, every 15 minutes. Join our mission here! 
• OLO's food ordering platform powers some of the largest restaurant chains and feeds millions of consumers. We're looking for Senior C# Software Engineers and DevOps Engineers to help us scale our system. Apply here.
• The AWS Relational Database Service (RDS) automates management of relational databases in the cloud. We have a wide variety of customers and are part of many mission-critical applications, like the ones built by the 2012 Obama re-election campaign. If you're interested in joining a fast-growing service and team, please send your resume to rds-jobs@amazon.com.
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs
• Aerospike is Hiring! You dream in C - and like it? Then join us as a Senior Distributed Systems Engineer or Client / Application Engineer. People covent your bag of tricks for troubleshooting systems and network issues? Join our Operations and QA team. See if these positions are a fit for you! 

Fun and Informative Events

• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 

Cool Products and Services

• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free. No sign-up required. http://aicache.com/deploy
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

There's not a lot  of talk about application architectures at the process level. You have your threads, pools of threads, and you have your callback models. That's about it. Languages/frameworks making a virtue out of simple models, like Go and Erlang, do so at the price of control. It's difficult to make a low latency well conditioned application when a power full tool, like work scheduling, is taken out of the hands of the programmer.

But that's not all there is my friend. We'll dive into different ways an application can be composed across threads of control.

Your favorite language may not give you access to all the capabilities we are going to talk about, but lately there has been a sort of revival in considering performance important, especially for controlling latency variance, so I think it's time to talk about these kind of issues. When it was do everything in the thread of a web server thread pool none of these issues really mattered. But now that developers are creating sophisticated networks of services, how you structure you application really does matter.

In the spirit of Level Scalability Solutions - The Conditioning Collection, we are going to talk about the pros and cons of some application architectures with an eye to making sure high priority works gets done with low enough latency, while dealing with deadlock and priority inheritance, while making the application developer's life as simple as possible...

Today we released a number of great enhancements to Windows Azure. These new capabilities include:

• Mobile Services: HTML5/JS (CORS) Client + PhoneGap + Windows Phone 7.5 + .NET Portable Library support
• Web Sites: Mercurial Source Control + Dropbox Deployment support
• HDInsight: New service that enables you to easily deploy and manage Hadoop Clusters on Azure

All of these improvements are now available to start using immediately (note: some services are still in preview). Below are more details on them:

Mobile Services: HTML5/JS Client (CORS), PhoneGap, Windows Phone 7.5

Today we are adding support to enable pure HTML5/JS clients (and PhoneGap apps) as well as Windows Phone 7.5 clients to use Windows Azure Mobile Services as a backend.  This comes in addition to the new Android SDK for Windows Azure Mobile Services we released two weeks ago (as well as the Windows 8, Windows Phone 8 and iOS support we had earlier).

HTML5/JS Clients

You can now connect both HTML5 web client apps as well as Apache Cordova/PhoneGap apps to your Mobile Services, and use Windows Azure for both data storage and authentication.  We are delivering this via:

• New Mobile Services web client library that supports IE8+ browsers, current versions of Chrome, Firefox, and Safari, plus PhoneGap 2.3.0+. It offers the same data querying and storage APIs support we have in other native SDKs, and allows easy user authentication via any of the four identity providers supported by Mobile Services (Microsoft Account, Google, Facebook, and Twitter). Please use the GitHub issue tracker to report any issues, and our forum to get help.

• Cross Origin Resource Sharing (CORS) support to enable your Mobile Service to accept cross-domain Ajax requests. You can now configure a whitelist of allowed domains for your Mobile Service using the Windows Azure management portal.

To get started, create a mobile service in the Windows Azure Management Portal and open the Quickstart tab. You can now select “HTML” and find the steps to create a new HTML5/JS client or add a backend to an existing one:

You can then continue with this tutorial for the remaining steps and build a simple HTML5 todo list app (that runs entirely in a browser) in under 5 minutes.

When deploying the HTML5 front-end app to a production environment, make sure to add the host name of the website you use to host it to your Windows Azure Mobile Services’ Cross-Origin Resource Sharing (CORS) whitelist using the Configure tab as shown below:

Visit the Windows Azure Mobile dev center and read this tutorial to learn more about working with server-side data, or this one if you want to learn more about authenticating users.

Windows Phone 7.5 Support and a new C# Client Library on NuGet

A few days ago we published a preview of our next version of the Mobile Services C# client library on NuGet.  The goal of this pre-release is to give Mobile Services developers an early look at the new features we are planning for our next C# SDK update and an opportunity to try them out ahead of time. Some of the great new features we have added include:

• Portable Library Support: We have consolidated our Windows 8 and Windows Phone 8 clients on top of a single codebase using Portable Libraries. This enables us to reach a variety of new client platforms, as well as enable you to call Mobile Services from your ASP.NET and .NET server backend.

• Windows Phone 7.5 support: With the move to Portable Libraries, we are also enabling support for Windows Phone 7.5

• Json.NET and HttpClient: We migrated our implementation to use the latest and most flexible HTTP components to enable maximum robustness and extensibility.

Note: Today’s drop is a pre-release. For production apps we recommend continuing to use the “stable” Mobile Service client libraries for .NET available for download here. 

Keep Giving Us Feedback

Please continue to visit our uservoice page to let us know what you’d like to see added next (today’s release added 3 of the top 5 asks in uservoice!). Email us to show off your app, and ask questions in our forum whenever you run into a problem. 

Web Sites: Mercurial and Dropbox Deployment Support

Today’s release also includes a number of deployment/publishing enhancements to Windows Azure Web Sites:

Mercurial Source Control Support

You can now use Mercurial (Hg) repositories when setting up continuous deployment of your Websites from your CodePlex or Bitbucket repositories.  This is in addition to the TFS, CodePlex, Git and GitHub source control provider support that we previously supported. 

Today’s release also includes improved UI that makes it even easier to setup deployment from source-control.  Simply click the “Setup deployment from source control” link on your web-site dashboard, and a new wizard will appear that makes it trivial to walkthrough setting up publishing endpoints using a variety of source control providers and sites.  For example, below is how you could choose to enable source code deployment from a public or private Mercurial (Hg) repository you might have on Bitbucket:

Dropbox Deployment Support

Windows Azure also now supports site/app deployment from Dropbox to Web sites, making website deployment as easy as copying files to a folder on your local computer.  To enable this from the Windows Azure management portal, click the “Set up deployment from source control” link on your Web site dashboard, choose Dropbox and authorize the connection, and then choose a Dropbox sub-folder to synchronize:

 

You can then simply copy your source files to the Dropbox sub-folder on your local computer and press the “Sync” button in the Windows Azure Portal to deploy the files.  Windows Azure will automatically build sources as needed, similar to Git or TFS based deployments.  Also, the deployment history tab in the portal will keep track of your deployments and enables you to re-deploy any previous deployment with the click of a button.

Watch this 2 minute screencast to see how easy it now is to deploy web sites to Windows Azure using Dropbox.

Improved UI for Managing Source Control Deployments

In addition to the new setup wizard for source control deployment, today’s Windows Azure release also includes some other nice enhancements to the source control UI.  Deployment history in the management portal now accurately reflects which source control provider is connected for continuous deployment, such as TFS, CodePlex, GitHub, or Bitbucket.  It is also now possible to disconnect from an already connected source provider on a web-site in order to set up a different one (previously you had to delete the site to do this).

TFS Certificate Renewal

It’s also now possible to renew the certificate used by Team Foundation Service for continuous deployment directly from the Windows Azure management portal. To do this, click the “Renew TFS certificate” link on either the Dashboard or Quick Start page.

Support for Regenerating the Publish Profile

Today you can download a publish profile from the Web Sites dashboard. Once that profile is downloaded, the credentials are basically good forever. We understand that this is not optimal. To address this, with today’s release we are introducing a new quick glance command in the dashboard called Reset publish profile credentials. When clicked, you will get a confirmation for resetting the credentials and the credentials are regenerated.

New HDInsight Server: Deploy and Manage Hadoop Clusters on Azure

Today we also released a public preview of the new HDInsight Service for Windows Azure. HDInsight provides everything you need to quickly deploy, manage and use Hadoop clusters running on Windows Azure.

If you have a Windows Azure account you can request access to the HDInsight Preview and then easily create an HDInsight cluster within the Windows Azure Management Portal. Within the Windows Azure Management Portal click the New button and select the new HDInsight service to create a Hadoop cluster.  Specify a name for the cluster, a password for logging into the cluster and the size of cluster you need:

Note: a storage account is required to create a cluster and in the current public preview the storage account must reside in the East US region. The Azure Storage account you associate with your cluster is where you will store the data that you will analyze in HDInsight.

HDInsight Clusters

A cluster will take a few minutes to create (as part of creating it will configure the necessary Virtual Machines that together make up your Hadoop cluster). The Hadoop components installed as part of an HDInsight cluster are outlined here. Once the cluster is created, you can drill into the dashboard view to see the cluster quick glance screen. This quick glance allows you to see the basic information about your cluster and gives you a simple method to connect to the cluster (just click the Manage button at the bottom of the dashboard).

When you connect to the cluster you’ll see a page that contains a number of tiles that provide information about the cluster and can be used to perform additional tasks:

The Create Job tile opens a MapReduce job submission form that you can use to submit MapReduce jobs as JAR files. The Interactive Console tile opens a console that lets you execute Javascript and Hive queries directly against your cluster.  The Samples title includes samples that you can use to get started.

Summary

The above features are now available to start using immediately (note: some of the services are still in preview).  If you don’t already have a Windows Azure account, you can sign-up for a free trial and start using them today.  Visit the Windows Azure Developer Center to learn more about how to build apps with it!

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu

I wanted to find a way to install ruby 2.0 on Ubuntu 12.04 via Chef, without going the rvm route, which is harder to automate. After several tries, I finally found a list of .deb packages which are necessary and sufficient as pre-requisites. Writing it down here for my future reference, and maybe it will prove useful to others out there.

I downloaded most of these packages from packages.ubuntu.com (if you google their names you'll find them), then I installed them via 'dpkg -i'. Here they are:


libffi5_3.0.9-1_amd64.deb
libssl0.9.8_0.9.8o-7ubuntu3.1_amd64.deb
zlib1g-dev_1.2.3.4.dfsg-3ubuntu4_amd64.deb

The actual ruby .deb package was built with fpm by my colleague Jeff Roberts courtesy of this blog post.

One of the best books I've read recently is Advice is for Winners, by Raul Valdes-Perez.   It's all about how to get advice for better decisions in work and life.  I’ve written a deep review on it:

Book Review: Advice is for Winners

It's a great book whether you are an advice seeker, or serve in a trusted advisor role.   It helps you with either role, because the author shares an in-depth look at what holds back people from taking advice, as well as the qualities that make an advisor more effective.

On a personal note, I've had to learn how to seek advice with skill, back when I first joined Microsoft.  I started out in Developer Support and it really was a team sport.  It was rare for any individual to have all the knowledge to address the complex issues that came our way.  Instead, the key was to be very good at finding the answers and expertise around the world.   It’s true that two-heads are better than one, and there is a lot of power in the collective perspective – if you know how to use it.

When I joined the Microsoft patterns & practices team, I had to learn how to be good at both seeking out experts as well as giving deep advice about how to put our platform together and make the most of it.   One of the biggest challenges I faced on a daily basis was conflicting advice from qualified experts.

At the end of the day, I learned how to use test cases to find and validate the answers and solutions.   To do this well, I need to use scenarios and context both to weed out generic or irrelevant advice, and to be able to test advice.  Interestingly, the key to finding a solution often involved being able to "repro" (reproduce) the problem or challenge.

Once you could "repro" the problem, you could share it with others and get their heads in the game.  Also, often while trying to create a repro, you would find out what the real problem was, or at least, get clarity in the decisions and assumptions.

Sometimes, trying to reproduce the problem wasn't practical, so instead, the goal would be to understand the context or scenario as best you could, and construct a skeletal solution in incremental steps.   This way, when somebody tries to duplicate the solution, if something doesn't work along the way, you can usually backtrack to the basic steps.  Effectively, you can gradually build up from a working foundation, and when a part of it, doesn't work, you can isolate it, and troubleshoot what's different about the particular context (such as security context, or configuration, etc.)

Back to the book … in Advice is for Winners, Raul provides a great distillation and synthesis on the art of getting advice with skill.  What I especially like about the book is that it very much matches what I’ve learned the hard way about giving and getting advice.   Raul does a fantastic job of helping you get over any limiting beliefs or mindset that might hold you back from seeking advice.   He also does a great job of articulating what holds us back from getting the advice we need.  

The backbone of the book is an actionable framework for getting advice that’s principle-based and easy to personalize.  If you aren’t sure how to approach people to ask for help, this framework will help you get over that.  If you aren’t sure how to deal with conflicting advice, the guidance will help you get over that, too.  If you aren’t sure what scenarios to even seek out advice, Raul provides very specific examples and stories.  To bottom line it, what you don’t know, can hurt you, and building your advice seeking skills can be a powerful investment that pays you back for the rest of your life in exponential ways that you can’t yet predict.

For a "movie-trailer” style book review of Advice is for Winners, see Book Review: Advice is for Winners.

Becoming a skilled advice seeker might be one of the best capabilities you can build to improve your personal effectiveness.

Hey, it's HighScalability time: 

• 0: # of Google Readers; 2.5 billion/day: new pieces of content added to Facebook; 2.7 billion/day: likes added to Facebook; 7PB/month: photos added to Facebook
• Quotable Quotes:
  • @cwgem: It seems like cutting down API access is the stock scalability answer these days
  • @abenik: @Prismatic surfaced this article on their architecture for me. How meta.
  • @NewsBlur: The waters are rocky now, but take note that I have some time to get things right. I'm working this week to get things stable, then scale.
  • @Pinboard: Just learned that Google Reader no longer offers direct JSON export. I guess they held the annual "What should we ruin next?" staff retreat
  • @DEVOPS_BORAT: You can not able have unlimit scalability without unlimit outage.
  • Jeff: Amazon RDS Scales Up - Provision 3 TB and 30,000 IOPS Per DB Instance
  • @migueldeicaza: Google recently hired all of the Twitter's scalability team to work on Google IO checkout.
  • @skamille: Interesting to consider the greatly diminished role of networked file systems in modern distributed computing
  • @vambenepe: The server huggers have regrouped. Now they’re VM huggers, ironically. Fighting PaaS with all their might.
  • @jezhumble: If the developers can't self-service everything they need programmatically through an API, it's not a private cloud.
  • @Bremmel: Foursquare users crawl the real world like Google's spiders crawl the web - Dennis Crowley
  • @mollstam: SimCity's API (and I'm guessing region storage) is on Amazon. How can it not be auto-scaling? How can it take three days to add servers?
  • @josephmartz: Scalability gurus: It's about low coupling merging with high cohesion. More encapsulation and f*ck scaling out. I just want one #node.
  • @SQLSniper: great recipe for #sqlserver scaling from @GlennAlanBerry precon :) "scale up is like pets, scale out is like cattle" 
• @NewsBlur's tweet feed is a great blow by blow of the crush that happened when Google Reader became a dead app walking. A signup a second...Now fetching millions of new feeds hourly...Suspended free accounts, premium accounts only....Prices increased...Redis suffered from memory corruption...Moved from one app server to 6...Dropped SES for Mail Gun...Hosting provider died...Bringing up PostgreSQL read slaves...DB server upgrades...Introduction of HAProxy...More app servers.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Just a quick note to say that I've added some new essays to my Software Architecture for Developers book. They are:

• Software architects should be master builders: this talks about the importance of software architects needing deep technology skills and working as a part of the team.
• Technology is not an implementation detail: this is about why you shouldn't leave technology decisions as a after-thought.
• The need for sketches: this explains the importance of sketches in communicating the overall software architecture, both inside and outside of the immediate team.
• Would you code it that way?: this talks about the key question you need to ask yourself whenever you're designing software and visualising that design through sketches.

Leanpub does have the ability to e-mail readers whenever a new version of the book is published, but I try to only do this once or twice a month so as not to spam everybody. There is a version history at the back of the book though, which will tell you when new essays have been added plus it hyperlinks right to them. Hope you enjoy the new content.

For the last few months I've been programming a system in Go, so I'm always on the lookout for information to feed my confirmation bias. An opportunity popped up when Iron.io wrote about their experience using Go to rewrite IronWorker, their ever busy job execution system, originally coded in Ruby.

The result:

Last week I spoke at QConLondon 2013 on the topic of "Modern Legacy Systems". Here are the slides and I'll update with a link to the video when QCon makes it available.

Thank you to everyone that attended and I had a couple of great conversations afterwards about the topic. I even had someone tell me that they were the 'other side' of one of the anecdotes I told - they were on the better of the two sides! The auditorium I spoke in (Churchill) was HUGE and had a maximum capacity of 700. I had under the capacity but this was the largest venue I've spoken in and don't mind admitting that I found it intimidating! Fortunately I had watched Damien Conway's excellent Instantly Better Presentations talk earlier in the day and the tips came in handy. I would really advise anyone giving presentations to watch the video when it is available.

I attended many fascinating presentations and a couple that stand out most were around technology and art - visual problem solving. This was a really enjoyable but I'm struggling to find an immediate use for what I learned in my job!

Also worth a mention was Jesper Richter-Reichhelm's talk on Painful Success. This was a review of an architectural mistake that was made early in a project and its resolution. The problem came down to an incorrectly chosen architectural archetype which became evident when the system was under load. I think this is very common, particularly for on-line applications where it is assumed that a website architecture is suitable. What impressed me was Jesper's honesty about the issue, which is necessary in finding a solution. Many people find admitting these errors impossible and I think we can all learn from the good example.

In my previous blog posts I was working on reading data from wordpress blogs using groovy. This is nice, but of course there was a reason why I needed this. I wanted to create a tool or better a plugin for Elasticsearch. This plugin should make it easier to check the state of you cluster, play around with facets and query your data.

On my employers blog I started a series of blog posts that explain this plugin. I go into details for the libraries I used: AngularJS, Twitter Bootstrap and of course elastic.js.

Check my employers blog post if you are interested.

Introducing a query tool as an elasticsearch plugin (part 1)/

The post Introducing a query tool for elasticsearch appeared first on Gridshore.

I am in no way a music expert, but when I listen to Symphony No. 4 by Charles Ives, I imagine it's what a complex software/hardware system might sound like if we could hear its inner workings. Ives uses a lot of riotously competing rhythms in this work. It can sound discordant, yet the effect is deeply layered and eventually harmonious, just like the systems we use, create, and become part of.

I was pointed to this piece by someone who said there were two conductors. I'd never heard of such a thing! So I was intrigued. The first version of the performance sounds and looks great, but it unfortunately does not use two conductors. The second version uses two conductors, but is unfortunately just a snippet.

It's strikingly odd to see two conductors, but I imagine different parts of our systems using different conductors too, running at different rhythms, sometimes slow, sometimes fast, sometimes there are outbursts, sometimes in vicious conflict. Yet conceptually it all stills seems to hang together. 

Charles Ives Symphony No. 4, BBC Symphony Orchestra/David Robertson, cond./Ralph van Raat, piano:

We talked about 42 Monster Problems That Attack As Loads Increase. And in The Aggregation Collection we talked about the value of prioritizing work and making smart queues as a way of absorbing and not reflecting traffic spikes.

Now we move on to our next batch of strategies where the theme is conditioning, which is the idea of shaping and controlling flows of work within your application...

Use Resources Proportional To a Fixed Limit

This is probably the most important rule for achieving scalability within an application. What it means:

Just a quick note to say thank you to everybody who came along to my workshop and/or talk at DevWeek 2013 last week. Here are links to the slides and photos.

• Slides from my "Effective architecture sketches" workshop
• Photos from my "Effective architecture sketches" workshop
• Slides from my "Software architecture for developers" talk

Thanks again.

Hey, it's HighScalability time: 

• Quotable Quotes:
  • @ibogost: Disabling features of SimCity due to ineffective central infrastructure is probably the most realistic simulation of the modern city.
  • antirez: The point is simply to show how SSDs can't be considered, currently, as a bit slower version of memory. Their performance characteristics are a lot more about, simply, "faster disks".
  • @jessenoller: I only use JavaScript so I can gain maximum scalability across multiple cores. Also unicorns. Paint thinner gingerbread
  • @liammclennan: high-scalability ruby. Why bother?
  • @scomma: Problem with BitCoin is not scalability, not even usability. It's whether someone will crack the algorithm and render BTC entirely useless.
  • @webclimber: Amazing how often I find myself explaining that scalability is not magical
  • @pneuman42: Game servers are the *worst* scalability problem. Most services start small and scale up over time, solving problems along the way
  • @jeffsussna: OH: "Amazon outages involve server auto-scaling failures. Microsoft outages involve credit card auto-renewal failures"
  • carlosthecharlie: Writing Map/Reduce jobs is like making debt payments on technical debt you don't yet owe
  • anonymous: *eye twitches* You maintain secondary indexes in dynamo db fields, managed in application code? Dude. DUDE!
• LinkedIn: Secrecy Doesn't Scale. Winston Churchill: Truth is so precious that she should always be attended by a bodyguard of lies.
• So eternal vigilance really can be crowdsourced: Bill Introduced to Re-Legalize Cellphone Unlocking.
• Engaging discussion with George Dyson: Turing’s Cathedral and the Dawn of the Digital Universe. Template based addressing. DNA is searched by template. You don't have to know the exact location of a protein and the match doesn't have to be exact. Google is template searching for data. He thinks this template idea is a third revolution in computing. Much more flexible and robust. Because of errors you have to build architectures that are more flexible and can deal with ambiguity, which is what nature does. Google as an Oracle Machine. Alan Turing said machines will never be intelligent unless they are allowed to make mistakes. Deterministic computing is limited. A non-deterministic element, an Oracle is required. Machines need to learn by making mistakes, tolerating mistakes, a learning from mistakes. Google is made up deterministic machines. We humans are in Google's loop to act as the non-deterministic signal, as Oracle Machines. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...