In NoSQL: Past, Present, Future Eric Brewer has a particularly fine section on explaining the often hard to understand ideas of BASE (Basically Available, Soft State, Eventually Consistent), ACID (Atomicity, Consistency, Isolation, Durability), CAP (Consistency Availability, Partition Tolerance), in terms of a pernicious long standing myth about the sanctity of consistency in banking.

Myth: Money is important, so banks must use transactions to keep money safe and consistent, right?

Reality: Banking transactions are inconsistent, particularly for ATMs. ATMs are designed to have a normal case behaviour and a partition mode behaviour. In partition mode Availability is chosen over Consistency.

Why? 1) Availability correlates with revenue and consistency generally does not. 2) Historically there was never an idea of perfect communication so everything was partitioned...

Twenty years ago today on 30th April 1993 CERN published a brief statement that made World Wide Web technology available on a royalty free basis and changed the world forever. Here's the innocuous piece of paper that shows this and that truly allowed Tim Berners-Lee, at the fantastic London 2012 Olympics opening ceremony to claim “this is for everyone”. Over the past twenty years the web has become imbedded in all of our lives in ways which most of us could never have dreamed of and has probably given many of us in the software industry quite a secure (and for some, lucrative) living during that time.

How fitting then that yesterday, almost 20 years to the day since CERN's historic announcement, IBM announced a new appliance called IBM MessageSight designed to help organizations manage and communicate with the billions of mobile devices and sensors found in systems such as automobiles, traffic management systems, smart buildings and household appliances, the so called Internet of Things.

I've no idea what this announcement means in terms of capabilities, other than what is available in the press release, however it is comforting to note that foundational to IBM MessageSight is its support of MQTT, which was recently proposed to become an OASIS standard, providing a lightweight messaging transport for communication in machine to machine (M2M) and mobile environments. Today more than ever enterprises and governments are demanding compliance with open standards rather than proprietary ones so it is good to see that platforms such as MessageSight will be adhering to such standards.

This morning we released the v2.0 update of the Windows Azure SDK for .NET. This is a major refresh of the Windows Azure SDK with some really great new features and enhancements.  These new capabilities include:

• Web Sites: Visual Studio Tooling updates for Publishing, Management, and for Diagnostics
• Cloud Services: Support for new high memory VM sizes, Faster Cloud Service publishing & Visual Studio Tooling for configuring and viewing diagnostics data
• Storage: Storage Client 2.0 is now included in new projects & Visual Studio Server Explorer now supports working with Storage Tables
• Service Bus: Updated client library with message pump programming model support, support for browsing messages, and auto-deleting idle messaging entities
• PowerShell Automation: Updated support for PowerShell 3.0, and lots of new PowerShell commands for automating Web Sites, Cloud Services, VMs and more.

All of these SDK enhancements are now available to start using immediately and the SDK can now be downloaded from the Windows Azure .NET Developer Center.  Like all of the other Windows Azure SDKs we provide, the Windows Azure SDK for .NET is a fully open source project (Apache 2 license) hosted on GitHub.

Below are more details on the new features and capabilities released today:

Web Sites: Improved Visual Studio Publishing

With today’s release we’ve made it even easier to publish Windows Azure Web Sites.  Just right-click on any ASP.NET Web Project (or Web Site) within Visual Studio to Publish it to Windows Azure:

This will bring up a publish profile dialog the first time you run it on a project: 

Clicking the import button will enable you to import a publishing profile (this is a one-time thing you do on a project – it contains the publishing settings for your site in Windows Azure). 

With previous SDK releases you had to manually download the publish profile file from the Windows Azure Management Portal.  Starting with today’s release you can now associate your Windows Azure Subscription within Visual Studio – at which point you can browse the list of sites in Windows Azure associated with your subscription in real-time, and simply select the one you want to publish to (with no need to manually download anything):

Then just select the Web Site on Windows Azure that you want to deploy your app to, hit ok, and your app will be live on Windows Azure in seconds.  You can then quickly republish again (also in seconds) without having to configure anything (all of the publish profile settings are persisted for later use). 

Web Sites: Management Support within the Visual Studio Server Explorer

Today’s SDK release also adds new support for managing Web Sites, deployed in the cloud with Windows Azure, through the Visual Studio Server Explorer.  When you associate your Windows Azure subscription with Visual Studio, you’ll now see all of your running web sites within Windows Azure in the Visual Studio Server Explorer:

In addition to listing your sites, you can also perform common operations on them like Starting/Stopping them (just right click on one to do this).  You can also use the View Settings command on a site to retrieve the live site configuration settings from Windows Azure:

When you do this you’ll be able to view and edit/save the live settings of the Web Site directly within Visual Studio.  These settings are being pulled in real-time from the running Web Site instance in the cloud within Windows Azure:

Changes you save here will be persisted immediately into the running instance within Windows Azure.  No need to redeploy the application nor even open the Windows Azure Management Portal. 

Web Sites: Streaming Diagnostic Logs

One of the really awesome new features in today’s release is support that enables you to stream your Windows Azure Web Site’s application logs directly into Visual Studio.  This is a super useful feature that enables you to easily debug your Web Site when it is running up in the cloud in Windows Azure. 

How to Enable Live Streaming of Diagnostic Logs

To try out this feature, we’ll first add a Trace statement to an ASP.NET Web application and publish it to Windows Azure (as a Web Site).  We’ll add the trace statement to our app using the standard System.Diagnostics tracing API in .NET.  We’ll use the Trace.TraceError() method to write out an error:

By default when we hit the Web Site this method will do nothing – because tracing is disabled by default on Web Sites. 

If we want to enable tracing on our Web Site (in order to debug something) we can do that through the Windows Azure Management Portal (click the Configuration tab within a Web Site to enable this in the portal).  Or alternatively we can now do this directly within Visual Studio using the View Settings command within Server Explorer (like we saw above):

Notice above how we are enabling Application Logging for our Web Site, and turning it on so that it logs all “Error” trace events.  Make sure “Error” is selected and then click the “Save” button to persist the setting to Windows Azure – at which point we can hit our Web Site again and this time our Trace Error statements will be saved.

To view the trace statements inside Visual Studio we then simply need to click on our Web Site within the Server Explorer and select the View Streaming Logs in Output Window command:

This will open our Visual Studio output window – which will display the Trace.TraceError() statements as they execute in our live site (there is only a ~2 second delay from the time it executes to the point it shows up in our Visual Studio output window – which is super convenient when trying to debug something):

When you are done debugging the issue, just right-click on the Web Site again and choose the Stop Viewing Logs command to stop the logs being sent to VS (and when you are done with the issue itself make sure to turn off logging entirely by going back to the settings window and disabling it):

The above support is super useful and makes it much easier to debug issues that only occur in a live Windows Azure environment.  For more information on this feature (and how to use it from the command-line) check out this blog from Scott Hanselman.

Note: You must have a /LogFiles/Application directory in your Windows Azure Web Site before you can stream the application logs to Visual Studio. This gets created the first time a trace statement gets written to disk – so you’ll want to make sure you execute a Trace statement first before opening up the log streaming view inside Visual Studio.  We’ll be making an update to Windows Azure Web Sites in the next week or two which will cause this directory to be automatically created for you – both for existing and new web sites.  This will enable you to start streaming the logs even before a trace operation has occurred.  Until then just make sure you have written one trace statement before you start the log streaming window in VS.

Cloud Services: Support for High Memory VM Instances

Two weeks ago we announced the general availability of our Windows Azure IaaS release.  Included as part of that release was support for creating large memory IaaS VMs using our new 4 core x 28GB RAM (A6) and 8 core x 56GB RAM (A7) VM sizes.

Starting with today’s Windows Azure SDK 2.0 for .NET release, you can also now deploy your Cloud Services to these same VM sizes:

For details on the VM sizes please refer to: http://msdn.microsoft.com/en-us/library/windowsazure/dn197896.aspx

Cloud Services: Faster Deployment Support with Simultaneous Update Option

Today’s release includes a number of enhancements to improve the deployment and update times of Cloud Services.

One of the new deployment options we now support is the ability to do a “Simultaneous Update” of a Cloud Service (we sometimes also refer to this as the “Blast Option”).  When you use this option we bypass the normal upgrade domain walk that is done by default with Cloud Services (where we upgrade parts of the Cloud Service sequentially to avoid ever bringing the entire service down) and we instead upgrade all roles and instances simultaneously. With today’s release this simultaneous update logic now happens within Windows Azure (on the cloud side).  This has the benefit of enabling the Cloud Service update to happen much faster.

Note that because it updates all roles simultaneously you want to be careful about using it in production for normal updates (otherwise users will experience app downtime).  But it is great for scenarios where you want to quickly update a dev or test environment (and don’t care about a short period of downtime between your updates), or if you need to blast out a critical app update fast in production and you are ok with a short availability impact.

To perform a Simultaneous Update using Visual Studio, select the “Advanced Settings” tab within the Cloud Service Publish wizard and choose the “Settings” link next to the Deployment Update checkbox:

This will launch a new dialog.  Within it you can now select the new “Simultaneous Update” option:

Once saved, the updates to this Cloud Service will be performed using this option and all roles and instances will be updated simultaneously.

Cloud Services: Improved Diagnostics Support

Today’s release also includes some major enhancements to our diagnostics support with Cloud Services.

Easily Configure Diagnostics

Visual Studio has enabled Windows Azure Diagnostics for several versions. With today’s Windows Azure .NET SDK release we are making it even easier to start with the right diagnostics collection plan and leverage the data it provides to find errors and other useful information about your live service.

You can right-click on a Cloud Service role within Visual Studio’s Solution Explorer to pull up Configuration about it:

Today’s SDK release includes an updated Diagnostics section within it:

You can use this updated Diagnostics section to configure how you want to collect and store errors captured by the default .NET trace listener and your Trace.TraceError() code – all without having to write any glue code to setup or initialize.  You can specify the collection plan you want to use at runtime: Errors Only [default], All Information or a Custom Plan.  The custom plan is pretty rich and enables fine grain control over error levels, performance counters, infrastructure logs, collection intervals and more.

The diagnostics plan you configure through the configuration UI above is persisted in a diagnostics.wadcfg XML file.  If you open the Cloud Service role node within the Server Explorer you can find it and optionally edit the settings directly within the text editor:

Because the file is saved with your source code it can be managed under source control. It is also deployed with your cloud service and can be changed post deployment without requiring an application redeploy (I cover how to enable this live update below).

View Diagnostics on a Live Service

With today’s release we are also making it really easy for developers to review the live diagnostics data from their Cloud Services directly within Visual Studio – as well as dynamically turn on or off more detailed diagnostic capturing on their Cloud Services without having to redeploy the Cloud Service (which makes it much easier to quickly debug live production issues).

For any published Cloud Service, you can now view a quick summary of live service errors and other important status by clicking the View Diagnostics Data command in Visual Studio – which is surfaced off of each role node within a Cloud Service in the Visual Studio Server Explorer:

Executing this command will query the diagnostics table for the Cloud Service within Windows Azure and list a quick summary view of recent data within it.  In the example below we can see that we forgot to update the app’s configuration pointing to our SQL DB and therefore our stored procedure calls are failing in the deployed service:

Even more detailed diagnostics data has been gathered and stored in the Cloud Service’s Diagnostics Storage account. Click the View all Data link to drill into it. This loads a new Windows Azure Storage Table viewer. You can make use of the Query Builder support in it to refine your view over the diagnostics data. In the following example we are filtering a window of time occurring after 5:48pm by querying over the TimeStamp(Virtual). This refers to the time it occurred in the service rather than the time the data was collected and transferred.

This makes it much easier for you to look through historical logs to try and identify what the issue is.

Update Diagnostics Settings on a Live Service

Visual Studio also now enables you to configure and update the diagnostics settings for running Cloud Service directly from Server Explorer.  Diagnostic configuration can be updated at any time without the need to add code to your project and without having to redeploy the Cloud Service (which makes it much easier to quickly debug live production issues).

To do this, use the Server Explorer –> Windows Azure Compute node to select a running role instance in Windows Azure, and then click the Update Diagnostics Settings command on it to configure the runtime diagnostics settings for it:

Selecting this command will bring up a dialog that allows you to view and edit the Diagnostics Settings for the role.  Note that we can dynamically change the application log collection settings, event logs, performance counters, Infrastructure logs (like IIS, etc), and more:

In this example we will collect information about available memory + CPU + Requests/sec on the role from a performance counter. We’ll do this by selecting the Performance Counters tab and selecting the appropriate counter within it.  In addition to selecting the performance counters we want to track, we also need to set a Transfer period (in minutes) and Buffer size (MB).  We’ll set these to be 1 minute and 1024 MB (if we don’t set these then the logs won’t be copied to our storage account):

When we click OK, the collection plan will immediately be applied to the live role instances, and we’ll start collecting the new data we specified.  Within about a minute we’ll see a new WADPerformanceCountersTable created in our storage account, and our performance monitor data will start to be collected in it:

Double clicking the above table would enable us to browse and review the performance monitor data. 

Being able to dynamically turn on/off this functionality at runtime (without having to redeploy the Cloud Service) is super useful.  If we wanted to change the collection plan long term for every subsequent deployment, we can just apply the configuration changes we make at runtime back in the role designer for the cloud service project (or check it into source control).  That way new Cloud Service deployments will get it by default.

More Information

The above diagnostics support is really powerful, and can be used to capture diagnostic data from any number of roles and instances within a Cloud Service (including both web and worker roles).  And it makes it even easier to debug and analyze issues within multi-tier deployments.

Note that the .NET Diagnostics Listener support to output trace statements to Windows Azure’s diagnostics agent is enabled by default when you create new Cloud Service projects within Visual Studio.  If you start with an existing ASP.NET Web Project and then later convert it to be a Cloud Service you’ll want to manually add the below trace listener registration code to your web.config file in order to enable the above diagnostics support:

<system.diagnostics>
  <trace>
    <listeners>
      <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
        name="AzureDiagnostics">
        <filter type="" />
      </add>
    </listeners>
  </trace>
</system.diagnostics>

Storage: Visual Studio Table Explorer

With the previous Windows Azure SDK 1.8 release we revamped the Visual Studio tooling support for Windows Azure Storage. This previous release focused on read/write features for the Windows Azure Storage Blob and Queue services.

With today’s Windows Azure SDK 2.0 release, you can also now create and delete Windows Azure Tables, and add/edit/delete table entities in them from the Visual Studio Server Explorer.  This saves you time and allows you to easily use Visual Studio to build apps that use Windows Azure Storage Tables.

Within the Visual Studio Server Explorer, simply right-click within the Windows Azure Storage node to create and name a new Table:

Once you have the table created, you can then optionally add entities to it directly within Visual Studio (just click the “Create Entity” button on the table designer):

You can also edit/delete existing entities within Tables:

We also now make it much easier to build Table queries - without requiring expertise with OData syntax - using a new Query Builder available as part of the Table tooling:

The above features make it much easier to use Windows Azure Storage Tables.

Service Bus: Updated Client Library

Today’s release also includes an updated Service Bus client library with several great new features:

• Message Browse Support: Message browsing enables you to view available messages in a queue without locking the message or performing an explicit receive operation on it. This is very useful for debugging scenarios, and in scenarios that involve monitoring.

• New Message Pump Programming Model: Today’s release also adds support for a new message pump programming model.  The Message Pump programming semantics are similar to an event-driven, or “push” based processing model and provides an alternative to the receive loop which we support today. This approach supports concurrent message processing, and enables processing messages at variable rates.

• Auto-delete for Idle Messaging Entities: Auto-delete enables you to set an interval after which an idle queue, topic, or subscription is automatically deleted.

PowerShell: Tons of new Automation Commands

With today’s release, Windows Azure PowerShell (which is a separate download) has moved to support PowerShell 3.0.  Today’s release also includes numerous new PowerShell cmdlets that enable you to automate Windows Azure Web Sites, Cloud Services, Virtual Machines, as well as application services including Service Bus and the Windows Azure Store. You can find the full change log here.

Below are a few examples of some of the new functionality provided:

Web Sites

You can now get streaming logs for both http and application logs from your PowerShell console via the following command:

>>> Get-AzureWebsiteLog <your website> –Tail 

Cloud Services

You can now use a faster deployment option by opting into a simultaneous upgrade option which will upgrade all web and worker roles in parallel:

>>> Set-AzureDeployment –Mode Simultaneous

Virtual Machines

You can now use the new high memory virtual machine A6 & A7 images with these two commands:

>>> New-AzureVM

>>> New-AzureQuickVM 

We also enabled PowerShell Remoting by default when you create a VM via PowerShell to enable you to easily run your PowerShell cmdlets or scripts against your newly created virtual machines in Azure.

Service Bus

You can now manage Service Bus namespaces via newly added cmdlets which allow you to create, list and remove Service Bus namespaces.

Windows Azure Store

You can now manage your Azure Store add-ons from PowerShell. You can list the available add-ons, purchase an add-on, view your purchased add-ons and also upgrade the plan on a purchased add-on.

For example, the below command would create and deploy a MongoDB service from MongoLab (one of our Windows Azure Store partners):

>>> New-AzureStoreAddOn myMongoDB –AddOn mongolab –plan free –Location “West US”

Storage

We now support blob CRUD operations via PowerShell which allow you to manage Storage blob containers, upload/download blob content, and copy blobs around. This enables you to create scripts to seed some initial data for your applications or check what is in your storage account quickly when you are developing your application.

Scaffolding cmdlets for Web/Worker Role

We have also added new cmdlets for scaffolding. You can now use Add-AzureWebRole and Add-AzureWorkerRole to create projects for general web/worker role. You can use New-AzureRoleTemplate to generate a customized role template which you can use in Add-AzureWebRole or Add-AzureWorkerRole via the –TemplateFolder parameter.

More Information

A few other updates/changes with today’s release:

• WindowsAzure.Diagnostics.dll no longer depends on WindowsAzure.StorageClient.dll. You will now be able to import and use the WindowsAzure.Storage 2.0 NuGet package in your application without introducing conflicts with Diagnostics.

• Windows Azure SDK 2.0 supports side by side with Windows Azure SDK 1.8 and 1.7 while dropping support for side by side with Windows Azure SDK 1.6. Therefore you will not be able to debug an SDK 1.6 service if SDK 2.0 is installed on the same machine.

• WindowsAzure.ServiceRuntime.dll, WindowsAzure.Configuration.dll and the caching assemblies are now built against the .Net framework 4.0 runtime. Therefore you will have to retarget your framework 3.5 application to 4.0 after migrating to Windows Azure SDK 2.0.

• We also recently published Windows Azure Cloud Service Support Policy which you can view in detail at http://support.microsoft.com/gp/azure-cloud-lifecycle-faq.

Learn More

You can also learn more about today’s SDK release, and see some demos of it in action, from my visit to this week’s latest Cloud Cover Show on Channel9:

Summary

Today’s release includes a bunch of great features that enable you to build even better cloud solutions.  If you don’t already have a Windows Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Windows Azure .NET Developer Center to learn more about how to build apps using today’s SDK release.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu

Who's Hiring?

• Want to build scalable systems that power the world's largest music streaming service? Spotify is looking for engineers for our backend infrastructure team. Apply now.
• At Evernote our vision is to help the world remember everything. If you want to work in a face paced, highly rewarding environment with some of the smartest engineers on the planet, then come join us! We are looking for Sr. Security Engineers and Sr. Operations Engineers/DevOps to join our operations team.
• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 
• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.
• Help build the platform that powers a better, fairer banking experience at Simple. Join a talented team that chooses its own tools; works across web, Android, iOS, and Ruby/Scala/Clojure backend apps; and develops a secure and scalable banking service on AWS. Learn more at careers.
• The AWS Relational Database Service (RDS) automates management of relational databases in the cloud. We have a wide variety of customers and are part of many mission-critical applications, like the ones built by the 2012 Obama re-election campaign. If you're interested in joining a fast-growing service and team, please send your resume to rds-jobs@amazon.com.
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI is happening on Sept. 12th-13th. Special, High Scalability Reader Rate: $50 off registration--now through September 10!
• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 

Cool Products and Services

• Install Aerospike Community Edition in 10 Minutes! To help you get up and running with our Community Edition as quickly as possible, we've created a tutorial video on AerospikeTV that provides guidance on installing your free database. Download our FREE Database and see how easy it is to install!
• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free. No sign-up required. http://aicache.com/deploy
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

What do Bill Gates, Steve Jobs, and Walt Disney teach us about building a culture of innovation?

A lot.

I put together a comprehensive collection of innovation quotes.   And by comprehensive, I mean more than 100 of the greatest thoughts on innovation, all at your finger tips.   You’ll hear from Edison, Mozart, Michael Porter, Peter Drucker, Seth Godin, and more.

And, to make the innovation quotes more meaningful, I’ve grouped them into useful categories, so you can flip through the sections you care about the most.   There’s a section on Action, Birthing Ideas, and Continuous Learning and Growth.  You’ll also find a section on Fear and Failure.  After all, success in innovation is often a numbers game.  Remember what Edison taught us.

Just because it’s a comprehensive collection of innovation quotes, doesn’t mean it’s complete, or that it’s a done deal.  There’s always room for improvement (and innovation.)  So if you have some favorite innovation quotes that I’ve left out, please let me know.  I want this collection to be truly insightful, and most importantly, actionable.

After all, what good are good ideas, if you can’t turn them into results.

And that’s the truth about innovation.

Enjoy.

This is a repost of part 2 (part 1) of an interview I did for the Boundary blog.

Boundary:  There’s another battle coming down the pike between Amazon (AWS) and Google (GCE). How should the CTO decide which one’s best?

Hoff: Given that GCE is still closed to public access we have very little common experience on which to judge. The best way to decide is as always, by running a few experiments. Pick a few representative projects, a representative team, implement the projects on both infrastructures, crunch some numbers, figure out the bigger picture and then select the one you wanted in the first place  .

Sebastian Stadil, founder of Scalr, recently wrote about his experiences on both platforms and found some interesting differences: AWS has a much richer set of services; GCE is on-demand only, so AWS can be cheaper; GCE has faster disk and faster network IO, especially between datacenters; GCE has faster boot times and can mount read-only partitions across multiple machines; and GCE shares images across regions...

In the last few years, we see the advent of highly distributed systems. Systems that have clusters with lots of servers are no longer the sole realm of the googles’ and facebooks’ of the world and we begin to see multi-node and big data systems in enterprises. e.g. I don’t think a company such as Nice (the company I work for) would release an hadoop based analytics platform and solutions, something we did just last week, 5-6 years ago.

So now that large(r) clusters are more prevalent, I thought it would be a good time to reflect on the fallacies of distributed computing and how/if they are relevant; should they be changed.
If you don’t know about the fallacies you can see the list and read the article I wrote about them at the link mentioned above. In a few words I’d just say that these are statement, originally drafted by Peter Deutsch, Tom Lyon and others in in 1991-2, about failed assumptions we are tempted to make when working on distributed systems which turn out as fallacies and cost us dearly.

So the fallacies help keep in mind that distributed systems are different, and they do seem to hold, even after the 20 years that passed. I think, however, that working with larger cluster we should also consider the following 3 as fallacies we’re likely to assume

• Instances are free
• Instances have identities
• Map/Reduce is a panacea

Instances are free
A lot of the new technologies of the big-data and noSQL era bring with them the promise of massive scalability. If you see a performance problem, you can just (a famous lullaby word) add another server. In most cases that is even true, you can indeed add more servers and get better performance. What these technologies don’t tell you is that instances have costs. More instances mean increased TCO starting from management effort monitoring, configuring etc, as well as operations cost either for the hardware; the rented space and electricity in a hosted solution or the usage by hours in a cloud environment. So from the development side of the fence the solution is easy – add more hardware. In reality sometimes it is better to make the effort and optimize your code/design. Just the other week we had a more than a 10 fold improvement in query performance by removing query parts that were no longer needed after a change in the data flow of the system – that was way cheaper than adding 2-3 more nodes to achieve the same results.

Instances have identities
I remember, sometime in Jurassic age, when I set up a network for the first time (A Novell Netware 3.11 if you must ask) it had just one server. Naturally that server was treated with a lot of respect. It had a a printer connected, it had a name, nobody could touch it but me. One server to rule all them clients. Moving on I had server farms, so just a list of random names began to be a problem so we started to use themes like gods, single malts (“can you reboot the Macallan please”) etc. Anyway, that’s all nice and dandy and if you are starting small with a (potentially) big data project you might be tempted to do something similar. If you are tempted – don’t. When you have tens of servers (and naturally even worst when you have hundreds or thousands) you no longer care about the individual server. You want to look at the world as pools of server types. you have a pool of data nodes in your hadoop cluster, a pool of application servers , a pool of servers running configuration x and another with configuration y. You’d need tools like abiquo and/or chef and/or ansible or similar products to manage this mess. But again, you won’t care much about XYZ2011 server and even it runs tomcat today, tomorrow it may make more sense to make it part of the cassandra cluster. What matters are the roles in the pools of resources and that the pool sizes will be enough to handle the capacity needed.

Map/Reduce is a panacea
Hadoop seems to be the VHS of large clusters. It might not be the ultimate solution, but it does seem to be the one that gets the most traction – a lot of vendors old (like IBM, Microsoft, Oracle etc.) and new (Hortonworks, Cloudera, Pivotal etc.) offer Hadoop distros and many other solutions offer Hadoop adaptors (Mongodb, Casandra, Vertica etc.) and Hadoop, well hadoop is about the distributed file system and, well, map/reduce.
Map/Reduce, which was introduced in 2004 by Google is an efficient algorithm for going over a large distributed data set without moving the data (map) and then producing aggregated or merged of results (reduce). Map/Reduce is great and it is a very useful paradigm applicable for a large set of problems.
However it shouldn’t be the only tool in your tool set as map/reduce is inefficient when there’s a need to do multiple iterations on the data (e.g. grpah processing) or when you have to do many incremental updates to the data but don’t need to touch all of it. Also there’s the matter of ad-hoc reports (which I’ll probably blog about separately) Google solved these in pregel, percolator and dremel in 2009/2010 and now the rest of the world is playing catchup as it did with map/reduce a few year ago – but even if the solutions are not mature yet, you should keep in mind that they are coming

Instances are free; Instances have identities; and map/reduce is a panacea – these are my suggested additions to the fallacies of distributed computing when talking about large clusters. I’d be happy to hear what you think and/or if there are other things to keep in mind that I’ve missed

This past January, more than 20,000 people got the book that’s changing lives, and changing the workplace:

Getting Results the Agile Way:  A Personal Results System for Work and life

You’re going to want to read this if you want to level up in work and life, or share it with a friend you know that you want to help give the edge.

I’m going to walk through how I use Agile Results  to show you how YOU can seriously and significantly amplify your impact, get better performance reviews, and spend more time doing what YOU enjoy.  (So, while this post might seem all about me, it’s really about you.)

I’m not going to make it look easy.  I’m going to make it real.  I care way more that you get the full power of the system in your hands so you can do amazing things and get exponential results.   Agile Results is not a fly-by-night.   It was more than ten years in the making.

Keep in mind, it’s an ultra-competitive world, and what you don’t know can hurt you.  On the flip side, what you do know can instantly boost your creativity, productivity, and impact in unfair ways.

Use Agile Results as your unfair advantage.

Now then, let’s roll up our sleeves and get to it.  But first, some context …

I use Agile Results as a personal productivity and time management system. 

In one line, it's my "personal results system for work and life." 

I also use it to lead distributed teams around the world.  I use it to drive high-impact projects, and for projects at home. 

This post is a detailed walkthrough of how I use Agile Results as a time management and productivity system for making things happen.

Before we dive into the details, I want to make an important point ...

The simplest way I use Agile Results is as follows:

I write down Three Wins that I want to accomplish for the day on paper.

Yes, that’s it, and it is that simple (to at least, that’s how simple it is to start using Agile Results.)

If ever I get off track (and I do), the simple way I get back on track with Agile Results, is I simply write down my three wins for the day, down on a piece of paper.  Agile Results is both forgiving and instantly useful.

The main goal of Agile Results is to help me spend more time where it counts.  I needed a light-weight and flexible system that I could use for myself or for others.  For several years, I had to build up a new team every six months.  I needed to build high-performance teams under the gun, as quickly as possible.  And, at the same time, I wanted work to be a place of self-expression, where you live your values, give your best where you have your best to give, and experience flow and continuous learning on a regular basis.

I needed to get "Special Forces" results, from individuals, and from the larger team.  So I needed a system that could stretch to fit ... either scale up for a team, or simply help an individual get exponential results.  I wanted it to be based on timeless and self-evident principles, rather than tools or fads.  And I wanted it to "play well with others" ... where if somebody already had an existing system, or favorite tools, Agile Results could just ride on top and help them get more of what they already use.

Above all, it had to be as simple as possible.

Having a system that’s as simple as possible, helps support you while you do the impossible.

With that in mind, let's dive in.  So here is how I use Agile Results ...

Daily Startup Routine

My favorite startup routine is:

1. Wake up
2. Throw on my shoes and run for 30 minutes
3. Take a shower
4. Eat breakfast slow
5. Take the back way to work, play my favorite songs, and figure out my three wins for the day

It's a simple routine.  I've learned that one of the keys is carving out time for what's important, first thing in the morning.  What I like about this routine is that it's not chaotic.  It's serene by design.  I've had chaotic startup patterns.  This is the one that I purposefully made the morning about exercising, eating, and setting the stage for a great day.  I don't turn on the TV.  I don't watch the news.  I don't check my computer.  All of that can wait until I'm in the office. 

It's how I charge up.

Monday Vision

Monday is all about vision for the week. 

For example, if the week were over, and you were looking back, what would be the three big things you want under your belt?

It's such a simple thing, but I make the most of the week, by starting with what I want out of the week.  On Monday mornings, my main starting point is Three Wins for the Week.  I identify the top Three Wins that would make this week great.  To do so, I jump ahead and imagine that if this was Friday, what would I want to rattle off as my three wins under my belt.  I do this on my way to work, while listening to my favorite songs.  I play around with possibilities.  I think of what big wins would look like.  I also think about the big, hairy problems need attention.  I try to balance between addressing pain, and acting on opportunities.

If I really get stuck, I try to think of the top three things that are top of mind that really need my attention.  If I'm going to invest the next week of my life, I want to make sure that I'm nailing the things that matter.

The key is that I use very simple words.  I'm effectively choosing labels for my wins. For example, "Vision is draft complete" is simple enough to say, and simple enough to remember.  If I can't say it, it's not sticky.

When I get to work, I scan my mail.  I think of my inbox as a stream of *potential* action.  I walk the halls to beat the street. I absorb what I learn against what I set out to do for the week.  If necessary, I readjust.  If I catch my manager, I do a quick sanity check to find out his Three Wins for the Week, and how I'm mapping to what's on the radar.

For each project on my plate, I have a simple list of work items.  This gives me "One Place to Look."  This also helps me identify the "Next Best Thing" to do.  It's this balance of the lists with what's top of mind, that keeps me grounded.  I try to support my mind, with just enough scaffolding, but let it do what it does best.  If I can identify the big outcomes for the week, I don't have to get caught up in the overhead of tracking minutia.

On my computer, I keep notepad open so that I can list my three wins at the top for the week, list my three wins for the day, any tasks or things on my mind below that.  It's important that I keep my mind fresh and ready for anything.  It's also where I do my brain dump at the end of the day ("Dump Your State"), which is simply a dump of anything on my mind or pending issues, so that I don't take work home with me, and I can pick up from where I left off, or start fresh the next day.

Daily Wins

Each day of the week, the most important thing I do at the start of the day, is identify Three Wins that I want for that day. I write them down.  I cross-check them against the Three Wins that I want for the week. 

First I brainstorm on what I want or need to achieve for the day.  This is just a rapid brain dump.  If I'm at my desk, I write it down on paper.  When I hone in on what seems to be my three key wins for the day, I say them out loud.  Verbalizing them is important, because it's how I simplify and internalize them.  Being able to say them, keeps them at my mental finger tips.  It's like having the scoreboard right in plain view.  I want them front and center so that I can use them to help me prioritize and focus throughout the day.

Worst Things First

I try to put my "Worst Things First", either in the start of the week, or the start of the day.  The worst thing is to have something looming over me all day or all week.  The other way I look at this is, if I jump my worst hurdle, then the rest of the day or the week is a glide-path.

If my worst thing is time consuming, then I might need to "Timebox" it, such as spend no more than an hour max on it.  If the work is intensive, I might tend to split it up, and work through it in 20 minute batches, and take 10 minute breaks.  If I'm on a roll, I might go straight for an hour.  If this is regular work that I need to do, that I really don't enjoy doing, then I try to either get it off my plate, or find a way to make it fun, or "Pair Up" with somebody.  I find somebody who loves to do what I hate doing, and see if they might like to show me, either why they love it, or how to do it better, faster, and easier.  This practice has taught me so many new tricks, and it's also helped me appreciate some of the deep skills that others are good at.

Power Hours

I know my peak times and my down times during the day.  For example, at around 11:00 AM, I have lunch on my mind, and 3:00 PM is effectively siesta time.

My best hours tend to be 8:00, 10:00, 2:00, and 4:00.

They are the hours where I am in the zone and firing on all cylinders.   I’m generally more “productive” earlier in the day, and more “creative” later in the day.   I don’t know all the reasons why, but what I do know is it’s a pattern.  And by knowing that pattern, I can leverage it.

What I do is I push my heaving lifting into those hours as best as I can.  I use my best horse-power to plow through my work and turn mountains into mole-hills.   When I don’t use those peak hours, somehow mole-hills turn into mountains, and it’s slow going.  It’s the difference in feeling between riding a wave, and pushing rocks uphill.

To get to this point, I simply had to notice during the week, when my best hours really are, not just when I want them to be.  Now that I know my best times for peak performance, I have to defend those hours as best I can, or at least know what I am trading off.

When it comes to defending your calendar, you need to know what’s worth it.  Once you know your best Power Hours, you know what’s worth it.

Aside from spending more time in my high ROI activities, and playing to my strengths, my Power Hours amplify my productivity more than any other way.

Creative Hours

This is the space of creative breakthroughs and innovation.  It’s not that I’m not creative throughout the day, but I generally have a pattern where I’m more creative at night, or in the quiet hours of the morning.  I’m also more creative on Fridays and Saturdays.

I can try to change the pattern, but I can also first notice the pattern and leverage what already exists.  If I know the times when I’m most creative, I can start to use this time to think and brainstorm more freely.

And, I do.

That’s how I come up with ways to do things better, faster, and cheaper.  It’s how I figure out ways to change the business, or ways to change my approach, and ways to take things to the next level.

When I’m in my creative zone, I do more exploration.  I follow my thoughts and play out “what if” scenarios.  I value the fact that my Creative Hours lead the ideas that help me learn and improve whatever I do.

A simple check, if I’m not flowing enough ideas or if I’m feeling too much nose-to-the grindstone, is I ask myself, “How many Creative Hours did I spend this week?”   If it’s not at least 2, I try to up the count.

Create Hours are my best way to decompress, absorb and synthesize, which ultimately leads to my greatest breakthroughs.

 

Daily Shutdown Routine

Day is done, gone the Sun.  From the lakes, from the hills, from the sky.

But how do you put it to rest?

I like a deliberate switch from work-mode to home-mode.  I don’t want to bring my work home with me, and have it seep into everything I do.  When I’m at work, I work hard (and play hard, too … especially because I treat work like play, and drive it with a passion.)

But when I shut down my work day, I need a way to unwind.

I found the best way to free my mind, is dump it down.   So I simply dump it to notepad, or my little yellow sticky pad.  Any open issues or challenges or things on my mind.  I can always pick them back up.  Or, I can let them go.

But the last thing I want is for a bunch of problems to be swirling around in my head.

Besides, if you stop swirling problems around in your head, you make space for creative insights, and the answers start to pop out of the woodwork.

Another pattern I’ve adopted is to use a metaphorical tree in my mind to hang my hat of problems on.  Again, I can always pick them up again tomorrow, but for now, I’ll stuff my problems in this hat, hang them on the tree, and free my mind.

Friday Reflection

What if every Friday you could get smarter about your productivity and effectiveness?

You can.

I know it sounds simple, and it is, but remember that one of the big keys in life is not just knowing what to do, but doing what you know.

Friday Reflection is a perfect chance to ask myself two simple questions:

1. What are three things going well?
2. What are three things to improve?

That’s how it starts.

I keep a simple recurring 20-minute appointment with myself for each Friday morning.   It’s often the most valuable 20 minutes I spend each week.  It’s where I actually reflect on my performance.  Not in a critical way, but a constructive way.  I explore with simple questions:

1. Am I biting off too much?
2. Am I biting off the right things?
3. Am I making the right impact?
4. Are there better activities I could spend more time on?
5. Are there soul-sucking activities that I could spend less time on?

Friday Reflection is how I learn to master my capacity and be more realistic about my own expectations.   I tend to over-estimate what’s possible in a week (and underestimate what’s possible in a month.)   This little feedback loop, helps me see the good, the bad, and the downright fugly.

The most important outcome of my Friday Reflection is, three things to try out next week to do a little better.

The little better adds up.

The main thing to keep in mind is that Friday Reflection gives you deeper insight into your strengths and weaknesses in a way that you instantly benefit from.   The key is to carry the good forward, and let the rest go, and to treat it as a continuous learning loop.

You only fail when you give up or stop learning or stop trying.

Monthly Focus

To make my month more meaningful and to add a dash of focus to it, I identify my Three Wins for the Month.  At the month level, I can take a step back and look at the bigger picture.   Asking myself, “What do I want under my belt when the month is over?” is a powerful and swift way to create clarity, and identify compelling outcomes.

Since I'm leading a team, I go a step further.  I think of Three Wins for the team.  Based on everything that's on our plate, I try to identify what the Three Wins for the team should be.  I try to figure out things that would be easy to share with my manager.  This makes it easy to check alignment, and it makes it easy for them to sell our impact up the management chain.  (Read – It helps you get better performance reviews.)

When I get to work, I send out a short mail to the team, with the subject line: WEEKLY WINS: 2012-07-23.  It's simply WEEKLY WINS, plus the current date.  I briefly summarize the drivers, the threats, and hot issues on our plate, then list the Three Wins identified.   I follow this by asking the team for their input, and whether we need to recalibrate.  At the bottom, I simply do an A-Z list of bulleted items to dump the full working set of work in flight.  It both helps people see what the full scope is, as well as help us rationalize whether we bit off the right things, and it helps people stay on top of all the work.  It's like a team To-Do list.  Sometimes it's a crazy list, but the three wins at the top, help keep our sanity and focus at all times.

It's a simple approach, but it works great for distributed teams, and it gives us something to go back and check at the end of the week, or throughout the week to remind ourselves of what we set out to do.

Since my manager adopted Agile Results too, he shares his three wins for the week to the team in a simple mail.  Folks across the team, simply add their wins for the week.  It's nothing formal ... it's more like a simple assertion of our intended victories.

During our team meeting, our manager goes around the team, and we share our three wins from last week, and our three wins we plan for this week.  This helps everybody across the team stay connected to what's going on.

Ten at Ten

I need to throw in this tip, because it’s the single most effective way I’ve found to get a team on the same page, and avoid a bunch of email.  And, it’s a simple way to create clarity, and avoid confusion.

It also builds the discipline of execution.

All you do is meet for ten minutes each day, Monday through Thursday.  I call it, Ten at Ten.

I found ten at ten to be one of the most effective times in the day to do a sync.  That said, because I always have distributed teams, I’ve had to vary this.   But for the most part, I like Ten at Ten as a reminder to have a quick sync up with the team, focused on creating clarity, debottlenecking any issues, and taking note of small wins and progress.

The way it works is this:

1. I schedule ten minutes for Monday through Thursday, at whatever time the team can agree to, but in the AM.
2. During the meeting, we go around and ask three simple questions:  1)  What did you get done?  2) What are you getting done today? (focused on Three Wins), and 3) Where do you need help?
3. We focus on the process (the 3 questions) and the timebox (10 minutes) so it’s a swift meeting with great results.   We put issues that need more drill-down or exploration into a “parking lot” for follow up.  We focus the meeting on status and clarity of the work, the progress, and the impediments.

You’d be surprised at how quickly people start to pay attention to what they’re working on and on what’s worth working on.  It also helps team members very quickly see each other’s impact and results.  It also helps people raise their bar, especially when they get to hear  and experience what good looks like from their peers.

Most importantly, it shines the light on little, incremental progress.  Progress is the key to happiness in work and life.

One thing I’ll point out is that the Monday meeting is actually 30 minutes, not 10 minutes, since it’s more of a level set for the week, and it’s a chance to figure out the Three Wins for the Week.

Well, there it is.

It might not look like a simple system for meaningful results, but when you think of all the synthesis it is effective.

The way to keep it simple is to always start simple.   Whenever you forget what to do, go back to the basics.  Simply ask yourself,

“What are Three Wins I want for today?”

- OR -

“What are Three Wins I want for this week?”

- OR -

“What are Three Wins I want for this month?”

- OR -

… if you’re feeling really bold, and want to go for the gold, “What are Three Wins I want for this year?”

Hopefully, this little walkthrough helps you easily see how you can apply Agile Results to your workflow, and get more out of the time you already spend.  If nothing else, remember this:

Value is the ultimate short-cut.

When you know what’s valued, you can target your effort.  When you know the high value activities, you can focus on those.

What Agile Results does is streamline your ability to flow value, for yourself and others. 

Pure and simple.

And that’s how getting results should be … elegance in action.

You Might Also Like

• 10 Big Ideas from Getting Results the Agile Way
• 30 Day Improvement Sprints: The Key to Making Impact, Changing Habits, and Rapid Learning
• A Quick Roundup of Review for Getting Results the Agile Way
• Mid-Year Review, Career, and Getting Ahead
• Weekly Outcomes for Teams and Leaders

This morning we released some great enhancements to Windows Azure. These new capabilities include:

• Virtual Networks: New Point-to-Site Connectivity (very cool!), Software VPN Device and Dynamic DNS Support
• Virtual Machines: Remote PowerShell and Linux SSH provisioning enhancements
• Cloud Services: Enable Remote Desktop Support Dynamically on Web/Worker Roles
• Ruby SDK: A new Windows Azure SDK support for Ruby

All of these improvements are now available to start using immediately (note: some services are still in preview). Below are more details on them:

Virtual Networks: New Point-to-Site Connectivity and Software VPN Device support

Last week we announced the general availability of Virtual Network support as part of our IaaS release.

Virtual Networking allows you to create a private, isolated network in Windows Azure and treat it as an extension of your on-premises datacenter. For example, you can assign private IP addresses to virtual machines inside a virtual network, specify a DNS, and securely connect it to your on-premises infrastructure using a VPN device in a site-to-site manner.

Here’s a visual representation of a typical site-to-site scenario through a secure Site-To-Site VPN connection:

Today, we are excited to announce that we’re expanding the capabilities of Virtual Networks even further to enable three new scenarios:

New Point-To-Site Connectivity

With today’s release we’ve added an awesome new feature that allows you to setup VPN connections between individual computers and a Windows Azure virtual network without the need for a VPN device. We call this feature Point-to-Site Virtual Private Networking. This feature greatly simplifies setting up secure connections between Windows Azure and client machines, whether from your office environment or from remote locations. 

It is especially useful for developers who want to connect to a Windows Azure Virtual Network (and to the individual virtual machines within it) from either behind their corporate firewall or a remote location. Because it is point-to-site they do not need their IT staff to perform any activities to enable it, and no VPN hardware needs to be installed or configured.  Instead you can just use the built-in Windows VPN client to tunnel to your Virtual Network in Windows Azure.  This tunnel uses the Secure Sockets Tunneling Protocol (SSTP) and can automatically traverse firewalls and proxies, while giving you complete security.

Here’s a visual representation of the new point-to-site scenarios now enabled:

In addition to enabling developers to easily VPN to Windows Azure and directly connect to machines, the new Point-to-Site VPN support enables some other cool new scenarios:

• Small businesses (or departments within an enterprise) who don’t have existing VPN devices and/or network expertise to manage VPN devices can now rely on the Point-to-Site VPN feature to securely connect to their Azure deployments. Because the VPN software to connect is built-into Windows it is really easy to enable and use.

• You can quickly set up secure connections without the involvement from the network administrator, even if your computers are behind a corporate proxy or firewall. This is great for cases where you are at a customer site or working in a remote location (or a coffee shop). 

How to Enable the Point-to-Site Functionality

With today’s release we’ve updated the Virtual Network creation wizard in the Portal so that you can now configure it to enable both ‘Site-to-Site’ and ‘Point-to-Site’ VPN options.  Create a Virtual Network using the “Custom Create” option to enable these options:

Within the Virtual Network Custom Create wizard you can now click a checkbox to enable either the Point-To-Site or Site-To-Site Connectivity options (or both at the same time):

On the following screens you can then specify the IP address space of your Virtual Network.  Once the network is configured, you will create and upload a root certificate for your VPN clients, start the gateway, and then download the VPN client package.  You can accomplish these steps using the “Quick Glance” commands on the Virtual Network dashboard as well as the “Create Gateway” button on the command-bar of the dashboard.  Read this tutorial on how to “Configure a Point-to-Site VPN in the Management Portal” for detailed instructions on how to do this.

After you finish installing the VPN client package on your machine, you will see a new connection choice in your Windows Networks panel.  Connecting to this will establish a secure VPN tunnel your Windows Azure Virtual Network:

Once you connect you will have full IP level access to all virtual machines and cloud services hosted in your Azure virtual network!  No hardware needs to be installed to enable it, and it works behind firewalls and proxy servers.  Additionally, with this feature, you don’t have to enable public RDP endpoints on virtual machines to connect to them - you can instead use the private IP addresses of your virtual private network to RDP to them through the secure VPN connection.

For details instructions on how to do all of the above please read our Tutorial on how to “Configure a Point-to-Site VPN in the Management Portal”

Software VPN Device support for Site-to-Site

With today’s release we are also adding software VPN device support to our existing ‘Site-to-Site VPN’ connectivity solution (which previously required you to use a hardware VPN device from Cisco or Juniper). Starting today we also now support a pure software based Windows Server 2012 ‘Site-to-Site’ VPN option.  All you need is a vanilla Windows Server 2012 installation. You can then run download and run a PowerShell script from the Windows Azure Management Portal that enables the Routing and Remote Access Service (RRAS) on the Windows Server and configures a Site-To-site VPN tunnel and routing table on it.  Sandrino Di Mattia has a step-by-step tutorial on how to do this here.

This allows you to enable a full site-to-site VPN tunnel that connects your on-premises network and machines to your virtual network within Windows Azure - without having to buy a hardware VPN device.

Dynamic DNS Support

With today’s release we have also relaxed restrictions around DNS server setting updates in virtual networks. You can now update the DNS server settings of a virtual network at any time without having to redeploy the virtual network and the VMs in them. Each VM in the virtual network will pick up the updated settings when the DNS is refreshed on that machine, either by renewing the DNS settings or by rebooting the instance.  This makes updates much simpler.

If you’re interested further in Windows Azure Virtual Networks, and the capabilities and scenarios it enables, you can find more information here.

Virtual Machines: Remote PowerShell and Linux SSH provisioning enhancements

Last week we announced the general availability of Virtual Machine support as part of our IaaS release. With today’s update we are adding two nice enhancements:

Support for Optionally Enabling Remote PowerShell on Windows Virtual Machines

With today’s update, we now enable you to configure whether remote PowerShell is enabled for Windows VMs when you provision them using the Windows Azure Management Portal. This option is now available when you create a Virtual Machine using the FROM GALLERY option in the portal:

The last step of the wizard now provides a checkbox that gives you the option of enabling PowerShell Remoting:

When the checkbox is selected the VM enables remote PowerShell, and a default firewall endpoint is created for the deployment.  This enables you to have the VM immediately configured and ready to use without ever having to RDP into the instance.

Linux SSH Provisioning

Previously, Linux VMs provisioned using Windows Azure defaulted to using a password as their authentication mechanism – with provisioning Linux VMs with SSH key-based authentication being optional. Based on feedback from customers, we have now made SSH key-based authentication the default option and allow you to omit enabling a password entirely if you upload a SSH key:

Cloud Services: Enabling Dynamic Remote Desktop for a Deployed Cloud Service

Windows Azure Cloud Services support the ability for developers to RDP into web and worker role instances.  This can be useful when debugging issues.

Prior to today’s release, developers had to explicitly enable RDP support during development – prior to deploying the Cloud Service to production.  If you forgot to enable this, and then ran into an issue in production, you couldn’t RDP into it without doing an app update and redeploy (and then waiting to hit the issue again).

With today’s release we have added support to enable administrators to dynamically configure remote desktop support – even when it was not enabled during the initial app deployment.  This ensures you can always debug issues in production and never have to redeploy an app in order to RDP into it.

How to Enable Dynamic Remote Desktop on a Cloud Service

Remote desktop can be dynamically enabled for all the role instances of a Cloud Service, or enabled for an individual role basis.  To enable remote desktop dynamically, navigate to the Configure tab of a cloud service and click on the REMOTE button:

This will bring up a dialog that enables you to enable remote desktop – as well as specify a user/password to login into it:

Once dynamically enabled you can then RDP connect to any role instance within the application using the username/password you specified for them.

Windows Azure SDK for Ruby

Windows Azure already has SDKs for .NET, Java, Node.js, Python, PHP and Mobile Devices (Windows 8/Phone, iOS and Android).  Today, we are happy to announce the first release of a new Windows Azure SDK for Ruby (v0.5.0).

Using our new IaaS offering you can already build and deploy Ruby applications in Windows Azure.  With this first release of the Windows Azure SDK for Ruby, you can also now build Ruby applications that use the following Windows Azure services:

• Storage: Blobs, Tables and Queues
• Service Bus: Queues and Topics/Subscriptions

If you have Ruby installed, just do a gem install azure to start using it.  Here are some useful links to learn more about using it:

• Ruby Developer Center
• GitHub repository
• Ruby gem
• Rdoc

Like all of the other Windows Azure SDKs we provide, the Windows Azure SDK for Ruby is a fully open source project hosted on GitHub. The work to develop this Ruby SDK was a joint effort between AppFog and Microsoft. I’d like to say a special thanks to AppFog and especially their CEO Lucas Carlson for their passion and support with this effort.

Summary

Today’s release includes a bunch of nice features that enable you to build even better cloud solutions.  If you don’t already have a Windows Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Windows Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu

Hey, it's HighScalability time:

 

• 100 Billion -  Neurons in The Human Brain, As Many Cells as Stars in the Milky Way; 10TB - Tumblr memcache
• Quoteable Quotes:
  • @thoward3: OH: "We make scalability a possibility.. You know, we make 'scalapossibilty'. "
  • Tesla: When wireless is perfectly applied the whole earth will be converted into a huge brain, which in fact it is, all things being particles of a real and rhythmic whole. We shall be able to communicate with one another instantly, irrespective of distance. Not only this, but through television and telephony we shall see and hear one another as perfectly as though we were face to face, despite intervening distances of thousands of miles; and the instruments through which we shall be able to do this will be amazingly simple compared with our present telephone. A man will be able to carry one in his vest pocket.
  • @ADTELLIGENCE: Data on the internet: Data of all of 1993 = Data of 1 second in 2013
  • Nassim Taleb: Man-made complex systems tend to develop cascades and runaway chains of reactions that decrease, even eliminate, predictability and cause outsized events. So the modern world may be increasing in technological knowledge, but, paradoxically, it is making things a lot more unpredictable.
  • The Bw-Tree: A B-tree for New Hardware Platforms: We believe that latch free techniques and state changes that avoid update-in-place are the keys to high performance on modern processors.
  • @rvirding: WhatsApp "Bigger Than Twitter" With Over 200M Monthly Active Users, 8B Inbound And 12B and they use #erlang
  • Jasper Fforde: There’s a lot to be said about merely having a hazy idea of what’s going on but generally reaching the right outcome by following broad policy outlines. In fact, I’ve a sneaky suspicion that it’s the only way of getting things done. Once the horror and unpredictability of unintended consequences gets a hold, even the best-intentioned and noblest of plans generally descend to mayhem, confusion and despair.
  • @enygma: I'm starting to think the Twitter unfollow bug is actually their way to handle scalability
  • @ndubaz: Spent last 2 days training with the Army's latest virtual trainers. More skeptical than ever of scalability and utility for light forces.
  • @bernardgolden: Airbnb workflow control system was 10K (!) lines of bash script.
• Scaling Deployment at Etsy by Daniel Schauenberg. 1.49 billion page views, 4,215,169 items sold, $94.7 million of goods sold, 22+ million members, 800,000+ active shops. LAMMP + Monolithic App + No Branching + Frequent deployment + lots more.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Joe Armstrong is a co-inventor of Erlang and general all around renaissance software tinkerer as shown by his excellent work on writing a C Compiler and his voluminous work on GitHub.

Given the success of Erlang it's probably no surprise that he wrote his thesis on the ground breaking ideas behind Erlang: Making reliable distributed systems in the presence of software errors.

Even if you have yet to join the cult of Erlang the principles behind Erlang are universal and well worth exploring for your own designs. Highly recommended.

Introduction:

If you find you can't keep up with the world around you, then break things down.  Breaking things down is the key to finishing faster.

Breaking things down is also the key to agility.

One of the toughest project management lessons I had to learn was breaking things down into more modular chunks.   When I took on a project, my goal was to make big things happen and change the world. 

After all, go big or go home, right?

The problem is you run out of time, or you run out of budget.  You even run out of oomph.  So the worst way to make things happen is to have a bunch of hopes, plans, dreams, and things, sitting in a backlog because they're too big to ship in the time that you've got.

Which brings us to the other key to agility ... ship things on a shorter schedule.

This re-trains your brain to chunk things down, flow value, chop dependencies down to size, learn, and, move on.

Best of all, if you miss the train, you catch the next train.

The trouble with technology, especially it seems computer technology, is that it keeps "improving".  I've written before about the ethics of the job that we as software architects do and whether or not we should always accept what we do without asking questions, not least of which should be, is this a technology step too far that I am building or being asked to build?

Three articles have caught my eye this week which have made me ponder this question again.

The first is from the technology watcher and author Nicholas Carr who talks about the Glass Collective, an an investment syndicate made up of three companies: Google Ventures, Andreessen Horowitz and Kleiner Perkins Caufield & Byers whose collective aim is to provide seed funding to entrepreneurs in the Glass ecosystem to help jump start their ideas.For those not in the know about Glass it is, according to the Google blog, all about "getting technology out of the way" and has the aim of building technology that is "seamless, beautiful and empowering". Glasses first manifestation is to be Internet-connected glasses that take photos, record video and offer hands-free Internet access right in front of a users' eyes.


Clearly the type of augmented reality that Glass opens up could have huge educational benefits (think of walking around a museum or art gallery and getting information on what you are looking at piped right to you as you look at different works of art) as well as very serious privacy implications. For another view on this read the excellent blog post from my IBM colleague Rick Robinson on privacy in digital cities.

In his blog post Carr refers to a quote from Marshall McLuhan, made a half century ago and now seeming quite prescient:
"Once we have surrendered our senses and nervous systems to the private manipulation of those who would try to benefit by taking a lease on our eyes and ears and nerves, we don’t really have any rights left."
The next thing to catch my eye (or actually several thousand things) was around the whole sorry tale of the Boston bombings. This post in particular from the Wall Street Journal discusses the role of Boston’s so called fusion center that "helps investigators scour for connections among potential suspects, by mining hundreds of law enforcement sources around the region, ranging from traffic violations, to jail records and criminal histories, along with public data like property records."

Whilst I doubt anyone would question the validity of using data in this way to track down people that have performed atrocities such as we saw in Boston, it does highlight just how much data is now collected on us and about us, much of which we have no control over of broadcasting to the world.

Finally, on a much lighter note, we learn that the contraceptive maker Durex has released their "long distance, sexy time fundawear". I'll let you watch the first live trial video of this at your leisure (warning, not entirely work safe) but let's just say here that it adds a whole new dimension to stroking the screen on your smartphone. I guess this one has no immediate privacy issues (providing the participants don't wear their Google Glass at the same time as playing in their fundawear at least) it does raise some interesting questions about how much we will let technology impinge on the most intimate part of our lives.

So where does this latest foray of mine into digital privacy take us and what conclusions, if any, can we draw? Back in 2006 IBM Fellow and Chief Scientist Jeff Jonas posted a comment on his blog called Responsible Innovation: Designing for Human Rights in which he asks two questions: what if we are creating technologies that go in the face of the Universal Declaration of Human Rights and what if systems are designed without the essential characteristics needed to support basic privacy and civil liberties principles?

Jeff argues that if technologies could play a role in any of the arrest, detention, exile, interference, attacks or deprivation mentioned in the Universal Declaration of Human Rights then they must support disclosure of the source upon which such invasions are predicated. He suggests that systems that could affect one’s privacy or civil liberties should have a number of design characteristics built in that allow for some level of auditability as well as ensuring accuracy of the data they hold. Such characteristics as, every data point is associated to its data source and every data point is associated to its author etc. Given this was written in 2006 when Facebook was only two years old and still largely confined to use in US universities this is a hugely prescient and thoughtful piece of insight (which is why Jeff is an IBM Fellow of course).
So, there is an idea. New technologies, when they come along should, be examined to ensure they have built in safeguards that mean such rights as are granted to us all in the Universal Declaration of Human Rights are not infringed or taken away from us. How would this be done and, more importantly of course, what bodies or organisations would we empower to ensure such safeguards were both effective and enforceable? No easy or straightforward answers here but certainly a topic for some discussion I believe.

Solving problems while saving money is always a problem. In Nobody ever got fired for using Hadoop on a cluster they give some counter-intuitive advice by showing a big-memory server may  provide better performance per dollar than a cluster:

1. For jobs where the input data is multi-terabyte or larger a Hadoop cluster is the right solution.
2. For smaller problems memory has reached a GB/$ ratio where it is technically and financially feasible to use a single server with 100s of GB of DRAM rather than a cluster. Given the majority of analytics jobs do not process huge data sets, a cluster doesn't need to be your first option. Scaling up RAM saves on programmer time, reduces programmer effort, improved accuracy, and reduces hardware costs.

 

This is a repost of part 1 of an interview I did for the Boundary blog.

Boundary: What is Facebook’s secret sauce for managing what’s got to be the biggest Big Data project, if you will, on the Web?

Hoff: From several presentations we’ve learned what Facebook insiders like Aditya Agarwal and Robert Johnson, both former Directors of Engineering, consider their secret sauce:

I’m always on the lookout for the best insight and action you can use for work and life.  I especially enjoy when I find somebody who is truly a thought leader, a giant in their space.

After all, I’m a big fan of helping everyone “stand on the shoulders of giants.”

Elizabeth is a giant (actually, more like a Titan) in the field of management.   She brings to the table more than 30 years of experience in the art and science of management.  She’s a former McKinsey partner, a holds a PhD from the Massachusetts Institute of Technology’s Sloan School of Management, and she is the author of McKinsey’s Marvin Bower, and The Definitive Drucker.

She knows her stuff.

So I asked her to share her stuff.

Elizabeth has written a powerful guest post for me on her best lessons learned in the art and science of management:

Management Lessons of a Lifelong Student, by Elizabeth Edersheim.

She reveals the secrets of the best managers and best leaders, and puts it right at your fingertips.  Every now and then you read something that changes your breadth or depth on a topic.   This is one of those posts.

It’s a wealth of insight and action.

Keep in mind that Elizabeth operates at multiple levels of management, so whether you are a line-leader or a CEO, Elizabeth has distilled some key insights you can immediate apply, or refine your thinking, or perhaps lead to a new “ah-ha” moment.

Enjoy, and may the best practices for management serve you well, whether you’re shaping your own business or the business around you.

This Tuesday, April 23, we’ll be hosting Windows AzureConf – a free online event for and by the Windows Azure community.  It will be streamed online from 9:00 AM - 5:00 PM PST via Channel 9, and you can watch it all for free.

I’ll be kicking off the event with a Windows Azure keynote in the morning (a great way to learn more about Windows Azure if you haven’t used it yet!). Following my talk the rest of the day will be full of excellent presentations from members of the Windows Azure community.  You can ask questions from them live and I think you’ll find the day an excellent way to learn more about Windows Azure – as well as hear directly from developers building solutions on it today.

Last year’s Windows AzureConf was a great success, and brought some awesome community members together to deliver some great content around Windows Azure. All of the sessions are available for on-demand viewing on the Windows AzureConf 2012 event page on Channel 9. Sessions from Windows AzureConf 2012 are still available for viewing online.

For more information including a schedule, speaker list or to register visit the Windows AzureConf website.

Hope to see you there!

Scott

P.S. We will also make the presentations available for download after the event in case you miss them.

A few years ago I was asked by one of our customers to help them make better use of their integration layer. Ever since then me and my team have been working on a framework in support of that. This is the fourth in a series of blogs on the development of our framework, and discusses the features it provides. The one that was announced last time, about building blocks, is momentarily postponed.

So far I’ve discussed the goals & challenges surrounding the development activities, but I’d like to focus more on the framework itself now, and what it brings to those that are using it.

As soon as a new party (be it service consumer or service provider) connects to our framework, it can profit directly from the wealth of functionality we deliver out-of-the-box. These ‘generic features’ are exactly what one would expect from a (logical) ESB, and are partly based on the Expanded Enterprise Service Bus Pattern.

As our project was scrum driven, features were developed only when they were needed. Sometimes, during the design & build phase, we discovered a better way of doing things, and sometimes the problem a feature was supposed to address was solved in a completely different way, outside of our scope. But in the end we managed to implement about 20 features, which can roughly be divided into five types: routing, robustness, security, transformation and data storage.

Routing

One of our main objectives is to get messages from A to B without A having to know where B is currently residing. To do this, we make extensive use of the WS-Addressing standard. One of the components in our framework, the routing service, uses the information in the message headers to decide what the next hop will be (hop in this case being another framework component). Most of the times a message is delivered to the backside as soon as it enters the integration layer, something we call simple routing.

However, as soon as some special activity needs to be performed (like data model transformation for example), the message is detoured to one of the framework components not connected to the outside world. We classify those as intermediate services, and they are agnostic of nature – which means they have no clue about the context in which they are called. The necessary information for the message to continue its path is part of the addressing headers as well, making this advanced routing.

A special kind of advanced routing is distribution, which makes it possible to send one message to several subscribers at the same time, using WS-Notification in its most basic form. Finally prioritized routing is a feature which makes sure that a message gets ahead of the rest so to speak – very useful when dealing with a customer waiting for service at a counter while there’s also a bulk load being processed.

Robustness

Of course it’s of eminent importance that nothing goes wrong when delivering the message, or that when it does we can at least deal with the situation (exception management). First thing that happens when we receive a message is that we check to see if it complies with the industry & design standards we enforce (technical validation). Sometimes the consumer/publisher doesn’t want to wait for the (functional) answer but still wants to be informed if his message was technically valid, in which case we send him a response stating just that (technical acceptance).

Two of our features deal with peak load: throttling makes sure the integration layer only takes in what it can handle, while buffering guarantees we don’t overwhelm the applications we connect to. Similar to the second one is postponed delivery which is used when we know beforehand the backside is not available.

But by far the most important of these types of feature is guaranteed delivery. We played around with a bidirectional variant (using WS-RM) but finally settled on an unidirectional implementation, meaning that before we send the message we first store it, and if we receive an HTTP error code we send it again.

Last (and actually least, as it’s hardly used) it’s also possible to have syntactic validation of outgoing messages. But as we like to follow Postel’s law (also known as the robustness principle) we feel it’s the consumer’s responsibility to make sure the message was valid to begin with (you can imagine this took some selling from our part). The only exception is when the payload is altered by the framework.

Security

Every application that wants to connect to the integration layer needs to make itself known using the WS-Security UsernameToken (authentication). For most of the services we expose that’s enough, in a few rare cases such an application has to have explicit permission (authorization).

Not used internally (only when we receive requests from certain external customers) is integrity, where we demand that certain parts of the message headers and payload are signed.

Transformation

Given the fact that not all applications connected to the framework ‘speak the same language’ (as mentioned in the previous blog), there’s an evident need for data model transformation – one of our most used features. A lot less popular is the split feature, which makes it possible to divide a big message into smaller parts.

Data storage

The last few features play a more supportive role to the ones already mentioned. We provide logging to be used during testing & bug-finding sessions and persistence when there’s a need to store the complete message. The latter is frequently used in combination with resending (which is necessary for the guaranteed delivery feature described above), but also in case of auditing requirements.

Conclusion

Most of these features have been around since the first version of our framework, and have proven their generic qualities over time. In a few cases we had to make some alterations and even now there are one or two features which we might implement differently in the future. There’s also a list of additional features but it’s rather short, which I take as a sign that what we have here is pretty complete.

That’s number four; next time I’ll talk a bit about the more specific deliverables we provide our project teams.

I was watching a video on Google Glass with Robert Scoble, and I couldn’t help but wonder about all the possibilities that technology can bring to the table.

Wearable computing bridges the gap between the real world and the things we see in Sci-Fi movies.

Of course, when we overlay information on our world, the key will be turning information into insight and action.  All change isn’t progress, and the market will flush out things faster than ever before.  And, to the victor go the spoils.

In the video, you can see how the Google Glass does a few basic things so far:

1. Take a picture
2. Record a video
3. Get directions to ...
4. Send a message to ...
5. Make a call to ...

The big limit in what it’s capable of, so far, seems to be the batter power.  And of course, a key concern was security.  It’s another reminder how in the software space, security and performance always play a role, even if they are behind the scenes.  In fact, that’s the irony of software security and performance, they are at their best when you don’t notice them.

Security and performance are often unsung heroes.

The big take away for me is that the game is on warp speed now.  By game, I mean, the business of software.  You can go from idea to market pretty fast.   So the big bottlenecks range from the right ideas, to the right people, to the right strategy, to the right execution.

But more importantly, the reminder is this:

Companies with smart people, data-driven insights, a culture of innovation, great software processes, customer focus, and reach around the world, can change the world -- at a faster pace than ever before.

Who knows what we’ll be wearing next?

We have 2 servers running HAProxy on Ubuntu 12.04. We want to set them up in an HA configuration, and for that we chose keepalived.

The first thing we did was look for an existing Chef cookbook for keepalived -- luckily, @jtimberman already wrote it. It's a pretty involved cookbook, probably one of the most complex I've seen. The usage instructions are pretty good though. In any case, we ended up writing our own wrapper cookbook on top of keepalived -- let's call it frontend-keepalived.

The usage documentation for the Opscode keepalived cookbook contains a role-based example and a recipe-based example. We took inspiration from both. In our frontend-keepalived/recipes/default.rb file we have:


include_recipe 'keepalived'

node[:keepalived][:check_scripts][:chk_haproxy] = {
  :script => 'killall -0 haproxy',
  :interval => 2,
  :weight => 2
}
node[:keepalived][:instances][:vi_1] = {
  :ip_addresses => '172.30.10.10',
  :interface => 'frontend_if',
  :track_script => 'chk_haproxy',
  :nopreempt => false,
  :advert_int => 1,
  :auth_type => :pass, # :pass or :ah
  :auth_pass => 'mypass'
}


This code overrides the default values for many of the attributes defined in the Opscode keepalived cookbook. It specifies the floating IP address that will be common between the 2 servers that will each run HAProxy (:ip_addresses). It also specifies the network interface where the multicast-based keepalived protocol (:interface) and the 'check script' which tests whether HAProxy is still running on each server.

However, we still needed a way to specify which of the 2 servers is the master and which is the backup (in keepalived parlance), as well as indicating priorities for each server. The usage document in the keep alived cookbook shows this as an example of using a single role to define the master and the backup:

override_attributes(
  :keepalived => {
    :global => {
      :router_ids => {
        'node1' => 'MASTER_NODE',
        'node2' => 'BACKUP_NODE'
      }
    }
  }
)

We couldn't get this to work (if somebody who did reads this, please leave a comment and tell me how you did it!). Instead, we defined 2 roles, one for the master and one for the backup. Here's the master role:


$ cat frontend-keepalived-master.rb
name "frontend-keepalived-master"
description "install keepalived and set state to MASTER"

override_attributes(
    "keepalived" => {
      "instance_defaults" => {
        "state" => "MASTER",
        "priority" => "101"
}
      }
)

run_list(
  "recipe[frontend-keepalived]"
)

Here's the backup role:


$ cat frontend-keepalived-backup.rb
name "frontend-keepalived-backup"
description "install keepalived and set state to BACKUP"

override_attributes(
    "keepalived" => {
      "instance_defaults" => {
        "state" => "BACKUP",
        "priority" => "100"
}
    }
)

run_list(
  "recipe[frontend-keepalived]"
)

Notice that we override 2 attributes, the state and the priority. The defaults for these are in the Opscode keepalived cookbook, under attributes/default.rb


default['keepalived']['instance_defaults']['state'] = 'MASTER'
default['keepalived']['instance_defaults']['priority'] = 100

This was useful in determining how to specify the stanza overriding them in our roles -- it made us see that we needed to specify the instance_defaults key under keepalived in the role files.

At this point, we added the master role to the Chef run_list of server #1 and the backup role to the Chef run_list of server #2. We had to do one more thing on each server (which we'll add to the default recipe of our frontend-keepalived cookbook): per this very helpful blog post on setting up HAProxy and keepalived, we edited /etc/systctl.conf and added:

net.ipv4.ip_nonlocal_bind=1
then applied it via 'sysctl -p'. This was needed so that HAProxy can listen on the keepalived-created 'floating IP' common to the 2 servers, which is not a real IP tied to an existing local network interface.

Once we ran chef-client on each of the 2 servers, we were able to verify that keepalived does its job by pinging the common floating IP from a 3rd server, then shutting down the network interface 'frontend_if' on each server, with no interruption in the ICMP responses sent from the floating IP. Our next step is to do some heavy-duty testing involving HTTP requests handled by HAProxy, and see that there is no interruption in service when we fail over from one HAProxy server to the other.

UPDATE

My colleague Zmer Andranigian discovered an attribute in the Opscode keepalived cookbook that deals with the sysctl setup. The default value for this attribute is:

default['keepalived']['shared_address'] = false

If this attribute is set to 'true' (for example in one of the 2 roles we defined above), then the keepalived cookbook will create a file called /etc/sysctl.d/60-ip-nonlocal-bind.conf containing:

net.ipv4.ip_nonlocal_bind=1

and will also set it in the running configuration of sysctl.

For reference, the role frontend-keepalived-master would contain the following attributes:


override_attributes(
    "keepalived" => {
      "instance_defaults" => {
        "state" => "MASTER",
        "priority" => "101"
}
      "shared_address" => "true"
   }
)