One of the best ways for making sense of a space is to have a lens for looking at it.  The productivity and results space are well-traversed and the body of knowledge is enormous.  That’s part of the problem.  Without an effective lens, it can be difficult to find, organize, and share the productivity strategies, tactics, etc.

Knowledge Areas
You can think of a “frame” or a “lens” as a set of knowledge areas that make it easier to learn a space.  Together, the knowledge areas form a constellation of knowledge.  For example, the PMBOK (Project Management Body of Knowledge) and SWEBOK (Software Engineering Body of Knowledge) use knowledge areas to cluster related topics, concepts, tasks, and terms to help share the information more effectively.  It’s a way to frame out the space.

Productivity Body of Knowledge
While working on Getting Results the Agile Way, one of the first things I needed to do was carve out the space into meaningful buckets.   By “framing out” the results and productivity space, I created a more effective lens to look at productivity.   This is how I created a “Productivity Body of Knowledge".   I named the collection of knowledge areas for productivity and results, the Results Frame.  Giving it a name and putting it into a simple table, made it easier to refer to and to share as a mental model with others.

The Results Frame (Productivity Knowledge Areas) 
Here is the Results Frame:

Hot Spot Description Action How you take action and manage your activities towards results. Efficiency and Effectiveness How you manage the cost and speed of your results, as well as how you manage the quality of your results. Energy Management How you manage your energy in terms of thinking, feeling, and doing, as well as how you take care of your eating, sleeping, and working out. Expectations How you set and reset expectations with yourself and others. Focus How you focus your time, energy, and attention. Goals and Objectives How you set meaningful goals and objectives for your results. Information Management How you organize and manage information, as well as avoid information overload. Learning How you find the lessons, improve, and correct course. Mindsets and Motivation How you get your head in the game. Planning How you map out the work to be done. Pioritizing How you choose what’s more important. Self-Awareness How to improve your knowledge about yourself in terms of achieving results. Self-Discipline How you correct your behavior. Task Management How you manage your tasks and action items. Time Management How you manage and schedule your time.

The key with these knowledge areas is that they are can contain insight and action.  They are great containers or buckets for productivity principles, patterns, and practices.  To create the buckets, I first gathered up all the “rocks” (the individual principles, patterns, practices, terms, concepts, etc.) , then I group the collections, and then I labeled the buckets.  This is the opposite of making up buckets and then looking to fill them.  I was more interested in creating buckets for proven practices and applied knowledge, rather than treating productivity as an abstract or academic exercise.

Not only did the Results Frame help with organizing my own body of knowledge for results and productivity, but it made it incredibly simple for me to very quickly parse just about any body of knowledge or significant work in the productivity space.  This frame also helped me quickly pressure test productivity systems against a more holistic view, as well as to find their more specific strengths and weaknesses.  Interestingly, you can also use the categories to help evaluate project management approaches and software development approaches.  The frame is useful whether you use it to organize your own knowledge base on productivity, or you use it for teams, or organization.  Don’t just take my word for it though … test drive it and you decide what works for you … you’re the ultimate expert on your context and scenario.

This is so funny I laughed until I cried! Definitely NSFW. OMG it's hilarious, but it's also not a bad overview of the issues. Especially loved: You read the latest post on HighScalability.com and think you are a f*cking Google and architect and parrot slogans like Web Scale and Sharding but you have no idea what the f*ck you are talking about. There are so many more gems like that.

Thanks to Alex Popescu for posting this on MongoDB is Web Scale. Whoever made this deserves a Webby.

These are some of the best ways I’ve found to master time management, get great results, improve your productivity, and amplify your impact:

1. Monthly Improvement Sprints. Use Monthly improvement sprints to cycle through things that you want to focus on. For example, focus on getting in shape in January; use February to learn a new skill. By using a month’s chunk of time, you give yourself enough of a timebox to achieve meaningful results. By using monthly themes, you give yourself a chance to cycle through a variety of your key interests and goals.
2. Balance your time across your Hot Spots. Balance your results across your meaningful buckets. For me, I use a life frame: mind, body, emotion, career, financial, relationships and fun. Each Hot Spot can be broken down into more Hot Spots. For example, my career bucket includes execution, thinking, administration, improvement and relationships.
3. Build a library of reference examples. Collect working examples to learn and model from. Actively looking for the positive examples of successful people around you helps keep your mind focused on success patterns. If you want to manage your time better, model from somebody who is effective. If you want to mange projects better, find somebody with a proven track record and learn from them. Keep in mind that what works for them, may not work for you, but there is no need to start from scratch.
4. Diversify your results. Think in terms of a portfolio of results. This means both producing results in different categories (such as relationships, career, and fun) as well as having some results you count on and some that are risks. Balance this with quitting when you aren’t going to get good at something, or you aren’t getting the return on investment. Diversify your results to avoid having all your eggs in one basket. For example, at work, you might have your flagship project that you can count on, but then add a couple of experimental projects to test the waters.
5. Establish a rhythm of results. Don't let the tail wag the dog. Factor when you create from when you release. This frees you up to focus on creation, without the immediate burden of production or release. Your release rate should match absorption rate and demand. Your production and release can occur at different times and at varying rates. For instance, you could write your eight blog posts on Sunday, then trickle them out over the week.
6. Find a way to flow value. Chunk your results down. Deliver incremental value to yourself or to others. Focus on value-delivered, not backlog burndown. Don't settle for being productive but ineffective. Focus on delivering value keeps you asking the right questions and making the right calls on priorities. Remember that backlogs tend to suffer from rot over time. If you focus on value delivered you won't miss windows of opportunity when they do appear. The other secret here is that focusing on value can be more energizing than tackling an overwhelming backlog, even if all you really changed is perspective.
7. Improve your network. Who you spend time with probably has the largest impact on getting results, personal growth, your quality of life, your career, etc. Here’s a tip: build a mind map of your personal and professional networks and see where you need to tune, prune, or grow. Your purpose is your guide, whether it’s seeing others’ perspectives to keep creative juices flowing, connecting with others you can model and learn from, or simply providing you with the support you need.
8. Make it a project. If you want to get something significant done, make it a project. This includes anything that takes several stages to complete or something that you know probably won’t get done otherwise. List the work to be done and estimate how long it will take.Allocate enough time, energy, and resources to accomplish the work and establish a timeline. Dedicate time to your project and see it through. It’s a simple but proven practice for achieving results. By giving something a start and an end, and by getting your head around the work, you dramatically improve your chances for success. By packaging up the work as a project, you can also look at it in terms of investment and ROI (return on investment). If you don’t think it will be worth the investment, you can make that call. Making it a project provides a lens you can both evaluate the opportunity and manage the work more effectively.
9. Stay flexible in your approach. Be flexible in the "how." If you have a compelling "what" and "why," you'll find the strategies. If something's not working, change your approach. A good sanity check is to ask yourself, "Is it effective?”
10. Sweeping. Periodically sweep up the mess you’ve left behind. Sometimes it’s easier to go back and clean things up than to try and get things right up front. It can be more efficient to batch and focus your time at a later point, than to try and keep things in order the whole time through. Consider sweeping as a deliberate strategy to maximize your energy by batching the work at a specific point in time. Sweeping as a practice gives you a chance to go back and improve things as well as integrate new learnings.

For more patterns and practices for improving results, see my book, Getting Results the Agile Way.

 With summer almost gone, it's time to fall into some good links...

• Hibari - distributed, fault tolerant, highly available key-value store written in Erlang. In this video Scott Lystig Fritchie gives a very good overview of the newest key-value store. 
• Tweets of Gold
  • lenidot: with 12 staff, @tumblr serves 1.5billion pageviews/month and 25,000 signups/day. Now that's scalability!
  • jmtan24: Funny that whenever a high scalability article comes out, it always mention the shared nothing approach
  • mfeathers: When life gives you lemons, you can have decades-long conquest to convert lemons to oranges, or you can make lemonade.
  • OyvindIsene: Met an old man with mustache today, he had no opinion on #noSQL. Note to myself: Don't grow a mustache, now or later. 
  • vlad003: Isn't it interesting how P2P distributes data while Cloud Computing centralizes it? And they're both said to be the future.
• You may be interested in a new DevOps Meetup organized by Dave Nielson, so you know it will be good.

The need for IT consolidation is most evident in two types of organizations. In the first group, IT grew organically with business over the decades, and survived changes of strategy, management, staff and vendor orientation. The second group of businesses capital groups are characterized by rapid growth through acquisitions (followed by attempts to integrate radically different IT environments). In both groups, their IT infrastructures have typically been pieced together over the past 20 (or more) years.

Read more on BigDataMatters.com

Consistent Hashing is a specific implementation of hashing that is well suited for many of today’s web-scale load balancing problems. Specifically, it can be seen in use in various caching solutions like Memcached and is applicable to NoSQL solutions as well. Consistent Hashing is used particularly because it provides a solution for the typical “hashcode mod n” method of distributing keys across a series of servers. It does this by allowing servers to be added or removed without significantly upsetting the distribution of keys, nor does it require that all keys be rehashed to accommodate the change in the number of servers.

You can read the full store here.

In Agile, everyone agrees on the concept that continuous improvement is a good thing. In Scrum and also in most Kanban practices we even have a ceremony to support this, namely The Retrospective. This periodically occurring meeting (often every other week) with the entire team plays a vital part in the process and in team [...]

patterns & practices Parallel Programming with Microsoft .NET is now available.  The book shows design patterns to help developers use the .NET 4 Task Parallel Library (TPL) to write parallel applications successfully.

Contents at a Glance

• Authors and Disclaimers
• Foreword
• Preface
• Acknowledgments
• Introduction
• Parallel Loops
• Parallel Tasks
• Parallel Aggregation
• Futures
• Dynamic Task Parallelism
• Pipelines
• Appendix A: Adapting Object-Oriented Patterns
• Appendix B: Debugging and Profiling Parallel Applications
• Appendix C: Technology Overview
• Glossary
• References

The Patterns
The book describes six key parallel patterns for data and task parallelism and how to implement them using the TPL.

The Book

• Free in HTML -  Parallel Programming with .NET (MSDN)
• Print / EBook -  Parallel Programming with .NET (O’Reilly)

The Code Samples

• Parallel Programming Code Samples (CodePlex)

The Talk

• Patterns of Parallel Programming (TechEd)

The Community

• Parallel Programming with .NET Community KB (CodePlex)

In this post I'll cover the difference between multi-core concurrency that is often referred to as Scale-Up and distributed computing that is often referred to as Scale-Out mode. 

more..

Source: Scale-out vs Scale-up (http://www.dzone.com/links/r/scaleout_vs_scaleup.html) by Nati Shalom

I blogged a while ago about MySQL fault-tolerance and disaster recovery techniques. At that time I was experimenting with the non-free InnoDB Hot Backup product. In the mean time I discovered Percona's XtraBackup (thanks Robin!). Here's how I tested XtraBackup for doing a hot backup and a restore of a MySQL database running Percona XtraDB (XtraBackup works with vanilla InnoDB too).

First of all, I use the following Percona .deb packages on a 64-bit Ubuntu Lucid EC2 instance:


# dpkg -l | grep percona
ii  libpercona-xtradb-client-dev      5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database development files
ii  libpercona-xtradb-client16        5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database client library
ii  percona-xtradb-client-5.1         5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database client binaries
ii  percona-xtradb-common             5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database common files (e.g. /etc
ii  percona-xtradb-server-5.1         5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database server binaries

I tried using the latest stable XtraBackup .deb package from the Percona downloads site but it didn't work for me. I started a hot backup with /usr/bin/innobackupex-1.5.1 and it ran for a while before dying with "InnoDB: Operating system error number 9 in a file operation." See this bug report for more details.
After unsuccessfully trying to compile XtraBackup from source, I tried XtraBackup-1.3-beta for Lucid from the Percona downloads. This worked fine.
Here's the scenario I tested against a MySQL Percona XtraDB instance running with DATADIR=/var/lib/mysql/m10 and a customized configuration file /etc/mysql10/my.cnf. I created and attached an EBS volume which I mounted as /xtrabackup on the instance running MySQL.
1) Take a hot backup of all databases under that instance:
/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --pasword=xxxxxx /xtrabackup
This will take a while and will create a timestamped directory under /xtrabackup, where it will store the database files from DATADIR. Note that the InnoDB log files are not created unless you apply step 2 below.
As the documentation says, make sure the output of innobackupex-1.5.1 ends with:
100901 05:33:12 innobackupex-1.5.1: completed OK!

2) Apply the transaction logs to the datafiles just created, so that the InnoDB logfiles are recreated in the target directory:

/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --password=xxxxxx --apply-log /xtrabackup/2010-09-01_05-21-36/

At this point, I tested a disaster recovery scenario by stopping MySQL and moving all files in DATADIR to a different location.

To bring the databases back to normal from the XtraBackup hot backup, I did the following:

1) Brought back up a functioning MySQL instance to be used by the XtraBackup restore operation:

i) Copied the contents of the default /var/lib/mysql/mysql database under /var/lib/mysql/m10/ (or you can recreate the mysql DB from scratch)

ii) Started mysqld_safe manually:

 mysqld_safe --defaults-file=/etc/mysql10/my.cnf

This will create the data files and logs under DATADIR (/var/lib/mysql/m10) with the sizes specified in the configuration file. I had to wait until the messages in /var/log/syslog told me that the MySQL instance is ready and listening for connections.

2) Copied back the files from the hot backup directory into DATADIR

Note that the copy-back operation below initially errored out because it tried to copy the mysql directory too, and it found the directory already there under DATADIR. So the 2nd time I ran it, I moved /var/lib/mysql/m10/mysql to mysql.bak. The copy-back command is:

/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --copy-back /xtrabackup/2010-09-01_05-21-36/

You can also copy the files from  /xtrabackup/2010-09-01_05-21-36/ into DATADIR using vanilla cp.

3) If everything went well in step 2, restart the MySQL instance to make sure everything is OK.

At this point, your MySQL instance should have its databases restored to the point where you took the hot backup. If that instance is used in replication, you will most likely need to adjust the master_log_file and master_log_position so that it gets back in sync with its master.

Note that XtraBackup can also run in a 'stream' mode useful for compressing the files generated by the backup operation. Details in the documentation.

Can you have your ACID cake and eat your distributed database too? Yes explains Daniel Abadi, Assistant Professor of Computer Science at Yale University, in an epic post, The problems with ACID, and how to fix them without going NoSQL, coauthored with Alexander Thomson, on their paper The Case for Determinism in Database Systems. We've already seen VoltDB offer the best of both worlds, this sounds like a completely different approach.

The solution, they propose, is: 

This is a behind the scenes look at my involvement in the creation of the MSDN Hub pages. 

The MSDN Hub pages you get to from the main “buttons” on the MSN home (pictured above.)   Specifically, these are the actual pages:

• Cloud Hub
• Desktop Hub
• Phone Hub
• Web Hub

The intent of the MSDN Hub pages to create some simple starting points for some of our stories on the Microsoft developer platform.  For example, you might want to learn the Microsoft cloud story, but you might not know the “building blocks” that make up the story (Windows Azure, SQL Azure, and Windows Azure platform AppFabric.)    A Hub page would be a way to share a simple overview of the story, a way to get started with the technology, common application paths and roadmaps, and where to go for more (usually the specific Developer Centers that would be a drill down for a specific technology.)

Why Was I Involved?
If you’re used to seeing me produce Microsoft Blue Books for patterns & practices, and focusing on architecture and design, security, and performance, it might seem odd that I was part of the team to create the MSDN Hub pages.   Actually, it makes perfect sense, and here’s why -- They needed somebody who had looked across the platform and technology stacks and could help put the story together.  Additionally:

• The purpose of the MSDN Hubs was to tell our platform story and put the platform leggos together in a meaningful way.  This is a theme I’ve had lots of practice with over the years on each of my patterns & practices projects.
• I was already working on the Windows Developer Center and the Windows IA (Information Architecture), and the .NET IA, so I was part of the right v-teams and regularly interacting with the key people making this happen.
• I shipped our platform playbook for the Microsoft Platform – the patterns & practices Application Architecture Guide, second edition.
• I had put together a map of our Microsoft application platform story, as well as created maps, matrixes, and drill downs on our stories for key clusters of our Microsoft technologies including the presentation technology stack, the data access technology stack, the workflow technology stack, and the integration technology stack, etc. 
• I had previously worked on specific projects to create a catalog to organize and share the patterns & practices catalog of assets. (Internally we called this the “the Catalog Project”.) 
• I had worked on an extensive catalog of app types, which served as the backbone for some downstream patterns & practices projects while influencing others, including factories, early attempts at MSF “app templates”,  our patterns & practices catalog (so we could  enable browsing our catalog by application type), and then of course, the Microsoft Application Architecture Guide.
• I teamed across product teams, support, field, industry experts, and customers to create a canonical set of app types for the App Arch Guide.  Here’s what Grady Booch, IBM tech fellow, had to say about the App Types work -- “an interesting language for describing a large class of applications.”  Naturally, this work fed into the MSDN Hub pages since we need to map out the most common application patterns, paths, and combos. 

My Approach
My approach was pretty simple.  I worked closely with a variety of team members including Kerby Kuykendall, Howard Wooten, Chris Dahl, John Boylan, Cyra Richardson, Pete M Brown, and Tim Teebken.  I started off working mostly with Kerby, but eventually I ended up working closest with Tim because he became my main point of contact for influencing and shaping the work.  That said, it was still a lot of mock ups, ad-hoc meetings, whiteboard discussions, and group meetings to shape the overall result.  Tim did a stellar job of integrating my feedback and recommendations, as well as sanity checking group decisions with me.

I also sanity checked things with customers, and I worked closely with folks on the Microsoft Developer Platform Evangelism team including Tim Sneath and Jaime Rodriguez.  They were passionate about having a way to tell our platform story, show common app pathy, and how to put our leggos together, and make technology choices.  I tried to surface this in the design and information model for the Hub pages.

The Hubs
For the Hubs, at one of our early meeting in November of 2009, I recommended a we use “deployment targets” as a way to help slice things up and keep it simple.  Specifically:

• Cloud
• Desktop
• Games
• Mobile
• Server
• Web

As you can see, it maps very well to the App Types set I created circa 2004, but I evolved it to account for a few things.  First, I included learnings from working on the App Arch guide (such as moving away from Rich Client to just “desktop.”) Second,  I tried to pin it more directly to physical deployment targets to keep it simple.  As a developer, you can write apps to target the Web (a “Web” browser app), a desktop (such as a Windows client, or Silverlight, or WPF, etc.), a game (game console), etc.   Third, I aligned with marketing efforts, such as recommending we use the “deployment target” metaphor and I renamed the “Mobile” bucket to “Phone” (which worked, because it extended the “deployment target” metaphor, was still easy to follow, and kept things simple.

I also kept the physical aspect of the “deployment target” metaphor loose.   For example, “Web” could run on server, or “desktop”, etc.  Instead, I wanted to bubble up interesting intersections of application types plus common deployment targets, and keep it simple.

The Server Hub
For the server hub, I recommended addressing our story from a few lenses.  First, we have server-side products that can be extended, such as SharePoint, Exchange, SQL Server, etc.  That lens is pretty straightforward.  Second, I recommended focusing on “Service.”  Here’s where it’s hard for folks to follow if they aren’t familiar with server-side development.  While you can lump “service” under “Cloud” (as a cloud developer, I can write a Web app, a service, etc.), the “service” story is a very special one.  It’s the evolution of our “middleware developers” and our “server-side developers.”  It’s the path that the COM builders and server-side component builders shifted to … a more message-based architecture over an object-based one, as well as a shift to replacing DCOM with HTTP.

So if we had a Server Hub, it realistically should address building on our server-side products/technologies (SharePoint, Exchange, SQL Server, AppFabric, etc.) and it should address “Services.”  Sure you could also lump SharePoint under Web or Services under Cloud, but you can also bubble up and give focus to some of the fundamental parts of our Microsoft application developer platform.

To be fair, a lot of folks moved around during the MSDN Hub page project, and as new folks came on board, the history, insights, and some of the work may have gotten lost.

How To Solve the Issue of Too Many Hubs
This was my suggestion for dealing with too many Hubs:

“I think one thing that helps to keep in mind is that different people will want different views – but I think it’s simpler to choose the most useful one across the broadest set of scenarios.   That’s why Burger King and McDonald’s have a quick simple visual menu of the most common options … then you can drill in for more with their detailed menu if needed.  I like that metaphor because it addresses the “simple” + “complete”  Platform is a pretty solid bet – with an orientation around “tribes” (I’ll walk you through when we sync live) – after all, we do competitive assessments against platforms and that’s where we need to win.”

I also made a few additional recommendations to deal with the problem of “simple” + “complete”:

1. Add an “Office/SharePoint”, and a “Server” (SQL Server, Windows Server, Exchange) – the Office/SharePoint platform tends to have a tribe of customers that speak the same language and share the same context … different than your everyday .NET dev.   It’s like BizTalk in that it’s a specialized space.
2. Use a carousel approach to feature the main 4, then a “view more…” pattern so show the full 6 or so top level – and leave breathing room.  I would go to a page that shows the full set at the top, but then shows the full set of products against a durable backdrop.  This would address the “AND” solution of both “Simple” and “Complete”

This would provide a “durable” + “expandable” … AND… “simple” + “complete” … and in the end, a “platform guidance” approach.

While I’m not a graphic artist, I had done some mockups to help illustrate the point …

Home with “View More >>”

View More … (full dev center against a durable backdrop, full tech stack, Roadmaps)

This is blowing up a section of the “Microsoft Developer Products and Technologies” map above that I had created to illustrate the point:

… etc.  and the map of course, continued with the technology stack, but used a robust backdrop.  The map would need to be vetted across product teams but that’s just the point … have a common map that shows our “catalog” of technologies that customers could easily browse, and that product teams stand behind, while providing simple jump points to either MSDN Developer Centers or Product Team sites.

The Information Model for the Hub Pages
I tried to be inclusive in the information model and I wanted to address and integrate customer pain points I’ve heard about how we tell our story over the years, as well as keep the pages simple and useful, based on what I know about customer usage patterns.   These are the main sections I recommended:

• Key Technologies
• Overview
• Common Paths
• Download and Install
• Build Your First Apps
• Arch / Design
• Roadmaps
• Developer Centers and Home Pages

A picture is worth a 1000 words (note this is a picture of the “Information Model” – NOT the page mockups):

My key customer scenarios and questions I used for test cases were:

1. Can I quickly make sense of the particular story? (cloud, web, etc.)
2. Can I figure out what technologies are relevant?
3. Can I figure out how to choose which technology combination to use?
4. Can I quickly install whatever I need to get up and running?
5. Can I build a quick Hello World app to get my feet wet and some quick confidence?
6. Can I figure out the roadmap for the technology or technologies?
7. Can I get a quick sense of the most common application patterns, app types, and design patterns?
8. Can I figure out my key architecture and design issues if necessary?
9. Can I figure out where to go for more? (which Dev Centers, which landing pages, which resources, etc.)

My key recommendations included:

• Provide  the most common application paths for the given stories (based on the App Arch guide and DPE feedback.)
• Start with text-based articles and if feasible, add video.
• If we have video overviews, then have the PG teams create them so that they stand behind them and connect with customers
• Keep the areas repeatable across the Hub pages
• For a metaphor, think of the Hubs as the centralizing story for our “building blocks” … and think of our building blocks as the technologies and the “Developer Centers”
• Name the articles “How To –“ and make them actionable and step-by-step
• Name the videos “Explained” if they explain a concept and “How To” if they provide a set of steps.
• Eventually create “Getting Started Guides” for each of the Hubs.

Tim Teebken and I had many late night discussions and drill-downs, which made the work interesting and exciting.  It’s always great to work with smart people.  We pushed each other, challenged each other, and ultimately we stayed on the same page on the journey.

Well, that’s the story in a nutshell.   That’s a behind the scenes look at the making of The Design of the MSDN Hubs and the role that I played.

If you have feedback on the information model, please feel fee to send my way.  You can use the contact form on my blog.

First of all, I want to emphasize that this is NOT a disaster recovery strategy I recommend. However, in a pinch, it might save your ass. Here's the scenario I have:

• 2 m1.large instances running Ubuntu 10.04 64-bit and the Percona XtraDB MySQL builds (for the record, the exact version I'm using is "Server version: 5.1.43-60.jaunty.11-log (Percona SQL Server (GPL), XtraDB 9.1, Revision 60")
• I'll call the 2 servers db101 and db201
• each server is running 2 MySQL instances -- I'll call them m1 and m2
• instance m1 on db101 and instance m1 on db201 are set up in master-master replication (and similar for instance m2)
• the DATADIR for m1 is /var/lib/mysql/m1 on each server; that file system is mounted from an EBS volume (and similar for m2)
• the configuration files for m1 are in /etc/mysql1 on each server -- that directory was initially a copy of the Ubuntu /etc/mysql configuration directory, which I then customized (and similar for m2)
• the init.d script for m1 is in /etc/init.d/mysql1 (similar for m2)

What I tested:

• I took a snapshot of each of the 2 EBS volumes associated with each of the DB servers (4 snapshots in all)
• I terminated the 2 m1.large instances
• I launched 2 m1.xlarge instances and installed the same Percona distribution (this was done via a Chef recipe at instance launch time); I'll call the 2 new instances xdb101 and xdb102
• I pushed the configuration files for m1 and m2, as well as the init.d scripts (this was done via fabric)
• I created new volumes from the EBS snapshots (note that these volumes can be created in any EC2 availability zone)
• On xdb101, I attached the 2 volumes created from the EBS snapshots on db101; I specified /dev/sdm and /dev/sdn as the device names (similar on xdb201)
• On xdb101, I created /var/lib/mysql/m1 and mounted /dev/sdm there; I also created /var/lib/mysql/m2 and mounted /dev/sdn there (similar on xdb201)
• At this point, the DATADIR directories for both m1 and m2 are populated with 'live files' from the moment when I took the EBS snapshot
• I made sure syslog-ng accepts UDP traffic from localhost (by default it doesn't); this is because by default in Ubuntu mysql log messages are sent to syslog --> to do this, I ensured that "udp(ip(127.0.0.1) port(514));" appears in the "source s_all" entry in /etc/syslog-ng/syslog-ng.conf

At this point, I started up the first MySQL instance on xdb101 via "/etc/init.d/mysql1 start". This script most likely will show [fail] on the console, because MySQL will not start up normally. If you look in /var/log/syslog, you'll see entries similar to:
Aug 31 18:03:21 xdb101 mysqld: 100831 18:03:21 [Note] Plugin 'FEDERATED' is disabled.Aug 31 18:03:21 xdb101 mysqld: InnoDB: The InnoDB memory heap is disabledAug 31 18:03:21 xdb101 mysqld: InnoDB: Mutexes and rw_locks use GCC atomic builtinsAug 31 18:03:22 xdb101 mysqld: 100831 18:03:22  InnoDB: highest supported file format is Barracuda.Aug 31 18:03:23 xdb101 mysqld: InnoDB: The log sequence number in ibdata files does not matchAug 31 18:03:23 xdb101 mysqld: InnoDB: the log sequence number in the ib_logfiles!Aug 31 18:03:23 xdb101 mysqld: 100831 18:03:23  InnoDB: Database was not shut down normally!Aug 31 18:03:23 xdb101 mysqld: InnoDB: Starting crash recovery.
If you wait a bit longer (and if you're lucky), you'll see entries similar to:
Aug 31 18:04:20 xdb101 mysqld: InnoDB: Restoring possible half-written data pages from the doublewriteAug 31 18:04:20 xdb101 mysqld: InnoDB: buffer...Aug 31 18:04:24 xdb101 mysqld: InnoDB: In a MySQL replication slave the last master binlog fileAug 31 18:04:24 xdb101 mysqld: InnoDB: position 0 15200672, file name mysql-bin.000015Aug 31 18:04:24 xdb101 mysqld: InnoDB: and relay log fileAug 31 18:04:24 xdb101 mysqld: InnoDB: position 0 15200817, file name ./mysqld-relay-bin.000042Aug 31 18:04:24 xdb101 mysqld: InnoDB: Last MySQL binlog file position 0 17490532, file name /var/lib/mysql/m1/mysql-bin.000002Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 InnoDB Plugin 1.0.6-9.1 started; log sequence number 1844705956Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Recovering after a crash using /var/lib/mysql/m1/mysql-binAug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Starting crash recovery...Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Crash recovery finished.
At this point, you can do "/etc/init.d/mysql1 restart" just to make sure that both stopping and starting that instance work as expected. Repeat for instance m2, and also repeat on server xdb201.
So....IF you are lucky and the InnoDB crash recovery process did its job, you should have 2 functional MySQL instances one each of xdb101 and xdb201. I tested this with several pairs of servers and it worked for me every time, but I hasten to say that YMMV, so DO NOT bet on this as your disaster recovery strategy!
At this point I still had to re-establish the master-master replication between m1 on xdb101 and m1 on xdb201 (and similar for m2). 
When I initially set up this replication between the original m1.large servers, I used something like this on both db101 and db201:
CHANGE MASTER TO MASTER_HOST='master1', MASTER_PORT=3306, MASTER_USER='masteruser', MASTER_PASSWORD='xxxxxx';"
The trick for me is that master1 points to db201 in db101's /etc/hosts, and vice-versa.
On the newly created xdb101 and xdb201, there are no entries for master1 in /etc/hosts, so replication is broken. Which is a good thing initially, because you want to have the MySQL instances on each server be brought back up without throwing replication into the mix.
Once I added an entry for master1 in xdb101's /etc/hosts pointing to xdb201, and did the same on xdb201, I did a 'stop slave; start slave; show slave status\G' on the m1 instance on each server. In all cases I tested, one of the slaves was showing everything OK, while the other one was complaining about   not being able to read from the master's log file. This was fairly simply to fix. Let's assume xdb101 is the one complaining. I did the following:

• on xdb201, I ran 'show master status\G' and noted the file name (for example "mysql-bin.000017") and the file position (for example 106)
• on xdb101, I ran the following command: "stop slave; change master to master_log_file='mysql-bin.000017', master_log_pos=106; start slave;"
• not a 'show slave status\G' on xdb101 should show everything back to normal

Some lessons:

• take periodic snapshots of your EBS volumes (at least 1/day)
• for a true disaster recovery strategy, use at least mysqldump to dump your DB to disk periodically, or something more advanced such as Percona XtraBackup; I recommend dumping the DB to an EBS volume and taking periodic snapshots of that volume
• the procedure I detailed above is handy when you want to grow your instance 'vertically' -- for example I went from m1.large to m1.xlarge

David Zinger has shared his take on Getting Results the Agile Way in a review on his blog at http://www.davidzinger.com.  You can check out David’s review of Getting Results the Agile Way at:

•  Employee Engagement and Getting Results the Agile Way

If you don’t know David, he’s a writer, educator, speaker, consultant, and all around good guy, that lives and breathes employee engagement, which is all about how individuals, teams, and leaders can be more engaged in the work that they do.   His passion and super skill is helping people get more out of their work, and unleash their passion on the job.

You can see David in action at The Employee Engagement Network and you can check out his amazingly concise and insightful book, Zengage.  In a nutshell, Zengage is a short powerful book to help you get more out of your work by getting more into your work.

Today was the final day of 30 Days of Getting Results, based on my book Getting Results the Agile Way.

It’s basically free training on many of the key skills for:

• Making meaning / Finding purpose
• Focus
• Productivity
• Time Management
• Energy Management

It’s the synthesis of what I’ve learned at Microsoft in the trenches and my travels in life, as well as from many very good mentors.  Mostly, it’s what I’ve learned from the school of hard knocks, trial and error, and necessity (I think of it as a collection of Microsoft Survival Skills in a box.)

Agile Results
Agile Results is a personal results system for work and life.  It’s a simple system for meaningful results that you can apply whenever you need it.  It’s a way to help you make the most of what you’ve got.  It’s a way to be the author of your life, and write your story forward.

Problems Addressed

• How to improve your personal productivity and personal effectiveness
• How to achieve work-life balance
• How to find your motivation and drive
• How to find your purpose and your passion
• How to deal with being overloaded or overwhelmed
• How to change a habit and make it stick
• How to focus and direct your attention with skill
• How to manage your time
• How to spend more time on the things that really matter to you
• How to play to your strengths and spend less time in weaknesses
• How to make the most of your your moments, days, weeks, months, and years

It’s been a fun ride and I’ve heard some amazing stories from people around the world.  I’ve enjoyed the emails and stories from people who have used the system to enhance their life, a day at a time, a story at a time.

30 Lessons on Getting Results the Agile Way

• Day 1 – Take a Tour of Getting Results the Agile Way
• Day 2 – Monday Vision – Use Three Stories to Drive Your Week
• Day 3 – Daily Outcomes – Use Three Stories to Drive Your Day
• Day 4 – Let Things Slough Off
• Day 5 – Hot Spots – Map Out What’s Important
• Day 6 – Friday Reflection – Identify Three Things Going Well and Three Things to Improve
• Day 7 – Setup Boundaries and Buffers
• Day 8 – Dump Your Brain to Free Your Mind
• Day 9 – Prioritize Your Day with MUST, SHOULD, and COULD
• Day 10 – Feel Strong All Week Long
• Day 11 – Reduce Friction and Create Glide Paths for Your Day
• Day 12 – Productivity Personas – Are You are a Starter or a Finisher?
• Day 13 – Triage Your Action Items with Skill
• Day 14 – Carve Out Time for What’s Important
• Day 15 – Achieve a Peaceful Calm State of Mind
• Day 16 – Use Metaphors to Find Your Motivation
• Day 17 – Add Power Hours to Your Week
• Day 18 – Add Creative Hours to Your Week
• Day 19 — Who are You Doing it For?
• Day 20 — Ask Better Questions, Get Better Results
• Day 21 – Carry the Good Forward, Let the Rest Go
• Day 22 – Design Your Day with Skill
• Day 23 — Design Your Week with Skill
• Day 24 – Bounce Back with Skill
• Day 25 – Fix Time. Flex Scope
• Day 26 – Solve Problems with Skill
• Day 27 – Do Something Great
• Day 28 – Find Your One Thing
• Day 29 – Find Your Arena for Your Best Results
• Day 30 – Take Agile Results to the Next Level

Key Links

• 30 Days of Getting Results (All 30 self-paced lessons and nuggets)
• The Book – Getting Results the Agile Way
• Getting Started with Agile Results

If you visit just one lesson, check out Day 27 – Do Something Great.   If you check out two, check out Day 10 – Feel Strong All Week Long.

Pomegranate is a novel distributed file system built over distributed tabular storage that acts an awful lot like a NoSQL system. It's targeted at increasing the performance of tiny object access in order to support applications like online photo and micro-blog services, which require high concurrency, high throughput, and low latency. Their tests seem to indicate it works:

We have demonstrate that file system over tabular storage performs well for highly concurrent access. In our test cluster, we observed linearly increased more than 100,000 aggregate read and write requests served per second (RPS). 

Rather than sitting atop the file system like almost every other K-V store, Pomegranate is baked into file system. The idea is that the file system API is common to every platform so it wouldn't require a separate API to use. Every application could use it out of the box.

The features of Pomegranate are:

• It handles billions of small files efficiently, even in one directory;
• It provide separate and scalable caching layer, which can be snapshot-able;
• The storage layer uses log structured store to absorb small file writes to utilize the disk bandwidth;
• Build a global namespace for both small files and large files;
• Columnar storage to exploit temporal and spatial locality;
• Distributed extendible hash to index metadata;
• Snapshot-able and reconfigurable caching to increase parallelism and tolerant failures;
• Pomegranate should be the first file system that is built over tabular storage, and the building experience should be worthy for file system community. 

Can Ma, who leads the research on Pomegranate, was kind enough to agree to a short interview.

This is a completely unscientific observation but over the years of being an architect I have observed the following characteristics in people that claim to be members of this profession.  I refer to these as the three C's of being an IT architect. Some people have only one of these but most have a mix of all three, with maybe one being dominant. The three C's are: 

• Creatives – These are the ideas people, that are keen to do new stuff. They are the people that build the solutions that address the business requirements. They have an intimate knowledge of technology (in their particular area of interest or concern) and want to use that technology. If you don't have these people in your project/team/organisation then nothing will actually get done. The downside of having too much of this attribute is that eventually you have to stop creating and ship something so you need to know when to stop. 
• Consumers – These are the people that use what the creatives create. Whilst they may not create anything new this type of architect is not without merit. They often combine what others have done in new and innovative ways. We sometimes refer to this activity as reuse, one of the Holy Grails of IT so it is not to be underestimated. If you don't have these people in your project/team/organisation then chances are the ideas from the creatives will not get fully realised. The downside of having too much of this attribute is that there is a limit to what you can build out of reusing stuff and eventually someone has to come up with some new ideas. 
• Connectors – These are the people that don't create or reuse but know people that can do these things and join the two together. Again this is not to be an underestimated skill. After all if a seller cannot find a buyer what's the point! If you don't have these people in your project/team/organisation then the two previous types won't find each other. The downside of having too much of this attribute is that you ain't going to do anything if all you do is push ideas of others around without creating or reusing things.

From my observations I reckon a ratio of Creatives to Consumers to Connectors needs to be something like 4:2:1.
Incidentally, my guess is that these actually apply to other professions as well but I have even less scientific evidence about those.

Here is the latest in my link-listing series.  Also check out my VS 2010 and .NET 4 series and ASP.NET MVC 2 series for other on-going blog series I’m working on.

[In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu]

.NET/WPF/EF

• 5 Little C#/.NET Wonders that Make Code Better: Nice blog post from James that highlights 5 nice language/framework tips you can take advantage of within your apps.  I’m betting a lot of people didn’t know about the StopWatch class.

• WPF Ribbon Control Released: The WPF team recently released the final V1 release of the WPF Ribbon control.  This makes it easy to build ribbon-based applications. Also read this WPF team blog post for lots of great details on what it enables.

• EF4 .Include() Method w/ Lambda Support: Ever wanted to use a Lambda expression instead of a string parameter when eagerly loading associations in EF4 using the Include() method?  This blog post shows you how you can.

ASP.NET

• Performance Tuning tricks for ASP.NET and IIS7 Part 1 and Part 2: Mads has a nice set of posts that detail some great performance optimization tips you can use with ASP.NET and IIS7.

• Better ViewState Management in ASP.NET 4 with the ViewStateMode Property: Scott Mitchell has a good article that discusses the new ViewStateMode property supported on ASP.NET 4 controls. This provides much more control over how viewstate is used within a page.

• Filtering Data using ASP.NET 4’s QueryExtender Control: Scott Mitchell has a nice article that talks about the new <asp:QueryExtender> control in ASP.NET 4 that makes enabling dynamic querying against a LinqDataSource or EntityDataSource really easy.

• Techniques to Avoid Duplicate URLs: Scott Mitchell has another great article that discusses techniques you can use within ASP.NET to avoid exposing duplicate URLs to the same content within your web-sites.  These can help improve your search engine traffic.

• Automatically pre-populate fields for Insert in ASP.NET 4: Scott Hunter has a nice blog post that discusses how to specify default values for insert with data controls in ASP.NET 4.

ASP.NET MVC

• MvcContrib Template Gallery: Download 58 pre-built ASP.NET MVC Template Gallery styles that you can use to customize the look and feel of your applications. All available under a Community Commons license.

• Step into the ASP.NET MVC Source Code with the VS Debugger: Nice blog post by Gunnar that describes how to use VS 2010’s built-in symbol server support to enable you to step into the ASP.NET MVC Framework source code when debugging.

• Installation Options for ASP.NET MVC: Jon has a good post that describes how to install ASP.NET MVC 2 on a server. There are a couple of options you can use for servers that don’t already have ASP.NET MVC installed – the easiest is to just enable the “Copy Local” flag on System.Web.Mvc.dll and install it in the \bin directory.

IIS Developer Express

• Visual Studio Support for IIS Express, SQL 4 Compact Edition, and Razor syntax: Channel 9 video with Damian Edwards and Scott Hanselman that discusses the support coming to enable IIS Express, SQL 4 CE, and Razor syntax support within Visual Studio 2010.

• How to Use IIS Express with VS today: Kamran has a nice blog post that describes how you can use IIS Express with Visual Studio today – without having to wait for the official Visual Studio patch to enable it.

Silverlight and Windows Phone 7

• Developers Roadmap for Windows Phone 7: The Windows Phone 7 team blogs about the release timeframe for Windows Phone 7 and the Visual Studio and Expression Blend tools for targeting it.  The official developer release will be on September 16th.

• Windows Phone 7 in 7 Minutes: Really nice set of 7 minute videos that enable you to quickly understand and learn Windows Phone 7 and the development fundamentals behind it.

• Windows Phone 7 Jumpstart Training: Nice list of free training sessions you can watch online to learn how to build Windows Phone 7 applications using Silverlight and XNA.

• Silverlight Panorama and Pivot Controls for Windows Phone 7: Jeff has a great post that discusses the awesome new Panorama and Pivot controls for Silverlight that will be part of the final Windows Phone 7 developer release.

• Using XNA from Silverlight on Windows Phone 7: Good blog post from Mike that demonstrates how to use XNA APIs from Silverlight applications on Windows Phone 7.  This post demonstrates how to use the XNA audio framework from Silverlight.

• Creating High Performance Silverlight Applications for Windows Phone 7: This is a great whitepaper written by my team that discusses best practices for building Silverlight applications that run well on Windows Phone 7.

• Performance of Silverlight on Windows Phone 7: A nice podcast with Scott Hanselman that discusses tips and techniques to enable smooth applications and great experiences on Windows Phone 7.

Book Recommendation

People often ask me for .NET book recommendations. Below is one book I always keep close by and find super-useful.  If you are looking for a good one-volume C# and .NET Base Class Library reference, C# 4.0 in a Nutshell a great one:

Hope this helps,

Scott

One of the top reported Microsoft Connect issues with VS 2010 has been an issue with the Find and Replace dialog – which grows 16px each time you use it (which is pretty annoying).

The Visual Studio team recently released a patch that fixes this issue. You can download and apply it here.

Hope this helps,

Scott

P.S. A few people reported issues installing the patch if they had an older version of the Silverlight 4 tools installed.  If you see an error message that says you need an update to Visual Studio to support Silverlight 4, you can fix it by installing the latest Silverlight 4 tools release.

Inspired by this book I discovered in the Tate Modern book shop this week I don't (yet) have 101 things I can claim I should have learned in IT Architecture School but this would certainly be my 10 things:

1. The best architectures are full of patterns. This from Grady Booch. Whilst there is an increasing  need to be innovative in the architectures we create we also need to learn from what has gone before. Basing architectures on well-tried and tested patterns is one way of doing this.
2. Projects that develop IT systems rarely fail for technical reasons. In this report the reasons for IT project failures are cited and practically all of them are because of human (communication) failures rather than real technical challenges. Learning point: effective IT architects need to have soft (people skills) as well as hard (technical skills). See my thoughts on this here.
3. The best architecture documentation contains multiple viewpoints. There is no single viewpoint that adequately describes an architecture. Canny architects know this and use viewpoint frameworks to organise and categorise these various viewpoints. Here's a paper myself and some IBM colleagues wrote a while ago describing one such viewpoint framework. You can also find out much more about this in the book I wrote with Peter Eeles last year.
4. All architecture is design but not all design is architecture. Also from Grady. This is a tricky one and alludes to the thorny issue of "what is architecture" and "what is design". The point is that the best practice of design (separation of concerns, design by contract, identification of clear component responsibilities etc) is also the practice of good architecture how architectures focus is on the significant elements that drive the overall shape of the system under development. For more on this see here.
5. A project without a system context diagram is doomed to fail. Quite simply the system context bounds the system (or systems) under development and says what is in scope and what is out. If you don't do this early you will spend endless hours later on arguing about this. Draw a system context early, get it agreed and print it out at least A2 size and pin it in highly visible places. See here for more discussion on this.
6. Complex systems may be complicated but complicated systems are not necessarily complex. For more discussion on this topic see my blog entry here.
7. Use architectural blueprints for building systems but use architectural drawings for communicating about systems. A blueprint is a formal specification of what is to be. This is best created using a formal modeling language such as UML or Archimate. As well as this we also need to be able to communicate our architectures to none or at least semi-literate IT people (often the people who hold the purse). Such communications are better done using drawings, not created using formal modeling tools but done with drawing tools. It's worth knowing the difference and when to use each.
8. Make the process fit the project, not the other way around. I'm all for having a 'proper' software delivery life-cycle (SDLC) but the first thing I do when deploying one on a project is customise it to my own purposes. In software development as in gentleman's suits there is no “one size fits all”. Just like you might think you can pick up a suit at Marks and Spencers that perfectly fits you can't. You also cannot take an off-the-shelf SDLC that perfectly fits your project. Make sure you customise it so it does fit.
9. Success causes more problems than failure.This comes from Clay Shirky's new book Cognitive Surplus. See this link at TED for Clay's presentation on this topic. You should also check this out to see why organisations learn more from failure than success. The point here is that you can analyse a problem to death and not move forward until you think you have covered every base but you will always find some problem or another you didn't expect. Although you might (initially) have to address more problems by not doing too much up front analysis in the long run you are probably going to be better off. Shipping early and benefitting from real user experience will inevitably mean you have more problems but you will learn more from these than trying to build the 'perfect' solution but running the risk of never sipping anything.
10. Knowing how to present an architecture is as important as knowing how to create one. Although this is last, it's probably the most important lesson you will learn. Producing good presentations that describe an architecture, that are targeted appropriately at stakeholders, is probably as important as the architecture itself. For more on this see here.