Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!


The Growth Mindset: A Key to Resilience, Motivation, and Achievement

Your mindset holds the key to realizing your potential.

Your mindset is your way of thinking, and your way of thinking can limit or empower you, in any number of ways.

In fact, according to Carol S. Dweck, author of Mindset: The New Psychology of Success, mindset is the one big idea that helps explain the following:

  • Why brains and talent don’t bring success
  • How they can stand in the way of it
  • Why praising brains and talent doesn’t foster self-esteem and accomplishment, but jeopardizes them
  • How teaching a simple idea about the brain raises grades and productivity
  • What all great CEOs, parents, teachers, athletes know

When Dweck was a young researcher, she was obsessed with understanding how people cope with failures, and she decided to study it by watching how students grapple with heard problems.

You’re Learning, Not Failing

One of Dweck’s key insights was that a certain kind of mindset could turn  a failure into a gift.

Via Mindset: The New Psychology of Success:

“What did they know?  They knew that human qualities, such as intellectual skills could be cultivated through effort.  And that’s what they were doing – getting smarter.  Not only weren’t they discouraged by failure, they didn’t even think they were failing.  They thought they were learning.”

Your Can Change Your IQ

Believe it or not, a big believer in the idea that you can use education and practice to fundamentally change your intelligence is Alfred Binet, the inventor of the IQ test.

Via Mindset: The New Psychology of Success:

“Binet, a Frenchman working in Paris in the early twentieth century, designed this test to identify children who were not profiting from the Paris public schools, so that new educational programs could be designed to get them back on track. Without denying individual differences in children’s intellects, he believed that education and practice could bring about fundamental changes in intelligence.”

Methods Make the Difference

Here is a quote from one of Binet’s major books,  Modern Ideas About Children:

"A few modern philosophers ... assert that an individual's intelligence is a fixed quantity, a quantity which cannot be increased.  We must protest  and react against this brutal pessimism ... With practice, training, and above all, method, we manage to increase our attention, our memory, our judgment and literally to become more intelligent than we were before."

Growth Mindset vs. Fixed Mindset

The difference that makes the difference in success and achievement is your mindset.  Specifically, a Growth Mindset is the key to unleashing and realizing your potential.

To fully appreciate what a Growth Mindset is, let’s contrast it by first understanding what a Fixed Mindset is.

According to Carol Dweck, a Fixed Mindset means that you fundamentally believe that intelligence and talent are fixed traits:

“In a fixed mindset, people believe their basic qualities, like their intelligence or talent, are simply fixed traits. They spend their time documenting their intelligence or talent instead of developing them. They also believe that talent alone creates success—without effort. They’re wrong.”

In contrast, according to Dweck, a Growth Mindset means that you fundamentally believe that you can develop your brains and talent:

“In a growth mindset, people believe that their most basic abilities can be developed through dedication and hard work—brains and talent are just the starting point. This view creates a love of learning and a resilience that is essential for great accomplishment. Virtually all great people have had these qualities.”

If you want to improve your motivation, set yourself up for success, and achieve more in life, then adopt and build a growth mindset.

Here are a few articles to help you get started:

3 Mindsets that Support You

5 Sources of Beliefs for Personal Excellence

6 Sources of Beliefs and Values

Growth Mindset Over Fixed Mindset

Training Mindset and Trusting Mindset

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For April 4th, 2014

Hey, it's HighScalability time:

The world ends not with a bang, but with 1 exaFLOP of bitcoin whimpers.
  • Quotable Quotes:
    • @EtienneRoy: Algorithm:  you must encode and leverage your ignorance, not only your knowledge #hadoopsummit - enthralling
    • Chris Brenny: A material is nothing without a process. While the constituent formulation imbues the final product with fundamental properties, the bridge between material and function has a dramatic effect on its perception and use.
    • @gallifreyan: Using AWS c1, m1, m2? @adrianco says don't. c3, m3, r3 are now better and cheaper. #cloudconnect #ccevent
    • @christianhern: Mobile banking in the UK: 1,800 transactions per MINUTE. A "seismic shift" that banks were unprepared for

  • While we are waiting for that epic article deeply comparing Google's Cloud with AWS, we have Adrian Cockcroft's highly hopped slide comparing the two. Google: no enterprise customers, no reservation options, need more regions and zones, need lower inter-zone latency, no SSD options. AWS: no per minute billing, need simpler discount options, need more regions and zones, no real PaaS strategy, not instance migration.
  • Technology has change us from a demo or die culture to a deploy or die culture. We see this in software, but it's happening in hardware too says, Joi Ito (MIT Media Lab), in this interview.
  • The Curmudgeon of Truth declares I reckon your message broker might be a bad idea: Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour." < Good discussion on Hacker News.
  • High phive this. Your application may already achieved a degree of consciousness. An information integration theory of consciousness: consciousness corresponds to the capacity of a system to integrate information.
  • People really really like to talk. Line, a messaging app that's big in Japansurpasses 400 Million Registered Users, 10 billion chat messages per day, 1.8 billion stickers per day, and over 12 million calls per day.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Enterprise WinRT Apps Build Roundup

DevHawk - Harry Pierson - Fri, 04/04/2014 - 16:40

Wow, it’s been a whirlwind couple of days down here in San Francisco @ Build 2014. It has certainly been a huge thrill for me, getting a chance to be a part of the day one keynote and getting 15 minutes of fame. However, as the conference winds down I wanted to pull together a summary of the stuff Microsoft announced that relates to enterprise app development and Windows 8.1 Update. I mean, it wasn’t all about my wardrobe choices…

The Windows for Business blog as a good summary post that hits the highlights. The stuff I wanted to specifically call out is:

  • We’ve changed the policy to allow side loaded apps to communicate with desktop apps. Literally every single enterprise customer, Microsoft dev consultant and enterprise technical sales rep I’ve spoken to in the last year has asked for this.
  • We’ve added a feature in Windows 8.1 Update to enable side loaded apps to run code outside of the App Container. This opens up side loaded apps to access the full power of Windows as well as all the existing code the enterprise may have in its portfolio
  • We’ve made it significantly easier to get side load rights. I’d go thru the specifics here, but Rocky Lhotka (who has been *very* vocal about the issues in this space) had a great summary: “For a maximum of around $100 virtually every organization (small to huge) can get a side loading unlock key for all their devices.”

If you want more information on how to take advantage of these new features for side loaded apps, here are some resources for you:

  • In addition to my 5 minutes in the keynote, I did a whole session where I drilled into more details on that demo. I also demos that used network loopback for interprocess communication.
  • John Vintzel and Cliff Strom had a session on deploying enterprise apps. As of this writing, the video isn’t online yet but it will be within a day or two at that URL.
  • We have published whitepapers on both Brokered WinRT Components and using network loopback in WinRT apps that go into more details on how to build solutions with this technology
  • Last but not least, we have a set of samples of sideloaded WinRT apps. This includes the keynote demo, another brokered component demo and the WCF & ASP.NET network loopback demos I did in my session. Note, the keynote demo sample is packaged oddly because of the way MSDN’s sample repo handles (or in this case doesn’t handle) VS solutions with multiple projects. When I get back to Redmond, I’m  going to see if there’s a better way to get this sample hosted.

I heard many times over the past two days from folks in person at the conference and via email, twitter, facebook, carrier pigeon, etc just how excited they are about these changes & features. As an engineer who spends most of his days in his office and or in meetings building this stuff, it is amazingly gratifying to hear directly from our users how much our work can help them.

Categories: Architecture, Programming

Leslie Lamport to Programmers: You're Doing it Wrong

Famous computer scientist Leslie Lamport is definitely not a worse is better kind of guy. In Computation and State Machines he wants to make the case that to get better programs we need to teach programmers to think better. And programmers will think better when they learn to think in terms of concepts firmly grounded in the language of mathematics.

I was disappointed that there was so much English in the paper. Surely it would have been more convincing if it was written as a mathematical proof. Or would it?

This whole topic has been argued extensively throughout thousands of years of philosophy. Mathematics has always been a strange attractor for those trying to escape a flawed human rationality. In the end as alluring as the utopia of mathematics is, it lacks a coherent theory of meaning and programming is not about rearranging ungrounded symbols, it's about manipulating and shaping meaning.

For programmers I think Ludwig Wittgenstein has the right sense of things. Meaning is derived by use within a community. Programs built and maintained by programmers is at bottom a community of effort.


For quite a while, I’ve been disturbed by the emphasis on language in computer science. One result of that emphasis is programmers who are C++ experts but can’t write programs that do what they’re supposed to. The typical computer science response is that programmers need to use the right programming/specification/development language instead of/in addition to C++. The typical industrial response is to provide the programmer with better debugging tools, on the theory that we can obtain good programs by putting a monkey at a keyboard and automatically finding the errors in its code.

I believe that the best way to get better programs is to teach programmers how to think better. Thinking is not the ability to manipulate language; it’s the ability to manipulate concepts. Computer science should be about concepts, not languages. But how does one teach concepts without getting distracted by the language in which those concepts are expressed? My answer is to use the same language as every other branch of science and engineering—namely, mathematics. But how should that be done in practice? This note represents a small step towards an answer. It doesn’t discuss how to teach computer science; it simply addresses the preliminary question of what is computation. 

Related Articles
Categories: Architecture

Blood Sweat & Code

DevHawk - Harry Pierson - Wed, 04/02/2014 - 20:17

CNNToday was a *HUGE* thrill as I got to present in the keynote at //build! I’ll have more on the specifics of Brokered WinRT Components later after my session, but apparently quite a big deal was made of my shirt. I ended up as an internet meme and on the CNN Live Blog!

A long, long time ago (back when I wrote Photoshop Plugins for Mac long before I joined Microsoft), I had a Metrowerks CodeWarrior t-shirt with the “Blood Sweat & Code” slogan on the back. I loved that slogan, but lost the shirt somewhere along the way. So a few months ago, I decided to make a new one – but this time with the purple and blue of Visual Studio’s brand instead of CodeWarrior yellow. When I got a chance to be a part of the //build keynote today, I figured it was a good wardrobe choice.

For those who want one of their own, I posted the design on Zazzle.

Categories: Architecture, Programming

Start Your Journey of Innovation and Build a Culture of Great Innovation

In todays world, the mantra is innovate or die.

You’re either climbing ahead or falling backward … there’s no hanging out in the middle.

And some folks are actually leap frogging ahead.

Disruptive innovation is keeping everybody on their toes.

Whether you are re-imagining you or your company, or you are driving innovation in your process, product, or capabilities, there are skills you can learn to be a lot more effective in your innovation efforts.

It’s a crazy world where a One-Man Band can write an app, serve it up on the Cloud, and change the world.  It’s also a strange world where a little idea can be a big shot heard round the world.   It’s a scary thing for businesses when a handful of developers can spin up a new service in the Cloud and instantly make a business obsolete.

What can you hold on to in this crazy world?   What can you latch on to, if you want to rise above the noise, and instead of getting washed out by a wave, be the one that makes the waves?

There are several things, but I’ll boil them down to this:

  1. Use your customer as the North Star (and remember that some customers are better for you than others)
  2. Share and scale your unique value to the world
  3. Adapt or die

What happens to a super successful business or a super effective person when the landscape changes under their feet?

It depends on how they adapt Smile

Nature favors the flexible.  Darwin taught us that.

You have to get your bold on, and embrace innovation as your shiny sword to do battle against challenge and change, but most importantly, to create the change that serves you, and those you serve.

I’m taking a fresh look at innovation, as well as going back through hard, expensive lessons I’ve learned in the past.  Whatever doesn’t kill us makes us stronger, so my battle scars are a healthy reminder of the lessons I’ve learned on how we can use innovation to leap frog ahead, as well as change the playing field (heck with changing the game, change the field and be the disruptor.)

Believe it or not, Peter Drucker was a wealth of wisdom when it comes to innovation.  Many of you know him as the wise and wonderful professor of business and guru of management.   But when you read through a lot of his work, he was incredibly insightful and pragmatic when it comes to creating a culture of innovation.

Innovation Nuggets to Get You Started on Your Innovation Journey

I’ve got a ton of innovation books, but one that I’m really liking lately is Out Think: How Innovative Leaders Drive Exceptional Outcomes, by G. Shawn Hunter.   I’ve been sharing some nuggets from the book, and it’s been reminding me what it takes to build a culture of innovation.

If you want to start your innovation journey, and create a culture of innovation, here are a few posts to help you on your way:

3 Key Questions to Challenge Yourself to Innovate

3 Keys for a Successful Innovation

A Superior Product is Not Built from It’s Features

Beware of Benchmarking Your Way to Mediocrity

Energized Differentiation Separates Brands from the Pack

High-Leverage Strategies for Innovation

How Great Leaders Build a Culture of Innovation and Change

Incremental Changes or Disruptive Innovation?

Innovate in Your Approach

Innovation Life Cycle

Innovation, Quantification, and Orchestration

The Innovative Team: Unleashing Creative Potential for Breakthrough Results

The Role of Process in Driving Reliable Innovation

Great Innovation Quotes to Inspire the Art of the Possible

If you need to remind yourself what innovation feels like or what’s possible, be sure to soak up some powerful words of wisdom:

Innovation Quotes

In my Innovation Quotes, I’ve also included a special section to light up what Bill Gates, Steve Jobs, and Walt Disney teach us about building a culture of innovation.

Let’s boldly go where we have not gone before.

Categories: Architecture, Programming

The Mullet Cloud Selection Pattern

In a recent thread on Hacker News one of the commenters mentioned that they use Digital Ocean for personal stuff, but use AWS for business.

This DO for personal and AWS for business split has become popular enough that we can now give it a name: the Mullet Cloud Selection Pattern - business on the front and party on the back.

Providers like DO are cheap and the lightweight composable container model has an aesthetic appeal to developers. Even though it seems like much of the VM infrastructure has to be reinvented for containers, the industry often follows the lead of developer preference.

The mullet is dead. Long live the mullet! Developers are ever restless, always eager to move onto something new. 

Categories: Architecture

Sponsored Post: Layer, The Factory, Airseed, Uber, ScaleOut Software, Couchbase, Tokutek, Booking, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • We’re looking for talented and driven engineers who are passionate about scalability and reliability to help us build Layer, the open communications layer for the Internet. Layer enables app developers to easily build secure, scalable messaging, voice and video features into any app. For more information and our full list of openings, please visit:

  • The Factory is seeking a collaborative, talented and entrepreneurial Sr. Front End Engineer. Backed by the co-founder of Skype+Rdio and led by Rdio's founding team, we're changing the way products are built and companies get launched (think incubator/accelerator without the nagging outside influence or funding/timing constraints). Our goal is to build a platform to better launch startups and opensource what we do along the way.

  • Airseed -- a Google Ventures backed, developer platform that powers single sign-on authentication, rich consumer data, and analytics -- is hiring lead backend and fullstack engineers (employees #4, 5, 6). More info here:

  • Join the team that scales Uber supply globally! Our supply engineering team is responsible for prototyping, building, and maintaining the partner-facing platform. We're looking for experienced back-end developers who care about developing highly scalable services. Apply at

  • We need awesome people @ - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Apply:

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • The Biggest MongoDB Event Ever Is On. Will You Be There? Join us in New York City June 23-25 for MongoDB World! The conference lineup includes Amazon CTO Werner Vogels and Cloudera Co-Founder Mike Olson for keynote addresses.  You’ll walk away with everything you need to know to build and manage modern applications. Register before April 4 to take advantage of super early bird pricing.
  • April 3 Webinar: The BlueKai Playbook for Scaling to 10 Trillion Transactions a Month. As the industry’s largest online data exchange, BlueKai knows a thing or two about pushing the limits of scale. Find out how they are processing up to 10 trillion transactions per month from Vice President of Data Delivery, Ted Wallace. Register today.
Cool Products and Services
  • As one of the fastest growing VoIP services in the world Viber has replaced MongoDB with Couchbase Server, supporting 100,000+ operations per second in the short term and 1,000,000+ operations per second in the long term for their third generation architecture.  See the full story on the Viber switch.

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

How WhatsApp Grew to Nearly 500 Million Users, 11,000 cores, and 70 Million Messages a Second

When we last visited WhatsApp they’d just been acquired by Facebook for $19 billion. We learned about their early architecture, which centered around a maniacal focus on optimizing Erlang into handling 2 million connections a server, working on All The Phones, and making users happy through simplicity.

Two years later traffic has grown 10x. How did WhatsApp make that jump to the next level of scalability?

Rick Reed tells us in a talk he gave at the Erlang Factory: That's 'Billion' with a 'B': Scaling to the next level at WhatsApp (slides), which revealed some eye popping WhatsApp stats:

What has hundreds of nodes, thousands of cores, hundreds of terabytes of RAM, and hopes to serve the billions of smartphones that will soon be a reality around the globe? The Erlang/FreeBSD-based server infrastructure at WhatsApp. We've faced many challenges in meeting the ever-growing demand for our messaging services, but as we continue to push the envelope on size (>8000 cores) and speed (>70M Erlang messages per second) of our serving system.

What are some of the most notable changes from two years ago?

  • Obviously much bigger in every dimension, except the number of engineers. More boxes, more datacenters, more memory, more users, and more scale problems. Handling this level of growth with so few engineers is what Rick is most proud of: 40 million users per engineer. This is part of the win of the cloud. Their engineers work on their software. The network, hardware, and datacenter is handled by someone else.

  • They’ve gone away from trying to support as many connections per box as possible because of the need to have enough head room to handle the overall increased load on each box. Though their general strategy of keeping down management overhead by getting really big boxes and running efficiently on SMP machines, remains the same.

  • Transience is nice. With multimedia, pictures, text, voice, video all being part of their architecture now, not having to store all these assets for the long term simplifies the system greatly. The architecture can revolve around throughput, caching, and partitioning.

  • Erlang is its own world. Listening to the talk it became clear how much of everything you do is in the world view of Erlang, which can be quite disorienting. Though in the end it’s a distributed system and all the issues are the same as in any other distributed system.

  • Mnesia, the Erlang database, seemed to be a big source of problem at their scale. It made me wonder if some other database might be more appropriate and if the need to stay within the Erlang family of solutions can be a bit blinding?

  • Lots of problems related to scale as you might imagine. Problems with flapping connections, queues getting so long they delay high priority operations, flapping of timers, code that worked just fine at one traffic level breaking badly at higher traffic levels, high priority messages not getting serviced under high load, operations blocking other operations in unexpected ways, failures causing resources issues, and so on. These things just happen and have to be worked through no matter what system you are using.

  • I remain stunned and amazed at Rick’s ability to track down and fix problems. Quite impressive.

Rick always gives a good talk. He’s very generous with specific details that obviously derive directly from issues experienced in production. Here’s my gloss on his talk…

Categories: Architecture

Stuff The Internet Says On Scalability For March 28th, 2014

Hey, it's HighScalability time:

Looks like a multiverse, if you can keep it.
  • Quotable Quotes:
    • @abt_programming: "I am a Unix Creationist. I believe the world was created on January 1, 1970 and as prophesized, will end on January 19, 2038" - @teropa
    • @demisbellot: Cloud prices are hitting attractive price points, one more 40-50% drop and there'd be little reason to go it alone.
    • @scott4arrows: Dentist "Do you floss regularly?" Me "Do you back up your computer data regularly?"
    • @avestal: "I Kickstarted the Oculus Rift, what do I get?" You get a lesson in how capitalism works.
    • @mtabini: “$20 charger that cost $2 to make.” Not pictured here: the $14 you pay for the 10,000 charger iterations that never made it to production.
    • @strlen: "I built the original assembler in JS, because it's what I prefer to use when I need to get down to bare metal." - Adm. Grace Hopper
    • tedchs: I'd like to propose a new rule for Hacker News: only if you have built the thing you're saying someone should save money by building themselves, may you say the person should build that thing.
    • lamby: Bezos predicted they would be good over the long term but said that he didn’t want to repeat “Steve Jobs’s mistake” of pricing the iPhone in a way that was so fantastically profitable that the smartphone market became a magnet for competition.
    • @PariseauTT: I feel a Netflix case study coming...everybody get your drinks ready...#AWSSummit
    • seanmccann: That's no different than startups of the past having to pay thousands to millions of dollars to setup servers. These days those same servers can be setup in minutes for a fraction of the cost. Timing is everything. Sometimes the current market pricing for a commodity is too expensive to make your business viable today but in the future that will not be the case. Just depends how long that will take. 
    • Petyr 'Littlefinger' Baelish: Chaos isn't a pit. Chaos is a ladder. Many who try to climb it fail and never get to try again. The fall breaks them. And some, are given a chance to climb. They refuse, they cling to the realm or the gods or love. Illusions. Only the ladder is real. The climb is all there is.
  • We turn those who first climb the peaks of tall mountains into heros. But what of the mountain? Mariana Mazzucato The Entrepreneurial State: Debunking Private vs. Public Sector Myths: Every feature of the iPhone was created, originally, by multi-decade government-funded research. From DARPA came the microchip, the Internet, the micro hard drive, the DRAM cache, and Siri. From the Department of Defense came GPS, cellular technology, signal compression, and parts of the liquid crystal display and multi-touch screen (joining funding from the CIA, the National Science Foundation, and the Department of Energy, which, by the way, developed the lithium-ion battery.) CERN in Europe created the Web. Steve Jobs’ contribution was to integrate all of them beautifully.

  • This is not an April Fools' joke, as the name might make a certain sort of mind consider. WebScaleSQL: MySQL goes web scale with contributions from MySQL engineering teams at Facebook, Google, LinkedIn, and Twitter. It includes lots of good work on the test suite, performance enhancements, and features to make scaling easier. So many forks at the table (MariaDB, Percona, webscale).

  • This is why we can't have nice code. Martin Sústrik argues In the Defense of Spaghetti Code, quite successfully I think, that it's often clearer to have one large 1500 line function than it is to have a refactored great big ball of mud. Commenters argue from a perfect world stance that if you do X from the start and then continue the same practices over the years and and hundreds of programmers then code can be perfected. Not so. The nature of many domains is they are just messy. At a certain level of abstraction messiness can be hidden, but when you are in the guts of a thing, messiness is preserved. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

"I'll Send You the Deck"

Software Architecture Zen - Pete Cripp - Fri, 03/28/2014 - 00:02
Warning, this is a rant!

I’m sure we’ve all been here. You’re in a meeting or on a conference call or just having a conversation with a colleague discussing some interesting idea or proposal which he or she has previous experience of and at some point they issue the immortal words “I’ll send you the deck”.

Categories: Architecture

Strategy: Cache Stored Procedure Results

Caching is not new of course, but I don't think I've heard of caching store procedure results before. It's like memoization in the database. Brent Ozar covers this idea in How to Cache Stored Procedure Results.

The benefits are the usual for doing work in the database, it doesn't take per developer per app work, just code it once in the stored proc and it works for everyone, everywhere, for all of time. The disadvantage is the usual as well, it adds extra load to a probably already busy database, so it should only be applied to heavy computations.

Brent positions this strategy as an emergency bandaid to apply when you need to take pressure off a database now. Developers can then work on moving the cache off the database and into its own tier. Interesting idea. And as the comments show the implementation is never as simple as it seems.

Categories: Architecture

Oculus Causes a Rift, but the Facebook Deal Will Avoid a Scaling Crisis for Virtual Reality

Facebook has been teasing us. While many of their recent acquisitions have been surprising, shocking is the only word adequately describing Facebook's 5 day whirlwind acquisition of Oculus, immersive virtual reality visionaries, for a now paltry sounding $2 billion.

The backlash is a pandemic, jumping across social networks with the speed only a meme powered by the directly unaffected can generate.

For more than 30 years VR has been the dream burning in the heart of every science fiction fan. Now that this future might finally be here, Facebook’s ownage makes it seem like a wonderful and hopeful timeline has been choked off, killing the Metaverse before it even had a chance to begin.

For the many who voted for an open future with their Kickstarter dollars, there’s a deep and personal sense of betrayal, despite Facebook’s promise to leave Oculus alone. The intensity of the reaction is because Oculus matters to people. It's new, it's different, it creates a better future. It's important in a way sending messages or taking pictures never can be.

Let’s use Andy Baio as a sane example of the loyal opposition:

I can palpably feel the oxygen sucked out of the room. Infinite bright possibilities from indie game devs, quietly shelved.

Jigsus backs this up on reddit:

Within the last hour EVERY friend I know was developing a rift game has canceled. That's around 11 projects just gone.

So WTF? Why in this reality would Oculus sell to Facebook? At first blush it makes little sense. A more mismatched couple could hardly be created in a fantasy immersive 3D game.

Yet there's a reason and a method here that's not only about about a sweet payoff. When you look deeper at the Facebook deal it’s about Oculus' existential need to scale. And scale is something Facebook is very, very good at.

Let’s explore why this deal makes more sense for Oculus than it might first appear and the central role scalability plays in the decision making...

Categories: Architecture

Architect Salary Survey

Software Architecture Zen - Pete Cripp - Tue, 03/25/2014 - 14:48
The first ever architecture specific salary survey has just been published in the UK by FMC Technology. In total over 1000 architects responded to the survey. The report looks at architect roles in six main areas:
  • Architecture Management
  • Enterprise Architecture
  • Business Architecture
  • Information Architecture
  • Application Architecture
  • Technology Architecture
It also looks at pay rises (in 2013), regional and industry differences as well as motivational factors
The results can be viewed here.

Also on my Wordpress blog here.
Categories: Architecture

Services, Microservices, Nanoservices – oh my!

Apparently there’s this new distributed architecture thing called microservices out and about – so last week I went ahead and read Martin Fowler’s & James Lewis’s extensive article on the subject . and my reaction to this was basically:

I guess it is easier to use a new name (Microservices) rather than say that this is what SOA actually meant – re

— Arnon Rotem-Gal-Oz (@arnonrgo) March 16, 2014

Similar arguments (nothing new here) were also expressed after Martin’s tweet of his article e.g. Clemens Vasters’ comment:

@martinfowler @boicy but these are the very principles of SOA before vendors does pushed the hub in the middle, i.e. ESB — Clemens Vasters (@clemensv) March 16, 2014

Or Steve Jones’ post “Microservices is SOA, for those who know what SOA is.”

Autonomy, smart endpoints, events etc. that the article talks about are all SOA concepts – If you happen to have read my book and you read this article you’ve probably identified these as patterns like inversion of communicationService Host,Service InstanceService Watchdog and others.

So microservices as they appear from Martin’s & James’s article is pretty much service orientation without some of the bad misconception that tied into the SOA moniker like WS*, ESBs as a must, etc. -perhaps that’s a good enough reason for a new name but personally I doubt it.

However, the story doesn’t end here there are various other opinions as to what microservices are such as Chris Ford’s view who says that

“My own opinion is that microservice architectures can be understood through a single abstract architectural constraint which can be interpreted along many different degrees of freedom.

X can be varied independently of the rest of the system.”

The idea that something is separate  and can be varied independently from the rest of the system is  good but I would hardly say it is a definition of anything or at least anything new. CSCI (Computer Software Configuration Item) which I first heard of as part  of  DOD-STD-2167A (published in 1988) essentially means the same thing a CSCI is a component that can be varied independently from the rest of the system. In 2167A eyes it also means a very detailed , waterfall,  documentation laden process,  which isn’t what anyone thinks service or microservices should entail but it does demonstrate that “being varied independently” doesn’t mean much.

I am sure some reader goes something like “but wait, we’re talking about micro-services here  - so they should also be small”  - indeed  there are posts like James Hughes’s on microservice with a quote like

“First things first what actually is a micro service? Well there really isn’t a hard and fast definition but from conversations with various people there seems to be a consensus that a micro service is a simple application that sits around the 10-100 LOC mark.”

(truth be told is that in the next sentence James says that LOC is an atrocious way to compare implementations but I thought it was worth repeating due to the use of the  words “there seems to be a consensus”)

So how can you have 100 LOC services? you can get there if you rely on frameworks (like Finagle or Sinatra James mention) generate serialization/deserialization code (protobuff, thrift, avro etc.)  - this is essentially building on a smart service host. Another example for this would be developing in Erlang with its supervisor hierarchies which also brings us to another way to reduce LOC by use languages that are less verbose (like the aforementioned Erlang,  python or scala vs. say, Java).

I would say., however, that if you find you have a 10  lines of code service, you are more likely than not, implementing a function as a service and you don’t have a real service micro or not- e.g. I can’t see you having a decentralized storage (and autonomy) as mentioned in Martin’s and Lewis’s article above or having monitoring and instrumentation that Hughes mention. 

You should also keep in mind that while better and cheaper networks allow us to push the limits – the fallacies of distributed computing still exist furthermore having a lot of small services that you need to manage along with the performance hits for serializations and deserializations, security etc. may very well mean that you moved from valid smaller “micro” services into the realm of headache which I called “Nano services

“Nanoservice is an antipattern where a service is too fine-grained. A nanoservice is a service whose overhead (communications, maintenance, and so on) outweighs its utility.”

So there we have it, for the most part microservices is just another name for the principle of SOA, anther name might have been appropriate in the hype days of SOA, but I think these days most of that vapor has cleared and people understand better what’s needed. Furthermore if we do want to name proper SOA by a new name I think microservices is a poor term as it leads toward the slippery slope into nano-services and 10 lines of code which are just your old web-service method executed by a fancy chic host using a hot serialization format.

Micro or not, services should be more useful than the overhear they incur

illustration by Les Chatfield

Categories: Architecture

Organisational Inertia - A Predictor for Success of Agile Transformations? (Part 2)

Xebia Blog - Tue, 03/25/2014 - 01:30

In part 1 (Organisational Inertia - Part 1) I have focussed on the question: 'Organisational Inertia - What is it?’. This blog addresses the question ‘How do we measure it?’.

I'll start from the definition of Organisational Inertia as defined in part 1. Then connect to existing models of Organisational Inertia and the relation to Agile teams and show how the analog with Physics is used to find a measure for the 'acceleration'. Then I'll combine these elements to provide a way of measuring inertia. Finally I'll provide basic examples.

Definition of Organisational Inertia

Following the analogy with physics I defined Organisational Inertia in Organisational Inertia - Part 1 as:


Organisational Inertia is an organisation’s capability to change after applying some force to it. Specifically, I define it as the ratio of the force applied and the speed of organisational change (acceleration) in reaction to some applied force, or in formula "Inertia (M) = Force / Acceleration".

We just shifted the problem of defining Organisational Inertia to defining ‘some applied force’ and ‘speed of change’. The next sections will provide an interpretation of these in the context of Agile teams and a model to describe 'some applied forces'.

Origin of the Forces Driving Organisational Change: Change Restraining and Change Facilitating Forces

Associated with Organisational Inertia are two competing forces as described in the Inertia model of Connor & Lake [Con88] (see also [Kin98]). This model describes the Change-restraining forces and the Change-facilitating forces.

The origin of the forces driving Organisational Inertia [Lar96]

The model recognises that there are forces that hinder change and that there are forces that facilitate change.

One of the Agile principles is to regularly reflect on how you are doing and improve. By doing so Agile teams not only improve over time but they change too. An Agile team that changes will force the environment to change with it.
For more detailed information on this, check out the ‘fitness landscape’ [App11] chapter 15 “How to improve everything”.

Another common way Agile teams force the environment to change is by pulling more work in than the environment is capable of delivering. Or, by pushing more working software into production and thereby literally pushing the limits of what the environment is capable to process. The latter mechanism drives organisations towards DevOps [Deb11]. The combination of applying a force to both the up- and down-stream teams is what drives the need for BusDevOps (Agile Delivery Model), see Agile Delivery.

Examples I often see in practise of how Agile teams force the organisation (or environment) to change include:

  • a faster delivery rate forces the environment to cope with more releases and simultaneously forces the business to keep up by supplying user stories at a higher rate,
  • raising impediments,
  • the business wants more features than IT can deliver (this often drives the need to transition form a traditional way of working to an agile way of working).

For a systemic view on cause and effect related to Organisational Inertia see e.g. [Lar96].

How (Agile) Teams Act As Change-Facilitating Forces

Agile teams continuously improve. The way they improve I divide in 3 types. With each type I associate a 'force'. The next section will go into details of quantifying them.

In Management 3.0 [App11] chapter 15 “How to improve everything” the concept of the ‘fitness landscape’ is explained. In this metaphor the team is part of the landscape (the environment of the team within the organisation) in which the team optimizes a certain variable, for instance the business value it creates. The optimum is symbolised by the highest mountain peak. Through improvements the team finds its way to the highest peak. Besides team intrinsic changes the team can also change the environment and thereby change the landscape by moving mountains towards the team instead of moving towards the mountain.

Following this metaphor there are three ways for a team to reach the top of the highest peak:
I) gradually improve (changes intrinsic to the team),
II) large improvements (kaikaku) and jump to other (nearby or remote) mountains, or
III) change the environment, i.e. moving the mountains to the team.

Gradual Improvements (Type I Changes)
Agile teams regularly evaluate how they are doing. A common practise is to perform a retrospective every two weeks. In the retrospective the team decides what to change and executes these improvements in the next sprint (or time period). With these improvements the team walks the ‘fitness landscape’ to the nearest mountain peak. The improvements may be gradual and internal to the team. By this we mean that it increases the velocity and does not force the environment of the team, i.e. the organisation, to change. The team does not exert a force to the organisation.

Large Improvements (Type II Changes)
It may also be the case that the improvements within the team either dramatically increase the velocity or many gradual improvements increases the velocity over a certain threshold. When this happens - in my experience it is not a question whether this will happen but rather ‘when’ - it will force the direct environment of the team to change with it.

For example, when teams start to deliver user stories faster the business needs to keep up by having enough user stories ready.

Another example is that operations needs to keep up by putting more features into production.

This way the team applies a force to the surrounding organisation that needs to change as well. The team is not just walking the ‘fitness landscape’ but forcing the landscape to change as well [App11].

Moving Mountains (Type III Changes)
Changing the landscapeThe third way for teams to apply a force is to raise impediments and have the organisation help them by addressing the most important impediments. By structurally solving impediments the organisation is changing the ‘fitness landscape’ [App11] by moving the mountain peaks to the team.

With 'structurally solving impediments' I mean that solving the impediments requires company policies to be changed.

An Organisation’s Speed of Change

The final part is to have a way of measuring the speed of change while the Agile team exerts a force upon the organisation. Before we can answer this question we need to identify how to observe that an organisation changes. Possible variables are:

  • the trend of the (average) resolution time of impediments,
  • change in the rate of bringing features to production,
  • change in the rate of taking work to the team(s),
  • company policies being changed.

Potentially this could become a long list. But are all these changes relevant? Ultimately it’s the bottom line that counts. It’s all about the outcome. Let’s try to come up with variables that are of value to the business. This is pretty close to the bottom line. At least close enough. Variables that come into mind include:

  • change in the rate of business value per unit of time,
  • change in rate of an organisation’s (agile) maturity rating,
  • change in rate of product incidents per unit time.

These are all variables that can be measured and are the (complex) outcome of the changes made.

The choice depends on the goal that you want to achieve with the (agile) transition. As long as this is communicated clearly to the organisation and progress is being made as measured using this or these variables, you are measuring the rate of success and there will be enough energy in the organisation to continue the transition and avoid Organisational Gravity kicking in.

Metrics for Organisational Inertia

From the aforementioned examples we recognise the following metrics for the 'forces' (Change Facilitating):

  • [F1] the number of raised impediments per time period, e.g. per sprint or per month,
    Force = <Number of Raised Impediments per Period>,
  • [F2] difference in rate (throughput) for pre-sprint, sprint and post-sprint parts in a so-called cumulative flow diagram,
    Force [F2a] = <Business Value Delivered in Production per Period> - <Work According to DoR Worth of Business Value per Period>, or
    Force [F2b] <Business Value Delivered in Production per Period> - <Work Completed according to DoD Worth of Business Value per Period>

Force [F2a] measures the 'strength' with which the (Scrum) team is pulling work from the Product Owner. A positive value means that the PO cannot keep up with the team.
Force [F2b] is similar for the IT-Ops - (Scrum) team interaction. Positive values indicate the (Scrum) team is pushing work for production at a faster rate than the organisation is capable of taking into production.

The associated metrics for measuring the 'acceleration' (speed of change) are:

  • [A1] the trend of the average number of resolved impediments per time period; count only resolved impediments that required changes in company policies,
    Acceleration = Δ(<Number of Resolved Impediments per Period>)/<Period>,
  • [A2] the trend or change in the rate of business value (outcome) delivered per time period,
    Acceleration = Δ(<Business Value Delivered in Production per Period>)/<Period>.

The metrics just given are readily available to Scrum teams and can easily be applied to any team, kanban type of system, or to any part of the total value stream.

Putting It All Together

Using the results of the previous section I quantify Organisational Inertia in terms of measurable variables available to (Agile) teams.

Gradual internal improvements that the team makes (Type I changes) often lead to Type II changes after a certain period of time. Therefore I only explicitly address Type II and Type III changes (as explained earlier).

Are We Moving the Mountains? (Type II Changes)
Taking the change facilitating forces as a basis the Organisational Inertia is derived as follows.

First, consider the metric for inertia (  M_\mathrm{org} M_\mathrm{org} ) based on business value, i.e. [F2b] and [A2] from the previous section. Combining these gives:

 \langle\text{Value delivered per Period}\rangle - \langle\text{Value ready (DoD) per period}\rangle = M_\mathrm{org}\times\Delta(\langle\text{Value delivered per period}\rangle)/\langle\text{Period}\rangle \langle\text{Value delivered per Period}\rangle - \langle\text{Value ready (DoD) per period}\rangle = M_\mathrm{org}\times\Delta(\langle\text{Value delivered per period}\rangle)/\langle\text{Period}\rangle

What does this formula tell us?

  • if the team’s delivery rate balances the organisation's rate of putting features into production no force is exerted and the organisation will not be forced to change; then also no change (Δ) in the delivery rate,
  • if the team delivers more functionality than the organisation can take into production, the team exerts a certain force upon the organisation which needs to undergo structural improvements causing a positive change (Δ) in the delivery rate,
  • for small values of the inertia (  M_\mathrm{org} M_\mathrm{org} ) the effect on the change in delivery rate is larger.

Note: A  unit of  measure for  M_\mathrm{org} M_\mathrm{org}  is ‘time’, e.g. 'weeks' or 'sprints' for Scrum teams.

Example 1. Suppose that the team delivers 5% more business value each sprint. Suppose further that they deliver 20% worth of value faster than can be taken to production. Then the inertia is ’4 sprint’ (20% divided by 5%).

Example 2. Suppose that no change in the rate of delivered business value is measured. Then the Organisational Inertia is infinitely large; under a change facilitating force it does not change.

The effect of Organisational Inertia on delivered business value

The effect of Organisational Inertia on delivered business value

Example 3. Suppose an inertia of ’4 sprint’ (Example 1). Further suppose that the team pushes 20% worth of features per sprint more than can be taken into production. Then it will take M_\mathrm{org}M_\mathrm{org} /20% = 20 sprints to double the production rate.

Example 4. As in the previous example, the graph to the right shows the same team but in an environment in which the inertia is twice the inertia of the other environment. The team in the environment with the least inertia delivers value at a rate that increases twice as fast as compared to the other team. This results in twice as much business value being generated.

Are the Mountains Moving? (Type III Changes)
Another possibility is to take impediments as a basis to define and measure the Organisational Inertia (  M_\mathrm{org} M_\mathrm{org} ):

 M_\mathrm{org} = \frac{F_1}{A_1} = \frac{\langle\text{Number of Raised Impediments per Period}\rangle}{\Delta(\langle\text{Number of Resolved Impediments per Period}\rangle)/\langle\text{Period}\rangle} M_\mathrm{org} = \frac{F_1}{A_1} = \frac{\langle\text{Number of Raised Impediments per Period}\rangle}{\Delta(\langle\text{Number of Resolved Impediments per Period}\rangle)/\langle\text{Period}\rangle}

Note: Only impediments that actually result in changes in the organisation should be taken into account.

Note: This can also be translated in terms of business value by estimating how much more business value the team would be able to put into production if the impediment is resolved.

Note: The unit of measure is ‘time’, e.f. 'weeks' or 'sprints'.


Organisational Inertia is supplementary to the concept of Organisational Gravity and an indicator for how well the team’s environment is facilitating the team’s growth. The two counter balancing driving forces of the speed of organisational change are the Change-restraining and Change-facilitating forces.

An indicator available to agile teams for the Change-restraining forces is the average number of solved impediments per time period. Change-facilitating forces are the forces that teams exert on their environment. Indicators available to agile teams are the amount of pulling from the business for more ready features and pushing completed work into production.

Using the interpretation of the formula  F=M A F=M A , well-known in physics, in terms of the Change-restraining and Change-facilitating forces as defined in the model for Organisational Inertia from Connor and Lake  [Con88], and identifying MM with  M_{\textrm{org}} M_{\textrm{org}}  expressions for Organisational Inertia are derived.

Once the organisation's inertia is known, this can serve as a prediction how long it will take to increase the organisation's delivery rate with a certain amount and is related to the business case of Agile Transformations.

References [App11] Management 3.0, Jurgen Appelo, 2011, Pearson Education, [Deb11] Devops: A Software Revolution in the Making, Patrick Debois, August 2011, Cutter IT Journal, Vol. 24, No. 8,, [Con88] Managing organization change, P.E. Connor & L.K. Lake, 1988, New York: Praeger, [Kin98] The development of an instrument for measuring organisational inertia, C Kinnear, G Roodt, Journal of Industrial Psychology, 1998, 24(2), 44-54; see, [Lar96] The Dynamics of Organisational Inertia, Survival and Change, Erik R. Larsen, Alessandro Lomi, 1996, System Dynamics conference,

7 Days of Agile Results: A Time Management Boot Camp for Productivity on Fire

An amazing thing happens when you become more focused and productive ...

You get more out of life.

It’s like you have 27 hours out of the day, while everyone else has 24, and they spend 8 of them sleeping, while you spend them dreaming of what’s to come next.

Too many folks have too much to do with too little time and they can't keep up.

We don't necessarily learn great time management skills in school or on the job, and we don't necessarily learn how to really blend our time, energy, and action to produce our best results.

That's where Agile Results steps in.

Agile Results it the underlying approach showcased in my best-selling book on time management, Getting Results the Agile Way.

It's a simple system for meaningful results.  It helps you cut through the clutter to get to what matters, and to use your best energy for your best work.  I put Agile Results together over a period of 10 years while testing principles, patterns, and practices that push the envelope in terms of high-performance, extreme productivity, work-life balance, stress management, and well-being.

Here’s What You Get by Using Agile Results:
  1. Proven practices to master time management, motivation, and personal productivity
  2. Discover the one way to stack the deck in your favor that’s authentic and works
  3. How to embrace change and get better results in any situation
  4. How to focus and direct your attention with skill
  5. How to use your strengths to create a powerful edge for getting results
  6. How to change a habit and make it stick
  7. How to never miss the things that matter most, and achieve better work-life balance
  8. How to spend more time doing the things you love
The 7 Day Boot Camp for Agile Results

I put together a simple time management book camp to help you just start using Agile Results.

For some case studies, stories, and testimonials see

If you need more depth beyond the 7 day time management book camp, then check out:

And, of course, there’s always the book:

If you’re already an Agile Results master, share this post and help somebody else set their productivity on fire.  Help friends, family, and colleagues reach a new level of awesome.

Categories: Architecture, Programming

Big, Small, Hot or Cold - Examples of Robust Data Pipelines from Stripe, Tapad, Etsy and Square

This is a guest repost by Pete Soderling, Founder at Hakka Labs, creating a community where software engineers come to grow.

In response to a recent post from MongoHQ entitled “You don’t have big data," I would generally agree with many of the author’s points.

However, regardless of whether you call it big data, small data, hot data or cold data - we are all in a position to admit that *more* data is here to stay - and that’s due to many different factors.

Perhaps primarily, as the article mentions, this is due to the decreasing cost of storage over time. Other factors include access to open APIs, the sheer volume of ever-increasing consumer activity online, as well as a plethora of other incentives that are developing (mostly) behind the scenes as companies “share” data with each other. (You know they do this, right?)

But one of the most important things I’ve learned over the past couple of years is that it’s crucial for forward thinking companies to start to design more robust data pipelines in order to collect, aggregate and process their ever-increasing volumes of data. The main reason for this is to be able to tee up the data in a consistent way for the seemingly-magical quant-like operations that infer relationships between the data that would have otherwise surely gone unnoticed - ingeniously described in the referenced article as correctly “determining the nature of needles from a needle-stack.”

But this raises the question - what are the characteristics of a well-designed data pipeline? Can’t you just throw all your data in Hadoop and call it a day?

As many engineers are discovering - the answer is a resounding "no!" We've rounded up four examples from smart engineers at Stripe, Tapad, Etsy & Square that show aspects of some real-world data pipelines you'll actually see in the wild.

How does Stripe do it?
Categories: Architecture

Stuff The Internet Says On Scalability For March 21st, 2014

Hey, it's HighScalability time:

Isaac Newton's College Notebook, such a slacker.
  • Quotable Quotes:
    • Chris Anderson: Petabytes allow us to say: ‘Correlation is enough.’
    • @adron: Back to writing the micro-services for my composite app with regionally distributed, highly available, key value crypto stores of unicorns!
    • DevOps Cafe: when the canary dies you don't buy a stronger canary.
    • Mark Pagel: Creativity, like evolution, is merely a series of thefts.
    • GitHub: Whoever has more capacity wins.
    • The Master: We balance probabilities and choose the most likely. It is the scientific use of the imagination.
    • Jonathan Ive: Steve, and I don’t recognize my friend in much of it. Yes, he had a surgically precise opinion. Yes, it could sting. Yes, he constantly questioned. ‘Is this good enough? Is this right?’ but he was so clever. His ideas were bold and magnificent. They could suck the air from the room. And when the ideas didn’t come, he decided to believe we would eventually make something great. And, oh, the joy of getting there!”
    • @joestump: Turns out the correct amount of RAM is always 16GB more than you have.

  • From a spec of sand does a pearl grow. In this case the oyster is Facebook, the irritant is PHP, and the pearl is Hack, a new improved version of PHP developed by Facebook. Hack now runs all of Facebook's front-end code. Lots of great comments on Hacker News and on Reddit with much lang on PHP hate. Realize Facebook is not trying to build the next perfect language. Historically they coded their web tier in PHP so they have a lot of code working in production. It would be insane to throw that all away. Over the years they've put a lot of work into making PHP more efficient. Hack continues that work by making PHP a better language, coupling PHP's convenient interpreted nature with static typing and many other features commonly found in modern programming languages. If PHP bothers you, Hack is not for you, but that's OK.

  • Dish Jump-Starts the Hopper With 8 Tuners and PS4 Support. Ultimate in local caching. Record all the things and then watch what you want later. Streaming access sucks. More local storage!

  • Cassandra Hits One Million Writes Per Second on Google Compute Engine on 300 VMs with 300 1TB Persistent Disk volumes, 15,000 clients, median latency of 10.3 ms and 95% completing under 23 ms. It took 1 hour and 10 minutes at a cost of $330 USD. All benchmark caveats apply. They wrote small records of 170 bytes. What happens when you write a large range of data sizes? When you are reading? Are you running queries? Are you backing up? Are indexes being updated? Is there contention? Etc, etc. Still, it's impressive.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Paper: Log-structured Memory for DRAM-based Storage - High Memory Utilization Plus High Performance

Most every programmer who gets sucked into deep performance analysis for long running processes eventually realizes memory allocation is the heart of evil at the center of many of their problems. So you replace malloc with something less worse. Or you tune your garbage collector like a fine ukulele. But there's a smarter approach brought to you from the folks at RAMCloud, a Stanford University production, which is a large scale, distributed, in-memory key-value database.

What they've found is that typical memory management approaches don't work and using a log structured approach yields massive benefits:

Performance measurements of log-structured memory in RAMCloud show that it enables high client through- put at 80-90% memory utilization, even with artificially stressful workloads. In the most stressful workload, a single RAMCloud server can support 270,000-410,000 durable 100-byte writes per second at 90% memory utilization. The two-level approach to cleaning improves performance by up to 6x over a single-level approach at high memory utilization, and reduces disk bandwidth overhead by 7-87x for medium-sized objects (1 to 10 KB). Parallel cleaning effectively hides the cost of cleaning: an active cleaner adds only about 2% to the latency of typical client write requests.

And for your edification they've written an excellent paper on their findings. Log-structured Memory for DRAM-based Storage:

Categories: Architecture