Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Strategy: Three Techniques to Survive Traffic Surges by Quickly Scaling Your Site

Matthew Might, as a first responder to a surprise traffic surge on his inexpensive linode hosted blog, took emergency steps that you might find useful in a similar situation:

  1. Find the bottleneck. Reloading the page in firebug showed the first page took 24 seconds to load and after that everything else loaded quickly. In retrospect this burst meant the site was thread limited as the CPU was idle.
  2. Cut image sizes in half with a shell script using ImageMagick's convert. Load time is now 12 seconds.
  3. Turn dynamic content into static content using a static index.html file  copied using the browser's "view source" feature. Load time is now 6 seconds.
  4. Added threads to the Apache configuration file. Load time is now 2 seconds. Crises averted.

Because of this quick thinking and quick action the patient survived to serve pages another day.

And in fine post-mortem tradition some of the future changes are: 

Categories: Architecture

10 Big Ideas from XYZ

I’m trying out a new way to do book reviews, to share more value in a better, faster, and easier way, with a predictable experience.  

My new approach is to focus on 10 big ideas.

Here’s an example:

10 Big Ideas from BRIEF

Side note – BRIEF is a powerful book with hard-core techniques for getting to the point and cutting through fluff.  If you struggle with being verbose, or rambling, this book will help you master the art of “Lean Communication.”

In my book reviews in the past, I shared the challenges the book solved, the structure of the book, and some “scenes” from the book, sort of like a “movie trailer.”   While that was effective in terms of really doing a book justice, I thought there was room for improvement.

I figured, Sources of Insight is all about, well, “insight.”   So then my best approach would be to focus on the big ideas in the books I read, and share that unique value in a simple to consume fashion.   I considered “3 Big Ideas” and “5 Big Ideas”, but they both seemed too small.  And more than 10 seemed too big.

10 Big Ideas seems like a healthy dose of insights to draw from a book.

I had actually considered this approach a long time ago, but I was worried that it would water things down too much.  Instead, I’m finding that it’s doing the exact opposite.  Using 10 Big Ideas as a constraint is a great forcing function to help me really synthesize and distill the essence of a book, and to really hone in on the most valuable takeaways.  

And it’s a great way to turn insight into action in a very repeatable way.

I already read and review books at a fast pace, but I think this new approach is going to help me get even better and faster at rapidly sharing insight and action.

I’m in the early stages, so if you have ideas or feedback on the 10 Big Ideas approach for my book reviews, please let me know.

Take 10 Big Ideas from BRIEF for a spin.  Kick the tires.   It will be worth your time.  If you master Big Idea #7, alone, you'll be ahead of the game when it comes to making your pitch, or presenting your ideas.

Lean Communication can be your differentiator in a noisy, crowded, information overloaded world.

Categories: Architecture, Programming

How to Deal with the TED Effect

Software Architecture Zen - Pete Cripp - Wed, 03/19/2014 - 00:19
Nancy Duarte, CEO of Duarte Design and author of the books Resonate: Present Visual Stories that Transform Audiences and slide:ology: The Art and Science of Creating Great Presentations has written a great blog post about what she refers to as the TED effect. The TED effect refers to the impact that the TED conferences have had on all of us who need to present as part of our daily lives.
Nancy’s basic assertion is that “in public speaking it’s no longer okay to be boring”. In the years BT (before TED) it was okay to deliver boring presentations because actually no one knew if you were being boring or not because most people’s bar for what constituted a good presentation was pretty low anyway. In the dark years of BT we would all just sit stoically through those presentations that bored us to death and missed the point completely because bad presentations were just an occupational hazard we all had to learn to deal with. If nothing else it gave us time to catch up on our email or quietly chatter away to a colleague in the back row.
Now though everything has changed! For anyone that has seen more than half a dozen TED talks we know that if we are not engaged within the first 30 seconds we are ready to walk. Not only that if we felt you were wasting our time we go onto Twitter or Facebook and tell the rest of the world how boring you were. If however you did engage us and managed to get across your idea in 18 minutes or under (the maximum time of a TED talk) then we will reward you by spreading your ideas and help you get them adopted and funded.
As technical people software architects often struggle with presentations simply because they are communicating technology so, by definition, that must be complicated and take loads of time with lots of slides containing densely populated text or diagrams that cannot be read unless you are sitting less than a metre from the screen. But, as Nancy Duarte has explained countless times in her books and her blog, it needn’t be like that, even for a die-hard techno-geek.
Here’s my take on on how to deal with the TED effect:
  1. Just because you are given an hour to present, don’t think you have to actually spend that amount of time talking. Use the TED 18 minute rule and try and condense your key points into that time. Use the rest of the time for discussion and exchange of ideas.
  2. Use handouts for providing more detail. Handouts don’t just have to be documents given out during the presentation. Consider writing up the detail in a blog post or similar and provide a link to this at the end of your talk.
  3. Never, ever present slides someone else has created. If a presentation is worth doing then it’s worth investing the time to make it your presentation.
  4. Remember the audience is there to see you speak and hear your ideas. Slides are an aid to get those ideas across and are not an end in their own right. If you’re just reading what’s on the presentation then so can the audience so you may as well not be there.
  5. The best talks are laid out like a book or a movie. They have a beginning, a middle and an end. It often helps to think of the end first (what is the basic idea or point you want to get across) and work backwards from there. As Steven Pressfield says in the book Do the Work, “figure out where you want to go; then work backwards from there”.
  6. Finally, watch as many TED talks as you can to see to see how they engage with the audience and get their ideas across. One of the key attributes you will see all the great speakers have is they are passionate about their subject and this really shines through in their talk. Maybe, just maybe, if you are not really passionate about what your subject you should not be talking about it in the first place?
Also posted in my Wordpress blog here.
Categories: Architecture

Sponsored Post: Airseed, Uber, ScaleOut Software, Couchbase, Tokutek, Logentries, Booking, Apple, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7

 

Who's Hiring?
  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.  
    • Senior Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives that drive projects forward to completion. Please apply here.
    • Software Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives that drive projects forward to completion. Please apply here.
    • Quality Assurance Engineer. The iOS Systems team is looking for a Quality Assurance engineer. In this role you will be expected to work hand-in-hand with the software engineering team to find and diagnose software defects. Please apply here.

  • Airseed -- a Google Ventures backed, developer platform that powers single sign-on authentication, rich consumer data, and analytics -- is hiring lead backend and fullstack engineers (employees #4, 5, 6). More info here: https://www.airseed.com/jobs

  • Join the team that scales Uber supply globally! Our supply engineering team is responsible for prototyping, building, and maintaining the partner-facing platform. We're looking for experienced back-end developers who care about developing highly scalable services. Apply at https://www.uber.com/jobs/4810.

  • We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Apply: http://www.booking.com/jobs.en-us.html

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • The Biggest MongoDB Event Ever Is On. Will You Be There? Join us in New York City June 23-25 for MongoDB World! The conference lineup includes Amazon CTO Werner Vogels and Cloudera Co-Founder Mike Olson for keynote addresses.  You’ll walk away with everything you need to know to build and manage modern applications. Register before April 4 to take advantage of super early bird pricing.

  • How to Scale MySQL for Big Data Applications. A Guide to Evaluating TokuDB on March 20th at 1pm ET. You can do more than you think with the MySQL you already have. Learn how to use MySQL or MariaDB in Big Data applications by simply upgrading the storage engine with TokuDB. Register now.

  • April 3 Webinar: The BlueKai Playbook for Scaling to 10 Trillion Transactions a Month. As the industry’s largest online data exchange, BlueKai knows a thing or two about pushing the limits of scale. Find out how they are processing up to 10 trillion transactions per month from Vice President of Data Delivery, Ted Wallace. Register today.
Cool Products and Services
  • As one of the fastest growing VoIP services in the world Viber has replaced MongoDB with Couchbase Server, supporting 100,000+ operations per second in the short term and 1,000,000+ operations per second in the long term for their third generation architecture.  See the full story on the Viber switch.

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • Log management made easy with Logentries Billions of log events analyzed every day to unlock insights from the log data the matters to you. Simply powerful search, tagging, alerts, live tail and more for all of your log data. Automated AWS log collection and analytics, including CloudWatch events. 

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Intuitively Showing How To Scale a Web Application Using a Coffee Shop as an Example

This is a guest repost by Sriram Devadas, Engineer at Vistaprint, Web platform group. A fun and well written analogy of how to scale web applications using a familiar coffee shop as an example. No coffee was harmed during the making of this post.

I own a small coffee shop.

My expense is proportional to resources
100 square feet of built up area with utilities, 1 barista, 1 cup coffee maker.

My shop's capacity
Serves 1 customer at a time, takes 3 minutes to brew a cup of coffee, a total of 5 minutes to serve a customer.

Since my barista works without breaks and the German made coffee maker never breaks down,
my shop's maximum throughput = 12 customers per hour.

 

Web server

Customers walk away during peak hours. We only serve one customer at a time. There is no space to wait.

I upgrade shop. My new setup is better!

Expenses
Same area and utilities, 3 baristas, 2 cup coffee maker, 2 chairs

Capacity
3 minutes to brew 2 cups of coffee, ~7 minutes to serve 3 concurrent customers, 2 additional customers can wait in queue on chairs.

Concurrent customers = 3, Customer capacity = 5

 

Scaling vertically

Business is booming. Time to upgrade again. Bigger is better!...

Categories: Architecture

Updated Speaker Lineup for The texas Association of Enterprise Architects Summit

Mike Walker's Blog - Sat, 03/15/2014 - 18:28

 Texas Association of Enterprise Architects Summit Mike Walker

Space Limited -- Register Now!

With the popularity and as we come closer to the event on March 20th seats become more limited. To be sure you have a spot in the audience to listen to luminaries like John Zachman and Jeanne Ross along with our local Enterprise Architects in the Texas area proving thought leadership that you shouldn’t miss. Make sure to RSVP soon. Space is limited and admission is first come first served.

Come join us at the Austin Renaissance in the Arboretum on March 20th, 2014 from 8:00am to 5:00pm with an evening social from approximately 4:30pm –7: 00pm. Costs of the event are highly subsidized by our sponsors for a low cost of $50.00 for members of the AEA and $100.00 for non-members of the AEA.

 RSVP for the Texas Association of Enterprise Architects Summit Mike Walker

 

Updated Agenda for the Day

We have an action packed agenda!

 Texas Association of Enterprise Architects Summit Mike Walker

Venue and Lodging Details

The summit will be held at:

RENAISSANCE AUSTIN | 9721 Arboretum Blvd | Austin, TX 78759

www.renaissancehotels.com

Making Reservations

We have negotiated discounted room rates at: $189 per room per night plus applicable tax and service fee. You have until Friday, March 14th before the discount expires. Please make your arrangements before then.

Booking Website:
https://resweb.passkey.com/Resweb.do?mode=welcome_ei_new&eventID=11150319

Questions?

info@TexasAEA.org

Categories: Architecture

Stuff The Internet Says On Scalability For March 14th, 2014

Hey, it's HighScalability time:


LifeExplorer Cells in 3D
  • Quotable Quotes:
    • The Master Switch: History shows a typical progression of information technologies: from somebody’s hobby to somebody’s industry; from jury-rigged contraption to slick production marvel; from a freely accessible channel to one strictly controlled by a single corporation or cartel—from open to closed system.
    • @adrianco: #qconlondon @russmiles on PaaS "As old as I am, a leaky abstraction would be awful..."
    • @Obdurodon: "Scaling is hard.  Let's make excuses."
    • @TomRoyce: @jeffjarvis the rot is deep... The New Jersey pols just used Tesla to shake down the car dealers.
    • @CompSciFact: "The cheapest, fastest and most reliable components of a computer system are those that aren't there." -- Gordon Bell
    • @glyph: “Eventually consistent” is just another way to say “not consistent right now”.
    • @nutshell: LinkedIn is shutting down access to their APIs for CRMs (unless you’re Salesforce or Microsoft). Support open APIs!
    • Tim Berners-Lee: I never expected all these cats.
    • @muratdemirbas: "Simple clear purpose&principles give rise to complex&intelligent behavior. Complex rules&regulations give rise to simple&stupid behavior."
    • @BonzoESC: “Duplication is far cheaper than the wrong abstraction.” @sandimetz @rbonales 
    • @BenedictEvans: Umeng: there are 700m active smartphones and tablets in China.

  • Scale matters object lesson number infinity: HBO Go Crashes During True Detective Finale. Perhaps make HBO Go available without a cable package and maybe you'll have money to scale the service? Think peak. But wait, Dan Rayburn says bandwidth was not the problem, it's other parts of the system, which is why Internet TV will never be as reliable as broadcast TV. Still, I'd like to cut the cord.

  • Turns out ecommerce over messaging works well...quite well. Retailers Are Striking Gold with Instagram: Fox and Fawn, items often sell out within minutes of the picture being posted on Instagram.

  • Even Facebook's infrastructure struggles when a new feature becomes an unexpected hit. That's the situation described in an engaging story: Looking back on “Look Back” videos. Look Back's are one minute videos generated from a user's pics and posts. For the release they planned on 187 Gbps more bandwidth and 25 petabytes of disk. To get the rendering done they highly parallelized the pipeline. CDNs were alerted. Internal tests on employees found a few bugs. Less storage was actually needed because the video could be regenerated so a high replication factor wasn't needed. Go time! The videos were an unexpected hit with a 40% reshare instead of the projected 10% reshare. It seems people like themselves...a lot. Overnight 30 teams cooperated to move tens of thousands of machines over to rendering. Good story. Though I'm disappointed it didn't have its own Look Back video. Stories are people too.

  • Why Google's services aren't really free. We all help train the beast. A Glimpse of Google, NASA & Peter Norvig + The Restaurant at the End of the Universe: Algorithms behave differently as they churn thru more data. For example in the figure, the Blue algorithm was better with a million training dataset. If one had stopped at that scale, one would think of optimizing that algorithm for better performance. But as the scale increased the purple algorithm started showing promise – in fact the blue one starts deteriorating at larger scale; In general, Google prefers algorithms that get better with data. Not all algorithms are like that, but Google likes to after the ones with this type of performance characteristic. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Getting to a software architecture quickly

Coding the Architecture - Simon Brown - Thu, 03/13/2014 - 18:02

In “Think big, act small” – what does it mean in architecture?, Viktor Grgic says, "Let’s simplify things by talking about concrete things we actually do in software projects." Inspired by this, here's a summary of what I consider to be a minimal set of software architecture practices. Or to put it another way, here's what I do to get to a software architecture quickly.

A minimal set of software architecture practices

The discussion about what I consider to be "just enough up front design" can be found in the sample of my "Software Architecture for Developers" book, but it's about...

  • Structure: I look to understand the significant structural elements and how they fit together, based upon the architectural drivers (e.g. a product backlog plus the important non-functional requirements and constraints). I achieve this by doing design and decomposition down to containers and components. There are other ways to decompose software systems, but I like thinking in terms of components/services. I'll often sketch out a domain model too, particularly if I'm dealing with a domain that is new to me or non-trivial. Oh, and yes, this does include choosing technology. Often a single technology will be identified, and at other times there may be some options that we would like to evaluate. And this latter point leads us to...
  • Risks: I like to identify and mitigate the highest priority risks - the things with a higher than normal risk of consequences if I don’t get them right. Good examples include not adequately catering for non-functional requirements or putting too much faith in an untried technology. Risk identification can be done with techniques like risk-storming and mitigated by prototypes, walking skeletons, etc.
  • Vision: I create and communicate a vision so that we, as a team, understand what is being built and how *we* are going to build it. Note the emphasis on *we* ... I typically write code too. I use a collection of simple software architecture sketches to illustrate this. You can transcribe them into an electronic tool if you need to, but even a bunch of sketches on the wall can help tremendously to provide some focus.

Just to be clear, this isn't about gathering all of the requirements in any detail and it's not about big design up front either. A few short workshops to gather requirements and do some collaborative design is all it takes to get to a decent starting point. Since I'm a visual person, I'll even start to draft up a context diagram during workshops with non-techie people when we're gathering requirements.

That's basically it to be honest. I may do more if needed, but I don't generally do less than this. This isn't specifically an "agile" thing either, as I've followed this same basic approach for software projects of all shapes and sizes. The whole point of this is to introduce some direction and put some foundations in place that we can build upon. I have no problem with the architecture changing once we start writing code, but it should be done for a reason and in a structured way. Structure, risks and vision ... it's all about having a starting point for the rest of the delivery process, however you slice it up.

Categories: Architecture

Paper: Scalable Eventually Consistent Counters over Unreliable Networks

Counting at scale in a distributed environment is surprisingly hard. And it's a subject we've covered before in various ways: Big Data Counting: How to count a billion distinct objects using only 1.5KB of Memory, How to update video views count effectively?, Numbers Everyone Should Know (sharded counters).

Kellabyte (which is an excellent blog) in Scalable Eventually Consistent Counters talks about how the Cassandra counter implementation scores well on the scalability and high availability front, but in so doing has "over and under counting problem in partitioned environments."

Which is often fine. But if you want more accuracy there's a PN-counter, which is a CRDT (convergent replicated data type) where "all the changes made to a counter on each node rather than storing and modifying a single value so that you can merge all the values into the proper final value. Of course the trade-off here is additional storage and processing but there are ways to optimize this."

And there's a paper you can count on that goes into more details: Scalable Eventually Consistent Counters over Unreliable Networks:

Categories: Architecture

Douglas Adams - 3 Rules that Describe Our Reactions to Technologies

Chris Dixon unearthed a great quote from Douglas Adams on the nature of technological adoption that unsurprisingly hits the mark in our ever changing and evolving world:

  1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
  2. Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
  3. Anything invented after you’re thirty-five is against the natural order of things

Some that come to mind: horse to car, index card to online search, PC to mobile, web to app, portal to messaging, Newton to Einstein, oil to electric, rock to rap, Aquinas to Bacon, buying to renting, files to streaming, network TV to cordkilling, broadcast to social, programming CPUs to programming biology, server to cloud, vm to container, wired to wireless, long read to TLDR, privacy to public to ephemeral, paper based news aggregation to digital aggregation, checks to online banking, gold to fiat to bitcoin, linear to exponential growth, large to small teams, to a world that ignores you to a world the responds to you, nation states to who knows what, a military of people to a Military of Things, and weekly versus binge watching. Any others?

Categories: Architecture

Building a Social Music Service Using AWS, Scala, Akka, Play, MongoDB, and Elasticsearch

This is a guest repost by Rotem Hermon, former Chief Architect for serendip.me, on the architecture and scaling considerations behind making a startup music service.

serendip.me is a social music service that helps people discover great music shared by their friends, and also introduces them to their “music soulmates” - people outside their immediate social circle that shares a similar taste in music.

Serendip is running on AWS and is built on the following stack: scala (and some Java), akka (for handling concurrency), Play framework (for the web and API front-ends), MongoDB and Elasticsearch.

Choosing the stack

One of the challenges of building serendip was the need to handle a large amount of data from day one, since a main feature of serendip is that it collects every piece of music being shared on Twitter from public music services. So when we approached the question of choosing the language and technologies to use, an important consideration was the ability to scale.

The JVM seemed the right basis for our system as for its proven performance and tooling. It's also the language of choice for a lot of open source system (like Elasticsearch) which enables using their native clients - a big plus.

When we looked at the JVM ecosystem, scala stood out as an interesting language option that allowed a modern approach to writing code, while keeping full interoperability with Java. Another argument in favour of scala was the akka actor framework which seemed to be a good fit for a stream processing infrastructure (and indeed it was!). The Play web framework was just starting to get some adoption and looked promising. Back when we started, at the very beginning of 2011, these were still kind of bleeding edge technologies. So of course we were very pleased that by the end of 2011 scala and akka consolidated to become Typesafe, with Play joining in shortly after.

MongoDB was chosen for its combination of developer friendliness, ease of use, feature set and possible scalability (using auto-sharding). We learned very soon that the way we wanted to use and query our data will require creating a lot of big indexes on MongoDB, which will cause us to be hitting performance and memory issues pretty fast. So we kept using MongoDB mainly as a key-value document store, also relying on its atomic increments for several features that required counters.
With this type of usage MongoDB turned out to be pretty solid. It is also rather easy to operate, but mainly because we managed to avoid using sharding and went with a single replica-set (the sharding architecture of MongoDB is pretty complex).

For querying our data we needed a system with full blown search capabilities. Out of the possible open source search solutions, Elasticsearch came as the most scalable and cloud oriented system. Its dynamic indexing schema and the many search and faceting possibilities it provides allowed us to build many features on top of it, making it a central component in our architecture.

We chose to manage both MongoDB and Elasticsearch ourselves and not use a hosted solution for two main reasons. First, we wanted full control over both systems. We did not want to depend on another element for software upgrades/downgrades. And second, the amount of data we process meant that a hosted solution was more expensive than managing it directly on EC2 ourselves.

Some numbers
Categories: Architecture

Satya Nadella on How the Key To Longevity is To Be a Learning Organization

Everything should be a startup.

Unless you’re a learning organization that actually uses what you learn to leapfrog ahead.

But the paradox is you can’t hold on too tightly to what you’ve learned in the past.  You have to be able to let things go.  Quickly.  And, you have to learn new things fast.  And, if you can create a learning organization with tight feedback loops, that’s the key to longevity.

Adapt or die.

But the typical challenge in a big organization, is rejecting the new, and embracing the old.   And that’s how the giants, the mighty fall.

Here is how Satya Nadella told us how to think about what longevity means in our business

“What does longevity mean in this business? Longevity in this business means, that you somehow take the core competency you have but start learning how to express it in different forms.

And that to me is the core strength.

It's not the manifestation in one product generation, or in one specific feature, or what have you, but if you culturally, right, if you sort of look at what excites me from an organizational capacity building, ... it's that learning ... the ability to be able to learn new things ... and have those new things actually accrue to what we have done in the past ...  or what we have done in the past accrues to new learnings ... and that feedback cycle is the only way I can see scale mattering in this business ... otherwise, quite frankly you would say, everything should be a startup ... everything should be a startup ... you would have a success, you would unwind, and everything should be a startup ... and if you're going to have a large organization, it better be a learning organization that knows how to take all the learning that it's had today and make it relevant in the future knowing that you'll have to unlearn everything, and that's the paradox of this business and I think that's what I want us to be going for.”

In my experience, if you don’t know where to start, a great place to start is get feedback.  If you don’t know who to get feedback from, then ask yourself, your organization, who do you serve?   Ask the customers or clients that you serve.

But balance what you learn with vision.  And balance it with analytics and insight on behaviors and actions.   Customers, and people in general, can say one thing, but do another, or ask for one thing, but mean something entirely different. 

Remember the words of Henry Ford:

“If I'd asked customers what they wanted, they would have said ‘a faster horse’.”

Expressing pains, needs, opportunities, and desired outcomes leaves a lot of room for interpretation.

Drive with vision, build better feedback loops, interpret well, and learn well, to survive and thrive in an ever-changing world.

You Might Also Like

Microsoft Explained: Making Sense of the Microsoft Platform Story

Satya Nadella is the New Microsoft CEO

Satya Nadella is All About Customer Focus, Employee Engagement, and Changing the World

Satya Nadella on How Success is a Mental Game

Satya Nadella on Live and Work a Meaningful Life

Satya Nadella on the Future is Software

Satya Nadella on Everyone Has to Be a Leader

Categories: Architecture, Programming

Let's Play a Game of Take It or Leave It - Game 1

The way this game is played is you read a few statements on some hot topics below. If you agree with a statement then you “take it”; if you disagree then you “leave it.” And if you are so moved please write a convincing comment as to why. Got it?

  1. Snowden vs. the State. Snowden represents true the spirit of freedom and is not a threat to all we hold dear.

  2. Walled Garden vs. Federated Freedom. The Walled Garden has won the last decade. The cycle of life will return the balance and federated services will once again win the day.

  3. Mobile + messaging vs. Le Web. Mobile + messaging is eating search and the web, changing the way things are found, discovered, and bought.

  4. Fiat vs. Cryptocurrency. BitCoin has had its 400 million dollars of fame, it’s on the way out, a tulip gone out of bloom.

  5. True Detective vs. The Field. True Detective is the best show on TV, ever. Wired and Breaking Bad need not apply.

Categories: Architecture

AngularJS e2e testing using ngMockE2E

Xebia Blog - Sat, 03/08/2014 - 10:10

For our project we needed a mock http backend that we could instruct to return predefined responses so we were able to e2e test our full application without having a 'real' backend. This way we do not have to account for any state in the backend and can run our tests in isolation.

Requirements
  • Reuse the mockdata object created for our unit tests
  • Reuse the config object in which we define the backend endpoints
  • Have little or no impact on the production code

The solution

First thing that is required is a second module that depends on your application's ng-app module and on ngMockE2E. So say your ngApp is called 'myApp' you start by defining a module myAppE2E:

angular.module('myAppE2E', ['myApp', 'ngMockE2E']);

This wrapper module will be responsible for instructing the ngMockE2E version of $httpBackend what to respond to which request. For this instruction we use the run method of the angular.Module type:

angular.module('myAppE2E', ['myApp', 'ngMockE2E']).run(function ($httpBackend, $cookies) {
  // let all views through (the actual html views from the views folder should be loaded)
  $httpBackend.whenGET(new RegExp('views\/.*')).passThrough();
 
  // Mock out the call to '/service/hello'
  $httpBackend.whenGET('/service/hello').respond(200, {message: 'world'});
  // Respond with 404 for all other service calls
  $httpBackend.whenGET(new RegExp('service\/.*')).respond(404)
});

As you see we created 2 testable scenarios here: 1 successful call and 1 failure scenario. Note that the failure scenario should be defined after the success scenario as the failure scenario has a more generic selector and would handle the success scenario as well. This is also the point where you can inject any objects used in your 'myApp' module such as a config object or a mockData object.

Now we have our module we need some non-intrusive way of injecting it into our index.html. For this we chose processHtml. We added the following section to the bottom of our index.html:

<!-- build:include:e2e e2e.html -->
<!-- /build -->

This section is replaced by the contents of the e2e.html file when processHtml is run in e2e mode and left to be in all other modes. The contents of the e2e.html file are as follows:

<!-- We need angular-mocks as it contains ngMockE2E -->
<script src="bower_components/angular-mocks/angular-mocks.js"></script>
<!-- This file contains the myAppE2E module responsible for priming the $httpBackend -->
<script src="test/e2e/util/myAppE2E.js"></script>
<!-- Replace the original ng-app attribute with the myAppE2E module name so that one is run -->
<script type="text/javascript">
    $('body').attr('ng-app', 'myAppE2E');
</script>

Now all that is left is instructing grunt to run processHtml in the e2e test mode. We first need to add the config to the initConfig section. Here we tell it to use index.html as input and create index_e2e.html as output:

grunt.initConfig({
    processhtml: {
      e2e: {
        files: {
          '<%= yeoman.app %>/index_e2e.html': ['<%= yeoman.app %>/index.html']
        }
      }
    },
// snipped of all sorts of other config
}

Next we simply enhance the (yeoman generated) e2e task to run procssHtml:e2e before running the server

 grunt.registerTask('e2e', [
    'clean:server',
    // Simply add the task:
    'processhtml:e2e',
    'concurrent:server',
    'autoprefixer',
    'connect:e2e',
    'karma:e2e'
  ]);

There you have it. No when you start grunt e2e and go to index_e2e.html you will have full control over the http responses and can write e2e tests that do not require you to take into account any state of the backend.

E2E mode - but no tests

Every now and then it is useful to be able to test the complete application in isolation without running the e2e tests. We created a special grunt task for this that behaves like the yeoman default 'server' task but with the ngMockE2E module running. This way, whenever you change a resource in your project it is processed immediately and you can see the results by refreshing instead of restarting the e2e task

  grunt.registerTask('e2enotest', [
    'clean:server',
    'processhtml:e2e',
    'concurrent:server',
    'autoprefixer',
    'connect:e2e',
    'watch'
  ]);

More on haproxy geolocation detection and CDN services

Agile Testing - Grig Gheorghiu - Fri, 03/07/2014 - 19:38
In a previous blog post I described a method to do geolocation detection with haproxy. The country detection was based on the user's client IP. However, if you have a CDN service in front of your load balancer, then the source IPs will all belong to the CDN server farm, and the closest such server to an end user may not be in the same country as the user. Fortunately, CDN services generally pass that end user IP address in some specific HTTP header, so you can still perform the geolocation detection by inspecting that header. For example, Akamai passes the client IP in a header called True-Client-IP.

In our haproxy.cfg rules detailed below we wanted to handle both the case where our load balancer is hit directly by end users (in case we bypass any CDN service), and the case where the load balancer is hit via a CDN.

1) We set our own HTTP headers containing the country code as detected by geolocation based on a) the source IP (this is so we can still look at the source IP in case we bypass the CDN and hit our load balancer directly) and b) the specific CDN header containing the actual client IP (True-Client-IP in the case of Akamai):

http-request set-header X-Country-Src %[src,map_ip(/etc/haproxy/geolocation.txt)]

http-request set-header X-Country-Akamai %[req.hdr_ip(True-Client-IP,-1),map_ip(/etc/haproxy/geolocation.txt)]

2) We set an ACL that is true if we detect the presence of the True-Client-IP header, which tells us that we are hit via Akamai:
acl acl_akamai_true_client_ip_header_exists req.hdr(True-Client-IP) -m found

3) We set an ACL that is true if we detect that the country of origin (obtained via Akamai's True-Client-IP) is US:

acl acl_geoloc_akamai_true_client_ip_us req.hdr(X-Country-Akamai) -m str -i US
4) We set an ACL that is true if we detect that the country of origin (obtained via the source IP of the client) is US:
acl acl_geoloc_src_us req.hdr(X-Country-Src) -m str -i US
5) Based on the ACLs defined above, we send non-US traffic to a specific backend, IF we are being hit via Akamai (ACL #2) AND we detected that the country of origin is non-US (negation of ACL #3) OR if we detected that the country of origin if non-US via the source IP (negation of ACL #4):

use_backend www-backend-non-us if acl_akamai_true_client_ip_header_exists !acl_geoloc_akamai_true_client_ip_us or !acl_geoloc_src_us

(note that the AND is implicit in the way haproxy looks at combinations of ACLs)
6) We also we an HTTP header called X-Country which our application inspects in order to perform country-specific logic. We first set this header to the X-Country-Src header set in rule #1, but we override it if we are getting hit via Akamai:
http-request set-header X-Country %[req.hdr(X-Country-Src)]
http-request set-header X-Country %[req.hdr(X-Country-Akamai)] if acl_akamai_true_client_ip_header_exists

This looks pretty complicated, but it works well :-)


Stuff The Internet Says On Scalability For March 7th, 2014

Hey, it's HighScalability time:


Twitter valiantly survived an Oscar DDoS attack by non-state actors.
  • Several Billion: Apple iMessages per Day along with 40 billion notifications and 15 to 20 million FaceTime calls. Take that WhatsApp. Their architecture? Hey, this is Apple, only the Shadow knows.
  • 200 bit quantum computer: more states than atoms in the universe; 10 million matches: Tinder's per day catch; $1 billion: Kickstarter's long tail pledge funding achievement
  • Quotable Quotes:
    • @cstross: Let me repeat that: 100,000 ARM processors will cost you a total of $75,000 and probably fit in your jacket pocket.
    • @openflow: "You can no longer separate compute, storage, and networking." -- @vkhosla #ONS2014
    • @HackerNewsOnion: New node.js co-working space has 1 table and everyone takes turns
    • @chrismunns: we're reaching the point where ease and low cost of doing DDOS attacks means you shouldn't serve anything directly out of your origin
    • @rilt: Mysql dead, Cassandra now in production using @DataStax python driver.
    • @CompSciFact: "No engineered structure is designed to be built and then neglected or ignored." -- Henry Petroski
    • Arundhati Roy: Revolutions can, and often have, begun with reading.
    • Brett Slatkin: 3D printing is to design what continuous deployment is to code.
  • Well Facebook got on that right quick: Facebook wants to use drones to blanket remote regions with Internet. We talked about a drone driven Internet back in January. This is good news IMHO. Facebook will have the resources to make this really happen. Hopefully. Maybe. Cross your fingers.

  • A vast hidden surveillance network runs across America, powered by the repo industry. This intelligence database was powered by individuals driving around and taking pictures of licence plates to track cars. Imagine how Google Glass will enable the tracking of people, without any three letter government agencies in the loop. Crowdsourcing is fun!

  • Francis Bacon way back in the 1700s was all over BigData with his ant, spider, and honey bee analogy:  Good scientists are not like ants (mindlessly gathering data) or spiders (spinning empty theories).  Instead, they are like bees, transforming nature into a nourishing product. This essay examines Bacon's "middle way" by elucidating the means he proposes to turn experience and insight into understanding.  The human intellect relies on "machines" to extend perceptual limits, check impulsive imaginations, and reveal nature's latent causal structure, or “forms.”

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

The Times They Are A-Changin'

Software Architecture Zen - Pete Cripp - Fri, 03/07/2014 - 00:05
Come senators, congressmen
Please heed the call
Don’t stand in the doorway
Don’t block up the hall
For he that gets hurt
Will be he who has stalled
There’s a battle outside and it is ragin’
It’ll soon shake your windows and rattle your walls
For the times they are a-changin’


So sang Bob Dylan in The Times They Are a-Changin' from his third album of the same name released in early 1964 which makes it 50 years old this year.

These are certainly epochal changing times as we all try to understand the combined forces that social, mobile, analytic and cloud computing are going to have on the world and how we as software architects react to them.

You may have noticed a lack of posts in this blog recently. This is partly due to my own general busyness but also due to the fact that I have been trying to understand and assimilate myself what impact these changes are likely to have on this profession of ours. Is it more of the same, just that the underlying technology is changing (again) or is it really a fundamental change in the way the world is going to work from now on? Whichever it is these are some of the themes I will be covering in upcoming posts in this (hopefully) reinvigorated blog.

As of this posting I plan to move my blog over to Wordpress which I have decided I prefer as a blogging platform. For a while I will maintain both until I build up readers on the new platform. At some point a will begin to just put links on here to my Wordpress blog and eventually stop updating this one altogether, just leaving it as an archive. Please join me therefore over at the other Software Architecture Zen.
Categories: Architecture

End of an era in my garage

Xebia Blog - Wed, 03/05/2014 - 19:27

This weekend I was forced to throw away a cardboard box. So what, I hear you think and I agree, but it being Sunday and me being in a hazy reflective Sunday afternoon state of mind (nono, no alcohol yet) and the box being the specific cardboard box it is (or rather was), I started thinking of the box’s significance for the future.

This is a picture of the cardboard box in question in it’s final state just before it got thrown into the recycler.
End of an era in my garage
As you can see it’s got a large sticker saying ‘Oracle’ above the word ‘KABELS’ written in my clunky handwriting. The word kabels doesn’t really matter, it just indicates the box was used to hold various lengths of electrical wire. Next to the Oracle sticker and the kabels text you’ll see a partially torn label from ‘Donelly Documentation Services’ in Ireland. It is in fact the box that was used to ship my very first set of Oracle manuals (for release 6 of the Database and probably 3.0 of Forms and more contemporary tools) in April 1992. Since that time the box used to hold manuals for consecutive releases of Oracle products stored in the trunk of my car. I used to take them with me to customers on various assignments.
Later I left Oracle and the box was used in two removals (it was of remarkably sturdy material) and ended in my garage holding bits and pieces of electrical wire.
Last Sunday I dragged it of a shelf and finally it’s corners tore spilling cables all over me and the garage floor. Exit box.

This is getting into a long winded intro, I know but there is an interesting similarity between the box and the changes in the tools I use in my professional life. After starting as an Oracle specialist I worked on Java software which became Oracle software later of course. Last Sunday I realized I was writing Java-ish software again for the first time in about a year. Android, but still.
I haven’t been using any of the software I grew up with for quite a while now. Relational databases are being replaced by key-value stores and even flat files. JEE servers changed into Spray and Akka. Scala and functional and reactive programming constructs rather than object oriented Java and its bolted-on lambda extensions. Angular-JS instead of, hey wait a minute, that looks like a fat client all over again (wondering what’s going to happen to it). Server software I work with is easy to install, unpacking a zip file instead of 500+ pages of manual guaranteed only on an outdated Linux version. Modern software runs on simple Linux boxes instead of high end hardware with 7 digit price tags. New and innovative solutions replace software with slow release cycles. Open source is definitely leading the industry.

I wonder, does the end of the box forebode the end of a whole class of software? And like the cardboard box, will some bits and pieces come back in other products? I’m just hoping that it won’t take 22 years to get rid of the last generation of software.

10 Things You Should Know About Running MongoDB at Scale

Guest post by Asya Kamsky, Principal Solutions Architect at MongoDB.

This post outlines ten things you need to know for operating MongoDB at scale based on my experience working with MongoDB customers and open source users:

  1. MongoDB requires DevOps, too. MongoDB is a database. Like any other data store, it requires capacity planning, tuning, monitoring, and maintenance. Just because it's easy to install and get started and it fits the developer paradigm more naturally than a relational database, don't assume that MongoDB doesn't need proper care and feeding. And just because it performs super-fast on a small sample dataset in development doesn't mean you can get away without having a good schema and indexing strategy, as well as the right hardware resources in production! But if you prepare well and understand the best practices, operating large MongoDB clusters can be boring instead of nerve-wracking.
  2. Successful MongoDB users monitor everything and prepare for growth. Tracking current capacity and capacity planning are essential practices in any database system, and MongoDB is no different. You need to know how much work your cluster is currently capable of sustaining and what demands will be placed on it during times of highest use. If you don't notice growing load on your servers you'll eventually get caught without enough capacity. To monitor your MongoDB deployment, you can use MongoDB Management Service (MMS) to visualize your operations by viewing the opscounters (operation counters) chart:
  3. The obstacles to scaling performance as your usage grows may not be what you'd expect. Having seen hundreds of users' deployments, the performance bottlenecks usually are (in this order):
Categories: Architecture

Five things every developer should know about software architecture

Coding the Architecture - Simon Brown - Wed, 03/05/2014 - 09:03

Now I may be biased, but a quick look at my calendar hints to me that there's a renewed and growing interest in software architecture. Although I really like much of the improvement the agile movement has provided to the software development industry, I still can't help feeling that there are a large number of teams out there who struggle with a lack of process. After all, not every team is staffed with rockstar engineers! Although we've moved away from heavy prescriptive process frameworks, at least they provided a starting point for many of the activities associated with software development ... and this includes software architecture.

Put very simply, software architecture plays a pivotal role in the delivery of successful software yet it's frustratingly neglected by many teams. Whether performed by one person or shared amongst the team, the architecture role exists on even the most agile of teams yet the balance of up front and evolutionary thinking often reflects aspiration rather than reality. The big problem is that software architecture has fallen out of favour over the past decade or so. Here are five things that every software developer should know about it.

1. Software architecture isn't about big design up front

Software architecture has traditionally been associated with big design up front and waterfall-style projects, where a team would ensure that every last element of the software design was considered before any code was written. Software architecture is basically about the high-level structure of a software system and how you get to an understanding of it. This is about the significant decisions that influence the shape of a software system rather than understanding how long every column in the database should be.

2. Every software team needs to consider software architecture

Regardless of the size and complexity of the resulting product, every software team needs to consider software architecture. Why? Put simply, bad things tend to happen if they don't! If software architecture is about structure and vision, not thinking about this tends to lead to poorly structured, internally inconsistent software systems that are hard to understand, hard to maintain and potentially don't satisfy one or more of the important non-functional requirements such as performance, scalability or security. Explicitly thinking about software architecture provides you with a way to introduce technical leadership and stacks the odds of a successful delivery in your favour.

3. The software architecture role is about coding, coaching and collaboration

The image that many people have of software architects is of traditional "ivory tower" software architects dictating instructions to an unsuspecting development team. It doesn't need to be like this though, with modern software architects preferring an approach that favours coding, coaching and collaborative design. The software architecture role doesn't necessarily need to be undertaken by a single person plus coding is a great way to understand whether the resulting architecture is actually going to work.

4. You don't need to use UML

Again, traditional views of software architecture often conjure up images of huge UML (Unified Modeling Language) models that attempt to capture every last drop of detail. While creating and communicating a common vision is important, you don't need to use UML. In fact, you could argue that UML isn't a great method for communicating software architecture anyway. If you keep a few simple guidelines in mind, lightweight "boxes and lines" style sketches are an effective way to communicate software architecture.

5. A good software architecture enables agility

There's a common misconception that "architecture" and "agile" are competing forces, there being a conflict between them. This simply isn't the case though. On the contrary, a good software architecture enables agility, helping you embrace and implement change. Good software architectures aren't created by themselves though, and some conscious effort is needed.

Categories: Architecture