Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Aeron: Do we really need another messaging system?

Do we really need another messaging system? We might if it promises to move millions of messages a second, at small microsecond latencies between machines, with consistent response times, to large numbers of clients, using an innovative design.  

And that’s the promise of Aeron (the Celtic god of battle, not the chair, though tell that to the search engines), a new high-performance open source message transport library from the team of Todd Montgomery, a multicast and reliable protocol expert, Richard Warburton, an expert on compiler optimizations, and Martin Thompson, the pasty faced performance gangster.

The claims are Aeron is already beating the best products out there on throughput and latency matches the best commercial products up to the 90th percentile. Aeron can push small 40 byte messages at 6 million messages a second, which is a very difficult case.

Here’s a talk Martin gave on Aeron at Strangeloop: Aeron: Open-source high-performance messaging. I’ll give a gloss of his talk as well as integrating in sources of information listed at the end of this article.

Martin and his team were in the enviable position of having a client that required a product like Aeron and was willing to both finance its development while also making it open source. So go git Aeron on GitHub. Note, it’s early days for Aeron and they are still in the heavy optimization phase.

The world has changed therefore endpoints need to scale as never before. This is why Martin says we need a new messaging system. It’s now a multi-everything world. We have multi-core, multi-socket, multi-cloud, multi-billion user computing, where communication is happening all the time. Huge numbers of consumers regularly pound a channel to read from same publisher, which causes lock contention, queueing effects, which causes throughput to drop and latency to spike. 

What’s needed is a new messaging library to make the most of this new world. The move to microservices only heightens the need:

As we move to a world of micro services then we need very low and predictable latency from our communications otherwise the coherence component of USL will come to rain fire and brimstone on our designs.

With Aeron the goal is to keep things pure and focused. The benchmarking we have done so far suggests a step forward in throughput and latency. What is quite unique is that you do not have to choose between throughput and latency. With other high-end messaging transports this is a distinct choice. The algorithms employed by Aeron give maximum throughput while minimising latency up until saturation.

“Many messaging products are a Swiss Army knife; Aeron is a scalpel,” says Martin, which is a good way to understand Aeron. It’s not a full featured messaging product in the way you may be used to, like Kafka. Aeron does not persist messages, it doesn’t support guaranteed delivery, nor clustering, nor does it support topics. Aeron won’t know if a client has crashed and be able to sync it back up from history or initialize a new client from history. 

The best way to place Aeron in your mental matrix might be as a message oriented replacement for TCP, with higher level services written on top. Todd Montgomery expands on this idea:

Aeron being an ISO layer 4 protocol provides a number of things that messaging systems can't and also doesn't provide several things that some messaging systems do.... if that makes any sense. Let me explain slightly more wrt all typical messaging systems (not just Kafka and 0MQ). 

One way to think more about where Aeron fits is TCP, but with the option of reliable multicast delivery. However, that is a little limited in that Aeron also, by design, has a number of possible uses that go well beyond what TCP can do. Here are a few things to consider: 

Todd continues on with more detail, so please keep reading the article to see more on the subject.

At its core Aeron is a replicated persistent log of messages. And through a very conscious design process messages are wait-free and zero-copy along the entire path from publication to reception. This means latency is very good and very predictable.

That sums up Aeron is nutshell. It was created by an experienced team, using solid design principles sharpened on many previous projects, backed by techniques not everyone has in their tool chest. Every aspect has been well thought out to be clean, simple, highly performant, and highly concurrent.

If simplicity is indistinguishable from cleverness, then there’s a lot of cleverness going on in Aeron. Let’s see how they did it...

Categories: Architecture

How To Build a Roadmap for Your Digital Business Transformation

Let’s say you want to take your business to the Cloud --  How do you do it?

If you’re a small shop or a startup, it might be easy to just swipe your credit card and get going.

If, on the other hand, you’re a larger business that wants to start your journey to the Cloud, with a lot of investments and people that you need to bring along, you need a roadmap.

The roadmap will help you deal with setbacks, create confidence in the path, and help ensure that you can get from point A to point B (and that you know what point B actually is.)  By building an implementable roadmap for your business transformation, you can also build a coalition of the willing to help you get their faster.  And you can design your roadmap so that your journey flows continuous business value along the way.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share how top leaders build better roadmaps for their digital business transformation.

Why You Need to Build a Roadmap for Your Digital Transformation

If you had infinite time and resources, maybe you could just wing it, and hope for the best.   A better approach is to have a roadmap as a baseline.  Even if your roadmap changes, at least you can share the path with others in your organization and get them on board to help make it happen.

Via Leading Digital:

“In a perfect world, your digital transformation would deliver an unmatched customer experience, enjoy the industry's most effective operations, and spawn innovative, new business models.  There are a myriad of opportunities for digital technology to improve your business and no company can entertain them all at once.  The reality of limited resources, limited attention spans, and limited capacity for change with force focused choices.  This is the aim of your roadmap.”

Find Your Entry Point

Your best starting point is a business capability that you want to exploit.

Via Leading Digital:

“Many companies have come to realize that before they can create a wholesale change within their organization, they have to find an entry point that will begin shifting the needle.  How? They start by building a roadmap that leverages existing assets and capabilities.  Burberry, for example, enjoyed a globally recognized brand and a fleet of flagship retail locations around the world.  The company started by revitalizing its brand and customer experience in stores and online.  Others, like Codelco, began with the core operational processes of their business.  Caesars Entertainment combined strong capabilities in analytics with a culture of customer service to deliver a highly personalized guest experience.  There is no single right way to start your digital transformation.  What matters is that you find the existing capability--your sweet spot--that will get your company off the starting blocks.

Once your initial focus is clear, you can start designing your transformation roadmap.  Which investments and activities are necessary to close the gap to your vision?  What is predictable, and what isn't? What is the timing and scheduling of each initiative? What are the dependencies between them?  What organizational resources, such as analytics skills, are required?”

Engage Practitioners Early in the Design

If you involve others in your roadmap, you get their buy-in, and they will help you with your business transformation.

Via Leading Digital:

“Designing your roadmap will require input from a broad set of stakeholders.  Rather than limit the discussion to the top team, engage the operational specialists who bring an on-the-ground perspective.  This will minimize the traditional vision-to-execution gap.  You can crowd-source the design.  Or, you can use facilitated workshops, as as 'digital days,' as an effective way to capture and distill the priorities and information you will need to consider.  We've seen several Digital Masters do both.

Make no mistake; designing your roadmap will take time, effort, and multiple iterations.  But you will find it a valuable exercise.  it forces agreement on priorities and helps align senior management and the people tasked to execute the program.  Your roadmap will become more than just a document.  If executed well, it can be the canvas of the transformation itself.  Because your roadmap is a living document, it will evolve as your implementation progresses.”

Design for Business Outcome, Not Technology

When you create your roadmap, focus on the business outcomes.   Think in terms of adding incremental business capabilities.   Don’t make it a big bang thing.   Instead, start small, but iterate on building business capabilities that take advantage of Cloud, Mobile, Social, and Big Data technologies.

Via Leading Digital:

“Technology for its own sake is a common trap.  Don't build your roadmap as a series of technology projects.  Technology is only part of the story in digital transformation and often the least challenging one.  For example, the major hurdles for Enterprise 2.0 platforms are not technical.  Deploying the platform is relatively straightforward, and today's solutions are mature.  The challenge lies in changing user behavior--encouraging adoption and sustaining engagement in the activities the platform is meant to enable.

Express your transformation roadmap in terms of business outcomes.  For example, 'Establish a 360-degree understanding of our customers.'  Build into your roadmap the many facets of organizational change that your transformation will require customer experiences, operational processes, employee ways of working, organization, culture, communication--the list goes on.  This is why contributions from a wide variety is so critical.”

There are lots of way to build a roadmap, but the best thing you can do is put something down on paper so that you can share the path with other people and start getting feedback and buy-in.

You’ll be surprised but when you show business and IT leaders a roadmap, it helps turn strategy into execution and make things real in people’s minds.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

Drive Business Transformation by Reenvisioning Operations

Drive Business Transformation by Reenvisioning Your Customer Experience

Dual-Speed IT Drives Business Transformation and Improves IT-Business Relationships

How To Build a Better Business Case for Digital Initiatives

How To Improve the IT-Business Relationship

How Leaders are Building Digital Skills

Management Innovation is at the Top of the Innovation Stack

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For November 14th, 2014

Hey, it's HighScalability time:


Spectacular rendering of the solar system to scale. (Roberto Ziche)

 

  • 700: number of low-orbit satellites in a sidecar cheap internet; 130 terabytes: AdRoll ad data processed daily; 15 billion: daily Weather Channel forecasts; 1 million: AWS customers
  • Quotable Quotes:
    • @benkepes: Each AWS data center has typically 50k to 80k physical servers. Up to 102Tbps provisioned networking capacity #reinvent
    • @scottvdp: AWS just got rid of infrastructure behind any application tier. Lambda for async distributed events, container engine for everything else.
    • @wif: AWS is handling 7 trillion DynamoDB requests per month in a single region. 4x over last year. same jitter. #reinvent
    • Philae: If my path was off by even half a degree the humans would have had to abort the mission.
    • Al Aho: Well, you can get a stack of stacks, basically. And the nested stack automaton has sort of an efficient way of implementing the stack of stacks, and you can think of it as sort of almost like a cactus. That's why some people are calling it cactus automata, at the time.
    • Gilt: Someone spent $30K on an Acura & LA travel package on their iPhone.
    • @cloudpundit: Gist of Jassy's #reinvent remarks: Are you an enterprise vendor? Do you have a high-margin product/service? AWS is probably coming for you.
    • @mappingbabel: Things coming out from the AWS #reinvent analyst summit - Amazon has minimum 3 million servers & lights up own globe-spanning fibre.
    • @cloudpundit: James Hamilton says mobile hardware design patterns are future of servers. Single-chip offerings, semiconductor-level innovation. #reinvent
    • @rightscale: RT @owenrog: AWS builds its own electricity substations simply because the power companies can't build fast enough to meet demand #reInvent
    • @timanderson: New C4 instances #reinvent up to 36 cores up to 16TB SSD
    • @holly_cummins: L1 cache is a beer in hand, L3 is fridge, main memory is walking to the store, disk access is flying to another country for beer. 
    • @ericlaw: Sample HTTP compression ratios observed on @Facebook: -1300%, -34.5%, -14.7%, -25.4%. ProTip: Don't compress two byte responses. #webperf
    • @JefClaes: It's not the concept that defines the invariants but the invariants that define the concept.

  • It's hard to imagine just a few short years ago AWS did not exist. Now it has 1 million customers, runs 70 million hours of software per month, and their AWS re:Invent conference has a robust 13,500 attendees. Re:Invent shows if Amazon is going to be disrupted, a lack of innovation will not be the cause. The key talking point is that AWS is not just IaaS anymore, AWS is a Platform. The underlying subtext is lock-in. Minecraft-like, Amazon is building out their platform brick by brick. Along with GCE, AWS announced a Docker based container service. Intel designed a special new cloud processor for AWS, which will be available in a new C4 instance type. There's Aurora, a bigger, badder MySQL. To the joy of many EBS is getting bigger and faster. The world is getting more reactive, S3 is emitting events. With less fan fare are an impressive suite of code deployment and management tools. There's also a key management service, a configuration manager, and a service catalog. Most provocative is Lambda, or PaaS++, which as the name suggests is the ability to call a function in response to events. Big deal? It could be, though quite limited currently, only supporting Node.js and a few event types. You can't, for example, terminate a REST request. What it could grow in to is promising, a complete abstraction layer over the cloud so any sense of machines and locations are removed. They aren't the first, but that hardly matters.

  • It's not a history of the Civil War. Or WWW I. Or the Dust Bowl. But An Oral History of Unix. Yes, that much time has passed. Interviewed are many names you'll recognize and some you've probably never heard of. A fascinating window into the deep deep past.

  • No surprise at all. Plants talk to each other using an internet of fungus: We suggest that tomato plants can 'eavesdrop' on defense responses and increase their disease resistance against potential pathogen...the phantom orchid, get the carbon they need from nearby trees, via the mycelia of fungi that both are connected to...Other orchids only steal when it suits them. These "mixotrophs" can carry out photosynthesis, but they also "steal" carbon from other plants...The fungal internet exemplifies one of the great lessons of ecology: seemingly separate organisms are often connected, and may depend on each other. 

  • How do you persist a 200 thousand messages/second data stream while guaranteeing data availability and redundancy? Tadas Vilkeliskis shows you how with Apache Kafka. It excels at high write rates, compession saves lots on network traffic, and a custom C++ http-to-kafka accommodates performance.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Speaking in Australia - YOW! 2014

Coding the Architecture - Simon Brown - Thu, 11/13/2014 - 10:37

For my final trip of the year, I'm heading to Australia at the end of this month for the YOW! 2014 series of conferences. I'll be presenting Agility and the essence of software architecture in Melbourne, Brisbane and Sydney. Plus I'll be running my Simple sketches for diagramming your software architecture workshop in Melbourne and Sydney. I can't wait; see you there!

Categories: Architecture

Simple sketches for diagramming your software architecture

Coding the Architecture - Simon Brown - Wed, 11/12/2014 - 23:02

If you’re working in an agile software development team at the moment, take a look around at your environment. Whether it’s physical or virtual, there’s likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress.

As an industry, we’ve become adept at visualising our software development process over the past few years – however, it seems we’ve forgotten how to visualise the actual software that we’re building. I’m not just referring to post-project documentation. This also includes communication during the software development process. Agile approaches talk about moving fast, and this requires good communication, but it’s surprising that many teams struggle to effectively communicate the design of their software.

Read more...

Categories: Architecture

【翻訳】マイクロサービス – 分散された大きな泥だんご

Coding the Architecture - Simon Brown - Wed, 11/12/2014 - 21:06

モノリシックがダメだからといって、マイクロサービスが解決策になるわけではない ソフトウェア開発業界は流行に左右されやすいという証拠に、今マイクロサービスが、いたるところで大騒ぎされています。”次の大ブーム”だと思う人もいるでしょう。また、(10年前に”上出来”と見なされたような)大型のSOA、サービス指向アーキテクチャが単に軽量化して進化したものだと捉える人もいるでしょう。私は現在のマイクロサービスアーキテクチャに関しては好意的に見ています。しかし、だからといってこのアーキテクチャは決して万能薬ではありません。言うまでもないことかもしれませんが、多くの人が間違った理由でマイクロサービスに飛び付いているように思えるのです。

Read more...

Categories: Architecture

Three Reasons You Probably Don’t Need Multi-Data Center Capabilities

This is a guest post by Nikhil Palekar, Systems Architect, FoundationDB

For many organizations that care a lot about strong consistency and low latency or haven’t already built a fault tolerant application tier on top of their database, adding a multiple data center (MDC) database implementation may create more complexity or unintended consequences than meaningful benefits. Why might that be?

Categories: Architecture

Announcing Open Source of .NET Core Framework, .NET Core Distribution for Linux/OSX, and Free Visual Studio Community Edition

ScottGu's Blog - Scott Guthrie - Wed, 11/12/2014 - 16:45

This week we are holding our Connect() developer event in New York City.  This is an event that is being streamed online for free, and it covers some of the great new capabilities coming with the Visual Studio 2015 and .NET 5 releases.  You can watch the event live as well as on-demand here.

I just finished giving the opening keynote of the event during which I made several big announcements:

Announcing the Open Sourcing of the .NET Core Runtime and Libraries

Over the last several years we have integrated more and more open source technology into our .NET, Visual Studio, and Azure offerings.  We have also begun to open source more of our own code and technology as well.

Earlier this year, at the Build 2014 conference, I announced the creation of the .NET Foundation – which is an independent organization designed to foster the development and collaboration of open source technologies for .NET.  We have now open sourced ASP.NET, EF, Web API, NuGet and the "Roslyn" C# and VB compilers under it. 

It has been great to see the energy and innovation in these technologies since we made the open source announcements. We continue to have dedicated Microsoft teams working on each of them (several of the teams have more developers than ever before).  The open source process is now enabling the teams to collaborate even more with other developers in the community, and every single one of the above projects have now accepted code contributions from developers outside Microsoft.  The combination is enabling an even richer flow of ideas, and even better products.

Open Sourcing the .NET Core Runtime and Libraries

Today I’m excited to announce that we are going even further, and will be open sourcing the .NET Core Runtime.  This will include everything needed to execute .NET code – including the CLR, Just-In-Time Compiler (JIT), Garbage Collector (GC), and core .NET base class libraries.

We are releasing the source under the MIT open source license and are also issuing an explicit patent promise to clarify users patent rights to .NET.  This morning, we published the public repository on GitHub where the project will be hosted: https://github.com/dotnet/corefx

Today’s source release includes many of the newer core .NET framework libraries (ImmutableCollections, SIMD, XML and MetadataReader).  These libraries are fully open, and are ready to accept contributions.  Over the next several weeks and months we will continue to transfer source (including the Core CLR which is not there right now but in the process of being moved) into the repository and likewise make it open for contributions.

What does this open sourcing mean?

Today’s open source announcement means that developers will have a fully supported, fully open source, fully cross platform .NET stack for creating server and cloud applications – including everything from the C#/VB compilers, to the CLR runtime, to the core .NET base class libraries, to the higher-level .NET Web, Data and API frameworks.

It is an exciting day for .NET, and the new open source process will allow the .NET teams in Microsoft to collaborate even more deeply with other developers around the world.  The result is going to be even better products for everyone. Announcing .NET Core Framework on Linux and OSX

Last month at a Cloud Event we held in San Francisco, Satya Nadella – our CEO – showed a slide like this one where he talked about how Microsoft loves Linux:

image

We’ve worked hard with Azure to make it a first-class cloud platform for Linux based applications, and shared how more than 20% of all VMs running on Azure are Linux based.  In fact, we now have 5 different Linux distributions officially supported for use on Azure – with full integration within our management portal and command-line extensibility.

Bringing Core .NET to Linux and OS X

Today I’m excited to announce the .NET side of our Linux support.  In addition to making the .NET server stack open-source, we are also going to release an official distribution of the .NET Core for Linux, as well as an official distribution of .NET Core for the Mac operation system as well.

This will enable you to build .NET server and cloud applications and run them on both Windows Server and Linux.  It is going to enable every developer – regardless of what operating system they use to develop or target – to use .NET. And to do so on a fully open source runtime.

We will be working closely with the Mono community as we complete our Linux port.  The Mono community have done a great job advancing .NET and Linux over the last decade.  Releasing the .NET Core source under an open source license is going to enable us to collaborate together much more closely going forward.  There are many Linux enhancements Mono has built that we would like to use, and likewise there are improvements Mono will be able to benefit from by being able to use the .NET source code.  Today’s set of announcements are a big win for everyone. Announcing Visual Studio Community Edition

I’m also excited to announce that we are launching a new free edition of Visual Studio today that will empower even more developers to build great apps and solutions.

The new Visual Studio Community 2013 edition is a full-featured IDE.  It supports multiple project types in one solution file in a single IDE, and has all of the productivity features and IDE extensibility capabilities (meaning you can use Xamarin, ReSharper, VsVim, and any other VSIX extension) that developers love in Visual Studio.

It is now available completely free for:

  • Any individual developer working on a commercial or non-commercial project
  • Any developer contributing to an open source project
  • Anyone in an academic research or course setting (e.g. students, teachers, classroom, online course)
  • Any non-enterprise organization with 5 or fewer developers working on a commercial/non-commercial project together

We are making it available for download starting today, and developers can download and start using it immediately.  There is no program you need to join to use it – simply visit www.visualstudio.com, click the download button, and you are good to go. 

It is going to enable even more developers to take advantage of Visual Studio and build even better applications.  We are looking forward to seeing what you build with it. Summary

It has never been a better time to be a software developer.  Software is what enables organizations to succeed in today’s digital environment.  It is what enables businesses to connect better with customers, to deliver amazing new experiences, to drive new revenue streams, and to run operations more efficiently.

Using the cloud, every software developer on the planet can now create and build solutions that can reach millions of users, with no upfront costs, powered by a cloud infrastructure that delivers completely global reach.  The impact an individual developer can now have has never been greater than it is today.

Our goal at Microsoft is to provide developers with the platform and tools that will make them incredibly successful.  It is a mission we have had since the very beginning of the company.  Today’s .NET open source, cross platform, and Visual Studio Community edition announcements will enable the development technology we build to be leveraged by an even wider range of developers.  We are really excited to see some of the new apps and solutions that are built with it.

In addition to the above announcements, we are also announcing and demoing tons of new features and services for the first time at our Connect() event streamed from New York.  You can watch the online presentations here.  Also read Soma’s blog post for a summary of some of the new VS 2015 and .NET 5 capabilities we announced this week.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: @scottgu omni

Categories: Architecture, Programming

Online training now available via Parleys

Coding the Architecture - Simon Brown - Wed, 11/12/2014 - 13:43

As announced during the opening keynote at Devoxx Belgium this morning, I'm delighted to say that Software Architecture for Developers is now available as an online/video training course via Parleys.

Parleys

Parleys

The online course includes the same content as the regular on-site 2-day version, with the exception of the hands-on software design and sketching exercises. From the course overview page...

A lightweight approach to software architecture is pivotal to successfully delivering software, and it can complement agile approaches rather than compete against them. After all, a good architecture enables agility and this doesn't happen by magic. "Software Architecture for Developers" is a practical and pragmatic guide to lightweight software architecture. You'll learn:


  • The essence of software architecture.
  • Why the software architecture role should include coding, coaching and collaboration.
  • The things that you *really* need to think about before coding.
  • How to visualise your software architecture using simple sketches.
  • A lightweight approach to documenting your software.
  • Why there is *no* conflict between agile and architecture.
  • What "just enough" up front design means.
  • How to identify risks with risk-storming.

I'm excited to be working with Parleys on this and I think they have an amazing platform for delivering online training. If you're thinking about creating an online course, I recommend taking a look at Parleys. The tooling behind the scenes used to put the course together is incredible. Many thanks to Carlo Waelens and the Parleys team for everything over the past few months - I hope this is the start of something big for you.

Categories: Architecture

Sponsored Post: Apple, Asana, Hypertable, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Sr. Software Developer. We are looking for a solid senior-level Java/C programmer who will be working on security software development. This software will provide the data protection, integrity, and service authentication services for iOS devices. Please apply here.
    • DevOps Software Engineer - Apple Pay, iOS Systems.  The iOS Systems team is looking for an outstanding DevOps software engineer to help make our rapidly growing platform manageable, scalable, and reliable using state of the art technologies and cutting edge system automation. Come join the team to strategize, architect, and build infrastructure to help our systems perform and scale. Please apply here
    • Sr. Software Engineer-iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of performance tuning Java applications make your heart leap? Would you like to work in a fast-paced environment where your technical abilities will be challenged on a day to day basis? Do you want your work to make a difference in the lives of millions of people? Please apply here.
    • Apple Pay - Site Reliability Engineer. You already know this… every issue counts. A single ticket can be the key to discovering an issue impacting thousands of people. And now that you’ve found it, you can’t wait to fix it. You also know that owning the quality of an application is about separating the signal from the noise. Finding that signal is what motives you. Now that you’ve found it, you’re next step is to role up the sleeves and start coding. As a member of the Apple Pay SRE team, you’re expected to not just find the issues, but to write code and fix them. Please apply here.
    • Senior Software Engineer -iOS Systems. This role demands the best and brightest engineers. The ideal candidate will be well rounded and offer a diverse skill set that aligns with key qualifications. Practical experience integrating with a diverse set of third-party APIs will also serve to distinguish you from other candidates. This is a highly cross functional role, and the typical team member's day to day responsibilities on the Carrier Services team. Please apply here

  • As a production-focused infrastructure engineer at Asana, you’ll be the  person who takes the lead on setting and achieving our stability and uptime  goals, architecting the production stack, defining the on-call experience, the  build process, cluster management, monitoring and alerting. Please apply  here.

  • Platform Software Engineer, Sprout Social, builds world-class social media management software designed and built for performance, scale, reliability and product agility. We pick the right tool for the job while being pragmatic and scrappy. Services are built in Python and Java using technologies like Cassandra and Hadoop, HBase and Redis, Storm and Finagle. At the moment we’re staring down a rapidly growing 20TB Hadoop cluster and about the same amount stored in MySQL and Cassandra. We have a lot of data and we want people hungry to work at scale. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you. http://www.aerospike.com/aerospike-training/
Cool Products and Services
  • Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at http://www.hypertable.com/. Connect with Hypertable: @hypertable--Blog.

  • FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.

  • Diagnose server issues from a single tab. The Scalyr log management tool replaces all your monitoring and analysis services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. It's a universal tool for visibility into your production systems. Log aggregation, server metrics, monitoring, alerting, dashboards, and more. Not just “hosted grep” or “hosted graphs,” but enterprise-grade functionality with sane pricing and insane performance. Trusted by in-the-know companies like Codecademy – try it free!

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

How Leaders are Building Digital Skills

Cloud, mobile, social, and big data are changing the game of business.

But to play the game well, leaders need to grow new skills.

In order to create new customer experiences and market-leading operational capabilities, leaders need to invest in digital skills.

Our Cloud-First, Mobile-First world provides unprecedented possibilities in terms of connectivity and compute resources for changing customer experiences, transforming the workforce, and transforming operations, and creating new business models.   Companies every day are building amazing solutions that integrate Cloud, Mobile, Social, and Big Data capabilities as well as what the Internet of Things brings to the table.   But to take advantage of these capabilities, you need leaders that grow and invest in a digital platform and in digital skills.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share how top leaders grow their digital skills.

Creating Great Customer Experiences Requires New Skills and New Ways of Working

Whether you want to reimagine your customer experience, or reimagine your operations, it takes new skills, and new ways of working.   Companies that don’t have the right digital skills struggle.   Worse, everybody is competing for the same skills, including social media analysts, mobile marketers, cloud architects, and data scientists.

Via Leading Digital:

“Creating great customer experiences or market-leading operational capabilities is more than technology challenge.  It's also an organizational challenge requiring new skills and new ways of working.  Yet, 77 percent of companies in our first year of research cited missing digital skills as a major hurdle to their digital transformation success. To compound the problem, most companies are chasing after similar skills--social media analysts, mobile marketers, cloud architects, or data scientists, to name a few.”

How Digital Masters are Building Skills

If you want to help your company become a Digital Master, or, if you want to be a high-performing leader, you need to invest in digital skills.  

Via Leading Digital:

“So what are Digital Masters doing differently when it comes to skills? First, they are investing.  Of the Digital Masters we surveyed, 82 percent are building the digital skills they need to support transformation efforts.  Only 40 perce3nt of nonmasters are doing so.

Second, Digital Masters are accelerating and creating  a gap.  Our survey research shows that the masters had greater digital skills than nonmasters, reporting 31 percent higher social media skills, 38 percent higher mobile skills, and 19 percent higher analytics skills.

But Digital Masters did not start with higher skills.  Burberry did not become excellent at digital marketing. and channels overnight.  CEO Ahrendts hired a new, dynamic marketing team whose members mirrored the behaviors of the millennial customer.  Nor did Caesars excel at delivering personalized customer experience solely because its CEO, Gary Loveman, has a PhD in economics from MIT. Caesars' executives actively incorporated quantitative skills into the marketing area.  In these companies, like other Digital Masters, top executives worked hard to build the digital skills they needed.”

The Line Between Technical Skills and Leadership Skills is Blurring Fast

The gap is huge but the lines blur fast.  There is a huge demand for people that are both business savvy and technology savvy.

Via Leading Digital:

“The skills difference extends beyond technology.  Digital Masters report 36 percent higher skills in digital leadership than nonmasters. Digital transformation requires changes to processes and thinking--changes that span your internal organizational silos.  'The clear delineation between technical skills and leadership skills in blurring fast.

The impact of digital technologies is now felt not only in the IT and technical departments, but also across the entire organization.  Digital transformation's need for cross-functional collaboration creates a huge demand for hybrid digital skills-- technical people who need to be more business savvy and businesspeople who need to be more technology savvy.  A retail executive explained: 'We are trying for the first time to work across the company.  That implies going through a new level of complexity in the organization, and requires people to manage and network differently.  That, I think, is the most important skills that needs to be developed.'”

Successful Leaders Will Have Business and Technical Skills

True hybrid professionals will be the leaders of tomorrow.

Via Leading Digital:

“The need for new skills can also result from the need to bridge the communication gap between digital and business competences.  One executive said, 'I need a charismatic quant--somebody who's an influencer and can carry his weight in a senior meeting, but at the same time, someone who can roll up his sleeves and look at data tables and build models and enjoy it.'

These bridging roles may soon become the responsibility of every manager. 'I believe,' said Markus Nordlin, CIO of Zurich Insurance, 'that the successful leaders of tomorrow, in any business or industry, are going to be true hybrid professionals who have spent some time in IT but have shifted to operations and vice-versa.'”

Digital Skills Create Competitive Advantage and Enable Digital Transformation

To keep up and get ahead, you need to master Digital Skills and be able to use them in a business savvy way.

Via Leading Digital:

“Aspiring Digital Masters are all chasing the same technical skills.  The shortage of digital skills is unprecedented.  In Europe alone, forecasts point to nearly a million vacancies for IT-related roles by 2015.  And globally, out of the 4.4 million big-data jobs to be created by 2015, only a third will be filled.

But by the same token, business professionals will increasingly need to be comfortable with digital tools and technologies to perform their core roles.  By 2015, research firm IDC expects that 90 percent of all jobs will require IT skills.  Some business functions are already adding technology skills to their mix.  Gartner reports that 70 percent of the companies they surveyed have a chief marketing technologist to support the digitization of the function.

This skills race won't slow down anytime soon.  Having the right digital skills is an important source of competitive advantage and a key enabler of digital transformation.  Companies that build skills faster will get ahead.

To win at the digital skills race, you will need to tap into multiple approaches--hiring, partnering, incubating, and the like.  It's not easy, as one executive explained: 'Our recruiters don't know where to go to find these people, and people with the right skills don't look to our kind of company for opportunities.'  HR organization will need to get up to speed quickly.  A recent Capgemini Consulting survey found that only 30 percent of HR functions were actively involved in digital skills development.  This needs to change.  Many Digital Masters have a carefully crafted plan to fight and win the talent race.”

All of the capabilities of Cloud, Mobile, Social, and Big Data are right at your fingertips.

Using these capabilities in meaningful ways takes a combination of business and technical skills, as well as great organizational change leadership skills.

If you can master business skills and combine them with great technical skills, you can lead you, your team, your organization, and others to change the world.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

Drive Business Transformation by Reenvisioning Operations

Drive Business Transformation by Reenvisioning Your Customer Experience

Dual-Speed IT Drives Business Transformation and Improves IT-Business Relationships

How To Improve the IT-Business Relationship

Management Innovation is at the Top of the Innovation Stack

Categories: Architecture, Programming

The neverending waveform of the full-stack developer

Xebia Blog - Tue, 11/11/2014 - 16:50

There was an article on Techcrunch a couple days ago which was linked in our internal mailing list the other day, titled The Rise And Fall Of The Full Stack Developer. I read it, and I just couldn't figure out why the title is about "the fall" of the full-stack developer (and I said as much on the mailing list, after which I was encouraged to write this blog post). In this post I'll try and explain why it's not the end, but just a stage in a recurring cycle

What I read was a waveform - the article describes that first you had low-level assembly, which is specialised but still pretty straightforward since it's just one platform and language. Then things started to get more complicated, with the larger web applications involving lots of experts in various fields (Java, database management, server and JVM management, to name a few from the article).

In or around 2005, there was a bit of a revolution going on. Internet access (and high-speed internet for that matter) became accessible to all, and with it, cheap webhosting - most notably, there was a boom in PHP development. What the article highlights is that that was an era where single or small teams of developers could build a web application from scratch, without needing to lug around all of that expert knowledge - effectively, a full-stack developer (storage, webhosting, a programming language (PHP, Python, Ruby, etc), HTML, CSS, Javascript, the whole stack).

But, as the article states, that isn't the full stack of today - added to that are things like machine learning, mobile development, more advanced and less traditional programming languages, frameworks and persistence solutions, etcetera. The conclusion of the author is that it's too much for one full-stack developer to take, that there's no way a full-stack developer can be an expert in all of those fields - and he basically renames the person that knows a bit of all of those technologies a full-stack integrator, and declares the full-stack developer dead.

But I didn't see that. What I saw was just history repeating - from simple and manageable, to complex and too much for one person to handle and know all about. From assembly and PHP, to Java enterprise software and Docker-contained AWS instances running a MongoDB and Scala REST interface to power your Android, iOS and single-page AngularJS-powered webapps (to name but a few buzzwords).

I'm sure the next 'simple' wave is already around the corner - maybe it's already here, lurking somewhere until some more influential guy goes "Hey guys, let's go back to the simpler, good old times where you wrote code and it Just Worked™!", where a wave of developer will follow - most likely a combination of younger people, new to the software development world, and older people that have been stuck in an overly complicated software development rut for far too long.

As for not being able to keep up with it all, this is why it's probably better to assemble teams out of T-shaped developers - full-stack developers with a broad knowledge set (and more importantly, broad interests and curiosity), but with at least one specialisation. This was extended within Xebia to pi-shaped developers, then made extreme to comb-shaped developers, but the latter is just a generalist like mentioned in the article - a jack of all trades, master of none. And it's important to realise, as a developer, that it's OK to not know everything, to miss some update, to not learn that fancy new programming language by people disgruntled by Java's slow development - there's simply too much information updates today, and trying to manage it all in the extremest forms can lead to serious problems. But the rapid development is a good thing, I might add, I don't think the software development world has ever moved as fast as it does today, and it's not nearly the end yet as long as more than half of the world's population doesn't have access to the internet and its boundless resources.

I think increasing complexity is just part of software development. I mean look at Facebook and Twitter - they were started using those simple, accessible tools like PHP and Ruby on Rails, which allowed them to get a flying start and adapt quickly to their huge growth (and with the former desperately clinging to their decision, despite lots of people telling them they should use a different technology), but despite that they still turned into some of the most complicated pieces of software ever. The important bit is to be able to manage all that, not so much a decision between a simple or complicated toolset - or having all of your developers know everything. Similarly, the current trend is microservices - again back to simple, one-purpose miniature applications that do one thing and do them right. But those will just end up being the next generation of huge, complicated software systems, given enough time. As a colleague stated, "Microservices are just hipster SOA".

As the saying goes, the more things change, the more they stay the same. The full-stack developer isn't dead and won't be going away any time soon, he'll just go by a different name depending on the times - J2EE Certified Software Engineer, full-stack developer, multidisciplinary expert, T-shaped developer, Xebian, chief of technology evangelisation, or whatever else the HR or marketing departments of the near future will come up with.

Nifty Architecture Tricks from Wix - Building a Publishing Platform at Scale

Wix operates websites in the long tale. As a HTML5 based WYSIWYG web publishing platform, they have created over 54 million websites, most of which receive under 100 page views per day. So traditional caching strategies don’t apply, yet it only takes four web servers to handle all the traffic. That takes some smart work.

Aviran Mordo, Head of Back-End Engineering at Wix, has described their solution in an excellent talk: Wix Architecture at Scale. What they’ve developed is in the best tradition of scaling is specialization. They’ve carefully analyzed their system and figured out how to meet their aggressive high availability and high performance goals in some most interesting ways.

Wix uses multiple datacenters and clouds. Something I haven’t seen before is that they replicate data to multiple datacenters, to Google Compute Engine, and to Amazon. And they have fallback strategies between them in case of failure.

Wix doesn’t use transactions. Instead, all data is immutable and they use a simple eventual consistency strategy that perfectly matches their use case.

Wix doesn’t cache (as in a big caching layer). Instead, they pay great attention to optimizing the rendering path so that every page displays in under 100ms.

Wix started small, with a monolithic architecture, and has consciously moved to a service architecture using a very deliberate process for identifying services that can help anyone thinking about the same move.

This is not your traditional LAMP stack or native cloud anything. Wix is a little different and there’s something here you can learn from. Let’s see how they do it...

Stats
Categories: Architecture

程序员必读之软件架构

Coding the Architecture - Simon Brown - Mon, 11/10/2014 - 16:40

A quick note to say that the Chinese translation of my Software Architecture for Developers book has now been released and is available from Turing Book Company, both as an e-book and hard-copy.

Software Architecture for Developers

I know there's demand for a hard-copy of the regular version, so I'll be doing this early next year, probably as a print-on-demand book from somewhere like Lulu, CreateSpace, etc.

Categories: Architecture

Dual-Speed IT Drives Digital Business Transformation and Improves IT-Business Relationships

Don’t try to turn all of your traditional IT into a digital unit.  

You’ll break both, or do neither well.

Instead,  add a Digital Unit.   Meanwhile, continue to simplify and optimize your traditional IT, but, at the same time, add a Digital Unit that’s optimized to operate in a Cloud-First, Mobile-First world.

This is the Dual-Speed IT approach, and, with this way, you can choose the right approach for the job and get the best of both worlds.

Some projects involve more extensive planning because they are higher-risk and have more dependencies.

Other projects benefit from a loose learning-by-doing method, with rapid feedback loops, customer impact, and testing new business waters.

And, over time, you can shift the mix.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are Digital Masters that created their digital visions and are driving business change.

Build Digital Skills Into One of Your Existing Business Units

You can grow one of your existing business units into a Digital Unit.  For example, marketing is a pretty good bet, given the customer focus and the business impact.

Via Leading Digital:

 

“Changing the IT-business relationship is well worth the effort, but doing so takes time.  Your company may not have the time to wait before starting your digital transformation.  Rather than improving the IT unit, some companies try to build digital skills into another unit, such as marketing.  They try to work around IT rather than with it.”

Don’t Mix Your Digital Unit with Your Traditional IT

Don’t throw away your existing IT or break it by turning it into something it’s not, too quickly.   Instead, leverage it for the projects where it makes sense, while also leveraging your new Digital IT unit.

Via Leading Digital:

“Although building digital skills is useful, trying to work around IT can be fraught with challenges, especially if people do not understand the reasons for IT's systematic, if sometimes ponderous, processes.  This kind of flanking action can waste money, make the digital platform more complex, and even worse, open the company to security and regulatory risks.”

Create a Dual-Speed IT to Support Both Traditional IT and Faster-Speed Digital Transformation

You can have the best of both worlds, while both evolving your traditional IT and growing your Digital Unit to thrive at Cloud speed.

Via Leading Digital:

“A better approach is to create a dual-speed IT structure, where one part of the IT unit continues to support traditional IT needs, while another takes on the challenge of operating at digital speed with the business.  Digital activities--especially in customer engagement--move faster than many traditional IT ones.  They look at design processes differently.  Where IT projects have traditionally depended on clear designs and well-structured project plans, digital activities often engage in test-and-learn strategies, trying features in real-life experiments and quickly adding or dropping them based on what they find.”

Optimize the Digital Unit for Digital World

Your Digital Unit needs to be very different from traditional IT in terms of the mindset and the approaches around the people, processes, and technology.

Via Leading Digital:

“In a dual-speed approach, the digital unit can develop processes and methods at clock-speeds more closely aligned with the digital world, without losing sight of the reasons that the old IT processes existed.  IT leaders can draw on informal relationships within the IT department to get access to legacy systems or make other changes happen.  Business leaders can use their networks to get input and resources.  Business and IT leaders can even start to work together in the kind of two-in-a-box leadership method that LBG and other companies have adopted.”

Choose the Right Leadership Both in the Business and in IT

To make it work and to make it work well, it takes partnerships on both sides.   The business and IT both need skin in the game.

Via Leading Digital:

“Building dual-speed IT units requires choosing the right leadership on both sides of the relationship.  Business executives need to be comfortable with technology and with being challenged by their IT counterparts.  IT leaders need to have a mind-set that extends beyond technology to encompass the processes and drivers of business performance.  Leaders from both sides need to be strong communicators who can slide easily between conversations with their business- or IT-focused people.”

Great IT Leaders Know When to Choose Traditional IT vs. the Digital Unit

With both options at your disposal, Great IT Leaders know how to choose the right approach for the job.   Some programs and projects will take a more traditional life-cycle or require heavier planning or more extensive governance and risk management, while other projects can be driven in a more lightweight and agile way.

Via Leading Digital:

“Dual-speed IT also requires perspective about the value of speed.  Not all digital efforts need the kind of fast-moving, constantly changing processes that digital customer-engagement processes can need.  In fact, the underlying technology elements that powered LBG's new platform, Asian Paints' operational excellence, and Nike's digital supply chain enhancements required the careful, systematic thinking that underpins traditional IT practices.  Doing these big implementations in a loose learning-by-doing method could be dangerous.  It could increase rework, waste money, and introduce security risks.  But once the strong digital platform is there, building new digital capabilities can be fast, agile, and innovative.  The key is to understand what you need in each type of project and how much room any project has to be flexible and agile.  Great IT leaders know how to do this.  If teamed with the right business leaders, they can make progress quickly and safely.”

Dual-Speed IT Requires New Processes within IT

It takes a shift in processes to do Dual-Speed IT.

Via Leading Digital:

“Dual-speed IT also takes new processes inside IT.  Few digital businesses have the luxury to wait for monthly software release cycles for all of their applications.  Digital-image hosting business Flickr, for example, aims for up to ten deployments per day, while some businesses require even more.  This continuous-deployment approach requires very tight discipline and collaboration between development, test, and operations people.  A bug in software, missed step in testing, or configuration problem in deployment can bring down a web site or affect thousands of customers.”

DevOps Makes Dual Speed IT Possible

DevOps blends development and operations into a more integrated approach that simplifies and streamlines processes to shorten cycle times and speed up fixes and feedback loops.

Via Leading Digital:

“A relatively new software-development method called DevOps aims to make this kind of disciplined speed possible.  It breaks down silos between development, operations, and quality assurance groups, allowing them to collaborate more closely and be more agile.  When done properly, DevOps improves the speed and reliability of application development and deployment by standardizing development environments.  It uses strong methods and standards, including synchronizing the tools used by each group.”

DevOps Can Help IT Release Software Better, Faster, Cheaper, and More Reliably

DevOps is the name of the game when it comes to shipping better, faster, cheaper and more reliably in a Cloud-First, Mobile-First world.

Via Leading Digital:

“DevOps relies heavily on automated tools to do tasks in testing, configuration control, and deployment—tasks that are both slow and error-prone when done manually.  Companies that use DevOps need to foster a culture where different IT groups can work together and where workers accept the rules and methods that make the process effective.  The discipline, tools, and strong processes of DevOps can help IT release software more rapidly and with fewer errors, as well as monitor performance and resolve process issues more effectively, than before.”

Driving Digital Transformation Takes a Strong Link Between Business and IT Executives

In order for your Digital Transformation to thrive, it takes building better bridges between the business leaders and the IT leaders.

Via Leading Digital:

“Whether your CIO takes it upon himself or herself to improve the IT-business relationship, or you decide to help make it happen, forging a strong link between business and IT executives is an essential part of driving digital transformation.  Strong IT-business relationships can transform the way IT works and the way the business works with it.  Through trust and shared understanding, your technology and business experts can collaborate closely, like at LBG, to innovate your business at digital speeds.  Without this kind of relationship, your company may become mired in endless requirements discussion, filing projects, and lackluster systems, while your competitors accelerate past you in the digital fast lane.”

If you want to thrive in the new digital economy while driving digital business transformation without breaking your existing business, consider adding Dual-Speed IT to your strategies and shift the mix from traditional IT to your Digital Unit over time.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

Drive Business Transformation by Reenvisioning Operations

Drive Business Transformation by Reenvisioning Your Customer Experience

How To Improve the IT-Business Relationship

Management Innovation is at the Top of the Innovation Stack

Think in a Series of Sprints, Not Marathons

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For November 7th, 2014

Hey, it's HighScalability time:


Google's new hyper efficient, hyper secure datacenters. (keynotelarge, m, o, r e)
  • 2 billion: containers started every week at Google; 86%: Apple's share of handset profits; 50: number of parallel processes in the brain; $12 million: cost for every additional second it takes customers to pay at a Walmart store; 800 terabytes: rendering for the movie Interstellar
  • Quotable Quotes:
    • @IFTTTRecipe: http://Highscalability.com  to email @ http://ift.tt/1omSSgs Send new highscalability.com articles to my mailbox
    • @chadfowler: I've decided to break up tomorrow's monolithic talk about microservices into a series of microtalks #gotober
    • Emily Balcetis: Attention can “be thought of as what you allow your eyes to look at.”
    • Adam Alter: researchers have shown that our names take root deep within our mental worlds, drawing us magnetically toward the concepts they embody
    • @mjpt777: Monitoring needs to reflect an issue within 10 seconds of it happening. Poll at 1 second granularity. - @adrianco #gotober
    • @adyranov: "Trust only benchmarks you've made yourself. All others are benchmarketing" #highload2014
    • @adrianco: #GOTOber @mjpt777 Aeron performance on a laptop - 20 million 40byte messages/sec. 90%ile latency of 7us. 100% 37us.
    • @mjpt777: Have your infrastructure optimised for speed of delivery rather than cost, stuff then gets done so fast you reduce cost. @adrianco #gotober
    • Seth Newburg: This is a first in mobile! This is a device running over an internal network, rather than just everything being connected to a CPU
    • @swardley: OpenStack vs AWS - IMHO OpenStack should have dominated but pisspoor strategic play makes it unlikely for 10-15yrs.
    • Rudiger Moller: There is another strong value in going off heap (even when using advanced collectors like Zing's): Datastructures expressed in Java frequently have a redundancy of 80-90%.
    • @zooko: Novice engineers have not yet grokked this: the number of modes or options in your system is the *exponent* in how hard it is to maintain.

  • OK Google...give me more cool functionality and lower prices. And they did. Google Cloud Platform Live: Introducing Container Engine, Cloud Networking and much more. This may take a while to process and put in some context. What Google is continuing to do is consumerize their internal services. The biggest example of this is opening up their container scheduling, management, and auto scaler services, while making it Docker compatible. If you want to run code as close as possible to your users you can now direct peer to any of Google's over 70 points of presence in 33 countries around the world. That's impressive. AWS has 14 and Azure has 11. Firebase is still behind Parse in features, but with Google's support look for it to blossom. The distributed debugger is truly impressive work. Game changing really. There's also a new improved Pub/Sub system. Slowly but surely the worm turns. 

  • A great list. I'd like to see what a similar list for the message dispatch and handlers would look like. Martin Thompson design principles for the Aeron messaging system: garbage free in steady state running; smart batching in the message path; wait-free algos in the message path; non-blocking IO in the message path; no exceptional cases in message path; apply the single writer principle; prefer unshared state; avoid unnecessary data copies. 

  • Nicely done André Staltz. The introduction to Reactive Programming you've been missing. If insight meditation has not been enough to guide your conversion from the object obsessed to the functionally enlightened then this article might be just what you are looking for. Or not.

  • This is cool, using Cassandra as an embedded database. Russian Facebook Ok.ru Utilizing Cassandra for 1 Million Operations per Second, Over 80TB in Largest Cluster: The way we use Casssandra is somewhat unusual – we don’t use thrift or netty based native protocol to communicate with Cassandra nodes remotely. Instead, we co-locate Cassandra nodes in the same JVM with business service logic, exposing not generic data manipulation, but business level interface remotely. This way we avoid extra network roundtrips within a single business transaction and use internal calls to Cassandra classes to get information faster. Also this helps us to do many little hacks on its internals, making huge gains on efficiency and ease of distributed servers development.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Mutation Testing: How Good are your Unit Tests?

Xebia Blog - Thu, 11/06/2014 - 20:56

You write unit tests for every piece of code you deliver. Your test coverage is close to 100%. So when the point comes when you have to make some small changes to the existing code, you feel safe and confident that your test suite will protect you against possible mistakes.
You make your changes, and all your tests still pass. You should be fairly confident now that you can commit your new code without breaking anything, right?

Well, maybe not. Maybe your unit tests were fooling you. Sure they covered every line of your code, but they could have performed the wrong assertions.
In this post I will introduce mutation testing. Mutation testing can help you find omissions in your unit tests.

Let's begin with a small example:

package com.xebia;

public class NameParser {
  public Person findPersonWithLastName(String[] names, String lastNameToFind) {
    Person result = null;
    for(int i=0; i <= names.length; i++) { // bug 1
      String[] parts = names[i].split(" ");
      String firstName = parts[0];
      String lastName = parts[1];
      if(lastName.equals(lastNameToFind)) {
        result = new Person(firstName, lastName);
        break;
      }
    }
    return result;
  }
}

NameParser takes a list of strings which consist of a first name and a last name, searches for the entry with a given last name, instantiates a Person object out of it and returns it.
Here is the Person class:

package com.xebia;

public class Person {
  private final String firstName;
  private final String lastName;

  public Person(String firstName, String lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
  }

  public String getFirstName() {
    return firstName;
  }

  public String getLastName() {
    return firstName; // bug 2
  }
}

You can see that there are two bugs in the code. The first one is in the loop in NameParser, which loops past the end of the names array. The second on is in Person, which mistakenly returns firstName in its getLastName method.

NameParser has a unit test:

package com.xebia;

import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.assertEquals;

public class NameParserTest {
  private NameParser nameParser;
  private String[] names;

  @Before
  public void setUp() {
    nameParser = new NameParser();
    names = new String[]{"Mike Jones", "John Doe"};
  }

  @Test
  public void shouldFindPersonByLastName() {
    Person person = nameParser.findPersonWithLastName(names, "Doe");
    String firstName = person.getFirstName();
    String lastName = person.getLastName();
    assertEquals("John", firstName);
  }
}

The unit tests covers the Person and NameParser code for 100% and succeeds!
It doesn't find the bug in Person.getLastName because it simply forgets to do an assertion on it. And it doesn't find the bug in the loop in NameParser because it doesn't test the case where the names list does not contain the given last name; so the loop always terminates before it has a chance to throw an IndexOutOfBoundsException.

Especially the last case is easy to overlook, so it would be nice if there existed a tool which could detect these kinds of problems.
And there is one: actually there are a couple. For this post I have chosen PIT, down at the end are links to some alternatives.

But first: what is mutation testing?

A mutation test will make a small change to your code and then run the unit test(s). Such a change is called a 'mutant'. If a change can be made and the unit tests still succeed, it will generate a warning saying that the mutant 'survived'.
The test framework will try to apply a number of predefined mutants at every point in your code where they are applicable. The higher the percentage of the mutants that get killed by your unit tests, the better the quality of your test suite.
Examples of mutants are: negating a condition in an If statement, changing a conditional boundary in a For loop, or throwing an exception at the end of a method.

Putting NameParser's testcase to the test with PIT

PIT stands for Parallel Isolated Test, which is what the project originally was meant for. But it turned out to be a much more interesting goal to do mutation testing, which required much of the same plumbing.

PIT integrates with JUnit or TestNG and can be configured with Maven, Gradle and others. Or it can be used directly as a plugin in Eclipse or IntelliJ.
I'm choosing for the last option: the IntelliJ plugin. The setup is easy: just install PITest from the plugin manager and you are ready to go. Once you're done, you'll find a new launch configuration option in the 'edit configurations' menu called PIT.

PIT launch configuration

You have to specify the classes where PIT will makes its mutations under 'Target classes'.
When we run the mutation test, PIT creates a Html report with the results for every class.
Here are the results for the NameParser class:

NameParser mutation testing results

As you can read under 'Mutations', PIT has been able to apply five code mutations to the NameParser class. Four of them resulted in a failing NameParserTest, which is exactly what we'd like to see.
But one of them did not: when the condition boundary in line 6, the loop constraint, was changed, NameParserTest still succeeded!
PIT changes loop constraints with a predefined algorithm; in this case, when the loop constraint was i <= names.length, it changed the '<=' into a '<'. Actually this accidentally corrected the bug in NameParser, and of course that didn't break the unit test.
So PIT found an omission in our unit test here, and it turned out that this omission even left a bug undetected!

Note that this last point doesn't always need to be the case. It could be that for the correct behavior of your class, there is some room for some conditions to change.
In the case of NameParser for instance, it could have been a requirement that the names list always contained an entry with the last name that was to be found. In that case the behavior for a missing last name would be unspecified and an IndexOutOfBoundsException would have been as good a result as anything else.
So PIT can only find strong indications of omissions in your unit tests, but they don't necessarily have to be.

And here are the results for the Person class:

Person mutation test results

PIT was able to do two mutations in the Person class; one in every getter method. Both times it replaced the return value with null. And as expected, the mutation in the getLastName method went undetected by our unit test.
So PIT found the second omission as well.

Conclusion

In this case, mutation testing would have helped us a lot. But there can still be cases where possible bugs can go unnoticed. In our code for example, there is no test in NameParser test that verifies the behavior when an entry in the names list does not contain both a first name and a last name. But PIT didn't find this omission.
Still it might make good sense to integrate mutation testing in your build process. PIT can be configured to break your Maven build if too many warnings are found.
And there's a lot more that can be configured as well, but for that I recommend to bring a visit to the website of PIT at www.pitest.org.

Alternatives

PIT is not the only mutation testing framework for Java, but it is the most popular and the one most actively maintained. Others are µJava and Jumble.
Although most mutation testing frameworks are written for Java, probably because it's so easy to dynamically change its bytecode, mutation testing is possible in some other languages as well: notable are grunt-mutation-testing for Javascript and Mutator, a commercial framework which is available for a couple of languages.

How To Improve the IT-Business Relationship

It’s possible to change IT from a poorly respected cost center to a high-functioning business partner.

Driving business transformation is a people, process, and technology thing.

Some people think they can change their business without IT.   The challenge is that technology is the enabler of significant business change in today’s digital landscape.  Cloud, Mobile, Social, and Big Data all bring significant capabilities to the table, and IT can hold the keys.

But the business doesn’t want to hear that.

Business Leaders Want to Hear About the WHY and WHAT of the Business

Business leaders don’t want to hear about the HOW of technology.

Business leaders want to hear about the impact on their business.   They want to hear about how predictive analytics can help them build a better pipeline, or target more relevant offers.   Business leaders want to hear about how they can create cross-sell/upsell opportunities in real-time.   And, business leaders want to hear about the business benefits and the KPI that will be impacted by choosing a particular strategy.

The reality is that the new Digital Masters of the emerging Digital Economy bring their IT with them, and in many cases, their IT even helps lead the business into the new Digital Frontier.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

How IT Can Change Its Game

While it takes work on both sides, IT can change it’s game by creating transparency around performance, roles, and value.  This includes helping employees think and talk differently about what they do.   IT can show very clearly how it delivers value for the money.  And IT can change the way IT and business leaders make investment decisions and assess the returns.

IT Needs to Speak Business

The CIO and everybody in IT, needs to speak the language of business.

Via Leading Digital:

“Poor relations between IT and business leaders can have many causes.  Sometimes it's the personality of the IT leader.  A common complaint among senior executives is that their CIO seems to speak a different language from the business.  Another is that the CIO doesn't seem to understand what's really important.  For example, a chemical company CIO we interviewed described how he communicates regularly with business executives about the innovative possibilities of digital technologies.  Yet none of his business executive peers (whom we interviewed separately) seemed to find the discussions credible.”

IT Needs to Deliver Better, Faster, and More Reliably than Outsourcing

It’s a competitive world and IT needs to continuously find ways to deliver solutions in a way that makes business sense.

Via Leading Digital:

“Sometimes the issue arises from IT's delivery capability.  According to Bud Mathaisel, who has served as CIO in several large companies, 'It starts with competence in delivering services reliably, economically, and at very high quality.  It is the absolute essential to be even invited into meaningful dialog about how you then build on that competence to do something very interesting with it.'  Unfortunately, some IT units today do not have this competence.  One business executive we interviewed said, 'IT is a mess.  It's costs are not acceptable.  It proposes things in nine or ten months, where external firms could do them in three to nine weeks.  We started offshoring our IT, and now our IT guys are trying to change.' A legacy of poor communication, byzantine decision processes, and broken commitments is no foundation on which to build a strong IT-business relationship.”

IT Needs a Good Digital Platform to Be High-Performing IT

In order to bet on IT, it needs to be high-performing.  And in order for IT to be high-performing, it needs to have a good digital platform.

Via Leading Digital:

“However, the fault doesn't always rest only with IT leaders.  In many cases, business executive share some of the blame ... high-performing IT requires a good digital platform, and good platforms require discipline.  If your approach to working with IT can be characterized by impatience, unreasonable expectations, or insisting on doing things your way, then you'll need to think about how to change your side of the relationship.”

CIOs Can Lead Digital Business Transformation

Key business transformation takes technology.  CIOs can help lead the business transformation, whether it's through shared goals with the business, creating a new governance mechanism, or creating a new shared digital unit.

Via Leading Digital:

“Regardless of the case, if your IT-business relationships are poor, it's essential to fix the problem.  A bank executive stated, 'IT has been brought closer to business during the last five years.  It is very important to success because man of the important transformations in our business are enabled by technology.'  With strong relationships, IT executives can help business executives meet their goals, and business executives listen when IT people suggest innovations.  Executives on both sides are willing to be flexible in creating new governance mechanisms or shared digital units.  At Codelco, Asian Paints, and P&G, the CIO even leads digital transformation for the company.”

CIOs Can Help Drive the Bus with the Executive Team

CIOs can help drive the bus, but it takes more than senior sponsorship.

Via Leading Digital:

“So, how can you start to improve your IT-business relationship?  Angela Ahrendts, CEO of Burberry, told her CIO he needed to help drive the bus with the executive team.  However, leadership changes or top-down mandates are only the start of the change.  Few CIOs can change the business by themselves, and not all business executives will climb on the bus with the CIO, even if the CEO demands it.”

Fix How You Communicate to Fix the IT-Business Relationship

Start by fixing how you communicate between the business and IT.

Via Leading Digital:

“Fixing the IT-business relationship can take time, as people learn how to trust each other and redefine the way they work together.  As with any struggling relationship, the best starting point is to fix the way you communicate.  Does IT really cost too much, or are costs reasonable, given what IT has to do? Is the IT unit really too bureaucratic, or do all of those procedures actually serve a useful purpose?  Are you a good partner to IT or a difficult one?  How can IT make it easier for you to get what you need, while still making sure things are done correctly?  What investments can help IT improve its technology, internal processes, cost-effectiveness, quality, or speed?”

Change IT from a Poorly Respected Cost Center to a High-Functioning Business Partner

It’s possible to change IT from a low performing cost center to a high-performing business partner.  Companies do it all the time, and MIT has the research.

Via Leading Digital:

“MIT research into IT turnarounds has identified a series of steps that can change IT from a poorly respected cost center to a high-functioning business partner.  The key change mechanism is transparency--around performance, roles, and value.  The first step is to help IT employees think, and talk, differently about what they do.  The second step proceeds to showing very clearly how well (or how poorly) IT delivers value for money--the right services at the right quality and right price, and where problems still exist.  And then the third step moves to changing the way IT and business leaders make investment decisions and assess the returns that projects deliver.  Through transparency around roles, performance, and investments, both sides can make smoother decisions and work together to identify and deliver innovation.”

If you’re part of a business that wants to change the world, start by reimagining how IT can help you achieve the art of the possible.

You Might Also Like

Building Better Business Cases for Digital Initiatives

Drive Business Transformation by Reenvisioning Your Customer Experience

Drive Business Transformation by Reenvisioning Your Operations

How Digital is Changing Physical Experiences

The Future of IT Leaders

Categories: Architecture, Programming

Swift Function Currying

Xebia Blog - Thu, 11/06/2014 - 12:55

One of the lesser known features of Swift is Function Currying. When you read the Swift Language Guide you won't find anything about curried functions. Apple only describes it in their Swift Language Reference. And that's a pity, since it's a very powerful and useful feature that deserves more attention. This post will cover the basics and some scenarios in which it might be useful to use curried functions.

I assume you're already somewhat familiar with function currying since it exists in many other languages. If not, there are many articles on the Internet that explain what it is and how it works. In short: you have a function that receives one or more parameters. You then apply one or more known parameters to that function without already executing it. After that you get a function reference to a new function that will call the original function with the applied parameters.

One situation in which I find it useful to use curried functions is with completion handlers. Imagine you have a function that makes a http request and looks something like this:

func doGET(url: String, completionHandler: ([String]?, NSError?) -> ()) {
    // do a GET HTTP request and call the completion handler when receiving the response
}

This is a pretty common pattern that you see with most networking libraries as well. We can call it with some url and do a bunch of things in the completion handler:

doGET("http://someurl.com/items?all=true", completionHandler: { results, error in
    self.results = results
    self.resultLabel.text = "Got all items"
    self.tableView.reloadData()
})

The completion handler can become a lot more complex and you might want to reuse it in different places. Therefore you can extract that logic into a separate function. Luckily with Swift, functions are just closures so we can immediately pass a completion handler function to the doGET function:

func completionHandler(results: [String]?, error: NSError?) {
    self.results = results
    self.resultLabel.text = "Got all items"
    self.tableView.reloadData()
}

func getAll() {
    doGET("http://someurl.com/items?all=true", completionHandler)
}

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, completionHandler)
}

This works well, as long as the completion handler should always do exactly the same thing. But in reality, that's usually not the case. In the example above, the resultLabel will always display "Got all items". Lets change that into "Got searched items" for the search request:

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, {results, error in
        self.completionHandler(results, error: error)
        self.resultLabel.text = "Got searched items"
    })
}

This will work, but it doesn't look very nice. What we actually want is to have this dynamic behaviour in the completionHandler function. We can change the completionHandler in such a way that it accepts the text for the resultLabel as a parameter and then returns the actual completion handler as a closure.

func completionHandler(text: String) -> ([String]?, NSError?) -> () {
    return {results, error in
        self.results = results
        self.resultLabel.text = text
        self.tableView.reloadData()
    }
}

func getAll() {
    doGET("http://someurl.com/items?all=true", completionHandler("Got all items"))
}

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, completionHandler("Got searched items"))
}

And as it turns out, this is exactly what we can also do using currying. We just need to add the parameters of the actual completion handler as a second parameters group to our function:

func completionHandler(text: String)(results: [String]?, error: NSError?) {
    self.results = results
    self.resultLabel.text = text
    self.tableView.reloadData()
}

Calling this with the first text parameter will not yet execute the function. Instead it returns a new function with the [String]?, NSError? as parameters. Once that function is called the completionHandler function is finally executed.

You can create as many levels of this currying as you want. And you can also leave the last parameter group empty just to get a reference to the fully applied function. Let's look at another example. We have a simple function that sets the text of the resultLabel:

func setResultLabelText(text: String) {
    resultLabel.text = text
}

And for some reason, we need to call this method asynchronously. We can do that using the Grand Central Dispatch functions:

dispatch_async(dispatch_get_main_queue(), {
    self.setResultLabelText("Some text")
})

Since the dispatch_async function only accepts a closure without any parameters, we need to create an inner closure here. If the setResultLabelText was a curried function, we could fully apply it with the parameter and get a reference to a function without parameters:

func setResultLabelText(text: String)() { // now curried
    resultLabel.text = text
}

dispatch_async(dispatch_get_main_queue(), setResultLabelText("Some text"))

But you might not always have control over such functions, for example when you're using third party libraries. In that case you cannot change the original function into a curried function. Or you might not want to change it since you already using it at many other places and you don't want to break anything. In that case we can achieve something similar by creating a function that creates the curried function for us:

// defined in global scope
func curry<T>(f: (T) -> (), arg: T)() {
    f(arg)
}

We can now use it as following:

func setResultLabelText(text: String) {
    resultLabel.text = text
}

dispatch_async(dispatch_get_main_queue(), curry(setResultLabelText, "Some text"))

Probably in this example it might be just as easy to go with the inner closure, but being able to pass around partial applied functions is very powerful and used in many programming languages already.

Unfortunately the last example also show a great drawback about the way currying is implemented in Swift: you cannot simply curry normal functions. It would be great to be able to curry any function that takes multiple parameters instead of having to explicitly create curried functions. Another drawback is that you can only curry in the defined order of parameters. That doesn't allow you to do reverse currying (e.g. apply only the last parameter) or even apply just any parameter you want, regardless of its position. Hopefully the Swift language will evolve in this and get more powerful currying features.

A Few Really Good Tech Podcasts for Your Listening Pleasure

Podcasts are back. Since I listen to a bunch of podcasts every week I was quite surprised to learn about their resurrection. But facial hair is back too, so I will revel in my short lived trendiness.

Why are podcasts back? One suggested reason is a little sad. People commute more than 25 hours a day and since cars are now faithful little bluetooth slaves, broadcasting podcasts over luxury car speakers is as easy as a smart technology and a fast cell network can make it. Podcasts are now a viable P2P radio replacement.

That’s the demand, means, motive, and opportunity side of things. What about supply?

Historically podcasts start in fire, the passion quickly burning to ash as podcasters learn making podcasts is a lot of hard work...and poorly unremunerated work at that. So podcasts have a high mortality rate.

What’s changed? Money. Strangely, people will exchange effort for money, so if podcasts can make money they will have the fuel they need to burn bright through the night.

Money from Subscriptions is New
Categories: Architecture