Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

SharePoint VPS solution

Microsoft SharePoint is an ideal solution for companies who have multiple offices and staff members who are on the move. Using SharePoint, documents and other materials can be easily shared with both colleagues and managers. Other features include advanced document management, which allows users to virtually check out a document, modify it or just read it, then check in the document again. This allows managers/company owners to see exactly when their staff members are working and just what they are doing. When combined with a highly customizable workflow management system and group calendars, SharePoint can improve the way in which your company functions and operates.

However, many organizations are observed to be failing with SharePoint implementation. So with this article, we are trying to make it simpler for organizations in-house IT administrators to help implement SharePoint over a virtual server environment.

Here we are going to see following key points:

Categories: Architecture

Ask HS: Design and Implementation of scalable services?

We have written agents deployed/distributed across the network. Agents sends data every 15 Secs may be even 5 secs. Working on a service/system to which all agent can post data/tuples with marginal payload. Upto 5% drop rate is acceptable. Ultimately the data will be segregated and stored into DBMS System (currently we are using MSQL).

Question(s) I am looking for answer

1. Client/Server Communication: Agent(s) can post data. Status of sending data is not that important. But there is a remote where Agent(s) to be notified if the server side system generates an event based on the data sent.

- Lot of advices from internet suggests using Message Bus (ActiveMQ) for async communication. Multicast and UDP are the alternatives.

2. Persistence: After some evaluation data to be stored in DBMS System.

- End of processing data is an aggregated record for which MySql looks scalable. But on the volume of data is exponential. Considering HBase as an option.

Looking if there are any alternatives for above two scenarios and get expert advice.

Categories: Architecture

Geolocation detection with haproxy

Agile Testing - Grig Gheorghiu - Tue, 01/14/2014 - 00:33
A useful feature for a web application is the ability to detect the user's country of origin based on their source IP address. This used not to be possible in haproxy unless you applied Cyril Bonté's geolocation patches (see the end of this blog post for how exactly to do that if you don't want to live on the bleeding edge of haproxy). However, the latest development version of haproxy (which is 1.5-dev21 at this time) contains geolocation detection functionality.

Here's how to use the geolocation detection feature of haproxy:

1) Generate text file which maps IP address ranges to ISO country codes

This is done using Cyril's haproxy-geoip utility, which is available in his geolocation patches. Here's how to locate and run this utility:
  • clone patch git repo: git clone https://github.com/cbonte/haproxy-patches.git
  • the haproxy-geoip script is now available in haproxy-patches/geolocation/tools
    • for the script to run, you need to have the funzip utility available on your system (it's part of the unzip package in Ubuntu)
    • you also need the iprange binary, which you can 'make' from its source file available in the haproxy-1.5-dev21/contrib/iprange directory; once you generate the binary, copy it somewhere in your PATH so that haproxy-geoip can locate it
  • run haproxy-geoip, which prints its output (IP ranges associated to ISO country codes) to stdout, and capture stdout to a file: haproxy-geoip > geolocation.txt
  • copy geolocation.txt to /etc/haproxy
2) Set custom HTTP header based on geolocation
For this, haproxy provides the map_ip function, which locates the source IP (the predefined 'src' variable in the line below) in the IP range in geolocation.txt and returns the ISO country code. We assign this country code to the custom X-Country HTTP header:
http-request set-header X-Country %[src, map_ip(/etc/haproxy/geolocation.txt)]
If you didn't want to map the source IP to a country code, but instead wanted to inspect the value of an HTTP header such as X-Forwarded-For, you could do this:
http-request set-header X-Country %[req.hdr_ip(X-Forwarded-For,-1), map_ip(/etc/haproxy/geolocation.txt)]
3) Use geolocation in ACLs
Let's assume that if the country detected via geolocation is not US, then you want to send the user to a different backend. You can do that with an ACL. Note that we compare the HTTP header X-Country which we already set above to the string 'US' using the '-m str' string matching functionality of haproxy, and we also specify that we want a case insensitive comparison with '-i US':
acl acl_geoloc_us req.hdr(X-Country) -m str -i USuse_backend www-backend-non-us if !acl_geoloc_us
If you didn't want to set the custom HTTP header, you could use the map_ip function directly in the definition of the ACL, like this:
acl acl_geoloc_us %[src, map_ip(/etc/haproxy/geolocation.txt)] -m str -i USuse_backend www-backend-non-us if !acl_geoloc_us
Speaking of ACLs, here's an example of defining ACLs based on the existence of a cookie and based on the value of the cookie then choosing a backend based on those ACLs:
acl acl_cookie_country req.cook_cnt(country_code) eq 1acl acl_cookie_country_us req.cook(country_code) -m str -i USuse_backend www-backend-non-us if acl_cookie_country !acl_cookie_country_us
And now for something completely different...which is what I mentioned in the beginning of this post: 
How to use the haproxy geolocation patches with the current stable (1.4) version of haproxy
a) Patch haproxy source code with gelocation patches, compile and install haproxy:
  • clone patch git repo: git clone https://github.com/cbonte/haproxy-patches.git
  • change to haproxy-1.4.24 directory
  • copy haproxy-1.4-geolocation.patc to the root of haproxy-1.4.24 
  • apply the patch: patch -p1 < haproxy-1.4-geolocation.patch
  • make clean
  • make TARGET=linux26
  • make install
b) Generate text file which maps IP address ranges to ISO country codes
  • install funzip: apt-get install unzip
  • create iprange binary
    • cd haproxy-1.4.24/contrib/iprange
    • make
    • the iprange binary will be created in the same folder. copy that to /usr/local/sbin
  • haproxy-geoip is located here: haproxy-patches/geolocation/tools
  • haproxy-geoip > geolocation.txt
  • copy geolocation.txt to /etc/haproxy 
c) Obtain country code based on source IP and use it in ACL
This is done via the special 'geolocate' statement and the 'geoloc' variable added to the haproxy configuration syntax by the geolocation patch:

geolocate src /etc/haproxy/geolocation.txt
acl acl-au geoloc eq AU
use_backend www-backend-au if acl-au

If instead of the source IP you want to map the value of the X-Forwarded-For header to a country, use:
geolocate hdr_ip(X-Forwarded-For,-1) /etc/haproxy/geolocation.txt

If you wanted to redirect to another location instead of using an ACL, use:
redirect location http://some.location.example.com:4567 if { geoloc AU }

That's it for now. I want to thank Cyril Bonté, the author of the geolocation patches, and Willy Tarreau, the author of haproxy, for their invaluable help and their amazingly fast responses to my emails. It's a pleasure to deal with such open source developers passionate about the software they produce.  Also thanks to my colleagues Zmer Andranigian for working on getting version 1.4 of haproxy to work with geolocation, and Jeff Roberts for working on getting 1.5-dev21 to work.
One last thing: haproxy-1.5-dev21 has been very stable in production for us, but of course test it thoroughly before deploying it in your environment.

NYTimes Architecture: No Head, No Master, No Single Point of Failure

Michael Laing, a Systems Architect at NYTimes, gave this great decription of their use of RabbitMQ and their overall architecture on the RabbitMQ mailing list. The closing sentiment marks this as definitely an architecture to learn from:

Although it may seem complex, Fabrik has simple components and is mostly principles and plumbing. The key point to grasp is that there is no head, no master, no single point of failure. As I write this I can see components failing (not RabbitMQ), and we are fixing them so they are more reliable. But the system doesn't fail, users can connect, and messages are delivered, regardless - all within design parameters.

Since it's short, to the point, and I couldn't say it better, I'll just reproduce several original sources here:

Categories: Architecture

Stuff The Internet Says On Scalability For January 10th, 2014

Hey, it's HighScalability time:


Run pneumatic tubes alongside optical fiber cables and we unite the digital and material worlds.
  • 1 billion: searches on DuckDuckGo in 2013
  • Quotable Quotes: 
    • pg: We [Hacker News] currently get just over 200k uniques and just under 2m page views on weekdays (less on weekends). 
      • rtm: New server: one Xeon E5-2690 chip, 2.9 GHz, 8 cores total, 32 GB RAM.
    • Kyle Vanhemert: The graph shows the site’s [Reddit] beginnings in the primordial muck of porn and programming.
    • Drake Baer: But it's not about knowing the most people possible. Instead of being about size, a successful network is about shape.
    • @computionist: Basically when you cache early to scale you've admitted you have no idea what you're doing.
    • Norvig's Law: Any technology that surpasses 50% penetration will never double again.  
    • mbell: Keep in mind that a single modern physical server that is decently configured (12-16 cores, 128GB of ram) is the processing equivalent of about 16 EC2 m1.large instances.
    • @dakami: Learned databases because grep|wc -l got slow. Now I find out that's pretty much map/reduce.
    • Martin Thompson: I think "Futures" are a poor substitute for being pure event driven and using state machines. Futures make failure cases very ugly very quickly and they are really just methadone for trying to wean programmers off synchronous designs :-) 
    • @wattersjames: Can your PaaS automate Google Compute Engine? If it can, you will find that it can create VMs in only 35 seconds, at most any scale.
    • Peter M. Hoffmann: Considering the inherent drive of matter to form ever more complex structures, life seems inevitable.

  • The marker for a new generation, like kids who will never know a card catalogue, Kodak film, pay phones, phone books, VHS tapes, typewriters, or floppy disks: Co-Founder Of Snapchat Admits He's Never Owned A Physical Server

  • Want the secret of becoming a hugely popular site? Make it fast and it will become popular. It's science. Are Popular Websites Faster?: No matter what distribution of websites you look at – the more popular websites trend faster. Even the slowest popular website is much faster than those that are less popular. On the web, the top 500 sites are nearly 1s faster (by the median), and on mobile it is closer to 1.5s faster. 

  • In 1956 we may not have had BigData, but BigStorage was definitely in. Amazing picture of IBM's 5 mega-byte drive weighing in at more than 2,000 pounds.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Extreme Goal Setting for 2014

When’s the last time you went for your personal Epic Win?   If it’s been a while, no worries.  Let’s go big this year.

I’ll give you the tools.

I realize time and again, that Bruce Lee was so right when he said, “To hell with circumstances; I create opportunities.”  Similarly, William B. Sprague told us, “Do not wait to strike till the iron is hot; but make it hot by striking.” 

And, Peter Drucker said, “The best way to predict the future is to create it.”   Similarly, Alan Kay said, "The best way to predict the future is to invent it."

Well then?  Game on!

By the way, if you’re not feeling very inspired, check out either my 37 Inspirational Quotes That Will Change Your Life, Motivational Quotes, or my Inspirational Quotes.  They are intense, and I bet you can find your favorite three.

As I’ve been diving deep into goal setting and goal planning, I’ve put together a set of deep dive posts that will give you a very in-depth look at how to set and achieve any goal you want.   Here is my roundup so far:

Brian Tracy on 12 Steps to Set and Achieve Any Goal

Brian Tracy on the Best Times for Writing and Reviewing Your Goals

Commit to Your Best Year Ever

Goal Setting vs. Goal Planning

How To Find Your Major Definite Purpose

How To Use 3 Wins for the Year to Have Your Best Year Ever

The Power of Annual Reviews for Achieving Your Goals and Realizing Your Potential

What Do You Want to Spend More Time Doing?

Zig Ziglar on Setting Goals

Hopefully, my posts on goal setting and goal planning save you many hours (if not days, weeks, etc.) of time, effort, and frustration on trying to figure out how to really set and achieve your goals.   If you only read one post, at least read Goal Setting vs. Goal Planning because this will put you well ahead of the majority of people who regularly don’t achieve their goals.

In terms of actions, if there is one thing to decide, make it Commit to Your Best Year Ever.

Enjoy and best wishes for your greatest year ever and a powerful 2014.

Categories: Architecture, Programming

Under Snowden's Light Software Architecture Choices Become Murky

Adrian Cockcroft on the future of Cloud, Open Source, SaaS and the End of Enterprise Computing:

Most big enterprise companies are actively working on their AWS rollout now. Most of them are also trying to get an in-house cloud to work, with varying amounts of success, but even the best private clouds are still years behind the feature set of public clouds, which is has a big impact on the agility and speed of product development

While the Snowden revelations have tattered the thin veil of trust secreting Big Brother from We the People, they may also be driving a fascinating new tension in architecture choices between Cloud Native (scale-out, IaaS), Amazon Native (rich service dependencies), and Enterprise Native (raw hardware, scale-up).

This tension became evident in a recent HipChat interview where HipChat, makers of an AWS based SaaS chat product, were busy creating an on-premises version of their product that could operate behind the firewall in enterprise datacenters. This is consistent with other products from Atlassian in that they do offer hosted services as well as installable services, but it is also an indication of customer concerns over privacy and security.

The result is a potential shattering of backend architectures into many fragments like we’ve seen on the front-end. On the front-end you can develop for iOS, Android, HTML5, Windows, OSX, and so on. Any choice you make is like declaring for a cold war power in a winner take all battle for your development resources. Unifying this mess has been the refuge of cloud based services over HTTP. Now that safe place is threatened.

To see why...

Categories: Architecture

Sponsored Post: Netflix, Logentries, Host Color, Booking, Apple, ScaleOut, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • Sr Software Engineer. The iOS Systems Team is looking for a Software Engineer to work on operations, tools development and support of worldwide iOS Device sales and activations. Please apply here
    • Sr. Security Software Developer. We are looking for an excellent programmer who's done extensive security programming. This individual will participate in various security projects from the start to the end. In addition to security concepts, it's important to have intricate knowledge of different flavors of Unix operating systems to develop code that's compact and optimal. Familiarity with key exchange protocols, authentication protocols and crypto analysis is a plus. Please apply here.
    • Senior Server Side Engineer. The Emerging Technology team is looking for a highly motivated, detail-oriented, energetic individual with experience in a variety of big data technologies.  You will be part of a fast growing, cohesive team with many exciting responsibilities related to Big Data, including: Develop scalable, robust systems that will gather, process, store large amount of data Define/develop Big Data technologies for Apple internal and customer facing applications. Please apply here.
    • Senior Server Side Engineer. The Emerging Technology team is looking for a highly motivated, detail-oriented, energetic individual with experience in a variety of big data technologies.  You will be part of a fast growing, cohesive team with many exciting responsibilities related to Big Data, including: Develop scalable, robust systems that will gather, process, store large amount of data Define/develop Big Data technologies for Apple internal and customer facing applications. Please apply here.
    • Senior Engineer: Emerging Technology. Apple’s Emerging Technology group is looking for a senior engineer passionate about exploring emerging technologies to create paradigm shifting cloud based solutions. Please apply here

  • The Netflix Cloud Performance Team is hiring. Help tackle the more complex scalability challenges emerging on the cloud today, wrangling tens of thousands of instances, handling billions of requests a day. We are searching for a Senior Performance Architect and a Senior Cloud Performance Tools Engineer

  • We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Apply: http://www.booking.com/jobs.en-us.html

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Your amazing event here.
Cool Products and Services
  • Log management made easy with Logentries Billions of log events analyzed every day to unlock insights from the log data the matters to you. Simply powerful search, tagging, alerts, live tail and more for all of your log data. Automated AWS log collection and analytics, including CloudWatch events. 

  • HostColorCloud Servers based on OpenNebula Cloud automation IaaS. Cloud Start features: 256 MB RAM; 1000 GB Premium Bandwidth; 10 GB fast, RAID 10 protected Storage; 1 CPU Core;  /30 IPv4 (4 IPs) and /48 IPv6 subnet and costs only $9.95/mo. Client could choose an OS (CentOS is default). 

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting

  • Rapidly Develop Hadoop MapReduce Code. With ScaleOut hServer™ you can use a subset of your Hadoop data and run your MapReduce code in seconds for fast code development and you don’t need to load and manage the Hadoop software  stack, it's a self-contained Hadoop MapReduce execution environment. To learn more check out www.scaleoutsoftware.com/prototypehadoop/

  • MongoDB Backup Free Usage Tier Announced. We're pleased to introduce the free usage tier to MongoDB Management Service (MMS). MMS Backup provides point-in-time recovery for replica sets and consistent snapshots for sharded systems with minimal performance impact. Start backing up today at mms.mongodb.com.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • Aerospike Capacity Planning Kit. Download the Capacity Planning Kit to determine your database storage capacity and node requirements. The kit includes a step-by-step Capacity Planning Guide and a Planning worksheet. Free download.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

How HipChat Stores and Indexes Billions of Messages Using ElasticSearch and Redis

This article is from an interview with Zuhaib Siddique, a production engineer at HipChat, makers of group chat and IM for teams.

HipChat started in an unusual space, one you might not think would have much promise, enterprise group messaging, but as we are learning there is gold in them there enterprise hills. Which is why Atlassian, makers of well thought of tools like JIRA and Confluence, acquired HipChat in 2012.

And in a tale not often heard, the resources and connections of a larger parent have actually helped HipChat enter an exponential growth cycle. Having reached the 1.2 billion message storage mark they are now doubling the number of messages sent, stored, and indexed every few months.

That kind of growth puts a lot of pressure on a once adequate infrastructure. HipChat exhibited a common scaling pattern. Start simple, experience traffic spikes, and then think what do we do now? Using bigger computers is usually the first and best answer. And they did that. That gave them some breathing room to figure out what to do next. On AWS, after a certain inflection point, you start going Cloud Native, that is, scaling horizontally. And that’s what they did.

But there's a twist to the story. Security concerns have driven the development of an on-premises version of HipChat in addition to its cloud/SaaS version. We'll talk more about this interesting development in a post later this week.

While HipChat isn’t Google scale, there is good stuff to learn from HipChat about how they index and search billions of messages in a timely manner, which is the key difference between something like IRC and HipChat. Indexing and storing messages under load while not losing messages is the challenge.

This is the road that HipChat took, what can you learn? Let’s see…

Categories: Architecture

Disaster Recovery and Planning

Coding the Architecture - Simon Brown - Mon, 01/06/2014 - 10:51

Maybe software developers are naturally optimistic but in my experience they rarely consider system failure or disaster scenarios when designing software. Failures are varied and range from the likely (local disk failure) to the rare (tsunami) and from low impact to fatal (where fatal may be the death of people or bankruptcy of a business).

Failure planning broadly fits into the following areas:

  1. Avoiding failure
  2. Failing safely
  3. Failure recovery
  4. Disaster Recovery

Avoiding failure is what a software architect is most likely to think about at design time. This may involve a number of High Availability (HA) techniques and tools including; redundant servers, distributed databases or real time replication of data and state. This usually involves removing any single point of failure but you should be careful to not just consider the software and hardware that it immediately runs on - you should also remove any single dependency on infrastructure such as power (battery backup, generators or multiple power supplies) or telecoms (multiple wired connections, satellite or radio backups etc).

Failing safely is a complex topic that I touched on recently and may not apply to your problem domain (although you should always consider if it does).

Failure recovery usually goes hand-in-hand with High Availability and ensures that when single components are lost they can be re-created/started to join the system. There is no point in having redundancy if components cannot be recovered as you will eventually lose enough components for the system to fail!

Disaster Recovery Scenarios and Planning

However, the main topic I want to discuss here is disaster recovery. This is the process that a system and its operators have to execute in order to recreate a fully operational system after a disaster scenario. This differs from a failure in that the entire system (potentially all the components but at least enough to render it inoperable) stops working. As I stated earlier, many software architects don't consider these scenarios but they can include:

  1. Power surge leading to a datacenter failure
  2. Flooding leading to the damage of machines located in a building's basement
  3. Communications failure leading to loss of connectivity
  4. Disgruntled staff member deliberately corrupting a system
  5. Staff member accidentally shutting down the system
  6. Terrorist action leading to the loss of a building with no re-entry possible

These are usually classified into either natural or man-made disasters. Importantly these are likely to cause outright system failure and require some manual intervention - the system will not automatically recover. Therefore an organisation should have a Disaster Recovery (DR) Plan for the operational staff to follow when this occurs.

A disaster recovery plan should consider a range of scenarios and give very clear and precise instructions on what to do for each of them. In the event of a disaster scenario the staff members are likely to be stressed and not thinking as clearly as they would otherwise. Keep any steps required simple and don't worry about stating the obvious or being patronising - remember that the staff executing the plan may not be the usual maintainers of the system.

Please remember that 'cloud hosted' systems still require disaster recovery plans! Your provider could have issues and you are still affected by scenarios that involve corrupt data and disgruntled staff. Can you roll-back your data store to a known point in the past before corruption?

Strategies

The aims and actions of any recovery will depend on the scenario that occurs. Therefore the scenarios listed should each refer to a strategy which contains some actions.

Before any strategy is executed you need to be able to detect the event has occurred. This may sound obvious but a common mistake is to have insufficient monitoring in place to actually detect it. Once detected there needs to be comprehensive notification in place so that all systems and people are aware that actions are now required.

For each strategy there has to be an aim for the actions. For example, do you wish to try to bring up a complete system with all data (no data loss) or do you just need something up and running? Perhaps missing data can be imported at a later time or maybe some permanent data-loss is tolerated? Does the recovered system have to provide full functionality or is an emergency subset sufficient?

This is hugely dependent on the problem domain and scenario but the key metrics are recovery point objectives (RPO) and recovery time objectives (RTO) along with level of service. Your RPO and RTO are key non-functional (quality) requirements and should be listed in your software architecture document. These metrics should influence your replication, backup strategies and necessary actions.

Business Continuity

The disaster recovery plans for the IT systems are actually a subset of the boarder 'business continuity' plans (BCP) that an organisation should have. This covers all the aspects of keeping an organisation running in the event of a disaster. BCP plans also includes manual processes, staff coverage, building access etc. You need to make sure that the IT disaster recovery plan fits into the business continuity plan and you state the dependencies between them.

There are a range of official standards covering Business Continuity Planning such as ISO22301, ISO22313 and ISO27031. Depending on your business and location you might have a legal obligation to comply with these or other local standards. I would strongly recommend that you investigate whether your organisation needs to be compliant - if you fail to do so then there could be legal consequences.

This is a complex topic which I have only touched upon - if it raises concerns then you may have a lot of work to do! If you don't know where to start then I'd suggest getting your team together and running a risk storming workshop.

Categories: Architecture

Speedy FitNesse roundtrips during development

Xebia Blog - Mon, 01/06/2014 - 10:11

FitNesse is an acceptance testing framework. It allows business users, testers and developers to collaborate on executable specifications (for example in BDD style and/or implementing Specification by Example), and allows for testing both the back-end and the front-end. Aside from partly automating acceptance testing and as a tool to help build a common understanding between developers and business users, a selection of the tests from a FitNesse test suite often doubles as a regression test suite.

In contrast to unit tests, FitNesse tests should usually be focused but still test a feature in an  'end-to-end' way. It is not uncommon for a FitNesse test to for example start mocked versions of external systems, start e.g. a Spring context and connect to a real test database rather than an in-memory one.

Running FitNesse during development

The downside of end-to-end testing is that setting up all this context makes running a single test locally relatively slow. This is part of the reason you should keep in mind the testing pyramid while writing tests, and write tests at the lowest possible level (though not lower).

Still, when used correctly, FitNesse tests can provide enormous value. Luckily, versions of FitNesse since 06-01-2014 make it relatively easy to significantly reduce this round-trip time.

A bit of background

Most modern FitNesse tests are written using the SLIM test system. When executing a test, a separate 'service' process is spun up to actually execute the code under test ('fixtures' and code-under-test). This has a couple of advantages: the classpath of the service process can be kept relatively clean - in fact, you can even use a service process written in an entirely different language, such as .Net or Ruby, as long as implements the SLIM protocol.

In the common case of using the Java SLIM service, however, this means spinning up a JVM, loading your classes into the classloader, and possibly additional tasks such as initializing part of your backend and mocking services. This can take a while, and slows down your development roundtrip, making FitNesse less pleasant to work with.

How to speed up your FitNesse round-trip times

One way to tremendously speed up test round-trip times is to, instead of initializing the complete context every time you run a test, start the SlimService manually and keep it running. When done from your IDE this allows you to take advantage of selective reloading of updated classes and easily setting breakpoints.

To locally use FitNesse in this way, put the FitNesse non-standalone jar on your classpath, and start the main method of fitnesse.slim.SlimService with parameters like '-d 8090': '-d' is to prevent the SlimService from shutting down after the first test disconnects, '8090' specifies the port number on which to listen.

Example: java -cp *yourclasspath* fitnesse.slim.SlimService -d 8090

Now, when starting the FitNesse web UI, use the 'slim.port' to specify the port to connect to and set 'slim.pool.size' to '1', and FitNesse will connect to the already-running SLIM service instead of spinning up a new process each time.

Example: java -Dslim.port=8090 -Dslim.pool.size=1 -jar fitnesse-standalone.jar -p 8000 -e 0 

We've seen improvements in the time it takes to re-run one of our tests from a typical ~15 seconds to about 2-3 seconds. Not only a productivity improvement, but more importantly this makes it much more pleasant to use FitNesse tests where they make sense.

The Most Effective CIOs in 2014

I was reading The Fruits of Innovation: Top 10 IT Trends in 2014, by Mark Harris.

Harris had this to say about the evolving role of the CIO:

“In the end, these leaders are now tasked to accurately manage, predict, execute and justify. Hence, the CIO’s role will evolve. Previously, CIOs were mostly technologists that were measured almost exclusively by availability and uptime. The CIO’s job was all about crafting a level of IT services that the company could count on, and the budgeting process needed to do so was a mostly a formality.”

Harris had this to say about the best qualities in a CIO:

“The most effective CIOs in 2014 will be business managers that understand the wealth of technology options now available, the costs associated with each as well as the business value of each of the various services they are chartered to deliver. He or she will map out a plan that delivers just the right amount service within their agreed business plan. Email, for instance, may have an entirely different value to a company than their online store, so the means to deliver these diverse services will need to be different. It is the 2014 CIO’s job to empower their organizations to deliver just the right services at just the right cost.”

That matches what I’ve been seeing.

CIOs need business acumen and the ability to connect IT to business impact.

Another way to think of it is, the CIO needs to help accelerate and realize business value from IT investments.

Value Realization is hot.

You Might Also Like

Stephen Kell on Value Realization

Blessing Sibanyoni on Value Realization

Paul Lidbetter on Value Realization

Martin Sykes on Value Realization

Mark Bestauros on Value Realization

Graham Doig on Value Realization

Categories: Architecture, Programming

How To Have Your Best Year Ever

There’s a little trick I learned about how to have your best year ever:

Commit to Your Best Year Ever

That’s it.

And, it actually works.

When you decide to have your best year ever, and you make it a mission, you find a way to make it happen.

You embrace the challenges and the changes that come you way. 

You make better choices throughout the year, in a way that moves you towards your best year ever.

A while back, our team, did exactly that.   We decided we wanted to make the coming year our best year ever.  We wanted a year we could look back on, and know that we gave it our best shot.  We wanted a year that mattered.  And we were willing to work for it.

And, it worked like a champ.

In fast, most of us go our best reviews at Microsoft.  Ever.

It’s not like it’s magic.  It works because it sets the stage.  It sets the stage for great expectations.  And, when you expect more, from yourself, or from the world, you start to look for and leverage more opportunities to make that come true.

It also helps you role with the punches.   You find ways to turnaround negative situations into more positive ones.  You find ways to take setbacks as learning opportunities to experience your greatest growth.  You look for ways to turn ordinary events into extraordinary adventures.

And when you get knocked down.  You try again.  Because you’re on a mission.

When you make it a mission to have your best year ever, you stretch yourself a little more.  You try new things.  You take old things to new heights.

But there’s a very important distinction here.   You have to own the decision. 

It has to be your choice.   YOU have to choose it so that you internalize it, and actually believe it, so that you actually act on it.

Otherwise, it’s just a neat idea, but you won’t live it.

And if you don’t live it, it won’t happen.

But, as soon as you decide that no matter what, this will be YOUR best year ever, you unleash your most resourceful self.

If you’ve forgotten what it’s like to go for the epic win, then watch this TED talk and my notes:

Go For the Epic Win

Best wishes for your best year.

Ever.

Categories: Architecture, Programming

Event Sourcing. Draw it

Think Before Coding - Jérémie Chassaing - Sat, 01/04/2014 - 22:58

Here is a drawing to show the interaction between the Decide and Apply functions:

Categories: Architecture, Requirements

Stuff The Internet Says On Scalability For January 3rd, 2014

Hey, it's HighScalability time, can you handle the truth?


Should software architectures include parasites? They increase diversity and complexity in the food web.
  • 10 Million: classic hockey stick growth pattern for GitHub repositories
  • Quotable Quotes:
    • Seymour Cray: A supercomputer is a device for turning compute-bound problems into IO-bound problems.
    • Robert Sapolsky: And why is self-organization so beautiful to my atheistic self? Because if complex, adaptive systems don’t require a blue print, they don’t require a blue print maker. If they don’t require lightning bolts, they don’t require Someone hurtling lightning bolts.
    • @swardley: Asked for a history of PaaS? From memory, public launch - Zimki ('06), BungeeLabs ('06), Heroku ('07), GAE ('08), CloudFoundry ('11) ...
    • @neil_conway: If you're designing scalable systems, you should understand backpressure and build mechanisms to support it.
    • Scott Aaronson...the brain is not a quantum computer. A quantum computer is good at factoring integers, discrete logarithms, simulating quantum physics, modest speedups for some combinatorial algorithms, none of these have obvious survival value. The things we are good at are not the same thing quantum computers are good at.
    • @rbranson: Scaling down is way cooler than scaling up.
    • @rbranson: The i2 EC2 instances are a huge deal. Instagram could have put off sharding for 6 more months, would have had 3x the staff to do it.
    • @mraleph: often devs still approach performance of JS code as if they are riding a horse cart but the horse had long been replaced with fusion reactor
  • Now we know the cost of bandwidth: Netflix’s new plan: save a buck with SD-only streaming
  • Massively interesting Stack Overflow thread on Why is processing a sorted array faster than an unsorted array? Compilers may grant a hidden boon or turn traitor with a deep deceit. How do you tell? It's about branch prediction.
  • Can your database scale to 1000 cores? Nope. Concurrency Control in the Many-core Era: Scalability and Limitations: We conclude that rather than pursuing incremental solutions, many-core chips may require a completely redesigned DBMS architecture that is built from ground up and is tightly coupled with the hardware.
  • Not all SSDs are created equal. Power-Loss-Protected SSDs Tested: Only Intel S3500 PassesWith a follow up. If data on your SSD can't survive a power outage it ain't worth a damn. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

xkcd: How Standards Proliferate:

The great thing about standards is there are so many to choose from. What is it about human nature that makes this so recognizably true?

Categories: Architecture

Paper: Nanocubes: Nanocubes for Real-Time Exploration of Spatiotemporal Datasets

How do you turn Big Data into fast, useful, and interesting visualizations? Using R and technology called Nanocubes. The visualizations are stunning and amazingly reactive. Almost as interesting as the technologies behind them.

David Smith wrote a great article explaining the technology and showing a demo by Simon Urbanek of a visualization that uses 32Tb of Twitter data. It runs smoothly and interactively on a single machine with 16Gb of RAM.  For more information and demos go to nanocubes.net.

David Smith sums it up nicely:

Despite the massive number of data points and the beauty and complexity of the real-time data visualization, it runs impressively quickly. The underlying data structure is based on Nanocubes, a fast datastructure for in-memory data cubes. The basic idea is that nanocubes aggregate data hierarchically, so that as you zoom in and out of the interactive application, one pixel on the screen is mapped to just one data point, aggregated from the many that sit "behind" that pixel. Learn more about nanocubes, and try out the application yourself (modern browser required) at the link below.

Abstract from Nanocubes for Real-Time Exploration of Spatiotemporal Datasets:

Consider real-time exploration of large multidimensional spatiotemporal datasets with billions of entries, each defined by a location, a time, and other attributes. Are certain attributes correlated spatially or temporally? Are there trends or outliers in the data? Answering these questions requires aggregation over arbitrary regions of the domain and attributes of the data. Many relational databases implement the well-known data cube aggregation operation, which in a sense precomputes every possible aggregate query over the database. Data cubes are sometimes assumed to take a prohibitively large amount of space, and to consequently require disk storage. In contrast, we show how to construct a data cube that fits in a modern laptop’s main memory, even for billions of entries; we call this data structure a nanocube. We present algorithms to compute and query a nanocube, and show how it can be used to generate well-known visual encodings such as heatmaps, histograms, and parallel coordinate plots. When compared to exact visualizations created by scanning an entire dataset, nanocube plots have bounded screen error across a variety of scales, thanks to a hierarchical structure in space and time. We demonstrate the effectiveness of our technique on a variety of real-world datasets, and present memory, timing, and network bandwidth measurements. We find that the timings for the queries in our examples are dominated by network and user-interaction latencies.
Categories: Architecture

Declaration of Interdependence

You might already know the Agile Manifesto:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

But do you know the Declaration of Interdependence:

  • We increase return on investment by making continuous flow of value our focus.
  • We deliver reliable results by engaging customers in frequent interactions and shared ownership.
  • We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
  • We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
  • We boost performance through group accountability for results and shared responsibility for team effectiveness.
  • We improve effectiveness and reliability through situationally specific strategies, processes and practices.

While the Agile Manifesto is geared toward Agile practitioners, the Declaration of Interdependence is geared towards Agile project leaders.

When you know the values that shape things, it helps you better understand why things are the way they are. 

Notice how you can read the Agile Manifesto as, “we value this more than that” and you can read the Declaration of Interdependence as “this benefit we achieve through this.”  Those are actually powerful and repeatable language patterns.  I’ve found myself drawing from those patterns over the years, whenever I was trying to articulate operating principles (which is a good name for principles that guide how you operate.)

You Might Also Like

3 Ways to Accelerate Business Value

Extreme Programming (XP) at a Glance

How We Adhered to the Agile Manifesto on the patterns & practices Team

The patterns & practices Way

Using MadLibs to Create Actionable Principles and Practices

Categories: Architecture, Programming

The Power of Annual Reviews for Personal Development

Talk about taking some things for granted.  Especially when it’s a love-hate relationship.  I’m talking about Annual Reviews. 

I didn’t realize how valuable they can be when you own the process and you line them up with your bigger goal setting for life.  I’ve done them for so long, in this way, that I forgot how much they are a part of my process for carving out a high-impact year.

I know I might do things a big differently in terms of how I do my review, so I highlighted key things in my post:

The Power of Annual Reviews for Achieving Your Goals and Realizing Your Potential

Note that if you hate the term Annual Review because it conjures up a bunch of bad memories, then consider calling it your Annual Retrospective.  If you’re a Scrum fan, you’ll appreciate the twist.

Here’s the big idea:

If you “own” your Annual Review, you can use taking a look back to take a leap forward.

What I mean is that if you are pro-active in your approach, and if you really use feedback as a gift, you can gain tremendous insights into your personal growth and capabilities.

Here’s a summary of what I do in terms of my overall review process:

  1. Take a Look Back.  In December, I take a look back.   For example, this would be my 2013 Year in Review.   What did I achieve?  What went well? What didn’t go well?  How did I do against my 3-5 key goals that really mattered.   I use The Rule of 3, so really, I care about 3 significant changes that I can tell a story around for the year (The value of a story is the value of the change, and the value of the change is the value of the challenge.)
  2. Take a Look Forward.  Also in December, I take a look ahead.  What are my 3-5 big goals that I want to achieve for this year?  I really focus on 3 wins for each year.  The key is to hone in on the changes that matter.  If it’s not a change, then it’s business as usual, and doesn’t really need my attention because it’s already a habit and I’m already doing it.
  3. Align Work + Life.  When the Microsoft mid-year process starts, I figure out what I want to achieve in terms of themes and goals for the year at work.  I’ve already got my bigger picture in mind.   Now it’s just a matter of ensuring alignment between work and life.  There’s always a way to create better alignment and better leverage, and that’s how we empower ourselves to flourish in work and life.

It’s not an easy process.  But that’s just it.  That’s what makes it worth it.  It’s a tough look at the hard stuff that matters.  The parts of the process that make it  a challenge are the opportunities for growth.   Looking back, I can see how much easier it is for me to really plan out a year of high-impact where I live my values and play to my strengths.  I can also see early warning signs and anticipate downstream challenges.  I know when I first started, it was daunting to figure out what a year might look like.  Now, it’s almost too easy.

This gives me a great chance up front to play out a lot of “What If?” scenarios.  This also gives me a great chance right up front to ask the question, if this is how the year will play out, is that the ride I want to be on?  The ability to plan out our future capability vision, design a better future, and change our course is part of owning our destiny.

In my experience, a solid plan at the right level, gives you more flexibility and helps you make smarter choices, before you become a frog in the boiling pot.

If you haven’t taken the chance to really own and drive your Annual Review, then consider doing an Annual Retrospective, and use the process to help you leap frog ahead.

Make this YOUR year.

You Might Also Like

2012 Year in Review

Anatomy of a High-Potential

Mid-Year Review, Career, and Getting Ahead

Performance Review Template

The Guerilla Guide to Getting a Better Performance Review at Microsoft

Categories: Architecture, Programming

Phoney Deadlines are Deadly for Achieving Results

Xebia Blog - Sat, 12/28/2013 - 17:55

Ever had deadlines that must be met causing short-term decisions to be made? Ever worked over time with your team to meet an important deadline after which the delivered product wasn’t used for a couple of weeks?

I believe we all know these examples where deadlines are imposed on the team for questionable reasons.

Yet, deadlines are part of reality and we have to deal with them. Certainly, there is business value in meeting them but they also have costs.

The Never Ending Story of Shifting Deadlines…..

Some time ago I was involved in a project for delivering personalised advertisements on mobile phones. At that time this was quite revolutionary and we didn’t know how the market would react. Therefore, a team of skilled engineers and marketeers was assembled and we set out to deliver a prototype in a couple of months and test it in real life, i.e. with real phones and in the mobile network. This was a success and we got the assignment to make it into a commercial product version 1.0.
At this time there was no paying customer for the product yet and we built it targeted at multiple potential customers.

For the investment to make sense the deadline for delivering version 1.0 was set to 8 months.

The prototype worked fine but how to achieve a robust product when the product is scaled to millions of subscribers and thousands of advertisements per second? What architecture to use? Should we build upon the prototype or throw it away and start all over with the acquired knowledge?

A new architecture required us to use new technology which would require training and time to get acquainted with it. Time we did not have as the deadline was only 8 months away. We double checked that the deadline can be moved to a later date. Of course, this wash’t possible as it would invalidate the business case. We decided to not throw away the prototype but enhance it further.

As the deadline was approached it became clear that we were not going to deliver a product 1.0. Causes were multiple: the prototype’s architecture did not quite fit the 1.0 needs, scope changed along the way as marketing got new insights from the changing market, the team grew in size, and the integration to other network components took time as well.
So, the deadline was extended with another 6 months.

The deadline got shifted 2 more times.

This felt really bad. It felt we let down both the company and the product owner by not delivering on time. We had the best people part on the team and already had a working proto type. How come we weren’t able to deliver? What happened? What could we do to prevent this from happening a third time?

Then the new product was going to be announced at a large Telecom conference. This is what the product (and the team) needed; we still got a deadline but this time there was a clear goal associated with the deadline, namely a demo version for attracting first customers! Moreover, there was a small time window for delivering the product; missing the deadline would mean an important opportunity was lost with severe impact to the business case. This time we made the deadline.

The conference was a success and we got our first customers; of course new deadlines followed and this time with clear goals related to the needs of specific customers.

The Effect Deadlines Have

Looking back, which always is easy, we could have finished the product much earlier if the initial deadline was set to a much later date. Certainly, there was value in being able to deliver a product very fast, i.e. having short-term deadlines. On the other hand there were also costs associated with these short-term deadlines including:

  • struggling with new technologies caused by not taking time to do the necessary trainings and take time to gain experience,
  • working with an architecture that does not quite fit causing more workarounds,
  • relatively simple tasks becoming more complex over time.

In this case the short-term deadline put pressure on the team to deliver in short time causing the team to take short-cuts along the way causing delays and refactoring at a later time. Over time less results will be delivered.
What makes this pattern hard to fix is that the action of imposing deadline will deliver short-term results and seems a good idea to get results from the team.

This pattern is known as ‘Shifting the Burden’ and is depicted below. In the previous example the root cause is not trusting the team to get the job done. The distrust is addressed by imposing a deadline as a way to ‘force’ the team to deliver.

deadline1

The balancing (B) loop on top is the short-term solution to the problem getting results from the team. The 'fear' of the lacking focus and therefore results leads to imposing a deadline and thereby increasing the focus (and results) of the team. The problem symptom will reduce but will reappear causing an 'addiction' to the loop of imposing deadlines.

The fundamental solution of trusting the team, prioritising and giving them goals is often overlooked. Also this fundamental solution is less evident and costs energy and effort from the organisation to implement. The short-term solution has unwanted side effects that in the long run - slashed arrow - have negative impact on the team results.

In the example above the fundamental solution consisted of setting and prioritising towards the goal of a successful demo at the conference. This worked because it was a short-term and realistic goal. Furthermore the urgency was evident to the team: there was not going to be a second chance in case this deadline was missed.

Another example
In practise I also encounter the situation in which deadlines are imposed is a team that seems to lack focus. The underlying problem is the lack of a (business) vision and goals. The symptom as experienced by the organisation is the lack of concrete results. In fact the team does work hard but does so by working on multiple goals at the same time. Here, clear goals and prioritising the work to be done first will help.

deadline2

Also in this example, the action of imposing deadline to ‘solve’ the problem has the unintended side effect of not addressing the underlying problem. This will make the problem of the team not delivering result reappear.

Goals & Deadlines

In the examples of the previous section the deadlines I call phoney deadlines. When in practise a deadline is imposed it usually also implies a fixed scope and fixed date.

Deadlines
Deadlines should be business case related and induce large costs if not met. For the deadlines in the above examples this is not the case.

Examples of deadlines that have associated large costs if not met, are:

  • associated with small market window of introducing a product (or set of features); the cost of missing the small time window is very high,
  • associated with implementation dates of laws; again missing these deadline severely harms the business,
  • hardware that becomes end of life,
  • support contracts the end.

Goals
In the story above the ‘real’ deadline actually was 2 years instead of the 8 months. In this case the deadline probably was used as a management tool, with all good intentions, to get the team focussed on producing results. Whereas in fact it caused short-cuts to be made by the team in order to meet the deadline.

Getting focus in teams is done by giving the team a goal: a series of subgoals leading to the end-goal [Bur13]. Establish a vision and derive a series of (sub)goals to realise the vision. Relate these goals to the business case. Story mapping is one of the tools available to define series of goals [Lit].

Conclusion

Avoid setting deadlines as a means to get results from a team. On the short-term this will give results but on the long run it negatively impacts the results that the organisation wants to achieve.

Reserve deadlines for events that have a very high Cost of Delay when missed, i.e. the cost of missing the deadline is very large.

Instead, set a vision (both for the organisation and product) that is consistent with the fundamental solution. In addition, derive a series of goals and prioritise to help team focus on achieving results. To derive a series of goals several techniques can be used like Story mapping and Goal-Impact Mapping.

References [Bur13] Daniel Burm, 2013, The Secret to 3-Digit Productivity Growth [Wik] Shifting the Burden, Wikipedia, Systems Thinking [Lit] Lithespeed, Story Mapping