Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Testing & QA

First experiences with OpenStack

Agile Testing - Grig Gheorghiu - Thu, 07/17/2014 - 21:37
We hit a big milestone this week, as we started to use OpenStack as a private cloud, intially just for QA/integration environments. Up to now we've been creating KVM machines semi-manually, which used to take minutes. Now we cut down that process to seconds, calling the Nova API from the command line, e.g.:

$ nova boot --image precise-image --flavor www --key_name mykey --nic net-id=3eafbd4f-0389-4c5b-93ba-7764742ee8cd www1.qa1

Once an instance is provisioned, we bootstrap it with Chef:

$ knife bootstrap www1.qa1.mydomain.com -x ubuntu --sudo -E qa1 -N www1.qa1 -r "role[base], role[www]"

Our internal network architecture is fairly complex, so my colleague Jeff Roberts spent quite some time bending OpenStack Neutron to his will (in conjunction with Open vSwitch) in order to support our internal VLANs. The OpenStack infrastructure has been stable so far, and it's just such a pleasure to do everything via an API and not to spin VMs up manually. Being back to working with a (private) cloud feels good.

This is just version 1.0 of our OpenStack rollout. Soon we'll start spinning up one environment at a time using chef-metal and fog  and we'll also integrate instance + environment spin-up with Jenkins. Exciting times ahead!

Measuring Coverage at Google

Google Testing Blog - Mon, 07/14/2014 - 20:45
By Marko Ivanković, Google Zürich

Introduction
Code coverage is a very interesting metric, covered by a large body of research that reaches somewhat contradictory results. Some people think it is an extremely useful metric and that a certain percentage of coverage should be enforced on all code. Some think it is a useful tool to identify areas that need more testing but don’t necessarily trust that covered code is truly well tested. Others yet think that measuring coverage is actively harmful because it provides a false sense of security.

Our team’s mission was to collect coverage related data then develop and champion code coverage practices across Google. We designed an opt-in system where engineers could enable two different types of coverage measurements for their projects: daily and per-commit. With daily coverage, we run all tests for their project, where as with per-commit coverage we run only the tests affected by the commit. The two measurements are independent and many projects opted into both.

While we did experiment with branch, function and statement coverage, we ended up focusing mostly on statement coverage because of its relative simplicity and ease of visualization.

How we measured
Our job was made significantly easier by the wonderful Google build system whose parallelism and flexibility allowed us to simply scale our measurements to Google scale. The build system had integrated various language-specific open source coverage measurement tools like Gcov (C++), Emma / JaCoCo (Java) and Coverage.py (Python), and we provided a central system where teams could sign up for coverage measurement.

For daily whole project coverage measurements, each team was provided with a simple cronjob that would run all tests across the project’s codebase. The results of these runs were available to the teams in a centralized dashboard that displays charts showing coverage over time and allows daily / weekly / quarterly / yearly aggregations and per-language slicing. On this dashboard teams can also compare their project (or projects) with any other project, or Google as a whole.

For per-commit measurement, we hook into the Google code review process (briefly explained in this article) and display the data visually to both the commit author and the reviewers. We display the data on two levels: color coded lines right next to the color coded diff and a total aggregate number for the entire commit.


Displayed above is a screenshot of the code review tool. The green line coloring is the standard diff coloring for added lines. The orange and lighter green coloring on the line numbers is the coverage information. We use light green for covered lines, orange for non-covered lines and white for non-instrumented lines.

It’s important to note that we surface the coverage information before the commit is submitted to the codebase, because this is the time when engineers are most likely to be interested in improving it.

Results
One of the main benefits of working at Google is the scale at which we operate. We have been running the coverage measurement system for some time now and we have collected data for more than 650 different projects, spanning 100,000+ commits. We have a significant amount of data for C++, Java, Python, Go and JavaScript code.

I am happy to say that we can share some preliminary results with you today:


The chart above is the histogram of average values of measured absolute coverage across Google. The median (50th percentile) code coverage is 78%, the 75th percentile 85% and 90th percentile 90%. We believe that these numbers represent a very healthy codebase.

We have also found it very interesting that there are significant differences between languages:

C++JavaGoJavaScriptPython56.6%61.2%63.0%76.9%84.2%

The table above shows the total coverage of all analyzed code for each language, averaged over the past quarter. We believe that the large difference is due to structural, paradigm and best practice differences between languages and the more precise ability to measure coverage in certain languages.

Note that these numbers should not be interpreted as guidelines for a particular language, the aggregation method used is too simple for that. Instead this finding is simply a data point for any future research that analyzes samples from a single programming language.

The feedback from our fellow engineers was overwhelmingly positive. The most loved feature was surfacing the coverage information during code review time. This early surfacing of coverage had a statistically significant impact: our initial analysis suggests that it increased coverage by 10% (averaged across all commits).

Future work
We are aware that there are a few problems with the dataset we collected. In particular, the individual tools we use to measure coverage are not perfect. Large integration tests, end to end tests and UI tests are difficult to instrument, so large parts of code exercised by such tests can be misreported as non-covered.

We are working on improving the tools, but also analyzing the impact of unit tests, integration tests and other types of tests individually.

In addition to languages, we will also investigate other factors that might influence coverage, such as platforms and frameworks, to allow all future research to account for their effect.

We will be publishing more of our findings in the future, so stay tuned.

And if this sounds like something you would like to work on, why not apply on our job site?

Categories: Testing & QA

GTAC is Almost Here!

Google Testing Blog - Tue, 07/01/2014 - 20:28
by The GTAC Committee

GTAC is just around the corner, and we’re all very busy and excited. I know we say this every year, but this is going to be the best GTAC ever! We have updated the GTAC site with important details:


If you are on the attendance list, we’ll see you on April 23rd. If not, check out the Live Stream page where you can watch the conference live and can get involved in Q&A after each talk. Perhaps your team can gather in a conference room and attend remotely.

Categories: Testing & QA

GTAC 2013 Wrap-up

Google Testing Blog - Tue, 07/01/2014 - 20:27
by The GTAC Committee

The Google Test Automation Conference (GTAC) was held last week in NYC on April 23rd & 24th. The theme for this year's conference was focused on Mobile and Media. We were fortunate to have a cross section of attendees and presenters from industry and academia. This year’s talks focused on trends we are seeing in industry combined with compelling talks on tools and infrastructure that can have a direct impact on our products. We believe we achieved a conference that was focused for engineers by engineers. GTAC 2013 demonstrated that there is a strong trend toward the emergence of test engineering as a computer science discipline across companies and academia alike.

All of the slides, video recordings, and photos are now available on the GTAC site. Thank you to all the speakers and attendees who made this event spectacular. We are already looking forward to the next GTAC. If you have suggestions for next year’s location or theme, please comment on this post. To receive GTAC updates, subscribe to the Google Testing Blog.

Here are some responses to GTAC 2013:

“My first GTAC, and one of the best conferences of any kind I've ever been to. The talks were consistently great and the chance to interact with so many experts from all over the map was priceless.” - Gareth Bowles, Netflix

“Adding my own thanks as a speaker (and consumer of the material, I learned a lot from the other speakers) -- this was amazingly well run, and had facilities that I've seen many larger conferences not provide. I got everything I wanted from attending and more!” - James Waldrop, Twitter

“This was a wonderful conference. I learned so much in two days and met some great people. Can't wait to get back to Denver and use all this newly acquired knowledge!” - Crystal Preston-Watson, Ping Identity

“GTAC is hands down the smoothest conference/event I've attended. Well done to Google and all involved.” - Alister Scott, ThoughtWorks

“Thanks and compliments for an amazingly brain activity spurring event. I returned very inspired. First day back at work and the first thing I am doing is looking into improving our build automation and speed (1 min is too long. We are not building that much, groovy is dynamic).” - Irina Muchnik, Zynx Health

Categories: Testing & QA

ThreadSanitizer: Slaughtering Data Races

Google Testing Blog - Mon, 06/30/2014 - 22:30
by Dmitry Vyukov, Synchronization Lookout, Google, Moscow

Hello,

I work in the Dynamic Testing Tools team at Google. Our team develops tools like AddressSanitizer, MemorySanitizer and ThreadSanitizer which find various kinds of bugs. In this blog post I want to tell you about ThreadSanitizer, a fast data race detector for C++ and Go programs.

First of all, what is a data race? A data race occurs when two threads access the same variable concurrently, and at least one of the accesses attempts is a write. Most programming languages provide very weak guarantees, or no guarantees at all, for programs with data races. For example, in C++ absolutely any data race renders the behavior of the whole program as completely undefined (yes, it can suddenly format the hard drive). Data races are common in concurrent programs, and they are notoriously hard to debug and localize. A typical manifestation of a data race is when a program occasionally crashes with obscure symptoms, the symptoms are different each time and do not point to any particular place in the source code. Such bugs can take several months of debugging without particular success, since typical debugging techniques do not work. Fortunately, ThreadSanitizer can catch most data races in the blink of an eye. See Chromium issue 15577 for an example of such a data race and issue 18488 for the resolution.

Due to the complex nature of bugs caught by ThreadSanitizer, we don't suggest waiting until product release validation to use the tool. For example, in Google, we've made our tools easily accessible to programmers during development, so that anyone can use the tool for testing if they suspect that new code might introduce a race. For both Chromium and Google internal server codebase, we run unit tests that use the tool continuously. This catches many regressions instantly. The Chromium project has recently started using ThreadSanitizer on ClusterFuzz, a large scale fuzzing system. Finally, some teams also set up periodic end-to-end testing with ThreadSanitizer under a realistic workload, which proves to be extremely valuable. When races are found by the tool, our team has zero tolerance for races and does not consider any race to be benign, as even the most benign races can lead to memory corruption.

Our tools are dynamic (as opposed to static tools). This means that they do not merely "look" at the code and try to surmise where bugs can be; instead they they instrument the binary at build time and then analyze dynamic behavior of the program to catch it red-handed. This approach has its pros and cons. On one hand, the tool does not have any false positives, thus it does not bother a developer with something that is not a bug. On the other hand, in order to catch a bug, the test must expose a bug -- the racing data access attempts must be executed in different threads. This requires writing good multi-threaded tests and makes end-to-end testing especially effective.

As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and more. It also natively understands atomic operations and thus can find bugs in lock-free algorithms (see e.g. this bug in the V8 concurrent garbage collector).

The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a -fsanitize=thread flag during compilation and linking. For Go programs, you simply need to add a -race flag to the go tool (supported on Linux, Mac and Windows).

Interestingly, after integrating the tool into compilers, we've found some bugs in the compilers themselves. For example, LLVM was illegally widening stores, which can introduce very harmful data races into otherwise correct programs. And GCC was injecting unsafe code for initialization of function static variables. Among our other trophies are more than a thousand bugs in Chromium, Firefox, the Go standard library, WebRTC, OpenSSL, and of course in our internal projects.

So what are you waiting for? You know what to do!
Categories: Testing & QA

Software Development Conferences Forecast June 2014

From the Editor of Methods & Tools - Thu, 06/26/2014 - 07:22
Here is a list of software development related conferences and events on Agile ( Scrum, Lean, Kanban) software testing and software quality, programming (Java, .NET, JavaScript, Ruby, Python, PHP) and databases (NoSQL, MySQL, etc.) that will take place in the coming weeks and that have media partnerships with the Methods & Tools software development magazine. AGILE2014, July 28 – August 1, Orlando, USA Agile on the Beach, September 4-5 2014, Falmouth in Cornwall, UK SPTechCon, September 16-19 2014, Boston, USA STARWEST, October 12-17 2014, Anaheim, USA JAX London, October 13-15 2014,London, UK Pacific Northwest ...

Kanban, Developer Career & Mobile UX in Methods & Tools Summer 2014 issue

From the Editor of Methods & Tools - Mon, 06/23/2014 - 14:54
Methods & Tools – the free e-magazine for software developers, testers and project managers – has just published its Summer 2014 issue that discusses objections to Kanban implementation, How to use a model to evaluate and improve mobile user experience, balancing a software development job and a meaningful life, Scrum agile project management tools, JavaScript unit testing and static analysis for BDD. Methods & Tools Summer 2014 contains the following articles: * Kanban for Skeptics * Using a Model To Systematically Evaluate and Improve Mobile User Experience * Developer Careers Considered Harmful * TargetProcess – ...

GTAC 2014: Call for Proposals & Attendance

Google Testing Blog - Sat, 06/21/2014 - 16:26
Posted by Anthony Vallone on behalf of the GTAC Committee

The application process is now open for presentation proposals and attendance for GTAC (Google Test Automation Conference) (see initial announcement) to be held at the Google Kirkland office (near Seattle, WA) on October 28 - 29th, 2014.

GTAC will be streamed live on YouTube again this year, so even if you can’t attend, you’ll be able to watch the conference from your computer.

Speakers
Presentations are targeted at student, academic, and experienced engineers working on test automation. Full presentations and lightning talks are 45 minutes and 15 minutes respectively. Speakers should be prepared for a question and answer session following their presentation.

Application
For presentation proposals and/or attendance, complete this form. We will be selecting about 300 applicants for the event.

Deadline
The due date for both presentation and attendance applications is July 28, 2014.

Fees
There are no registration fees, and we will send out detailed registration instructions to each invited applicant. Meals will be provided, but speakers and attendees must arrange and pay for their own travel and accommodations.

Update : Our contact email was bouncing - this is now fixed.



Categories: Testing & QA

Quote of the Month June 2014

From the Editor of Methods & Tools - Fri, 06/20/2014 - 06:39
A UX team that deals with only the details of radio buttons and check boxes is committing a disservice to its organization. Today UX groups must deal with strategy. Source: Institutionalization of UX (2nd Edition), Eric Schaffer & Apala Lahiri, Addison-Wesley

Setting up the hostname in Ubuntu

Agile Testing - Grig Gheorghiu - Fri, 06/13/2014 - 23:04
Most people recommend setting up the hostname on a Linux box so that:

1) running 'hostname' returns the short name (i.e. myhost)
2) running 'hostname -f' returns the FQDN (i.e. myhost.prod.example.com)
3) running 'hostname -d' returns the domain name (i.e prod.example.com)

After experimenting a bit and also finding this helpful Server Fault post, here's what we did to achieve this (we did it via Chef recipes, but it amounts to the same thing):

  • make sure we have the short name in /etc/hostname:
myhost

(also run 'hostname myhost' at the command line)
  • make sure we have the FQDN as the first entry associated with the IP of the server in /etc/hosts:
10.0.1.10 myhost.prod.example.com myhost myhost.prod
  • make sure we have the domain name set up as the search domain in /etc/resolv.conf:
search prod.example.com

Reboot the box when you're done to make sure all of this survives reboots.


Software Development Linkopedia June 2014

From the Editor of Methods & Tools - Thu, 06/12/2014 - 17:08
Here is our monthly selection of interesting knowledge material on programming, software testing and project management.  This month you will find some interesting information and opinions about Agile project estimation, JavaScript frontend testing and refactoring, retrospectives, software configuration, Test-Driven Development, software tools and NoSQL databases. Blog: Calculating Velocity FAQ Blog: Writing Testable Frontend Javascript Part 1 – Anti-patterns and their fixes Blog: Writing Testable Frontend Javascript Part 2 – Refactor away anti-patterns Blog: Creating options by slicing features – #NoEstimates technique Article: Agile Retrospectives: Why They Matter Article: First Sketches of an App: Planning the Design ...

GTAC 2014 Coming to Seattle/Kirkland in October

Google Testing Blog - Wed, 06/04/2014 - 20:21
Posted by Anthony Vallone on behalf of the GTAC Committee

If you're looking for a place to discuss the latest innovations in test automation, then charge your tablets and pack your gumboots - the eighth GTAC (Google Test Automation Conference) will be held on October 28-29, 2014 at Google Kirkland! The Kirkland office is part of the Seattle/Kirkland campus in beautiful Washington state. This campus forms our third largest engineering office in the USA.



GTAC is a periodic conference hosted by Google, bringing together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It’s a great opportunity to present, learn, and challenge modern testing technologies and strategies.

You can browse the presentation abstracts, slides, and videos from last year on the GTAC 2013 page.

Stay tuned to this blog and the GTAC website for application information and opportunities to present at GTAC. Subscribing to this blog is the best way to get notified. We're looking forward to seeing you there!

Categories: Testing & QA

10 Articles About the People Side of Software Development

From the Editor of Methods & Tools - Wed, 06/04/2014 - 14:58
In their seminal book Peopleware, Tom DeMarco and Tim Lister wrote “The major problems of our work are not so much technological as sociological in nature.” Computer science curriculum are full of courses about how to code sorting algorithms or how to write a compiler, but most of us never had any training on the people aspect of our software development which involve communication and negotiation. This is a selection of articles that will interest those who want to improve their knowledge about the people side of software development. They discuss ...

Testing on the Toilet: Risk-Driven Testing

Google Testing Blog - Sat, 05/31/2014 - 00:10
by Peter Arrenbrecht

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.

We are all conditioned to write tests as we code: unit, functional, UI—the whole shebang. We are professionals, after all. Many of us like how small tests let us work quickly, and how larger tests inspire safety and closure. Or we may just anticipate flak during review. We are so used to these tests that often we no longer question why we write them. This can be wasteful and dangerous.

Tests are a means to an end: To reduce the key risks of a project, and to get the biggest bang for the buck. This bang may not always come from the tests that standard practice has you write, or not even from tests at all.

Two examples:

“We built a new debugging aid. We wrote unit, integration, and UI tests. We were ready to launch.”

Outstanding practice. Missing the mark.

Our key risks were that we'd corrupt our data or bring down our servers for the sake of a debugging aid. None of the tests addressed this, but they gave a false sense of safety and “being done”.
We stopped the launch.

“We wanted to turn down a feature, so we needed to alert affected users. Again we had unit and integration tests, and even one expensive end-to-end test.”

Standard practice. Wasted effort.

The alert was so critical it actually needed end-to-end coverage for all scenarios. But it would be live for only three releases. The cheapest effective test? Manual testing before each release.

A Better Approach: Risks First

For every project or feature, think about testing. Brainstorm your key risks and your best options to reduce them. Do this at the start so you don't waste effort and can adapt your design. Write them down as a QA design so you can point to it in reviews and discussions.

To be sure, standard practice remains a good idea in most cases (hence it’s standard). Small tests are cheap and speed up coding and maintenance, and larger tests safeguard core use-cases and integration.

Just remember: Your tests are a means. The bang is what counts. It’s your job to maximize it.

Categories: Testing & QA

Let’s test at Let’s Test

James Bach’s Blog - Fri, 05/23/2014 - 07:55

I’ve been telling people that the best conference I know for thinking testers is Let’s Test (followed closely by CAST, which I will also be at, this year, in New York). Let’s Test was created by people who experienced CAST and wanted to be even more dedicated to Context-Driven testing principles.

Now, I’m here in Stockholm once again to be with the most interesting testers in Europe. I’m not done with my presentations, yet. But I still have a couple of days.

(I will presenting a new model of what it means to be an excellent observer, together with one or two observation challenges for participants. And Pradeep Soundararajan and I will be presenting a tutorial on reviewing a specification by testing it.)

Let’s Test is not for the faint of heart. Events go on day and night. I suffer from terrible jet lag, so I probably won’t be seen after dinner. But for you crazy kids, it’s a great place to try a testing exercise, or present one.

(Note: I’m being paid to teach at Let’s Test. I don’t get a percentage of the gate, though– I get paid the same whether anyone shows up or not.)

Australia Let’s Test

I will also be in Australia for the first Let’s Test happening down there, in September. There are some interesting testers in Oz. I’m sure they will all be there. It will be the first great party of ambitious intellectual testers that I know of in the history of Australian testing.

Anne-Marie Charrett and I will be doing our Coaching Testers tutorial, which is the only time this year we will teach it together.

“Intellectual” testers?

Why do I keep saying that? Because the state of the practice in testing is for testers NOT to read about their craft, NOT to study social science or know anything about the proper use of statistics or the meaning of the word “heuristic”, and NOT to challenge the now 40 year stale ideas about making testing into factory work that lead directly to mass outsourcing of testing to lowest bidder instead of the most able tester.

Intellectual testers are not the most common type of tester.

The ISTQB and similar programs require your stupidity and your fear in order to survive. And their business model is working. They don’t debate us for the same reason that HP made billions of dollars selling bad test tools by pitching them to non-testers who had more money than wisdom. Debating us would spoil their racket.

So, don’t be like that. Be smart.

I’ll see you at Let’s Test.

Categories: Testing & QA

Affordance Open-Space at XP Day

Mistaeks I Hav Made - Nat Pryce - Tue, 12/03/2013 - 00:20
Giovanni Asproni and I facilitated an open-space session at XP Day on the topic of affordance in software design. Affordance is "a quality of an object, or an environment, which allows an individual to perform an action". Don Norman distinguishes between actual afforance and perceived affordance, the latter being "those action possibilities that are readily perceivable by an actor". I think this distinction applies as well to software programming interfaces. Unless running in a sandboxed execution environments, software has great actual affordance, as long as we're willing to do the wrong thing when necessary: break encapsulation or rely on implementation-dependent data structures. What makes programming difficult is determining how and why to work with the intentions of those who designed the software that we are building upon: the perceived affordance of those lower software layers. Here are the notes I took during the session, cleaned up a little and categorised into things that provide help affordance, hinder affordance or can both help and hinder. Helping Repeated exposure to language you understand results in fluency. Short functions Common, consistent language. Domain types. Good error messages Things that behave differently should look different. e.g. convention for ! suffix in Scheme and Ruby for functions that mutate data. Able to explore software: able to put "probes" into/around the software to experiment with its behaviour know where to put those probes TDD, done properly, is likely to make most important affordances visible simpler APIs ability to experiment and probe software running in isolation. Hindering Two ways software has bad affordance: can't work out what it does (poor perceived affordance) it behaves unexpectedly (poor actual affordance) (are these two viewpoints on the same thing?) Null pointer exceptions detected far from origin. Classes named after design patterns incorrectly. e.g. programmer did not understand the pattern. Factories that do not create objects Visitors that are really internal iterators Inconsistencies can be caused by unfinished refactoring which code style to use? agree in the team spread agreement by frequent pair rotation Names that have subtly different meanings in natural language and in the code. E.g. OO design patterns. Compare with FP design patterns that have strict definitions and meaningless names (in natural language). You have to learn what the names really mean instead of rely on (wrong) intuition. Both Helping and Hindering Conventions e.g. Ruby on Rails alternative to metaphor? but: don't be magical! hard to test code in isolation If you break convention, draw attention to where it has been broken. Naming types after patterns better to use domain terms than pattern names? e.g. CustomerService mostly does stuff with addresses. Better named AddressBook, can see it also has unrelated behaviour that belongs elsewhere. but now need to communicate the system metaphors and domain terminology (is that ever a bad thing?) Bad code style you are familiar with does offer perceived affordance. Bad design might provide good actual affordance. e.g. lots of global variables makes it easy to access data. but provides poor perceived affordance e.g. cannot understand how state is being mutated by other parts of the system Method aliasing: makes code expressive common in Ruby but Python takes the opposite stance - only one way to do anything Frameworks: new jargon to learn new conventions provide specific affordances if your context changes, you need different affordances and the framework may get in the way. Documentation vs Tests actual affordance of physical components is described in a spec documents and not obvious from appearance. E.g. different grade of building brick. Is this possible for software? docs are more readable tests are more trustworthy best of both worlds - doctests? Used in Go and Python How do we know what has changed between versions of a software component? open source libraries usually have very good release notes why is it so hard in enterprise development?
Categories: Programming, Testing & QA

Multimethods

Mistaeks I Hav Made - Nat Pryce - Thu, 10/10/2013 - 00:30
I have observed that... Unless the language already provides them, any sufficiently complicated program written in a dynamically typed language contains multiple, ad hoc, incompatible, informally-specified, bug-ridden, slow implementations of multimethods. (To steal a turn of phrase from Philip Greenspun).
Categories: Programming, Testing & QA

Thought after Test Automation Day 2013

Henry Ford said “Obstacles are those frightful things you see when take your eyes off the goal.” After I’ve been to Test Automation Day last month I’m figuring out why industrializing testing doesn’t work. I try to put it in this negative perspective, because I think it works! But also when is it successful? A lot of times the remark from Ford is indeed the problem. People tend to see obstacles. Obstacles because of the thought that it’s not feasible to change something. They need to change. But that’s not an easy change.

After attending the #TAD2013 as it was on Twitter I saw a huge interest in better testing, faster testing, even cheaper testing by using tools to industrialize. Test automation has long been seen as an interesting option that can enable faster testing. it wasn’t always cheaper, especially the first time, but at least faster. As I see it it’ll enable better testing. “Better?” you may ask. Test automation itself doesn’t enable better testing, but by automating regression tests and simple work the tester can focus on other areas of the quality.

images

And isn’t that the goal? In the end all people involved in a project want to deliver a high quality product, not full of bugs. But they also tend to see the obstacles. I see them less and less. New tools are so well advanced and automation testers are becoming smarter and smarter that they enable us to look beyond the obstacles. I would like to say look over the obstacles.

At the Test Automation Day I learned some new things, but it also proved something I already know; test automation is here to stay. We don’t need to focus on the obstacles, but should focus on the goal.

Categories: Testing & QA

The Toyota Way: The need for doing it right the first time

After WWII Toyota started developing its Toyota Production System (TPS); which was identified as ‘Lean’ in the 1990s. Taiichi Ohno, Shigeo Shingo and Eiji Toyoda developed the system between 1948 and 1975. In the myth surrounding the system it was not inspired by the American automotive industry, but from a visit to American supermarkets, Ohno saw the supermarket as model for what he was trying to accomplish in the factor and perfect the Just-in-Time (JIT) production system. While accomplishing this low inventory levels were a key outcome of the TPS, and an important element of the philosophy behind its system is to work intelligently and eliminate waste so that only minimal inventory is needed.

As TPS and Lean have their own principles as outlined by Toyota:

  • Long-term Philosophy
  • Right process will produce the right results
  • Value to organization by developing people
  • Solving root problems drives organizational learning

As these principles were summed up and published by Toyota in 2001, by naming it “The Toyota Way 2001”. It consists the above named principles in two key areas: Continuous Improvement, and Respect for People.

the-toyota-way

The principles for a continuous improvement include establishing a long-term vision, working on challenges, continual innovation, and going to the source of the issue or problem. The principles relating to respect for people include ways of building respect and teamwork. When looking at the ALM all these principles come together in the ‘first time right’ approach already mentioned. And from Toyota’s view they were outlined as followed:

  • The right process will produce the right results
    • Create continuous process flow to bring problems to the surface.
    • Use the ‘pull’ system to avoid overproduction (kanban)
    • Level out the workload (heijunka).
    • Build a culture of stopping to fix problems, to get quality right from the first (jidoka)
  • Continuously solving root problems drives organizational learning
    • Go and see for yourself to thoroughly understand the situation (Genchi Genbutsu);
    • Make decisions slowly by consensus, thoroughly considering all options (nemawashi); implement decisions rapidly;
    • Become a learning organization through relentless reflection (hansei) and continuous improvement (kaizen).
Let’s do it right now!

As the economy is changing and IT is more common sense throughout ore everyday life the need for good quality software products has never been this high. Software issues create bigger and bigger issues in our lives. Think about trains that cannot ride due to software issues, bank clients that have no access to their bank accounts, and people oversleeping because their alarm app didn’t work on their iPhone. As Capers Jones [Jones, 2011] states in his 2011 study that “software is blamed for more major business problems than any other man-made product” and that “poor quality has become one of the most expensive topics in human history”. The improvement of software quality is a key topic for all industries.

 Right the first time vs jidoka

In both TPS and Lean autonomation or jidoka are used. Autonomation can be described as ‘intelligent autonomation’, it means that when an abnormal situation arises the ‘machine’ stops and fix the abnormality. Autonomation prevents the production of defective products, eliminates overproduction, and focuses attention on understanding the problem and ensuring that it never recurs; a quality control process that applies the following four principles:

  • Detect the abnormality.
  • Stop.
  • Fix or correct the immediate condition.
  • Investigate the root cause and install a countermeasure.
Find defects as early as possible

In other words autonomation helps to get quality right the first time perfectly. With IT projects being different from the Toyota car production line, ‘perfectly’ may be a bit too much, but the process around quality assurance should be the same:

  • Find the defect.
  • Stop.
  • Fix or correct the error.
  • Investigate the root cause and take countermeasures.

The defect should be found as early as possible to be fixed as early as possible. And as with Lean and TPS the reason behind this is to make it possible to address the identification and correction of defects immediately in the process.

Categories: Testing & QA

When will you start with test automation?

I just came back from vacation and when I started again I noticed a slight change in resource requests I now see coming by; as almost all requests are with a statement around test automation. In the last two days I had two separate sessions around test automation tools. Has test automation all of a sudden become more important? Did people follow up on my last post, where I state that tools are a prerequisite in testing today, or actually: yesterday!

If you missed the latest cycle in new tools for test automation you’re either an ostrich with your head in the ground (“sorry vacation was in Southern Africa”), or you just simply still afraid. Afraid of change that test automation would cannibalize your manual test execution.

Test automation is not anymore that it takes over test execution in a very complex and unmanageable way. No it offers higher efficiency on test design, test execution, but also more options to test certain non-functional parts of applications – that could not be done without those tools – like security and performance, and virtual environments to do end-2-end tests without test environments that are down all the time. Tools are now also offering more support for testing mobile solutions. Tools are everywhere!

automated-testing

Test automation offers us testers the opportunity to do more, faster, les risky, and cheaper. I set these words specifically in that order. Test automation is often seen as a way to do test cheaper. You can, but you also can do more, for instance:

  • Let the tool do the checks and you explore the application further;
  • Setup a virtual test environment that doesn’t go down after 1 hour of use and test more in the same time;
  • Create and execute more test cases by generating and executing them automatically.
  • Get higher quality by really doing a thorough regression test, instead of a simple check, to find integration errors.

There are enough reasons to work on test automation and I don’t see why not. I think now it is even time for the next step in test automation. What that is time will tell, but I look forward to hearing that at the Test Automation Day in June. Where Bryan Bakker will tell more on this next step in his presentation “Design for Testability – the next step in Test Automation”. After the congress I’ll post my ideas here.

Categories: Testing & QA