Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Software Development Blogs: Programming, Software Testing, Agile Project Management
Methods & Tools
Testing & QA
By Aaron Jacobs
Google JS Test is a JavaScript unit testing framework that runs on the V8 JavaScript Engine, the same open source project that is responsible for Google Chrome’s super-fast JS execution speed. Google JS Test is used internally by several Google projects, and we’re pleased to announce that it has been released as an open source project.
Features of Google JS Test include:
By Jim Reardon
The test plan is dead!
Well, hopefully. At a STAR West session this past week, James Whittaker asked a group of test professionals about test plans. His first question: “How many people here write test plans?” About 80 hands shot up instantly, a vast majority of the room. “How many of you get value or refer to them again after a week?” Exactly three people raised their hands.
That’s a lot of time being spent writing documents that are often long-winded, full of paragraphs of details on a project everyone already knows to get abandoned so quickly.
A group of us at Google set about creating a methodology that can replace a test plan -- it needed to be comprehensive, quick, actionable, and have sustained value to a project. In the past few weeks, James has posted a few blogs about this methodology, which we’ve called ACC. It's a tool to break down a software product into its constituent parts, and the method by which we created "10 Minute Test Plans" (that only take 30 minutes!)
Comprehensive
The ACC methodology creates a matrix that describes your project completely; several projects that have used it internally at Google have found coverage areas that were missing in their conventional test plans.
Quick
The ACC methodology is fast; we’ve created ACC breakdowns for complex projects in under half an hour. Far faster than writing a conventional test plan.
Actionable
As part of your ACC breakdown, risk is assessed to the capabilities of your appliciation. Using these values, you get a heat map of your project, showing the areas with the highest risk -- great places to spend some quality time testing.
Sustained Value
We’ve built in some experimental features that bring your ACC test plan to life by importing data signals like bugs and test coverage that quantify the risk across your project.
Today, I'm happy to announce we're open sourcing Test Analytics, a tool built at Google to make generating an ACC simple -- and which brings some experimental ideas we had around the field of risk-based testing that work hand-in-hand with the ACC breakdown.
Defining a project’s ACC model.
Test Analytics has two main parts: first and foremost, it's a step-by-step tool to create an ACC matrix that's faster and much simpler than the Google Spreadsheets we used before the tool existed. It also provides visualizations of the matrix and risks associated with your ACC Capabilities that were difficult or impossible to do in a simple spreadsheet.
A project’s Capabilities grid.
The second part is taking the ACC plan and making it a living, automatic-updating risk matrix. Test Analytics does this by importing quality signals from your project: Bugs, Test Cases, Test Results, and Code Changes. By importing these data, Test Analytics lets you visualize risk that isn't just estimated or guessed, but based on quantitative values. If a Component or Capability in your project has had a lot of code change or many bugs are still open or not verified as working, the risk in that area is higher. Test Results can provide a mitigation to those risks -- if you run tests and import passing results, the risk in an area gets lower as you test.
A project’s risk, calculated as a factor of inherent risk as well as imported quality signals.
This part's still experimental; we're playing around with how we calculate risk based on these signals to best determine risk. However, we wanted to release this functionality early so we can get feedback from the testing community on how well it works for teams so we can iterate and make the tool even more useful. It'd also be great to import even more quality signals: code complexity, static code analysis, code coverage, external user feedback and more are all ideas we've had that could add an even higher level of dynamic data to your test plan.
An overview of test results, bugs, and code changes attributed to a project’s capability. The Capability’s total risk is affected by these factors.
You can check out a live hosted version, browse or check out the code along with documentation, and of course if you have any feedback let us know - there's a Google Group set up for discussion, where we'll be active in responding to questions and sharing our experiences with Test Analytics so far.
Long live the test plan! 
By Anthony F. Voellm (aka Tony the perfguy)
It’s amazing what has happened in the field of test in the last 20 years... a lot of “art” has turned into “science”. Computer scientists, engineers, and many other disciplines have worked on provable systems and calculus, pioneered model based testing, invented security fuzz testing, and even settled on a common pattern for unit tests called xunit. The xunit pattern shows up in open source software like JUnit as well as in Microsoft development test tools.
With all this innovation in test, there’s no wonder test is dead. The situation is no different from the late 1800’s when patents were declared dead. Everything had been invented. So now that everything in test has been invented, it’s dead.
Well... if you believe everything in test has been invented then please stop reading now :)
As an aside: “Test is dead” was a keynote at the Google Test Automation Conference (GTAC) in 2011. You can watch that talk and many other GTAC test talks on YouTube, and I definitely recommend you check them out here. Talks span a wide range of topics ranging from GUI Automation to Cloud.
What really excites me these days is that we have closed a chapter on test. A lot of the foundation of writing and testing great software has been laid (examples at the beginning of the post, tools like Webdriver for UI, FIO for storage, and much more), which I think of as Testing 1.0. We all use Testing 1.0 day in and day out. In fact at Google, most of the developers (called Software Engineers or SWEs) do the basic Testing 1.0 work and we have a high bar on quality. Knuth once said "Be careful about using the following code -- I've only proven that it works, I haven't tested it."
This brings us to the current chapter in test which I call Testing 1.5. This chapter is being written by computer scientists, applied scientists, engineers, developers, statisticians, and many other disciplines. These people come together in the Software Engineer in Test (SET) and Test Engineer (TE) roles at Google. SET/TEs focus on; developing software faster, building it better the first time, testing it in depth, releasing it quicker, and making sure it works in all environments. We often put deep test focus on Security, Reliability and Performance. I sometimes think of the SET/TE’s as risk assessors whose role is to figure out the probability of finding a bug, and then working to reduce that probability. Super interesting computer science problems where we take a solid engineering approach, rather than a process oriented / manual / people intensive based approach. We always look to scale with machines wherever possible.
While Testing 1.0 is done and 1.5 is alive and well, it’s Testing 2.0 that gets me up early in the morning to start my day. Imagine if we could reinvent how we use and think about tests. What if we could automate the complex decisions on good and bad quality that humans are still so good at today? What would it look like if we had a system collecting all the “quality signals” (think: tests, production information, developer behavior, …) and could predict how good the code is today, and what it most likely will be tomorrow? That would be so awesome...
Google is working on Testing 2.0 and we’ll continue to contribute to Testing 1.0 and 1.5. Nothing is static... keep up or miss an amazing ride.
Peace.... Tony
Special thanks to Chris, Simon, Anthony, Matt, Asim, Ari, Baran, Jim, Chaitali, Rob, Emily, Kristen, Annie, and many others for providing input and suggestions for this post. 
By James Whittaker
Anything in software development that takes ten minutes or less to perform is either trivial or is not worth doing in the first place. If you take this rule of thumb at face value, where do you place test planning? Certainly it takes more than 10 minutes. In my capacity as Test Director at Google I presided over teams that wrote a large number of test plans and every time I asked how long one would take I was told “tomorrow” or “the end of the week” and a few times, early in the day, I was promised one “by the end of the day.” So I’ll establish the task of test planning to be of the hours-to-days duration.
As to whether it is worth doing, well, that is another story entirely. Every time I look at any of the dozens of test plans my teams have written, I see dead test plans. Plans written, reviewed, referred to a few times and then cast aside as the project moves in directions not documented in the plan. This begs the question: if a plan isn’t worth bothering to update, is it worth creating in the first place?
Other times a plan is discarded because it went into too much detail or too little; still others because it provided value only in starting a test effort and not in the ongoing work. Again, if this is the case, was the plan worth the cost of creating it given its limited and diminishing value?
Some test plans document simple truths that likely didn’t really need documenting at all or provide detailed information that isn’t relevant to the day to day job of a software tester. In all these cases we are wasting effort. Let’s face facts here: there is a problem with the process and content of test plans.
To combat this, I came up with a simple task for my teams: write a test plan in 10 minutes. The idea is simple, if test plans have any value at all then let’s get to that value as quickly as possible.
Given ten minutes, there is clearly no room for fluff. It is a time period so compressed that every second must be spent doing something useful or any hope you have of actually finishing the task is gone. This was the entire intent behind the exercise from my point of view: boil test planning down to only the essentials and cut all fat and fluff. Do only what is absolutely necessary and leave the details to the test executors as opposed to the test planners. If I wanted to end the practice of writing test plans that don’t stand the test of time, this seemed a worthwhile exercise.
However, I didn’t tell the people in the experiment any of this. I told them only: here is an app, create a test plan in 10 minutes or less. Remember that these people work for me and, technically, are paid to do as I tell them. And, again technically I am uniquely positioned to begin termination procedures with respect to their Google employment. On top of that I am presuming they have some measure of respect for me, which means they were likely convinced I actually thought they could do it. This was important to me. I wanted them to expect to succeed!
As preparation they could spend some time with the app in question and familiarize themselves with it. However, since many of the apps we used (Google Docs, App Engine, Talk Video, etc.) were tools they used every week, this time was short.
So here's how the task progressed:
They started, did some work and when ten minutes passed I interrupted them. They stated they weren't done yet. I responded by telling them they were out of time, nice try, here's a different problem to work on. 10 minutes later, the same thing happened and I changed the problem again. They began working faster and trying different angles, things that were too time consuming or not worth the effort got jettisoned really quick!
In each case, the teams came up with techniques that helped speed things along. They chose to jot down lists and create grids over writing long paragraphs of prose. Sentences … yes, paragraphs … no. They wasted little time on formatting and explanations and chose instead to document capabilities. Indeed, capabilities or what the software actually does, were the one commonality of all the plans. Capabilities were the one thing that all the teams gravitated toward as the most useful way to spend the little time they were given.
The three things that emerged as most important:
1. Attributes the adverbs and adjectives that describe the high level concepts testing is meant to ensure. Attributes such as fast, usable, secure, accessible and so forth.
2. Components the nouns that define the major code chunks that comprise the product. These are classes, module names and features of the application.
3. Capabilities the verbs that describe user actions and activities.
None of the teams finished the experiment in the 10 minutes allotted. However, in 10 minutes they were all able to get through both the Attributes and Components (or things that served a similar purpose) and begin documenting Capabilities. At the end of an additional 20 minutes most of the experiments had a large enough set of Capabilities that it would have been a useful starting point for creating user stories or test cases.
Which, at least to me, made the experiment a success. I gave them 10 minutes and hoped for an hour. They had 80% of the work complete in 30 minutes. And really isn’t 80% enough? We know full well that we are not going to test everything so why document everything? We know full well that as we start testing, things (schedules, requirements, architecture, etc.) are going to change so insisting on planning precision when nothing else obeys such a calling for completeness seems out of touch with reality.
80% complete in 30 minutes or less. Now that’s what I call a 10 minute test plan!
By Anthony Vallone
If you haven’t noticed, Google has been launching many public APIs recently. These APIs are empowering mobile app, desktop app, web service, and website developers by providing easy access to Google tools, services, and data. In the past couple of years, we have invested heavily in building a new API infrastructure for our APIs. Before this infrastructure, our teams had numerous technical challenges to solve when releasing an API: scalability, authorization, quota, caching, billing, client libraries, translation from external REST requests to internal RPCs, etc. The new infrastructure generically solves these problems and allows our teams to focus on their service. Automating testing of these new APIs turned out to be quite a large problem. Our solution to this problem is somewhat unique within Google, and we hope you find it interesting.
System Under Test (SUT)
Let’s start with a simplified view of the SUT design:
A developer’s application uses a Google-supplied API Client Library to call Google API methods. The library connects to the API Infrastructure service and sends the request. Part of the request defines the particular API and version being used by the client. This service is knowledgeable of all Google APIs, because they are defined by API Configuration files. These files are created by each API providing team. Configuration files declare API versions, methods, method parameters, and other API settings. Given an API request and information about the API, the API Infrastructure Service can translate the request to Google’s internal RPC format and pass it to the correct API Provider Service. This service then satisfies the request and passes the response back to the developer’s app via the API Infrastructure Service and API Client Library.
Now, the Fun Part
As of this writing, we have released 10 language-specific client libraries and 35 public APIs built on this infrastructure. Also, each of the libraries need to work on multiple platforms. Our test space has three dimensions: API (35), language (10), and platform (varies by lib). How are we going to test all the libraries on all the platforms against all the APIs when we only have two engineers on the team dedicated to test automation?
Step 1: Create a Comprehensive API
Each API uses different features of the infrastructure, and we want to ensure that every feature works. Rather than use the APIs to test our infrastructure, we create a Test API that uses every feature. In some cases where API configuration options are mutually exclusive, we have to create API versions that are feature-specific. Of course, each API team still needs to do basic integration testing with the infrastructure, but they can assume that the infrastructure features that their API depends on are well tested by the infrastructure team.
Step 2: Client Abstraction Layer in the Tests
We want to avoid creating library-specific tests, because this would lead to mass duplication of test logic. The obvious solution is to create a test library to be used by all tests as an abstraction layer hiding the various libraries and platforms. This allows us to define tests that don’t care about library or platform.
Step 3: Adapter Servers
When a test library makes an API call, it should be able to use any language and platform. We can solve this by setting up servers on each of our target platforms. For each target language, create a language-specific server. These servers receive requests from test clients. The servers need only translate test client requests into actual library calls and return the response to the caller. The code for these servers is quite simple to create and maintain.
Step 4: Iterate
Now, we have all the pieces in place. When we run our tests, they are configured to run over all supported languages and platforms against the test API:
Test Nirvana Achieved
We have a suite of straightforward tests that focus on infrastructure features. When the tests run, they are quick, reliable, and test all of our supported features, platforms, and libraries. When a feature is added to the API infrastructure, we only need to create one new test, update each adapter server to handle a new call type, and add the feature to the Test API.
ScriptCover makes Javascript coverage analysis easy
By Ekaterina Kamenskaya, Software Engineer in Test, YouTube
Today we introduce the Javascript coverage analysis tool, ScriptCover. It is a Chrome extension that provides line-by-line Javascript code coverage statistics for web pages in real time without any user modifications required. The results are collected both when the page loads and as users interact with it. The tool reports details about total web page coverage and for each external/internal script, as well as annotated code sources with individually highlighted executed lines.
Short report in Chrome extension’s popup, detailing both overall scores and per-script coverage.
Main features:
Sample of annotated source code from detailed report. First two columns are line number and number of times each instruction has been executed.
Here are the benefits of ScriptCover over other existing tools:
Want to try it out? Install ScriptCover and let us know what you think.
We envision many potential features and improvements for ScriptCover. If you are passionate about code coverage, read our documentation and participate in discussion group. Your contributions to the project’s design, code base and feature requests are welcome! 
Today we introduce the Javascript coverage analysis tool, ScriptCover. It is a Chrome extension that provides line-by-line Javascript code coverage statistics for web pages in real time without any user modifications required. The results are collected both when the page loads and as users interact with it. The tool reports details about total web page coverage and for each external/internal script, as well as annotated code sources with individually highlighted executed lines.
Short report in Chrome extension’s popup, detailing both overall scores and per-script coverage.
Main features:
- Report current and previous total Javascript coverage percentages and total number of instrumented code instructions.
- Report Javascript coverage per individual instruction for each internal and external script.
- Display detailed reports with annotated Javascript source code.
- Recalculate coverage statistics while loading the page and on user actions.
Sample of annotated source code from detailed report. First two columns are line number and number of times each instruction has been executed.
Here are the benefits of ScriptCover over other existing tools:
- Per instructions coverage for external and internal scripts: The tool formats original external and internal Javascript code from ‘<script>’ tags to ideally place one instruction per line and then calculates and displays Javascript coverage statistics. It is useful even when the code is compressed to one line.
- Dynamic: Users can get updated Javascript coverage statistics while the web page is loading and while interacting with the page.
- Easy to use: Users with different levels of expertise can install and use the tool to analyse coverage. Additionally, there is no need to write tests, modify the web application’s code, save the inspected web page locally, manually change proxy settings, etc. When the extension is activated in a Chrome browser, users just navigate through web pages and get coverage statistics on the fly.
- It’s free and open source!
Want to try it out? Install ScriptCover and let us know what you think.
We envision many potential features and improvements for ScriptCover. If you are passionate about code coverage, read our documentation and participate in discussion group. Your contributions to the project’s design, code base and feature requests are welcome!
Categories: Testing & QA
Google JS Test, now in Open Source
By Aaron Jacobs
Google JS Test is a JavaScript unit testing framework that runs on the V8 JavaScript Engine, the same open source project that is responsible for Google Chrome’s super-fast JS execution speed. Google JS Test is used internally by several Google projects, and we’re pleased to announce that it has been released as an open source project.
Features of Google JS Test include:
- Extremely fast startup and execution time, without needing to run a browser.
- Clean, readable output in the case of both passing and failing tests.
- An optional browser-based test runner that can simply be refreshed whenever JS is changed.
- Style and semantics that resemble Google Test for C++.
- A built-in mocking framework that requires minimal boilerplate code (e.g. no $tearDown or$verifyAll calls), with style and semantics based on the Google C++ Mocking Framework.
- A system of matchers allowing for expressive tests and easy to read failure output, with many built-in matchers and the ability for the user to add their own.
Categories: Testing & QA
RPF: Google's Record Playback Framework
By Jason Arbon
At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.
Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.
We were wrong, but in a good way!
We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?
During that first week of trying out RPF, Wensi found:
We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.
Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.
RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.
Please join us in maintaining BITE/RPF, and be nice to Po Hu and Joel Hynoski who are driving this work forward within Google. 
At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.
Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.
We were wrong, but in a good way!
We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?
During that first week of trying out RPF, Wensi found:
- 77% of the features in Webstore were testable by RPF
- Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
- The RPF scripts caught 6 functional regressions and many more intermittent server failures.
- Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
- RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
- Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
- Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.
We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.
Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.
RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.
Please join us in maintaining BITE/RPF, and be nice to Po Hu and Joel Hynoski who are driving this work forward within Google.
Categories: Testing & QA
Take a BITE out of Bugs and Redundant Labor
In a time when more and more of the web is becoming streamlined, the process of filing bugs for websites remains tedious and manual. Find an issue. Switch to your bug system window. Fill out boilerplate descriptions of the problem. Switch back to the browser, take a screenshot, attach it to the issue. Type some more descriptions. The whole process is one of context switching; from the tools used to file the bug, to gather information about it, to highlight problematic areas, most of your focus as the tester is pulled away from the very application you’re trying to test.
The Browser Integrated Testing Environment, or BITE, is an open source Chrome Extension which aims to fix the manual web testing experience. To use the extension, it must be linked to a server providing information about bugs and tests in your system. BITE then provides the ability to file bugs from the context of a website, using relevant templates.
When filing a bug, BITE automatically grabs screenshots, links, and problematic UI elements and attaches them to the bug. This gives developers charged with investigating and/or fixing the bug a wealth of information to help them determine root causes and factors in the behavior.
When it comes to reproducing a bug, testers will often labor to remember and accurately record the exact steps taken. With BITE, however, every action the tester takes on the page is recorded in JavaScript, and can be played back later. This enables engineers to quickly determine if the steps of a bug repro in a specific environment, or whether a code change has resolved the issue.
Also included in BITE is a Record/Playback console to automate user actions in a manual test. Like the BITE recording experience, the RPF console will automatically author javascript that can be used to replay your actions at a later date. And BITE’s record and playback mechanism is fault tolerant; UI automation tests will fail from time to time, and when they do, it tends to be for test issues, rather than product issues. To that end, when a BITE playback fails, the tester can fix their recording in real-time, just by repeating the action on the page. There’s no need to touch code, or report a failing test; if your script can’t find a button to click on, just click on it again, and the script will be fixed! For those times when you do have to touch the code, we’ve used the Ace (http://ace.ajax.org/) as an inline editor, so you can make changes to your javascript in real-time.
Check out the BITE project page at http://code.google.com/p/bite-project. Feedback is welcome at bite-feedback@google.com. Posted by Joe Allan Muharsky from the Web Testing Technologies Team (Jason Stredwick, Julie Ralph, Po Hu and Richard Bustamante are the members of the team that delivered the product). 
The Browser Integrated Testing Environment, or BITE, is an open source Chrome Extension which aims to fix the manual web testing experience. To use the extension, it must be linked to a server providing information about bugs and tests in your system. BITE then provides the ability to file bugs from the context of a website, using relevant templates.
When filing a bug, BITE automatically grabs screenshots, links, and problematic UI elements and attaches them to the bug. This gives developers charged with investigating and/or fixing the bug a wealth of information to help them determine root causes and factors in the behavior.
When it comes to reproducing a bug, testers will often labor to remember and accurately record the exact steps taken. With BITE, however, every action the tester takes on the page is recorded in JavaScript, and can be played back later. This enables engineers to quickly determine if the steps of a bug repro in a specific environment, or whether a code change has resolved the issue.
Also included in BITE is a Record/Playback console to automate user actions in a manual test. Like the BITE recording experience, the RPF console will automatically author javascript that can be used to replay your actions at a later date. And BITE’s record and playback mechanism is fault tolerant; UI automation tests will fail from time to time, and when they do, it tends to be for test issues, rather than product issues. To that end, when a BITE playback fails, the tester can fix their recording in real-time, just by repeating the action on the page. There’s no need to touch code, or report a failing test; if your script can’t find a button to click on, just click on it again, and the script will be fixed! For those times when you do have to touch the code, we’ve used the Ace (http://ace.ajax.org/) as an inline editor, so you can make changes to your javascript in real-time.
Check out the BITE project page at http://code.google.com/p/bite-project. Feedback is welcome at bite-feedback@google.com. Posted by Joe Allan Muharsky from the Web Testing Technologies Team (Jason Stredwick, Julie Ralph, Po Hu and Richard Bustamante are the members of the team that delivered the product).
Categories: Testing & QA
Unleash the QualityBots
By Richard Bustamante
Are you a website developer that wants to know if Chrome updates will break your website before they reach the stable release channel? Have you ever wished there was an easy way to compare how your website appears in all channels of Chrome? Now you can!
QualityBots is a new open source tool for web developers created by the Web Testing team at Google. It’s a comparison tool that examines web pages across different Chrome channels using pixel-based DOM analysis. As new versions of Chrome are pushed, QualityBots serves as an early warning system for breakages. Additionally, it helps developers quickly and easily understand how their pages appear across Chrome channels.
QualityBots is built on top of Google AppEngine for the frontend and Amazon EC2 for the backend workers that crawl the web pages. Using QualityBots requires an Amazon EC2 account to run the virtual machines that will crawl public web pages with different versions of Chrome. The tool provides a web frontend where users can log on and request URLs that they want to crawl, see the results from the latest run on a dashboard, and drill down to get detailed information about what elements on the page are causing the trouble.
Developers and testers can use these results to identify sites that need attention due to a high amount of change and to highlight the pages that can be safely ignored when they render identically across Chrome channels. This saves time and the need for tedious compatibility testing of sites when nothing has changed.
We hope that interested website developers will take a deeper look and even join the project at the QualityBots project page. Feedback is more than welcome at qualitybots-discuss@googlegroups.com. Posted by Ibrahim El Far, Web Testing Technologies Team (Eriel Thomas, Jason Stredwick, Richard Bustamante, and Tejas Shah are the members of the team that delivered this product) 
Are you a website developer that wants to know if Chrome updates will break your website before they reach the stable release channel? Have you ever wished there was an easy way to compare how your website appears in all channels of Chrome? Now you can!
QualityBots is a new open source tool for web developers created by the Web Testing team at Google. It’s a comparison tool that examines web pages across different Chrome channels using pixel-based DOM analysis. As new versions of Chrome are pushed, QualityBots serves as an early warning system for breakages. Additionally, it helps developers quickly and easily understand how their pages appear across Chrome channels.
QualityBots is built on top of Google AppEngine for the frontend and Amazon EC2 for the backend workers that crawl the web pages. Using QualityBots requires an Amazon EC2 account to run the virtual machines that will crawl public web pages with different versions of Chrome. The tool provides a web frontend where users can log on and request URLs that they want to crawl, see the results from the latest run on a dashboard, and drill down to get detailed information about what elements on the page are causing the trouble.
Developers and testers can use these results to identify sites that need attention due to a high amount of change and to highlight the pages that can be safely ignored when they render identically across Chrome channels. This saves time and the need for tedious compatibility testing of sites when nothing has changed.
We hope that interested website developers will take a deeper look and even join the project at the QualityBots project page. Feedback is more than welcome at qualitybots-discuss@googlegroups.com. Posted by Ibrahim El Far, Web Testing Technologies Team (Eriel Thomas, Jason Stredwick, Richard Bustamante, and Tejas Shah are the members of the team that delivered this product)
Categories: Testing & QA
Using wrapper cookbooks in Chef
Not sure if this is considered a Chef best practice or not -- I would like to get some feedback, hopefully via constructive comments on this blog post. But I've started to see this pattern when creating application-specific Chef cookbooks: take a community cookbook, include it in your own, and customize it for your specific application.
A case in point is the haproxy community cookbook. We have an application that needs to talk to a Riak cluster. After doing some research (read 'googling around') and asking people on Twitter (because Tweeps are always right), it looks like the preferred way of putting a load balancer in front of Riak is to run haproxy on each application server that needs to talk to Riak, and have haproxy listen on 127.0.0.1 on some port number, then load balance those requests to the Riak backend. Here is an example of such an haproxy.cfg file.
So what I did was to create a small cookbook called haproxy-riak, add a default.rb recipe file that just calls
include_recipe haproxy
and then customize the haproxy.cfg file via a template file in my new cookbook (I actually didn't do it via the template yet, only via a hardcoded cookbook file, but my colleague and Chef guru Jeff Roberts is working on templatizing it).
I also added
depends haproxy
to metadata.rb.
I think this is pretty much all that's needed in order to create this 'wrapper' cookbook. This cookbook can then be used by any application that needs to talk to Riak. As a matter of fact, we (i.e. Jeff) are thinking that we should have such a cookbook per application (so we would call it app1-haproxy-riak for example) just so we can do things like search for different Riak clusters that we may have for different types of applications, and populate haproxy.cfg with the search results.
In any case, I would be curious to find out if other people are using this 'pattern', or if they found other ways to apply the DRY principle in their Chef cookbooks. Please leave a comment!
A case in point is the haproxy community cookbook. We have an application that needs to talk to a Riak cluster. After doing some research (read 'googling around') and asking people on Twitter (because Tweeps are always right), it looks like the preferred way of putting a load balancer in front of Riak is to run haproxy on each application server that needs to talk to Riak, and have haproxy listen on 127.0.0.1 on some port number, then load balance those requests to the Riak backend. Here is an example of such an haproxy.cfg file.
So what I did was to create a small cookbook called haproxy-riak, add a default.rb recipe file that just calls
include_recipe haproxy
and then customize the haproxy.cfg file via a template file in my new cookbook (I actually didn't do it via the template yet, only via a hardcoded cookbook file, but my colleague and Chef guru Jeff Roberts is working on templatizing it).
I also added
depends haproxy
to metadata.rb.
I think this is pretty much all that's needed in order to create this 'wrapper' cookbook. This cookbook can then be used by any application that needs to talk to Riak. As a matter of fact, we (i.e. Jeff) are thinking that we should have such a cookbook per application (so we would call it app1-haproxy-riak for example) just so we can do things like search for different Riak clusters that we may have for different types of applications, and populate haproxy.cfg with the search results.
In any case, I would be curious to find out if other people are using this 'pattern', or if they found other ways to apply the DRY principle in their Chef cookbooks. Please leave a comment!
Testacular - Spectacular Test Runner for JavaScript
By Vojta Jína
[NOTE: After this post was published, Testacular was renamed Karma.]
“Testacular has changed my life. Now I test-drive everything.”
-- Matias Cudich, YouTube on TV team lead
At Google we believe in testing. On the AngularJS team, it’s even worse - we are super crazy about testing. Every feature of the framework is designed with testability in mind.
We found that we were struggling with existing tools, so we decided to write our own test runner. We wanted a test runner that would meet all of our needs for both quick development and continuous integration -- a truly spectacular test runner. We've called it Testacular.
Let's walk through some mental tests for what we believe makes for an ideal test runner...
it(‘should be fast’)
In order to be productive and creative you need instant feedback. Testacular watches the files in your application. Whenever you change any of them, it immediately executes the specified tests and reports the results. You never have to leave your text editor.
This enables a new way of developing. Instead of moving back and forth between the editor and the browser, you can simply stay in the editor and experiment. You instantly see the results at the command line whenever your changes are saved.
Besides that, our experience says that if test execution is slow, people don’t write tests. Testacular eliminates many barriers that keep folks from writing tests. When developers get instant feedback from their tests, the tests become an asset rather than annoyance.
it(‘should use real browsers’)
JavaScript itself is pretty consistent between different browsers, so one could potentially test browser code in non-browser environments like Node.js. Unfortunately, that’s not the case with the DOM APIs. AngularJS does a lot of DOM manipulation, and we need to be sure that it works across browsers. Executing tests on real browsers is a must.
And because Testacular communicates with browsers through a common protocols (eg. HTTP or WebSocket), you can test not only on desktop browsers but also on other devices such as mobile phones and tablets. For instance, the YouTube team uses Testacular to run continuous integration builds on PlayStation 3.
Another advantage of using real browsers is that you can use any of the tools that the browser provides. For example, you can jump into a debugger and step through your test.
it(‘should be reliable/stable’)
To be honest, most of these ideas were already implemented in JsTD almost three years ago. I think these are truly great ideas. Unfortunately, the implementation was flaky. It’s very easy to get JsTD into an inconsistent state, so you end up restarting it pretty much all day.
Testacular solves that. It can run for days, without restarting. That’s because every test run is executed in a fresh iframe. It reconnects browsers that have lost their connection to the server. And yep, it can gracefully recover from other issues, like syntax errors in the code under test.
We'd like to invite you to take Testacular for a spin. You can learn a bit more in this screencast. Please let us know what you think!
The project is open sourced and developed on GitHub.
[NOTE: After this post was published, Testacular was renamed Karma.]
“Testacular has changed my life. Now I test-drive everything.”
-- Matias Cudich, YouTube on TV team lead
At Google we believe in testing. On the AngularJS team, it’s even worse - we are super crazy about testing. Every feature of the framework is designed with testability in mind.
We found that we were struggling with existing tools, so we decided to write our own test runner. We wanted a test runner that would meet all of our needs for both quick development and continuous integration -- a truly spectacular test runner. We've called it Testacular.
Let's walk through some mental tests for what we believe makes for an ideal test runner...
it(‘should be fast’)
In order to be productive and creative you need instant feedback. Testacular watches the files in your application. Whenever you change any of them, it immediately executes the specified tests and reports the results. You never have to leave your text editor.
This enables a new way of developing. Instead of moving back and forth between the editor and the browser, you can simply stay in the editor and experiment. You instantly see the results at the command line whenever your changes are saved.
Besides that, our experience says that if test execution is slow, people don’t write tests. Testacular eliminates many barriers that keep folks from writing tests. When developers get instant feedback from their tests, the tests become an asset rather than annoyance.
it(‘should use real browsers’)
JavaScript itself is pretty consistent between different browsers, so one could potentially test browser code in non-browser environments like Node.js. Unfortunately, that’s not the case with the DOM APIs. AngularJS does a lot of DOM manipulation, and we need to be sure that it works across browsers. Executing tests on real browsers is a must.
And because Testacular communicates with browsers through a common protocols (eg. HTTP or WebSocket), you can test not only on desktop browsers but also on other devices such as mobile phones and tablets. For instance, the YouTube team uses Testacular to run continuous integration builds on PlayStation 3.
Another advantage of using real browsers is that you can use any of the tools that the browser provides. For example, you can jump into a debugger and step through your test.
it(‘should be reliable/stable’)
To be honest, most of these ideas were already implemented in JsTD almost three years ago. I think these are truly great ideas. Unfortunately, the implementation was flaky. It’s very easy to get JsTD into an inconsistent state, so you end up restarting it pretty much all day.
Testacular solves that. It can run for days, without restarting. That’s because every test run is executed in a fresh iframe. It reconnects browsers that have lost their connection to the server. And yep, it can gracefully recover from other issues, like syntax errors in the code under test.
We'd like to invite you to take Testacular for a spin. You can learn a bit more in this screencast. Please let us know what you think!
The project is open sourced and developed on GitHub.
Categories: Testing & QA
Google Test Analytics - Now in Open Source
By Jim Reardon
The test plan is dead!
Well, hopefully. At a STAR West session this past week, James Whittaker asked a group of test professionals about test plans. His first question: “How many people here write test plans?” About 80 hands shot up instantly, a vast majority of the room. “How many of you get value or refer to them again after a week?” Exactly three people raised their hands.
That’s a lot of time being spent writing documents that are often long-winded, full of paragraphs of details on a project everyone already knows to get abandoned so quickly.
A group of us at Google set about creating a methodology that can replace a test plan -- it needed to be comprehensive, quick, actionable, and have sustained value to a project. In the past few weeks, James has posted a few blogs about this methodology, which we’ve called ACC. It's a tool to break down a software product into its constituent parts, and the method by which we created "10 Minute Test Plans" (that only take 30 minutes!)
Comprehensive
The ACC methodology creates a matrix that describes your project completely; several projects that have used it internally at Google have found coverage areas that were missing in their conventional test plans.
Quick
The ACC methodology is fast; we’ve created ACC breakdowns for complex projects in under half an hour. Far faster than writing a conventional test plan.
Actionable
As part of your ACC breakdown, risk is assessed to the capabilities of your appliciation. Using these values, you get a heat map of your project, showing the areas with the highest risk -- great places to spend some quality time testing.
Sustained Value
We’ve built in some experimental features that bring your ACC test plan to life by importing data signals like bugs and test coverage that quantify the risk across your project.
Today, I'm happy to announce we're open sourcing Test Analytics, a tool built at Google to make generating an ACC simple -- and which brings some experimental ideas we had around the field of risk-based testing that work hand-in-hand with the ACC breakdown.
Defining a project’s ACC model.
Test Analytics has two main parts: first and foremost, it's a step-by-step tool to create an ACC matrix that's faster and much simpler than the Google Spreadsheets we used before the tool existed. It also provides visualizations of the matrix and risks associated with your ACC Capabilities that were difficult or impossible to do in a simple spreadsheet.
A project’s Capabilities grid.
The second part is taking the ACC plan and making it a living, automatic-updating risk matrix. Test Analytics does this by importing quality signals from your project: Bugs, Test Cases, Test Results, and Code Changes. By importing these data, Test Analytics lets you visualize risk that isn't just estimated or guessed, but based on quantitative values. If a Component or Capability in your project has had a lot of code change or many bugs are still open or not verified as working, the risk in that area is higher. Test Results can provide a mitigation to those risks -- if you run tests and import passing results, the risk in an area gets lower as you test.
A project’s risk, calculated as a factor of inherent risk as well as imported quality signals.
This part's still experimental; we're playing around with how we calculate risk based on these signals to best determine risk. However, we wanted to release this functionality early so we can get feedback from the testing community on how well it works for teams so we can iterate and make the tool even more useful. It'd also be great to import even more quality signals: code complexity, static code analysis, code coverage, external user feedback and more are all ideas we've had that could add an even higher level of dynamic data to your test plan.
An overview of test results, bugs, and code changes attributed to a project’s capability. The Capability’s total risk is affected by these factors.
You can check out a live hosted version, browse or check out the code along with documentation, and of course if you have any feedback let us know - there's a Google Group set up for discussion, where we'll be active in responding to questions and sharing our experiences with Test Analytics so far.
Long live the test plan!
Categories: Testing & QA
Hermetic Servers
By Chaitali Narla and Diego Salas
Consider a complex and rich web app. Under the hood, it is probably a maze of servers, each performing a different task and most talking to each other. Any user action navigates this server maze on its round-trip from the user to the datastores and back. A lot of Google’s web apps are like this including GMail and Google+. So how do we write end-to-end tests for them?
The “End-To-End” Test
An end-to-end test in the Google testing world is a test that exercises the entire server stack from a user request to response. Here is a simplified view of the System Under Test (SUT) that an end-to-end test would assert. Note that the frontend server in the SUT connects to a third backend which this particular user request does not need.
One of the challenges to writing a fast and reliable end-to-end test for such a system is avoiding network access. Tests involving network access are slower than their counterparts that only access local resources, and accessing external servers might lead to flakiness due to lack of determinism or unavailability of the external servers.
Hermetic Servers
One of the tricks we use at Google to design end-to-end tests is Hermetic Servers.
What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! This is a special case of the more general “hermetic” concept which applies to an isolated system not necessarily on a single machine.
Why is it useful to have a hermetic server? Because if your entire SUT is composed of hermetic servers, it could all be started on a single machine for testing; no network connection necessary! The single machine could be a physical or virtual machine.
Designing Hermetic Servers
The process for building a hermetic server starts early in the design phase of any new server. Some things we watch out for:
Using Hermetic Servers in tests
Let’s take the SUT shown earlier and assume all the servers in it are hermetic servers. Here is how an end-to-end test for the same user request would look:
The end-to-end test does the following steps:
This end-to-end test is more reliable because it uses no network connection. It is faster because everything it needs is available in-memory or in the local hard disk. We run such tests on our continuous builds, so they run at each changelist affecting any of the servers in the SUT. If the test fails, the logging module helps track where the failure occurred in the SUT.
We use hermetic servers in a lot of end-to-end tests. Some common cases include
Conclusion
Hermetic servers do have some limitations. They will increase your test’s runtime since you have to start the entire SUT each time you run the end-to-end test. If your test runs with limited resources such as memory and CPU, hermetic servers might push your test over those limits as the server interactions grow in complexity. The dataset size you can use in the in-memory datastores will be much smaller than production datastores.
Hermetic servers are a great testing tool. Like all other tools, they need to be used thoughtfully where appropriate.
Consider a complex and rich web app. Under the hood, it is probably a maze of servers, each performing a different task and most talking to each other. Any user action navigates this server maze on its round-trip from the user to the datastores and back. A lot of Google’s web apps are like this including GMail and Google+. So how do we write end-to-end tests for them?
The “End-To-End” Test
An end-to-end test in the Google testing world is a test that exercises the entire server stack from a user request to response. Here is a simplified view of the System Under Test (SUT) that an end-to-end test would assert. Note that the frontend server in the SUT connects to a third backend which this particular user request does not need.
One of the challenges to writing a fast and reliable end-to-end test for such a system is avoiding network access. Tests involving network access are slower than their counterparts that only access local resources, and accessing external servers might lead to flakiness due to lack of determinism or unavailability of the external servers.
Hermetic Servers
One of the tricks we use at Google to design end-to-end tests is Hermetic Servers.
What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! This is a special case of the more general “hermetic” concept which applies to an isolated system not necessarily on a single machine.
Why is it useful to have a hermetic server? Because if your entire SUT is composed of hermetic servers, it could all be started on a single machine for testing; no network connection necessary! The single machine could be a physical or virtual machine.
Designing Hermetic Servers
The process for building a hermetic server starts early in the design phase of any new server. Some things we watch out for:
- All connections to other servers are injected into the server at runtime using a suitable form of dependency injection such as commandline flags or Guice.
- All required static files are bundled in the server binary.
- If the server talks to a datastore, make sure the datastore can be faked with data files or in-memory implementations.
- Make sure those connection points which our test won’t exercise have appropriate fakes or mocks to verify this non-interaction.
- Provide modules to easily populate datastores with test data.
- Provide logging modules that can help trace the request/response path as it passes through the SUT.
Using Hermetic Servers in tests
Let’s take the SUT shown earlier and assume all the servers in it are hermetic servers. Here is how an end-to-end test for the same user request would look:
The end-to-end test does the following steps:
- starts the entire SUT as shown in the diagram on a single machine
- makes requests to the server via the test client
- validates responses from the server
This end-to-end test is more reliable because it uses no network connection. It is faster because everything it needs is available in-memory or in the local hard disk. We run such tests on our continuous builds, so they run at each changelist affecting any of the servers in the SUT. If the test fails, the logging module helps track where the failure occurred in the SUT.
We use hermetic servers in a lot of end-to-end tests. Some common cases include
- Startup tests for servers using Guice to verify that there are no Guice errors on startup.
- API tests for backend servers.
- Micro-benchmark performance tests.
- UI and API tests for frontend servers.
Conclusion
Hermetic servers do have some limitations. They will increase your test’s runtime since you have to start the entire SUT each time you run the end-to-end test. If your test runs with limited resources such as memory and CPU, hermetic servers might push your test over those limits as the server interactions grow in complexity. The dataset size you can use in the in-memory datastores will be much smaller than production datastores.
Hermetic servers are a great testing tool. Like all other tools, they need to be used thoughtfully where appropriate.
Categories: Testing & QA
Test Engineers @ Google
By Yvette Nameth
At Google, we’re very big into highlighting individuals’ strengths and using them to make teams and products better. However, we frequently get asked “What do Test Engineers (aka TEs) do?” I pause when I get this question since it’s hard to speak for my peers - I test Google Maps rendering, which is just one small portion of what Google’s Test Engineers test.
In order to get a clearer picture of what Test Engineers are responsible for, I chatted with three of my colleagues. We were able to identify the underlying Test Engineers’ similarities, while highlighting the differences.
So what common themes do Test Engineers specialize in at Google?
We’re product experts:
Test Engineers need to become a “go-to” person for how their product works and integrates with other Google products. (You aren’t expected to have this before working with a product, but you need to figure out how to become one on any product you work on!) TEs need to understand use cases and contracts with other services, products, and features. We aren’t expected to write unit tests for other engineer’s code; instead we ensure product quality on the functional and integration aspects of the product.
We’re flexible:
Test Engineers are required to switch tasks and re-prioritize frequently. From unplanned catastrophes, to shifting launch calendars, to people asking us questions, our work is filled with interrupts. We determine how to ensure quality in the face of the interrupts.
We also modify our tests based on the pace of the development and understand that there is no one right way to test a product. Test Engineers adapt tools to meet their needs and understand when a tool just can’t get the job done.
We’re clear communicators:
We have to be able to communicate via test plans, design docs, bugs, email and code. Every day we work with a wide variety of people in different roles: Software Engineers, Software Engineers in Test, Product Managers, Usability Researchers, Designers, Legal Counsel, etc. We need to address these different audiences to make sure we’re either gathering the information that will help us build better strategies or presenting feedback that will help influence the product.
We’re good at coordination:
We are people who use our “in between the product and user” status to coordinate integration testing efforts between products. We may coordinate manual testing efforts by our manual testers; or we may make sure that test gaps are being addressed by “someone” (Test Engineer, Software Engineer in Test, or Software Engineer). We put our product knowledge and communication together with a bit of coordination and make sure that bugs are looked at and the product is getting tested hourly / daily.
We have impact:
Google Test Engineers have big impact. We hold responsibility thinking of ways that our products could fail in “real scenarios”; and then we add tests to make sure that the worst won’t come to pass.
How big is this? Well, in my case, I’m responsible for making sure Google Maps represents a map that is useful to my relatives in the middle of rural Montana as well as my friends living in London, Paris or Sydney. When you add to that the billions of other users in different regions, speaking different languages and using the map for different reasons, I know that my testing is impacting their ability to get around and find out information about the physical world around them safely.
We code:
The other most common question is “Do you write code?” The answer is yes; Test Engineers at Google do code.
The three aspects that generally differentiate what a Test Engineer does day-to-day depend on the following:
Individual’s Strengths & Interests:
Everyone is different and every TE has different passions, strengths and areas of expertise. Thankfully, Google’s a big enough company that many different areas of testing are available, and we gravitate to testing products we like. All TEs start with core competencies in testing, coding, and algorithms. How a TE applies this knowledge varies.
The Type of Product:
Desktop, web app or mobile? Frontend or backend? The technologies that our products use and run on create a lot of variation in what and how we test.
The Product’s History / Lifecycle:
Early concept products don’t resemble those that exist in production. And the amount of testing that a product already has will determine what testing the TE is focused on. We work creating a test roadmap that parallels the product’s development cycle and addresses any testing gaps.
If you still want to know what the day in the life of a Test Engineer entails, we’ll never be able to give you a general answer for that. Instead I suggest that you check out what Alan Faulkner is doing or ask the next Google Test Engineer you meet.
Interested in joining the ranks of Test Engineers (or Software Engineers in Test)? Check out http://goo.gl/2RDKj
About the Contributors:
Albert Drona has been at Google for 5 years and is currently working on Google Maps for Mobile.
Jatin Shah has been at Google for 9 months on Google+.
Mohammad Khan has been at Google for 7 years and is currently working on Google+ releases.
About the Author:
Yvette Nameth has been at Google for 5 years and is currently working on Google Maps rendering.
At Google, we’re very big into highlighting individuals’ strengths and using them to make teams and products better. However, we frequently get asked “What do Test Engineers (aka TEs) do?” I pause when I get this question since it’s hard to speak for my peers - I test Google Maps rendering, which is just one small portion of what Google’s Test Engineers test.
In order to get a clearer picture of what Test Engineers are responsible for, I chatted with three of my colleagues. We were able to identify the underlying Test Engineers’ similarities, while highlighting the differences.
So what common themes do Test Engineers specialize in at Google?
We’re product experts:
Test Engineers need to become a “go-to” person for how their product works and integrates with other Google products. (You aren’t expected to have this before working with a product, but you need to figure out how to become one on any product you work on!) TEs need to understand use cases and contracts with other services, products, and features. We aren’t expected to write unit tests for other engineer’s code; instead we ensure product quality on the functional and integration aspects of the product.
We’re flexible:
Test Engineers are required to switch tasks and re-prioritize frequently. From unplanned catastrophes, to shifting launch calendars, to people asking us questions, our work is filled with interrupts. We determine how to ensure quality in the face of the interrupts.
We also modify our tests based on the pace of the development and understand that there is no one right way to test a product. Test Engineers adapt tools to meet their needs and understand when a tool just can’t get the job done.
We’re clear communicators:
We have to be able to communicate via test plans, design docs, bugs, email and code. Every day we work with a wide variety of people in different roles: Software Engineers, Software Engineers in Test, Product Managers, Usability Researchers, Designers, Legal Counsel, etc. We need to address these different audiences to make sure we’re either gathering the information that will help us build better strategies or presenting feedback that will help influence the product.
We’re good at coordination:
We are people who use our “in between the product and user” status to coordinate integration testing efforts between products. We may coordinate manual testing efforts by our manual testers; or we may make sure that test gaps are being addressed by “someone” (Test Engineer, Software Engineer in Test, or Software Engineer). We put our product knowledge and communication together with a bit of coordination and make sure that bugs are looked at and the product is getting tested hourly / daily.
We have impact:
Google Test Engineers have big impact. We hold responsibility thinking of ways that our products could fail in “real scenarios”; and then we add tests to make sure that the worst won’t come to pass.
How big is this? Well, in my case, I’m responsible for making sure Google Maps represents a map that is useful to my relatives in the middle of rural Montana as well as my friends living in London, Paris or Sydney. When you add to that the billions of other users in different regions, speaking different languages and using the map for different reasons, I know that my testing is impacting their ability to get around and find out information about the physical world around them safely.
We code:
The other most common question is “Do you write code?” The answer is yes; Test Engineers at Google do code.
The three aspects that generally differentiate what a Test Engineer does day-to-day depend on the following:
Individual’s Strengths & Interests:
Everyone is different and every TE has different passions, strengths and areas of expertise. Thankfully, Google’s a big enough company that many different areas of testing are available, and we gravitate to testing products we like. All TEs start with core competencies in testing, coding, and algorithms. How a TE applies this knowledge varies.
The Type of Product:
Desktop, web app or mobile? Frontend or backend? The technologies that our products use and run on create a lot of variation in what and how we test.
The Product’s History / Lifecycle:
Early concept products don’t resemble those that exist in production. And the amount of testing that a product already has will determine what testing the TE is focused on. We work creating a test roadmap that parallels the product’s development cycle and addresses any testing gaps.
If you still want to know what the day in the life of a Test Engineer entails, we’ll never be able to give you a general answer for that. Instead I suggest that you check out what Alan Faulkner is doing or ask the next Google Test Engineer you meet.
Interested in joining the ranks of Test Engineers (or Software Engineers in Test)? Check out http://goo.gl/2RDKj
About the Contributors:
Albert Drona has been at Google for 5 years and is currently working on Google Maps for Mobile.
Jatin Shah has been at Google for 9 months on Google+.
Mohammad Khan has been at Google for 7 years and is currently working on Google+ releases.
About the Author:
Yvette Nameth has been at Google for 5 years and is currently working on Google Maps rendering.
Categories: Testing & QA
Conversation with a Test Engineer
By Alan Myrvold
Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused on the migration component of Bid Manager, which transitions advertiser information from Invite Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office.
Are you a Test Engineer, or a Software Engineer in Test, and what’s the difference?
Right now, I’m a Test Engineer, but the two roles can be very similar. As a Test Engineer, you’re more focused on the overall quality of the product and speed of releases, while a Software Engineer in Test might focus more on test frameworks, automation, and refactoring code for testability. I think of the difference as more of a shift in focus and not capabilities, since both roles at Google need to be able to write production quality code. Example test engineering tasks I worked on are introducing an automated release process, identifying areas for the team to improve code coverage, and reducing the manual steps needed to validate data correctness.
What is a typical day for you?
When I get in, I look at any code reviews I need to respond to, look for any production bugs from technical account managers that are high priority, and then start writing code. In my current role, I focus my development effort on improving the efficiency and coverage of our large scale integration tests and frameworks. I also work on adding additional features to our product that improve our testability. I typically spend anywhere from 50% to 75% of my time either writing code or participating in code reviews.
Do you write only test code?
No, I write a lot of code that is included in the product as well. One of the great things about being an SET or TE at Google is that you can write product code as easily as test code. I write both. My test code focuses on improving test frameworks and enabling developers to write integration tests. The production code that I write focuses on increasing the verification of external inputs. I also focus on adding features that improve testability. This pushes more quality features into the product itself rather than relying on test code for correctness.
What programming languages do you use?
Both the test and product code are mostly Java. Occasionally I use Python or C++ too.
How much time to do you spend doing manual testing?
Right now, with the role I am in, I spend less than 5% of my time doing manual testing. Although some exploratory testing helps develop product knowledge and find risky areas, it doesn’t scale as a repeated process. There are a small amount of manual steps and I focus on ways to help reduce this so our team does not spend our time doing repeated manual steps as part of our data migration.
Do you write unit tests for code that isn’t yours?
At Google, the responsibility of testing is shared across all product engineers, not just Test Engineers. Everyone is responsible for writing unit tests as well as integration tests for their components. That being said, I have written unit tests for components that are outside of what I developed but that has been to illustrate how to write a unit test for said component. This component usually involved a abnormally complex set of code or to illustrate using a new mocking framework, such as Mockito.
What do you like about working on the Google advertising products?
I like the challenges of the scalability problems we need to solve, from handling massive amounts of data to dealing with lots of real time ad requests that need to be responded to in milliseconds. I also like the impact, since the products affect a lot of people. It’s rewarding to work on stuff like that.
How is testing at Google different from your experience at other companies?
I feel the role is more flexible at Google. There are fewer SET’s and TE’s in my group at Google per developer, and you have the flexibility to pick what is most important. For example, I get to write a lot of production code to fix bugs, make the code more testable, and increasing the visibility into errors encountered during our data migrations. Plus, developers at Google spend a lot of time writing tests, so testing isn’t just my responsibility.
How does the Google Kirkland office differ from the headquarters in Mountain View?
What I really like about the offices at Google is that each of them has their own local feel and personality. Google encourages this! For instance, the office here in Kirkland has a climbing wall, boats and all the conference rooms in our building are named after local bands in the area. The office in Seattle has kayaks and the New York office has an actual food truck in its cafeteria.
What’s the future of testing at Google?
I think the future is really bright. We have a lot of flexibility to make a big impact on quality, testability and improving our release velocity. We need to release new features faster and with good quality. The problems that we face are complex and at an extreme scale. We need engineering teams focused on ensuring that we have efficient ways to simulate and test. There will always be a need for testers and developers that focus on these areas here at Google.
Interested in test jobs at Google? 
Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused on the migration component of Bid Manager, which transitions advertiser information from Invite Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office.
Are you a Test Engineer, or a Software Engineer in Test, and what’s the difference?Right now, I’m a Test Engineer, but the two roles can be very similar. As a Test Engineer, you’re more focused on the overall quality of the product and speed of releases, while a Software Engineer in Test might focus more on test frameworks, automation, and refactoring code for testability. I think of the difference as more of a shift in focus and not capabilities, since both roles at Google need to be able to write production quality code. Example test engineering tasks I worked on are introducing an automated release process, identifying areas for the team to improve code coverage, and reducing the manual steps needed to validate data correctness.
What is a typical day for you?
When I get in, I look at any code reviews I need to respond to, look for any production bugs from technical account managers that are high priority, and then start writing code. In my current role, I focus my development effort on improving the efficiency and coverage of our large scale integration tests and frameworks. I also work on adding additional features to our product that improve our testability. I typically spend anywhere from 50% to 75% of my time either writing code or participating in code reviews.
Do you write only test code?
No, I write a lot of code that is included in the product as well. One of the great things about being an SET or TE at Google is that you can write product code as easily as test code. I write both. My test code focuses on improving test frameworks and enabling developers to write integration tests. The production code that I write focuses on increasing the verification of external inputs. I also focus on adding features that improve testability. This pushes more quality features into the product itself rather than relying on test code for correctness.
What programming languages do you use?
Both the test and product code are mostly Java. Occasionally I use Python or C++ too.
How much time to do you spend doing manual testing?
Right now, with the role I am in, I spend less than 5% of my time doing manual testing. Although some exploratory testing helps develop product knowledge and find risky areas, it doesn’t scale as a repeated process. There are a small amount of manual steps and I focus on ways to help reduce this so our team does not spend our time doing repeated manual steps as part of our data migration.
Do you write unit tests for code that isn’t yours?
At Google, the responsibility of testing is shared across all product engineers, not just Test Engineers. Everyone is responsible for writing unit tests as well as integration tests for their components. That being said, I have written unit tests for components that are outside of what I developed but that has been to illustrate how to write a unit test for said component. This component usually involved a abnormally complex set of code or to illustrate using a new mocking framework, such as Mockito.
What do you like about working on the Google advertising products?
I like the challenges of the scalability problems we need to solve, from handling massive amounts of data to dealing with lots of real time ad requests that need to be responded to in milliseconds. I also like the impact, since the products affect a lot of people. It’s rewarding to work on stuff like that.
How is testing at Google different from your experience at other companies?
I feel the role is more flexible at Google. There are fewer SET’s and TE’s in my group at Google per developer, and you have the flexibility to pick what is most important. For example, I get to write a lot of production code to fix bugs, make the code more testable, and increasing the visibility into errors encountered during our data migrations. Plus, developers at Google spend a lot of time writing tests, so testing isn’t just my responsibility.
How does the Google Kirkland office differ from the headquarters in Mountain View?
What I really like about the offices at Google is that each of them has their own local feel and personality. Google encourages this! For instance, the office here in Kirkland has a climbing wall, boats and all the conference rooms in our building are named after local bands in the area. The office in Seattle has kayaks and the New York office has an actual food truck in its cafeteria.
What’s the future of testing at Google?
I think the future is really bright. We have a lot of flexibility to make a big impact on quality, testability and improving our release velocity. We need to release new features faster and with good quality. The problems that we face are complex and at an extreme scale. We need engineering teams focused on ensuring that we have efficient ways to simulate and test. There will always be a need for testers and developers that focus on these areas here at Google.
Interested in test jobs at Google?
Categories: Testing & QA
A Virtual Battlefield... with Bugs: Ship Wars @ Google Kirkland
By Anthony F. Voellm (aka Tony the @p3rfguy / G+) and Emily Bedont
On Wednesday, October 24th, while sitting under the Solar System, 30 software engineers from the Greater Seattle area came together at Google Kirkland to partake in the first ever Test Edition of Ship Wars. Ship Wars was created by two Google Waterloo engineers, Garret Kelly and Aaron Kemp, as a 20% project. Yes, 20% time does exist at Google! The object of the game is to code a spaceship that will outperform all others in a virtual universe - algorithm vs algorithm.
The Kirkland event marked the 7th iteration of the program which was also recently done in NYC. Kirkland however was the first time that the game had been customized to encourage exploratory testing. In the case of "Ship Wars the Test Edition," we planted 4 bugs that the engineering participants were awarded for finding. Well, we ran out of prizes and were quickly reminded that when you put a lot of testing minded people in a room, many bugs will be unveiled! One of the best unveiled bugs was not one of the four planted in the simulator. When turning your ship 90 degrees, the ship actually turned -90 degrees. Oops!
Participants were encouraged to test their spaceship built on their own machine or a Google Chromebook. While the coding was done in the browser, the simulator and web server were run on Google Compute Engine. Throughout the 90 minutes, people challenged other participants to duels. Head-to-head battles took place on Chromebooks at the front of the room. There were many accolades called out but in the end, there could only be one champion who would walk away with a brand spankin’ new Nexus7. Check out our video of the evening’s activities.
Sounds fun, huh? We sure hope our participants, including our first place winner shown receiving the Nexus 7 from Garret, enjoyed the evening! Beyond the battles, our guests were introduced to the revived Google Testing Blog, heard firsthand that GTAC will be back in 2013, learned about testing at Google, and interacted with Googlers in a "Googley" environment. Achievement unlocked.
Special thanks to all the Googlers that supported the event!
On Wednesday, October 24th, while sitting under the Solar System, 30 software engineers from the Greater Seattle area came together at Google Kirkland to partake in the first ever Test Edition of Ship Wars. Ship Wars was created by two Google Waterloo engineers, Garret Kelly and Aaron Kemp, as a 20% project. Yes, 20% time does exist at Google! The object of the game is to code a spaceship that will outperform all others in a virtual universe - algorithm vs algorithm.
The Kirkland event marked the 7th iteration of the program which was also recently done in NYC. Kirkland however was the first time that the game had been customized to encourage exploratory testing. In the case of "Ship Wars the Test Edition," we planted 4 bugs that the engineering participants were awarded for finding. Well, we ran out of prizes and were quickly reminded that when you put a lot of testing minded people in a room, many bugs will be unveiled! One of the best unveiled bugs was not one of the four planted in the simulator. When turning your ship 90 degrees, the ship actually turned -90 degrees. Oops!
Participants were encouraged to test their spaceship built on their own machine or a Google Chromebook. While the coding was done in the browser, the simulator and web server were run on Google Compute Engine. Throughout the 90 minutes, people challenged other participants to duels. Head-to-head battles took place on Chromebooks at the front of the room. There were many accolades called out but in the end, there could only be one champion who would walk away with a brand spankin’ new Nexus7. Check out our video of the evening’s activities.
Sounds fun, huh? We sure hope our participants, including our first place winner shown receiving the Nexus 7 from Garret, enjoyed the evening! Beyond the battles, our guests were introduced to the revived Google Testing Blog, heard firsthand that GTAC will be back in 2013, learned about testing at Google, and interacted with Googlers in a "Googley" environment. Achievement unlocked.
Special thanks to all the Googlers that supported the event!
Categories: Testing & QA
Testing 2.0
By Anthony F. Voellm (aka Tony the perfguy)
It’s amazing what has happened in the field of test in the last 20 years... a lot of “art” has turned into “science”. Computer scientists, engineers, and many other disciplines have worked on provable systems and calculus, pioneered model based testing, invented security fuzz testing, and even settled on a common pattern for unit tests called xunit. The xunit pattern shows up in open source software like JUnit as well as in Microsoft development test tools.
With all this innovation in test, there’s no wonder test is dead. The situation is no different from the late 1800’s when patents were declared dead. Everything had been invented. So now that everything in test has been invented, it’s dead.
Well... if you believe everything in test has been invented then please stop reading now :)
As an aside: “Test is dead” was a keynote at the Google Test Automation Conference (GTAC) in 2011. You can watch that talk and many other GTAC test talks on YouTube, and I definitely recommend you check them out here. Talks span a wide range of topics ranging from GUI Automation to Cloud.
What really excites me these days is that we have closed a chapter on test. A lot of the foundation of writing and testing great software has been laid (examples at the beginning of the post, tools like Webdriver for UI, FIO for storage, and much more), which I think of as Testing 1.0. We all use Testing 1.0 day in and day out. In fact at Google, most of the developers (called Software Engineers or SWEs) do the basic Testing 1.0 work and we have a high bar on quality. Knuth once said "Be careful about using the following code -- I've only proven that it works, I haven't tested it."
This brings us to the current chapter in test which I call Testing 1.5. This chapter is being written by computer scientists, applied scientists, engineers, developers, statisticians, and many other disciplines. These people come together in the Software Engineer in Test (SET) and Test Engineer (TE) roles at Google. SET/TEs focus on; developing software faster, building it better the first time, testing it in depth, releasing it quicker, and making sure it works in all environments. We often put deep test focus on Security, Reliability and Performance. I sometimes think of the SET/TE’s as risk assessors whose role is to figure out the probability of finding a bug, and then working to reduce that probability. Super interesting computer science problems where we take a solid engineering approach, rather than a process oriented / manual / people intensive based approach. We always look to scale with machines wherever possible.
While Testing 1.0 is done and 1.5 is alive and well, it’s Testing 2.0 that gets me up early in the morning to start my day. Imagine if we could reinvent how we use and think about tests. What if we could automate the complex decisions on good and bad quality that humans are still so good at today? What would it look like if we had a system collecting all the “quality signals” (think: tests, production information, developer behavior, …) and could predict how good the code is today, and what it most likely will be tomorrow? That would be so awesome...
Google is working on Testing 2.0 and we’ll continue to contribute to Testing 1.0 and 1.5. Nothing is static... keep up or miss an amazing ride.
Peace.... Tony
Special thanks to Chris, Simon, Anthony, Matt, Asim, Ari, Baran, Jim, Chaitali, Rob, Emily, Kristen, Annie, and many others for providing input and suggestions for this post.
Categories: Testing & QA
Why Are There So Many C++ Testing Frameworks?
By Zhanyong Wan - Software Engineer
These days, it seems that everyone is rolling their own C++ testing framework, if they haven't done so already. Wikipedia has a partial list of such frameworks. This is interesting because many OOP languages have only one or two major frameworks. For example, most Java people seem happy with either JUnit or TestNG. Are C++ programmers the do-it-yourself kind?
When we started working on Google Test (Google’s C++ testing framework), and especially after we open-sourced it, people began asking us why we were doing it. The short answer is that we couldn’t find an existing C++ testing framework that satisfied all our needs. This doesn't mean that these frameworks were all poorly designed or implemented. Rather, many of them had great ideas and tricks that we learned from. However, Google had a huge number of C++ projects that got compiled on various operating systems (Linux, Windows, Mac OS X, and later Android, among others) with different compilers and all kinds of compiler flags, and we needed a framework that worked well in all these environments and ccould handle many different types and sizes of projects.
Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.
To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will fail to compile. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects (in case you are curious, Google Test doesn’t require exceptions or run-time type identification by default; when these language features are turned on, Google Test will try to take advantage of them and provide you with more utilities, like the exception assertions.).
Why not just write a portable framework, then? Indeed, that's a top design goal for Google Test. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.
So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments, making it hard to write portable C++ code. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.
Another reason is that some limitations of C++ make it impossible to implement certain features really well, and different people chose different ways to workaround the limitations. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some frameworks require you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some disagree. Also, there are different ways to devise the macros and they involve different trade-offs, so the result is not clear cut.
Let’s see some actual code to understand how Google Test solves the test registration problem. The simplest way to add a test is to use the TEST macro (what else would we name it?):
TEST(Subject, HasCertainProperty) {
… testing code goes here …
}
This defines a test method whose purpose is to verify that the given subject has the given property. The macro automatically registers the test with Google Test such that it will be run when the test program (which may contain many such TEST definitions) is executed.
Here’s a more concrete example that verifies a Factorial() function works as expected for positive arguments:
TEST(FactorialTest, HandlesPositiveInput) {
EXPECT_EQ(1, Factorial(1));
EXPECT_EQ(2, Factorial(2));
EXPECT_EQ(6, Factorial(3));
EXPECT_EQ(40320, Factorial(8));
}
Finally, many C++ testing framework authors neglected extensibility and were happy just providing canned solutions, so we ended up with many solutions, each satisfying a different niche but none general enough. A versatile framework must have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing. Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test, we try to make it easy to expand your testing vocabulary by defining custom assertions that generate informative error messages. For instance, here’s a naive way to verify that an int value is in a given range:
bool IsInRange(int value, int low, int high) {
return low <= value && value <= high;
}
...
EXPECT_TRUE(IsInRange(SomeFunction(), low, high));
The problem is that when the assertion fails, you only know that the value returned by SomeFunction() is not in range [low, high], but you have no idea what that return value and the range actually are -- this makes debugging the test failure harder.
You could provide a custom message to make the failure more descriptive:
EXPECT_TRUE(IsInRange(SomeFunction(), low, high))
<< "SomeFunction() = " << SomeFunction()
<< ", not in range ["
<< low << ", " << high << "]";
Except that this is incorrect as SomeFunction() may return a different answer each time. You can fix that by introducing an intermediate variable to hold the function’s result:
int result = SomeFunction();
EXPECT_TRUE(IsInRange(result, low, high))
<< "result (return value of SomeFunction()) = " << result
<< ", not in range [" << low << ", " << high << "]";
However this is tedious and obscures what you are really trying to do. It’s not a good pattern when you need to do the “is in range” check repeatedly. What we need here is a way to abstract this pattern into a reusable construct.
Google Test lets you define a test predicate like this:
AssertionResult IsInRange(int value, int low, int high) {
if (value < low)
return AssertionFailure()
<< value << " < lower bound " << low;
else if (value > high)
return AssertionFailure()
<< value << " > upper bound " << high;
else
return AssertionSuccess()
<< value << " is in range ["
<< low << ", " << high << "]";
}
Then the statement EXPECT_TRUE(IsInRange(SomeFunction(), low, high)) may print (assuming that SomeFunction() returns 13):
Value of: IsInRange(SomeFunction(), low, high)
Actual: false (13 < lower bound 20)
Expected: true
The same IsInRange() definition also lets you use it in an EXPECT_FALSE context, e.g. EXPECT_FALSE(IsInRange(AnotherFunction(), low, high)) could print:
Value of: IsInRange(AnotherFunction(), low, high)
Actual: true (25 is in range [20, 60])
Expected: false
This way, you can build a library of test predicates for your problem domain, and benefit from clear, declarative test code and descriptive failure messages.
In the same vein, Google Mock (our C++ mocking framework) allows you to easily define matchers that can be used exactly the same way as built-in matchers. Also, we have included an event listener API in Google Test for people to write plug-ins. We hope that people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful.
Perhaps one day we will solve the C++ testing framework fragmentation problem, after all. :-)
These days, it seems that everyone is rolling their own C++ testing framework, if they haven't done so already. Wikipedia has a partial list of such frameworks. This is interesting because many OOP languages have only one or two major frameworks. For example, most Java people seem happy with either JUnit or TestNG. Are C++ programmers the do-it-yourself kind?
When we started working on Google Test (Google’s C++ testing framework), and especially after we open-sourced it, people began asking us why we were doing it. The short answer is that we couldn’t find an existing C++ testing framework that satisfied all our needs. This doesn't mean that these frameworks were all poorly designed or implemented. Rather, many of them had great ideas and tricks that we learned from. However, Google had a huge number of C++ projects that got compiled on various operating systems (Linux, Windows, Mac OS X, and later Android, among others) with different compilers and all kinds of compiler flags, and we needed a framework that worked well in all these environments and ccould handle many different types and sizes of projects.
Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.
To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will fail to compile. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects (in case you are curious, Google Test doesn’t require exceptions or run-time type identification by default; when these language features are turned on, Google Test will try to take advantage of them and provide you with more utilities, like the exception assertions.).
Why not just write a portable framework, then? Indeed, that's a top design goal for Google Test. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.
So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments, making it hard to write portable C++ code. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.
Another reason is that some limitations of C++ make it impossible to implement certain features really well, and different people chose different ways to workaround the limitations. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some frameworks require you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some disagree. Also, there are different ways to devise the macros and they involve different trade-offs, so the result is not clear cut.
Let’s see some actual code to understand how Google Test solves the test registration problem. The simplest way to add a test is to use the TEST macro (what else would we name it?):
TEST(Subject, HasCertainProperty) {
… testing code goes here …
}
This defines a test method whose purpose is to verify that the given subject has the given property. The macro automatically registers the test with Google Test such that it will be run when the test program (which may contain many such TEST definitions) is executed.
Here’s a more concrete example that verifies a Factorial() function works as expected for positive arguments:
TEST(FactorialTest, HandlesPositiveInput) {
EXPECT_EQ(1, Factorial(1));
EXPECT_EQ(2, Factorial(2));
EXPECT_EQ(6, Factorial(3));
EXPECT_EQ(40320, Factorial(8));
}
Finally, many C++ testing framework authors neglected extensibility and were happy just providing canned solutions, so we ended up with many solutions, each satisfying a different niche but none general enough. A versatile framework must have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing. Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test, we try to make it easy to expand your testing vocabulary by defining custom assertions that generate informative error messages. For instance, here’s a naive way to verify that an int value is in a given range:
bool IsInRange(int value, int low, int high) {
return low <= value && value <= high;
}
...
EXPECT_TRUE(IsInRange(SomeFunction(), low, high));
The problem is that when the assertion fails, you only know that the value returned by SomeFunction() is not in range [low, high], but you have no idea what that return value and the range actually are -- this makes debugging the test failure harder.
You could provide a custom message to make the failure more descriptive:
EXPECT_TRUE(IsInRange(SomeFunction(), low, high))
<< "SomeFunction() = " << SomeFunction()
<< ", not in range ["
<< low << ", " << high << "]";
Except that this is incorrect as SomeFunction() may return a different answer each time. You can fix that by introducing an intermediate variable to hold the function’s result:
int result = SomeFunction();
EXPECT_TRUE(IsInRange(result, low, high))
<< "result (return value of SomeFunction()) = " << result
<< ", not in range [" << low << ", " << high << "]";
However this is tedious and obscures what you are really trying to do. It’s not a good pattern when you need to do the “is in range” check repeatedly. What we need here is a way to abstract this pattern into a reusable construct.
Google Test lets you define a test predicate like this:
AssertionResult IsInRange(int value, int low, int high) {
if (value < low)
return AssertionFailure()
<< value << " < lower bound " << low;
else if (value > high)
return AssertionFailure()
<< value << " > upper bound " << high;
else
return AssertionSuccess()
<< value << " is in range ["
<< low << ", " << high << "]";
}
Then the statement EXPECT_TRUE(IsInRange(SomeFunction(), low, high)) may print (assuming that SomeFunction() returns 13):
Value of: IsInRange(SomeFunction(), low, high)
Actual: false (13 < lower bound 20)
Expected: true
The same IsInRange() definition also lets you use it in an EXPECT_FALSE context, e.g. EXPECT_FALSE(IsInRange(AnotherFunction(), low, high)) could print:
Value of: IsInRange(AnotherFunction(), low, high)
Actual: true (25 is in range [20, 60])
Expected: false
This way, you can build a library of test predicates for your problem domain, and benefit from clear, declarative test code and descriptive failure messages.
In the same vein, Google Mock (our C++ mocking framework) allows you to easily define matchers that can be used exactly the same way as built-in matchers. Also, we have included an event listener API in Google Test for people to write plug-ins. We hope that people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful.
Perhaps one day we will solve the C++ testing framework fragmentation problem, after all. :-)
Categories: Testing & QA
The Economics of Car Rental
We’re flying back to the UK for a week later this month, so once again we’re forced to rent a car. Both sets of parents live too far from public transport for it to be a realistic option, even assuming the trains in the UK break the habit of a lifetime and work for a change. (Rail replacement bus service? What sadistic bastard thought of that?)
It got me thinking about the outrageous cost of hiring a car for the week. To get a car large enough for the three of us and all our luggage (who knew we needed to take a kitchen sink for a 1 week trip?), it will cost us over £200 for the week. I could practically buy a car more cost effectively than that… if only there was some way of everyone clubbing together to buy a car that you can drive when you need it. Scaled up, you’d have… a car rental company. Are they just ripping us off? Or does it plausibly cost that much to hire cars out? I made up some numbers to see if it would shed any light on the situation.
New price £24,000.00 Price @ 3 yrs £12,000.00 Annual running costs £2,000.00 Cost over 3 yrs £18,000.00A fairly low-spec, family sized car would cost about £24,000 to buy brand new. Assuming you kept it for 3 years, and allowing for insurance and up-keep etc, I reckon you’d need to make about £6,000 a year to cover it. Is that a lot?
Annual cost £6,000.00 Average utilisation 50% Min daily cost £32.97Assuming the car is being used 50% of the time, you’d need to charge about £30/day to make your £6,000 a year back. Which works out about the same as I’m actually paying – so it’s probably not a million miles wide of the mark. Which is slightly depressing – unless you buy an older car, and risk spending more money on maintaining it and repairing it… it’s difficult to see how you could rent a car for less than they currently charge.
Although I’m hiring a car for the week, I don’t actually need the car for 7 full days of 24 hours. Maximum, I’ll probably be in the car for 12 hours across that week. If only there was a way that I could rent a car for an hour at a time? (Zipcar and the like only serve big cities, which is useless when you’re visiting the sticks)
Self-Driving CarsWell, maybe this is where self-driving cars could massively change things. With a self-driving car, it wouldn’t matter that I’m parking up in the middle of nowhere. It can scoot itself off into the nearest city to continue ferrying people about. I’d only need to pay for the time I actually use it.
Annual cost £6,000.00 Average utilisation 25% Min hourly cost £2.75Obviously the utilisation would be lower (I can’t imagine many people will need carrying about in the middle of the night). But instead of paying £230 for a week’s car parking, I could instead pay £33 for 12 hours of driving. In theory, the car would still be paid for over three years – but the cost to individual users would be massively lower.
Not Just for HolidaysAt those prices, could it even replace my main car?
Annual mileage 12000 miles Average monthly mileage 1000 miles Average speed 40 mph Average monthly time 25 hours Average monthly cost £68.68If I was paying for the exact same car, to spend the bulk of it’s time parked outside my house, it would cost £500 per month. But renting it by the hour is a fraction of that. Of course, if everyone did this, you’d need a massive number of cars to cover the peak demand at rush hour – but the utilisation could drop to 5% and it would still be cheaper than owning the car outright. That’s each car in use for just over 1 hour each day, given that’s the average UK daily commute that seems easily achievable.
So, it’s 2013, I only have one question:
Where’s my goddamned self driving car?
Categories: Programming, Testing & QA
The 10 Minute Test Plan
By James Whittaker
Anything in software development that takes ten minutes or less to perform is either trivial or is not worth doing in the first place. If you take this rule of thumb at face value, where do you place test planning? Certainly it takes more than 10 minutes. In my capacity as Test Director at Google I presided over teams that wrote a large number of test plans and every time I asked how long one would take I was told “tomorrow” or “the end of the week” and a few times, early in the day, I was promised one “by the end of the day.” So I’ll establish the task of test planning to be of the hours-to-days duration.
As to whether it is worth doing, well, that is another story entirely. Every time I look at any of the dozens of test plans my teams have written, I see dead test plans. Plans written, reviewed, referred to a few times and then cast aside as the project moves in directions not documented in the plan. This begs the question: if a plan isn’t worth bothering to update, is it worth creating in the first place?
Other times a plan is discarded because it went into too much detail or too little; still others because it provided value only in starting a test effort and not in the ongoing work. Again, if this is the case, was the plan worth the cost of creating it given its limited and diminishing value?
Some test plans document simple truths that likely didn’t really need documenting at all or provide detailed information that isn’t relevant to the day to day job of a software tester. In all these cases we are wasting effort. Let’s face facts here: there is a problem with the process and content of test plans.
To combat this, I came up with a simple task for my teams: write a test plan in 10 minutes. The idea is simple, if test plans have any value at all then let’s get to that value as quickly as possible.
Given ten minutes, there is clearly no room for fluff. It is a time period so compressed that every second must be spent doing something useful or any hope you have of actually finishing the task is gone. This was the entire intent behind the exercise from my point of view: boil test planning down to only the essentials and cut all fat and fluff. Do only what is absolutely necessary and leave the details to the test executors as opposed to the test planners. If I wanted to end the practice of writing test plans that don’t stand the test of time, this seemed a worthwhile exercise.
However, I didn’t tell the people in the experiment any of this. I told them only: here is an app, create a test plan in 10 minutes or less. Remember that these people work for me and, technically, are paid to do as I tell them. And, again technically I am uniquely positioned to begin termination procedures with respect to their Google employment. On top of that I am presuming they have some measure of respect for me, which means they were likely convinced I actually thought they could do it. This was important to me. I wanted them to expect to succeed!
As preparation they could spend some time with the app in question and familiarize themselves with it. However, since many of the apps we used (Google Docs, App Engine, Talk Video, etc.) were tools they used every week, this time was short.
So here's how the task progressed:
They started, did some work and when ten minutes passed I interrupted them. They stated they weren't done yet. I responded by telling them they were out of time, nice try, here's a different problem to work on. 10 minutes later, the same thing happened and I changed the problem again. They began working faster and trying different angles, things that were too time consuming or not worth the effort got jettisoned really quick!
In each case, the teams came up with techniques that helped speed things along. They chose to jot down lists and create grids over writing long paragraphs of prose. Sentences … yes, paragraphs … no. They wasted little time on formatting and explanations and chose instead to document capabilities. Indeed, capabilities or what the software actually does, were the one commonality of all the plans. Capabilities were the one thing that all the teams gravitated toward as the most useful way to spend the little time they were given.
The three things that emerged as most important:
1. Attributes the adverbs and adjectives that describe the high level concepts testing is meant to ensure. Attributes such as fast, usable, secure, accessible and so forth.
2. Components the nouns that define the major code chunks that comprise the product. These are classes, module names and features of the application.
3. Capabilities the verbs that describe user actions and activities.
None of the teams finished the experiment in the 10 minutes allotted. However, in 10 minutes they were all able to get through both the Attributes and Components (or things that served a similar purpose) and begin documenting Capabilities. At the end of an additional 20 minutes most of the experiments had a large enough set of Capabilities that it would have been a useful starting point for creating user stories or test cases.
Which, at least to me, made the experiment a success. I gave them 10 minutes and hoped for an hour. They had 80% of the work complete in 30 minutes. And really isn’t 80% enough? We know full well that we are not going to test everything so why document everything? We know full well that as we start testing, things (schedules, requirements, architecture, etc.) are going to change so insisting on planning precision when nothing else obeys such a calling for completeness seems out of touch with reality.
80% complete in 30 minutes or less. Now that’s what I call a 10 minute test plan!
Categories: Testing & QA
Google Developer Day 2011
By James Whittaker
Google Developer Day is gearing up for a fantastic fall season of tours that crawl the continents. And a surprise this year ... yours truly will be the keynote for the Developer Day in Sao Paulo Brazil and Buenos Aires Argentina in September.
Google Developer Day is a deep dive into the future of Web, Mobile and Cloud technologies crafted specifically for software engineering professionals. And this year we are adding the element of Social to tie it all together. Google+ is only the start.
If you are attending, please stop by and say hello!
Click here for more information about dates and agenda. 
Google Developer Day is gearing up for a fantastic fall season of tours that crawl the continents. And a surprise this year ... yours truly will be the keynote for the Developer Day in Sao Paulo Brazil and Buenos Aires Argentina in September.
Google Developer Day is a deep dive into the future of Web, Mobile and Cloud technologies crafted specifically for software engineering professionals. And this year we are adding the element of Social to tie it all together. Google+ is only the start.
If you are attending, please stop by and say hello!
Click here for more information about dates and agenda.
Categories: Testing & QA
Testing Google's New API Infrastructure
By Anthony Vallone
If you haven’t noticed, Google has been launching many public APIs recently. These APIs are empowering mobile app, desktop app, web service, and website developers by providing easy access to Google tools, services, and data. In the past couple of years, we have invested heavily in building a new API infrastructure for our APIs. Before this infrastructure, our teams had numerous technical challenges to solve when releasing an API: scalability, authorization, quota, caching, billing, client libraries, translation from external REST requests to internal RPCs, etc. The new infrastructure generically solves these problems and allows our teams to focus on their service. Automating testing of these new APIs turned out to be quite a large problem. Our solution to this problem is somewhat unique within Google, and we hope you find it interesting.
System Under Test (SUT)
Let’s start with a simplified view of the SUT design:
A developer’s application uses a Google-supplied API Client Library to call Google API methods. The library connects to the API Infrastructure service and sends the request. Part of the request defines the particular API and version being used by the client. This service is knowledgeable of all Google APIs, because they are defined by API Configuration files. These files are created by each API providing team. Configuration files declare API versions, methods, method parameters, and other API settings. Given an API request and information about the API, the API Infrastructure Service can translate the request to Google’s internal RPC format and pass it to the correct API Provider Service. This service then satisfies the request and passes the response back to the developer’s app via the API Infrastructure Service and API Client Library.
Now, the Fun Part
As of this writing, we have released 10 language-specific client libraries and 35 public APIs built on this infrastructure. Also, each of the libraries need to work on multiple platforms. Our test space has three dimensions: API (35), language (10), and platform (varies by lib). How are we going to test all the libraries on all the platforms against all the APIs when we only have two engineers on the team dedicated to test automation?
Step 1: Create a Comprehensive API
Each API uses different features of the infrastructure, and we want to ensure that every feature works. Rather than use the APIs to test our infrastructure, we create a Test API that uses every feature. In some cases where API configuration options are mutually exclusive, we have to create API versions that are feature-specific. Of course, each API team still needs to do basic integration testing with the infrastructure, but they can assume that the infrastructure features that their API depends on are well tested by the infrastructure team.
Step 2: Client Abstraction Layer in the Tests
We want to avoid creating library-specific tests, because this would lead to mass duplication of test logic. The obvious solution is to create a test library to be used by all tests as an abstraction layer hiding the various libraries and platforms. This allows us to define tests that don’t care about library or platform.
Step 3: Adapter Servers
When a test library makes an API call, it should be able to use any language and platform. We can solve this by setting up servers on each of our target platforms. For each target language, create a language-specific server. These servers receive requests from test clients. The servers need only translate test client requests into actual library calls and return the response to the caller. The code for these servers is quite simple to create and maintain.
Step 4: Iterate
Now, we have all the pieces in place. When we run our tests, they are configured to run over all supported languages and platforms against the test API:
Test Nirvana Achieved
We have a suite of straightforward tests that focus on infrastructure features. When the tests run, they are quick, reliable, and test all of our supported features, platforms, and libraries. When a feature is added to the API infrastructure, we only need to create one new test, update each adapter server to handle a new call type, and add the feature to the Test API.
Categories: Testing & QA
Welcome to the Next Generation of Google Testing
By Anthony Vallone
Wow... it has been a long time since we’ve posted to the blog. This past year has been a whirlwind of change for many test teams as Google has restructured leadership with a focus on products. Now that the dust has settled, our teams are leaner, more focused, and more effective. We have learned quite a bit over the past year about how best to tackle and manage test problems at monumental scale. The next generation of test teams at Google are looking forward to sharing all that we have learned. Stay tuned for a revived Google Testing Blog that will provide deep insight into our latest testing technologies and strategies.
Wow... it has been a long time since we’ve posted to the blog. This past year has been a whirlwind of change for many test teams as Google has restructured leadership with a focus on products. Now that the dust has settled, our teams are leaner, more focused, and more effective. We have learned quite a bit over the past year about how best to tackle and manage test problems at monumental scale. The next generation of test teams at Google are looking forward to sharing all that we have learned. Stay tuned for a revived Google Testing Blog that will provide deep insight into our latest testing technologies and strategies.
Categories: Testing & QA
Be Strongly Agile: Show Your Weaknesses
I am frequently amused when the same people who believe that it is not possible to get the requirements for a product correct up front, and therefore want to use Scrum for development, will in the next breath explain how they aren’t quite ready to start using Scrum because they haven’t worked out all of the details of their Scrum approach! This type of thinking is poorly aligned with fundamental Scrum principles.
When employing Scrum, you shouldn’t worry about getting things perfect up front. You can’t! And trying to be perfect ...