By James Whittaker

Possessing a bill of materials means that we understand the overall size of the testing problem. Unfortunately, the size of most testing problems far outstrips any reasonable level of effort to solve them. And not all of the testing surface is equally important. There are certain features that simple require more testing than others. Some prioritization must take place. What components must get tested? What features simply cannot fail? What features make up the user scenarios that simply must work?

In our experience it is the unfortunate case that no one really agrees on the answers to these questions. Talk to product planners and you may get a different assessment than if you talk to developers, sales people or executive visionaries. Even users may differ among themselves. It falls on testers to act as the user advocates and find out how to take into account all these concerns to prioritize how testing resources will be distributed across the entire testing surface.

The term commonly used for this practice is risk analysis and at Google we take information from all the projects stakeholders to come up with overall numerical risk scores for each feature. How do we get all the stakeholders involved? That's actually the easy part. All you need to do is assign numbers and then step back and have everyone tell you how wrong you are. We've found being visibly wrong is the best way to get people involved in the hopes they can influence getting the numbers right! Right now we are collecting this information in spreadsheets. By the time GTAC rolls around the tool we are using for this should be in a demonstrable form.

I blogged a while ago about MySQL fault-tolerance and disaster recovery techniques. At that time I was experimenting with the non-free InnoDB Hot Backup product. In the mean time I discovered Percona's XtraBackup (thanks Robin!). Here's how I tested XtraBackup for doing a hot backup and a restore of a MySQL database running Percona XtraDB (XtraBackup works with vanilla InnoDB too).

First of all, I use the following Percona .deb packages on a 64-bit Ubuntu Lucid EC2 instance:


# dpkg -l | grep percona
ii  libpercona-xtradb-client-dev      5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database development files
ii  libpercona-xtradb-client16        5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database client library
ii  percona-xtradb-client-5.1         5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database client binaries
ii  percona-xtradb-common             5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database common files (e.g. /etc
ii  percona-xtradb-server-5.1         5.1.43-xtradb-1.0.6-9.1-60.jaunty.11 Percona SQL database server binaries

I tried using the latest stable XtraBackup .deb package from the Percona downloads site but it didn't work for me. I started a hot backup with /usr/bin/innobackupex-1.5.1 and it ran for a while before dying with "InnoDB: Operating system error number 9 in a file operation." See this bug report for more details.
After unsuccessfully trying to compile XtraBackup from source, I tried XtraBackup-1.3-beta for Lucid from the Percona downloads. This worked fine.
Here's the scenario I tested against a MySQL Percona XtraDB instance running with DATADIR=/var/lib/mysql/m10 and a customized configuration file /etc/mysql10/my.cnf. I created and attached an EBS volume which I mounted as /xtrabackup on the instance running MySQL.
1) Take a hot backup of all databases under that instance:
/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --pasword=xxxxxx /xtrabackup
This will take a while and will create a timestamped directory under /xtrabackup, where it will store the database files from DATADIR. Note that the InnoDB log files are not created unless you apply step 2 below.
As the documentation says, make sure the output of innobackupex-1.5.1 ends with:
100901 05:33:12 innobackupex-1.5.1: completed OK!

2) Apply the transaction logs to the datafiles just created, so that the InnoDB logfiles are recreated in the target directory:

/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --password=xxxxxx --apply-log /xtrabackup/2010-09-01_05-21-36/

At this point, I tested a disaster recovery scenario by stopping MySQL and moving all files in DATADIR to a different location.

To bring the databases back to normal from the XtraBackup hot backup, I did the following:

1) Brought back up a functioning MySQL instance to be used by the XtraBackup restore operation:

i) Copied the contents of the default /var/lib/mysql/mysql database under /var/lib/mysql/m10/ (or you can recreate the mysql DB from scratch)

ii) Started mysqld_safe manually:

 mysqld_safe --defaults-file=/etc/mysql10/my.cnf

This will create the data files and logs under DATADIR (/var/lib/mysql/m10) with the sizes specified in the configuration file. I had to wait until the messages in /var/log/syslog told me that the MySQL instance is ready and listening for connections.

2) Copied back the files from the hot backup directory into DATADIR

Note that the copy-back operation below initially errored out because it tried to copy the mysql directory too, and it found the directory already there under DATADIR. So the 2nd time I ran it, I moved /var/lib/mysql/m10/mysql to mysql.bak. The copy-back command is:

/usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql10/my.cnf --user=root --copy-back /xtrabackup/2010-09-01_05-21-36/

You can also copy the files from  /xtrabackup/2010-09-01_05-21-36/ into DATADIR using vanilla cp.

3) If everything went well in step 2, restart the MySQL instance to make sure everything is OK.

At this point, your MySQL instance should have its databases restored to the point where you took the hot backup. If that instance is used in replication, you will most likely need to adjust the master_log_file and master_log_position so that it gets back in sync with its master.

Note that XtraBackup can also run in a 'stream' mode useful for compressing the files generated by the backup operation. Details in the documentation.

I’ll be traveling and training this fall. Take note of where I’ll be, because if I come near where you are and want some relatively free consulting, all you have to do is take me to dinner.

I’ll do almost anything for free food.

September

• Reston, Virginia
• San Diego, California

October

• Stockholm, Sweden
• Tartu, Estonia
• Cluj-Napoca, Romania
• Bucharest, Romania

(The event in Cluj-Napoca is a one-day public seminar that introduces my Rapid Testing methodology through a series of puzzle challenges and lecture. I will also take any and all questions about testing. Click here to sign up for that.)

November

• Tallinn, Estonia

December

• England
• (tentatively) Singapore

(I have one class to teach in England. I’m interested in doing something more, if there is any interest. Email me.)

My full schedule is published here.

First of all, I want to emphasize that this is NOT a disaster recovery strategy I recommend. However, in a pinch, it might save your ass. Here's the scenario I have:

• 2 m1.large instances running Ubuntu 10.04 64-bit and the Percona XtraDB MySQL builds (for the record, the exact version I'm using is "Server version: 5.1.43-60.jaunty.11-log (Percona SQL Server (GPL), XtraDB 9.1, Revision 60")
• I'll call the 2 servers db101 and db201
• each server is running 2 MySQL instances -- I'll call them m1 and m2
• instance m1 on db101 and instance m1 on db201 are set up in master-master replication (and similar for instance m2)
• the DATADIR for m1 is /var/lib/mysql/m1 on each server; that file system is mounted from an EBS volume (and similar for m2)
• the configuration files for m1 are in /etc/mysql1 on each server -- that directory was initially a copy of the Ubuntu /etc/mysql configuration directory, which I then customized (and similar for m2)
• the init.d script for m1 is in /etc/init.d/mysql1 (similar for m2)

What I tested:

• I took a snapshot of each of the 2 EBS volumes associated with each of the DB servers (4 snapshots in all)
• I terminated the 2 m1.large instances
• I launched 2 m1.xlarge instances and installed the same Percona distribution (this was done via a Chef recipe at instance launch time); I'll call the 2 new instances xdb101 and xdb102
• I pushed the configuration files for m1 and m2, as well as the init.d scripts (this was done via fabric)
• I created new volumes from the EBS snapshots (note that these volumes can be created in any EC2 availability zone)
• On xdb101, I attached the 2 volumes created from the EBS snapshots on db101; I specified /dev/sdm and /dev/sdn as the device names (similar on xdb201)
• On xdb101, I created /var/lib/mysql/m1 and mounted /dev/sdm there; I also created /var/lib/mysql/m2 and mounted /dev/sdn there (similar on xdb201)
• At this point, the DATADIR directories for both m1 and m2 are populated with 'live files' from the moment when I took the EBS snapshot
• I made sure syslog-ng accepts UDP traffic from localhost (by default it doesn't); this is because by default in Ubuntu mysql log messages are sent to syslog --> to do this, I ensured that "udp(ip(127.0.0.1) port(514));" appears in the "source s_all" entry in /etc/syslog-ng/syslog-ng.conf

At this point, I started up the first MySQL instance on xdb101 via "/etc/init.d/mysql1 start". This script most likely will show [fail] on the console, because MySQL will not start up normally. If you look in /var/log/syslog, you'll see entries similar to:
Aug 31 18:03:21 xdb101 mysqld: 100831 18:03:21 [Note] Plugin 'FEDERATED' is disabled.Aug 31 18:03:21 xdb101 mysqld: InnoDB: The InnoDB memory heap is disabledAug 31 18:03:21 xdb101 mysqld: InnoDB: Mutexes and rw_locks use GCC atomic builtinsAug 31 18:03:22 xdb101 mysqld: 100831 18:03:22  InnoDB: highest supported file format is Barracuda.Aug 31 18:03:23 xdb101 mysqld: InnoDB: The log sequence number in ibdata files does not matchAug 31 18:03:23 xdb101 mysqld: InnoDB: the log sequence number in the ib_logfiles!Aug 31 18:03:23 xdb101 mysqld: 100831 18:03:23  InnoDB: Database was not shut down normally!Aug 31 18:03:23 xdb101 mysqld: InnoDB: Starting crash recovery.
If you wait a bit longer (and if you're lucky), you'll see entries similar to:
Aug 31 18:04:20 xdb101 mysqld: InnoDB: Restoring possible half-written data pages from the doublewriteAug 31 18:04:20 xdb101 mysqld: InnoDB: buffer...Aug 31 18:04:24 xdb101 mysqld: InnoDB: In a MySQL replication slave the last master binlog fileAug 31 18:04:24 xdb101 mysqld: InnoDB: position 0 15200672, file name mysql-bin.000015Aug 31 18:04:24 xdb101 mysqld: InnoDB: and relay log fileAug 31 18:04:24 xdb101 mysqld: InnoDB: position 0 15200817, file name ./mysqld-relay-bin.000042Aug 31 18:04:24 xdb101 mysqld: InnoDB: Last MySQL binlog file position 0 17490532, file name /var/lib/mysql/m1/mysql-bin.000002Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 InnoDB Plugin 1.0.6-9.1 started; log sequence number 1844705956Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Recovering after a crash using /var/lib/mysql/m1/mysql-binAug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Starting crash recovery...Aug 31 18:04:24 xdb101 mysqld: 100831 18:04:24 [Note] Crash recovery finished.
At this point, you can do "/etc/init.d/mysql1 restart" just to make sure that both stopping and starting that instance work as expected. Repeat for instance m2, and also repeat on server xdb201.
So....IF you are lucky and the InnoDB crash recovery process did its job, you should have 2 functional MySQL instances one each of xdb101 and xdb201. I tested this with several pairs of servers and it worked for me every time, but I hasten to say that YMMV, so DO NOT bet on this as your disaster recovery strategy!
At this point I still had to re-establish the master-master replication between m1 on xdb101 and m1 on xdb201 (and similar for m2). 
When I initially set up this replication between the original m1.large servers, I used something like this on both db101 and db201:
CHANGE MASTER TO MASTER_HOST='master1', MASTER_PORT=3306, MASTER_USER='masteruser', MASTER_PASSWORD='xxxxxx';"
The trick for me is that master1 points to db201 in db101's /etc/hosts, and vice-versa.
On the newly created xdb101 and xdb201, there are no entries for master1 in /etc/hosts, so replication is broken. Which is a good thing initially, because you want to have the MySQL instances on each server be brought back up without throwing replication into the mix.
Once I added an entry for master1 in xdb101's /etc/hosts pointing to xdb201, and did the same on xdb201, I did a 'stop slave; start slave; show slave status\G' on the m1 instance on each server. In all cases I tested, one of the slaves was showing everything OK, while the other one was complaining about   not being able to read from the master's log file. This was fairly simply to fix. Let's assume xdb101 is the one complaining. I did the following:

• on xdb201, I ran 'show master status\G' and noted the file name (for example "mysql-bin.000017") and the file position (for example 106)
• on xdb101, I ran the following command: "stop slave; change master to master_log_file='mysql-bin.000017', master_log_pos=106; start slave;"
• not a 'show slave status\G' on xdb101 should show everything back to normal

Some lessons:

• take periodic snapshots of your EBS volumes (at least 1/day)
• for a true disaster recovery strategy, use at least mysqldump to dump your DB to disk periodically, or something more advanced such as Percona XtraBackup; I recommend dumping the DB to an EBS volume and taking periodic snapshots of that volume
• the procedure I detailed above is handy when you want to grow your instance 'vertically' -- for example I went from m1.large to m1.xlarge

By James Whittaker

When are you finished testing? It’s the age old quality question and one that has never been adequately answered (other than the unhelpful answer of never). I argue it never will be answered until we have a definition of the size of the testing problem. How can you know you are finished if you don’t fully understand the task at hand?

Answers that deal with coverage of inputs or coverage of code are unhelpful. Testers can apply every input and cover every line of code in test cases and still the software can have very serious bugs. In fact, it’s actually likely to have serious bugs because inputs and code cannot be easily associated with what’s important in the software. What we need is a way to identify what parts of the product can be tested, a bill of materials if you will, and then map our actual testing back to each part so that we can measure progress against the overall testing goal.

This bill of materials represents everything that can be tested. We need it in a format that can be compared with actual testing so we know which parts have received enough testing and which parts are suspect.

We have a candidate format for this bill of materials we are experimenting with at Google and will be unveiling at GTAC this year.

Most of the testing world is managed around artifacts: test cases, test documents, bug reports. If you look at any “test management” tool, you’ll see that the artifact-based approach permeates it. “Test” for many people is a noun.

For me test is a verb. Testing is something that I do, not so much something that I create. Testing is the act of exploration of an unknown territory. It is casting questions, like Molotov cocktails, into the darkness, where they splatter and burst into bright revealing fire.

How to Manage Such a Process?

My brother Jon and I created a way to control highly exploratory testing 10 years ago, called session-based test management (SBTM). I recently returned from an intense testing project in Israel, where I used SBTM. But I also experimented with a new idea: thread-based test management (TTM).

Like many of my new ideas, it’s not really new. It’s the christening (with words) and sharpening (with analysis) of something many of us already do. The idea is this: organize management around threads of activity rather than test sessions or artifacts.

Thread-based testing is a generalized form of session-based testing, in that sessions are a form of thread, but a thread is not necessarily a session. In SBTM, you test in uninterrupted blocks of time that each have a charter. A charter is a mission for that session; a light sort of commitment, or contract. A thread, on the other hand, may be interrupted, it may go on and on indefinitely, and does not imply a commitment. Session-based testing can be seen as an extension of thread-based testing for especially high accountability and more orderly situations.

I define a thread as a set of one or more activities intended to solve a problem or achieve an objective. You could think of a thread as a very small project within a project.

Why Thread-Based Test Management?

Because it can work under even the most chaotic and difficult conditions. The only formalism required for TBTM is a list of threads. I use this form of test management when I am dropped into a project with as little a day or two to get it done.

What Does Thread-Based Test Management Looks Like?

It’s simple. Thread-based test management looks like a todo list, except that we organize the todo items into an outline that matches the structure of the testing process. Here’s a mocked-up example:

Test Facilities

• Power meter calibration method
• Backup test jig validation
• Create standard test images

Test Strategy

• Accuracy Testing
  • Sampling strategy
  • Preliminary-testing
  • Log file analysis program
• Transaction Flow Testing
• Essential Performance Testing
• Safety Testing
  • warnings and errors FRS review
  • tool for forcing errors
• Compliance Testing
• Test Protocol V1.0 doc.

Test Management

• Change protocol definition
• Build protocol definition
• Test cycle protocol definition
• Bug reporting protocol definition
• Bug triage
• Fix verifications

This outline describes the high level threads that comprise the test project. I typically use a mind-mapping program like MindManager to organize and present them.

So, you should be thinking, “Is that it? Todo lists?” right about now. Well, no. That’s not it. But that’s one face of it.

What Else Does Thread-Based Test Management Look Like?

It looks like testers gathered around a todo list, talking about what they are going to work on that afternoon. Then they split up and go to work. Several times day they might come together like that. If the team is not co-located, then this meeting is done over instant messaging, email, or perhaps through a wiki.

Is That All it Looks Like?

Well, there is also the status report. Whether written or spoken, the thread-based test management version of a status report lists the threads, who is working on the threads, and the outlook for each thread. It typically also includes an issues list.

Other documentation may be produced, of course. TBTM doesn’t tell you what documents to create. It simply tells you that threads are the organizing principle we use for managing the project.

Where Do Threads Come From?

Threads are first spawned from our model of the testing problem. The Satisfice Heuristic Test Strategy Model is an example of such a model. By working through those lists, we get an idea of the kinds of testing we might want to do: those are the first of the threads. After that, threads might be created in many ways, including splitting off of existing threads as we gain a deeper understanding of what testing needs to be done. Of course, in an Agile environment, each user story kicks off a new testing thread.

Which Threads Do We Work On?

Think priority and progress. We might frequently drop threads, switch threads, and pick them up again. In general, we work on the highest priority threads, but we also work on lower priority threads many times, when we see the possibility for quick and inexpensive progress. If I’m trying to finish a sanity check on the new build, I might interrupt that to discuss the status of a particular known bug if the developer happens to wander by.

Major ongoing threads often become attached to specific people. For instance “client testing” or “performance testing” often become full-time jobs. Testing itself, after all, can be thought of as a thread so challenging to do well, and so different from programming, that most companies have seen fit to hire dedicated testers.

How Do Threads End?

A thread ends either in a cut or knot. Cutting a thread means to cancel that task. A knot, however, is a milestone; an achievement of some kind. This is exactly the meaning of the phrase “tying up the loose ends” and marks either the end of the thread (or group of threads) or a good place to drop it for a while.

How Do We Estimate Work?

In thread-based test management, there is no special provision or method for estimating work, except that this is done on a thread-by-thread basis. Session-based test management may be overlaid onto TBTM in order to estimate work in terms of sessions.

How Do We Evaluate Progress?

In thread-based test management, there is no special provision or method for evaluating progress, either, except that this is done on a thread-by-thread basis, and status reports may be provided frequently, perhaps at the end of each day. Session-based test management is also helpful for that.

So What?

This form of management is actually quite common. But, to my knowledge, no one has yet named and codified it. Without a convenient way to talk about it, we have a hard time explaining and justifying it. Then when the “process improvement” freaks come along, they act like there’s no management happening at all. This form of management has been “illegible” up to now (meaning that it’s there but no one notices it) and my brother and I are going to push to make it fully legible and respectable in the testing arena.

From now on, when asked about my approach to test management, I can say “I practice Rapid Testing methodology, which I track in either a thread-based or session-based manner, depending on the stage of the project, and in a risk-based manner at all times.”

How is TBTM Any Different From Using a TO-DO List?

Michel Kraaij questions the substance of TBTM by wondering how it’s different from the age-old idea of a “to-do” list? See his post here.

This is a good question. Yes, TBTM is different than just using a to-do list, but even so, I don’t think I’ve ever read an article about to-do list based test management (TDBTM?). Most textbooks focus on artifacts, not the activity of testing. Thread-based test management is trying to capture the essence of managing with to-do lists, plus some other things in addition to that.

The main additional element, beyond just making a to-do list, is that a traditional to-do list contains items that can be “done”, whereas many threads might not ever be “done.” They might be cut (abandoned) or knotted (temporarily parked at some level of completion). Some threads maybe tied up with a bow and “done” like a normal task, but not the main ones that I’m thinking of. As I practice testing, for instance, I’m rarely “done” with test strategy. I tinker with the test strategy all the way through the project. That’s why it makes sense to call it a thread.

Another thing to recognize is that the main concern of TBTM is how to know what to put on your thread list. The answer to that invokes the entire framework of Rapid Software testing. So, yeah, it’s more than having an outline of threads, which does look very much like a to-do list– it’s the activity (and skills) of making the list and managing it. If you want to talk about to-do list based test management, then you would have to invent that lore as well. You couldn’t just say “make a to-do list” and claim to have communicated the methodology.

[You can find Jonathan's take on TBTM here.]

[I credit Sagi Krupetski, the test lead on my recent project, for helping me get this idea. His clockwork status reporting and regular morning question "Where are we on the project and what do you think you need to work on today?" caused me to see the thread structure of our project clearly for the first time. He's back on the market now (Chicago area), in case you need a great tester or test manager.]

In his book “Succeeding with Agile”, Mike Cohn present nine questions that you should ask for a current or proposed team. Questions should be asked iteratively… until you answer “yes” to each. Here are the questions:
* Does the structure accentuate the strengths, shore up the weaknesses, and support the motivations of the team members?
* Does the structure minimize the number of people required to be on two teams (and avoid having anyone on three)?
* Does the structure maximize the amount of time that teams will remain together?
* Are component teams used only in limited and easily justifiable cases?
* Will you be able to feed most teams with two pizzas?
* Does the structure minimize the number of communication paths between teams?
* Does the structure encourage teams to communicate who wouldn’t otherwise do so?
* Does the design support a clear understanding of accountability?
* Did team members have input into the design of the team?

Besides the cultural bias (in Italy, one pizza will usually feed one person but it might be different in the USA…), you could use these questions for every project that you are currently running or plan to run.

Reference: “Succeeding with Agile”, Mike Cohn, Addison-Wesley, 262 pages, IBSN 978-0-321-57936-2

Get more details on this book or buy it on amazon.com
Get more details on this book or buy it on amazon.co.uk

Here is a list of software development related conferences and events that will take place in September and that have media partnerships with Methods & Tools:

* Mobile Application Stores, September 7 2010, Zurich, Switerland

* iqnite 2010 Schweiz, September 21 2010, Zurich, Switzerland

* Lean & Kanban 2010 Europe, September 23-24 2010, Antwerp, Belgium

* Software Testing Analysis & Review Conference, September 26—October 1 2010, San Diego, USA

* iPhone/iPad DevCon, September 27-29 2010, San Diego, USA

* iqnite Nordic, September 29-30 2010, Stockholm, Sweden

* iqnite United Kingdom, October 4 2010, London, UK

* Agile Eastern Europe, October 8-9, Kyiv, Ukraine

Find more software development conferences

Our software development tools directory has now categorized more than 2000 tools. From project management and unit testing tools to NoSQL databases, you can find tools used in every software development activity, as the 10′000 monthly visitors do, searching by programming language, running platforms or another of our classification criteria.

If you are the committer of an open source project or the marketing manager of a commercial tool, do not hesitate to add your tool. It is free and you will be able to post press releases to communicate about new versions.

Munin is a great tool for resource visualization. Sometimes though installing a 3rd party Munin plugin is not as straightforward as you would like. I have been struggling a bit with one such plugin, munin-mysql, so I thought I'd spell it out for my future reference. My particular scenario is running multiple MySQL instances on various port numbers (3306 and up) on the same machine. I wanted to graph in particular the various InnoDB metrics that munin-mysql supports. I installed the plugin on various Ubuntu flavors such as Jaunty and Lucid.

Here are the steps:

1) Install 2 pre-requisite Perl modules for munin-mysql: IPC-ShareLite and ache-Cache

2) git clone http://github.com/kjellm/munin-mysql

3) cd munin-mysql; edit Makefile and point PLUGIN_DIR to the directory where your munin plugins reside (if you installed Munin on Ubuntu via apt-get, that directory is /usr/share/munin/plugins)

4) make install --> this will copy the mysql_ Perl script to PLUGIN_DIR, and the mysql_.conf file to /etc/munin/plugin-conf.d

5) Edit /etc/munin/plugin-conf.d/mysql_.conf and customize it with your specific MySQL information.

For example, if you run 2 MySQL instances on ports 3306 and 3307, you could have something like this in mysql_.conf:


[mysql_3306_*]
env.mysqlconnection DBI:mysql:mysql;host=127.0.0.1;port=3306
env.mysqluser myuser1
env.mysqlpassword mypassword1

[mysql_3307_*]
env.mysqlconnection DBI:mysql:mysql;host=127.0.0.1;port=3307
env.mysqluser myuser2
env.mysqlpassword mypassword2

6) Run "/usr/share/munin/plugins/mysql_ suggest" to see what metrics are supported by the plugin. Then proceed to create symlinks in /etc/munin/plugins, adding the port number and the metric name as the suffix.

For example, to track InnoDB I/O metrics for the MySQL instance running on port 3306, you would create this symlink:

ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_3306_innodb_io

(replace 3306 with 3307 to track this metric for the other MySQL instance running on port 3307)

Of course, it's easy to automate this by a simple shell script.

7) Restart munin-node and wait 10-15 minutes for the munin master to receive the information about the new metrics.

Important! If you need to troubleshoot this plugin (and any Munin plugin), do not make the mistake of simply running the plugin script directly in the shell. If you do this, it will not read the configuration file(s) correctly, and it will most probably fail. Instead, what you need to do is to follow the "Debugging Munin plugins" documentation, and run the plugin through the munin-run utility. For example:


# munin-run mysql_3306_innodb_io
ib_io_read.value 34
ib_io_write.value 57870
ib_io_log.value 8325
ib_io_fsync.value 55476

One more thing: you should probably automate all these above steps. I have most of it automated via a fabric script. The only thing I do by hand is to create the appropriate symlinks for the specific port numbers I have on each server.
That's it! Enjoy staring for hours at your brand new MySQL metrics!

By James Whittaker

Each year, about this time, we say goodbye to our summer interns and bid them success in the upcoming school year. Every year they come knowing very little about testing and leave, hopefully, knowing much more. This is not yet-another-plea to universities to teach more testing, instead it is a reflection on how we teach ourselves.

I like to experiment with metaphors that help people "get it." From attacks to tools to tours to the apocalypse, I've seen my fair share. This summer, I got a lot of aha moments from various interns and new hires likening testing to cooking. We're chefs with no recipes, just a list of ingredients. We may all end up making a different version of Testing Cake, but we better at least be using the same set of ingredients.

What are the ingredients? I'll list them here over the next couple of weeks. Please feel free to add your own and I'll hope you don't steal my thunder by getting them in faster than I. Right now I have a list of 7.

Ingredient 1: Product expertise

Developers grow trees, testers manage forests. The level of focus of an individual developer should be on the low level concerns of building reliable and secure components. Developers must maintain intellectual mastery from the UI to low level APIs and memory usage of the features they code. We don’t need them distracted and overwhelmed with system wide product expertise duties as well.

Testers manage system wide issues and rarely have deep component knowledge. As a manager of the forest, we can treat any individual tree abstractly. Testers should know the entire landscape understanding the technologies and components involved but not actually taking part in their construction. This breadth of knowledge and independence of insight is a crucial complement to the developer’s low level insights because testers must work across components and tie together the work of many developers when they assess overall system quality.

Another way to think about this is that developers are the domain experts who understand the problem the software is solving and how it is being solved. Testers are the product experts who focus on the breadth of technologies used across the entire product.

Testers should develop this product expertise to the extent that they cannot be stumped when asked questions like "how would I do this?" with their product. If I asked one of my Chrome testers any question about how to do anything with Chrome concerning installation, configuration, extensions, performance, rendering ... anything at all ... I expect an answer right away. An immediate, authoritative and correct answer. I would not expect the same of a developer. If I can stump a tester with such a question then I have cause for concern. If there is a feature none of us know about or don't know completely then we have a feature that might escape testing scrutiny. No, not on our watch!

Product expertise is one ingredient that must be liberally used when mixing Testing Cake.

Blog: Categorizing the Cloud …

Blog: Patterns and Practices for Improving Personal Productivity, Time Management, and Effectiveness

Blog: Earned Value v. Earned Schedule

Blog: A List of Coding Standard Websites

Blog: TDD at the System Scale

Site: Real QA Manifesto

Humour: My husband is a programmer; I have no idea what that means.

Article: jQuery Test-Driven Development

Article: Are We Headed to Abilene?

Tool: Flerry is a Flex-Java bridge for Adobe AIR 2.0

Tool: Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests

Video: Managing Ruby Teams

Video: How to Cope with Communication Problems in an Agile Project?

Video: Continuous Integration, Pipelines and Deployment

Find more interesting links on the software development links directory, the software development tools directory, the software development articles directory, the software development blogs aggregator or the software development videos directory.

An exploratory DOCTOR is known as… a “doctor.”
An exploratory WELDER is known as… a “welder.”
An exploratory PILOT is known as… a “pilot.”
An exploratory WRITER is known as… a “writer.”
An exploratory SCIENTIST is known as… a “scientist.”
An exploratory TRUCK DRIVER is known as… a “truck driver.”

A non-exploratory doctor is known as… “irresponsible.”
A non-exploratory welder is known as… “irresponsible.”
A non-exploratory pilot is known as… “killed in a plane crash.”
A non-exploratory writer known as… “a plagiarist.”
A non-exploratory scientist is known as… “a tobacco company scientist.” (also “Creationist”)
A non-exploratory truck driver is known as… “lost.”

Can you spot the pattern here?

There is no such thing as an “exploratory tester” except inasmuch as a good tester obviously can and will do exploration as a basic part of his work.

This book is composed of papers previously written by Watts Humphrey. The people and management aspects of software development are often neglected in books and this one is a good source to start thinking about them… and improving our practice. The book is structured in four parts: managing your projects, managing your teams, managing your boss and managing yourself. In each part, it presents both general principles and real life examples or stories taken from Watts Humphrey career. This makes the book very easy to read as we can connect the theory to situations that we have met in our professional life.

Read the complete review on Sofware Development Books

Reference: “Reflections on Management – How to Manage Your Software Projects, Your Teams, Your Boss, and Yourself”, Watts S. Humphrey and William R. Thomas Addison-Wesley, 260 pages, IBSN 978-0-321-71153-3

Get more details on this book or buy it on amazon.com
Get more details on this book or buy it on amazon.co.uk

In Growing Object-Oriented Software, Guided by Tests, we advocate starting development by writing a system test (or tests). This caused more controversy than I expected. System tests have the reputation of being slow (not entirely avoidable, I admit), difficult to automate reliably and difficult to diagnose when they fail. However, I find that many teams follow a TDD process at the unit-level, but do post-hoc testing at the system level, and so don't use system tests as a source of design feedback. For me, the key feature of the TDD process is that when something is hard to test, that's a driver to go back and adjust the design until it is easy to test. At the system-test scale, tests become unreliable when they cannot observe the activity of the system well enough to reliably synchronise their execution with that of the system. Failures are hard to understand if the tests cannot gather and filter information about the activity of the system during execution of the test. Defects will break test isolation and cause cascading failures if the tests cannot wait until the system becomes passive and restore it to a known good state when the system fails a test. To reliably test a system, we must be able to... determine what the system is doing. detect when it has stopped doing it. detect when the system has failed. explain why has it failed. restore the system to a known good state. If our system tests are unreliable, that's a sign that we need to add interfaces to our system through which tests can better observe, synchronise with and control the activity of the system. Those changes turned out to be exactly what we need to better manage the systems we built. We used the same interfaces that the system exposed to the tests to build automated and manual support tools. It seems obvious in hindsight: to reliably test manage a system, we must be able to... determine what the system is doing. detect when it has stopped doing it. detect when the system has failed. explain why has it failed. restore the system to a known good state. Design Feedback from TDD TDD at the unit scale guides the design of the code to make the system easier to modify, because it is easier to test code that is organised into loosely coupled, cohesive units that have clear responsibilities. My hunch is that TDD at the system scale works in a similar way, guiding the design of the architecture to make the system easier to manage, because it requires the system have machine-readable interfaces through which tools can observe and control its activity.

This is just a quick post that I hope will save some people some headache when they try to customize their MySQL setup on Ubuntu. I've spent some quality time with this problem over the weekend. I tried in vain for hours to have MySQL read its configuration files from a non-default location on an Ubuntu 9.04 server, only to figure out that it was all AppArmor's fault.

My ultimate goal was to run multiple instances of MySQL on the same host. In the past I achieved this with MySQL Sandbox, but this time I wanted to use MySQL installed from Debian packages and not from a tarball of the binary distribution, and MySQL Sandbox has some issues with that.

Here's what I did: I copied /etc/mysql to /etc/mysql0, then I edited /etc/mysql0/my.cnf and modified the location of the socket file, the pid file and the datadir to non-default locations. Then I tried to run:

/usr/bin/mysqld_safe --defaults-file=/etc/mysql0/my.cnf

At this point, /var/log/daemon.log showed this error:

mysqld[25133]: Could not open required defaults file: /etc/mysql0/my.cnf
mysqld[25133]: Fatal error in defaults handling. Program aborted

It took me as I said a few hours trying all kinds of crazy things until I noticed lines like these in /var/log/syslog:

kernel: [18593519.090601] type=1503 audit(1281847667.413:22): operation="inode_permission" requested_mask="::r" denied_mask="::r" fsuid=0 name="/etc/mysql0/my.cnf"
 pid=4884 profile="/usr/sbin/mysqld"

This made me realize it's AppArmor preventing mysqld from opening non-default files. I don't need AppArmor on my servers, so I just stopped it with 'service apparmor stop' and chkconfig-ed it off....at which point every customization I had started to work perfectly.
At least 2 lessons:
1) when you see mysterious, hair-pulling errors, check security-related processes on your server: iptables, AppArmor, SELinux etc.
2) check all log files in /var/log -- I was focused on daemon.log and didn't notice the errors in syslog quickly enough
Google didn't help when I searched for "mysqld Could not open required defaults file". I couldn't find any reference to AppArmor, only to file permissions.

We invite you to propose a session for this leading software development conference. We have a long tradition of high quality sessions covering many aspects of software development, from programming languages (e.g., Java, C#, Python, Erlang, Haskell, Ruby, Groovy, C, C++, etc.), and technologies (libraries, frameworks, databases, etc.) to subjects about the wider development environment such as testing, architecture and design, development process, analysis, patterns, project management, and softer aspects such as team building, communication and leadership.

Sessions may be either tutorial-based, presentations of case studies, or take the form of interactive workshops. We are always open to novel formats, so please contact us with your idea. The standard length of a session is 90 minutes, with some exceptions. In order to allow less experienced speakers to speak at the conference without the pressure of filling a full 90 minutes, we reserve a number of shorter 45 minute sessions.

http://accu.org/index.php/conferences

Google has recently bought Instantiations, the editor of Java, GWT, Struts and Ajax software development tools. On the Instantiations web site, the company says: “Yes it’s true. Instantiations’ award-winning Java and Ajax development tools and our incredible Eclipse team have been acquired by Google. We are all very excited about taking our technology and team to the next level – and there is no bigger step up than Google! We very much appreciate your patronage and interest through the years. As part of Google, we look forward to continuing to work with you. Please stay tuned for exciting new announcements coming soon on the Google Web Toolkit blog. Current Java and Ajax product customers — we are committed to making this a seamless transition for you. For a short period of time, new downloads of our products will be unavailable while we make the transition, but our service continues uninterrupted for those who have support agreements in place.”

The Smalltalk business will remain independent and their new web site is http://st.instantiations.com/. Mike Taylor will continue as President and CEO of the new Smalltalk-focused Instantiations. Founder Eric Clayberg will join Google, but will also continue as a Director/Board Member, technical advisor and major shareholder of Instantiations. John O’Keefe will assume an even stronger role guiding the technical development and advancement of VAST.

This acquisition is however not mentioned neither in Google Official Blog or Google Press Releases . On an Instantiations forum posting titled “Seamless migration to Google” , a customer complains that “Unless I’ve missed something the website is no longer available and the support emails just bounce back ? What am I doing wrong ?” It seems that the communication was lacking not only to the external world but also towards Instantiations customers. Mail to my instantiations.com contact is effectively bouncing…

I can understand that this acquisition may be a minor action in Google exciting life. For instance the press release section of Google doesn’t mention also that will stop deploying the Google Wave technology. This is what you call an “update” in PR language ;o)

You can get a lot of benefits from a consistent architectural style but it is difficult to ensure that the architectural style is not violated when the programming language does not enforce it. I'm hoping that a vivid system metaphor will help explain the style to people reading the code and making architectural violations feel wrong because they break the metaphor. Communicating Sequential Processes My latest project has some awkward concurrency issues. We're using a proprietary analytics library that is not thread safe, receive high-volume market data feeds through vendor APIs that call into our code from their own threads, and have to look up reference data from slow, remote services without delaying the processing of unrelated market data events, which must all be processed on one thread because that analytics library is not thread safe. This has led us to structure the system as multiple event-driven, concurrent threads. The market data threads feed events to the core processing thread, which is the only thread allowed to call into the analytics library. The core processing thread does not perform any blocking I/O: if it needs more data to process an event, it emits an event requesting that data; other threads react to that event by loading the data and sending it to to the core thread in another event. This style of program can be pretty confusing if you don't carefully keep track of the various state machines and the communication that can go on between them. To help manage these issues, we've adopted an architectural style inspired by the Conic programming language. Conic In the Conic language a program is organised into lightweight processes (known as task modules) that communicate by asynchronous message passing. In contrast to the Actor model, adopted by Erlang, processes do not address one another directly. A Conic process has exitports through which it sends messages and entryports through which it receives messages. At runtime, a processes exitports are connected to the entryports of other processes. The interface of a component is defined as a set of named, typed entry- and exitports. Conic modules are composed into group module types that instantiate submodules, connect them together and expose some of their ports at the interface of the group module. From outside, there's no way to know if a module is a primitive task module or a composite group module. A program is a group module that has no unsatisfied inputs or outputs. Conic strictly separates behaviour and structure. A group module has no behaviour of its own. Its behaviour is only defined by the composition of component modules within it and the ports of its component modules that it exposes at its interface. These constraints lead to an architecture that acts as a helpful conceptual framework in which to decompose complex concurrent behaviour into event-driven parts, or, conversely, to compose those parts into systems with sophisticated behaviour. Those parts and compositions of parts are easy to test (or even model check) because they are context independent with clearly defined inputs and outputs. It is easy to understand the system's architecture from the configuration code that defines the system. In fact, you can generate visualisations of the architecture, like the diagrams above, that have a one-to-one correspondence with the code. And the small set of primitives (modules, inputs, outputs & links) makes for a very simple meta-model that makes it easy to write reflective code that manipulates any module or system in a generic way: for example, to insert interceptors for management & monitoring or unit testing. A Circuit Metaphor In Conic this constraint is enforced by the language. However, our system is written in Java. There's nothing in the language to stop programmers adding behaviour to group modules and violating the model. We must adhere to the architectural style through convention. The constraints imposed by the style are not really obvious from the terminology used by Conic: "task module", "group module", "entryport", "exitport", etc. When I was introduced to Conic, the style was described to me as "software integrated circuits", so I've used a circuit metaphor in the Java implementation to better express the important constraints. We define "components", rather than "modules", and compose components into "circuits" rather than "group modules". Components have inputs and outputs and a circuit "connects", rather than "links", outputs to inputs. The circuit metaphor has guided the naming of other parts of the system. For example, we have JMX MBeans that install "wiretaps" between components to let us remotely monitor message flow. This is an expanded version of a story I posted to the Software Metaphor mailing list. Image by Andrew Magill. Used under the Creative Commons Attribution 2.0 Generic license.

By Philip Zembrod
In my quest to explore TDD I recently found another propery of TDD-written code that I hadn't expected: When reviewing or just reading such code, it's often best to first read the tests.
When I look at new code or a code change, I ask: What is this about? What is it supposed to do? Questions that tests often have a good answer for. They expose interfaces and state use cases. This is cool, I thought, and decided to establish test-first reading as my code-reviewing routine. Of course this just applies the specification aspect of tests: Reading the specs before reading the code.
Only it didn't always work. From some tests I just failed to learn the point and intention of the tested code. Often, though not always, these were tests that were heavy with mocks and mock expectations.
Mocks aren't always a helpful tool, was my first conclusion. The phrase "Good mocks, bad mocks" popped up in my mind. I began to appreciate fakes again - and the people who write them. But soon I realized that this was about more than mocks vs. fakes vs. dummies vs. other Friends You Can Depend On. I was really looking at how well tests fulfill their role as specification.
TDD teaches that tests are a better specification than prose. Tests are automatically enforced, and get stale less easily. But not all tests work equally well as specification! That's what test driven code reviewing taught me.
I began to call them well-specifying tests and poorly-specifying tests. And the specification aspect isn't just some additional benefit, it's a really crucial property of tests. The more I thought about it, the more I saw: It is connected to a lot of things that first weren't obvious to me:

• If tests are poorly-specifying, then possibly the tested product is poorly specified or documented. After all, it's the tests that really make sure how a product behaves. If they don't clearly state what they test, then it's less clear how the product works. That's a problem.

• Well-specifying tests are more robust. If a test just does and verifies things of which the architect or product manager will readily say "yes, we need that" then the test will survive refactorings or new features. Simply because "yes, we need that." The test's use case is needed, its conditions must hold. It needn't be adapted to new code, new code must pass it. False positives are less likely.

Corollary: Well-specifying tests have higher authority. If a test fails, a natural reaction is to ask "is this serious?" If a test is poorly-specifying, if you don't really understand what it is testing, then you may say "well, maybe it's nothing". And you may even be right! If a test is well-specifying, you'll easily see that its failing is serious. And you'll make sure the code gets fixed.
I'm now thinking about an authority rank between 0 and 1 as a property of tests. It could be used to augment test coverage metrics. Code that is just covered by poorly-specifying tests would have poor authority coverage, even if the coverage is high. Quantifying an authority rank would be a conceptual challenge, of course, but part of it could be how well test driven code reviewing works with a given test.
P.S. If anyone suspects that I'm having some fun inventing terms beginning with "test driven," I'll plead guilty as charged. :-)