Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Software Development Blogs: Programming, Software Testing, Agile Project Management
Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Oliver Erlewein is an automation specialist. He’s respected in the Context-Driven Testing community of New Zealand and has been an agitator pushing back against the ISTQB. After some years of frustration with bad management he finally went independent. Now he’s back to full-time work. He posted the following as a comment:
Starting 2014, I have given up my self-employment and joined a (sort of) start up. I didnâ€™t think I was ever going back to being employed but this was worth it. I have found a company that respects my professionalism and listens to what I say, where I am responsible for what I produce and get the full control of how to go about it. I and the task I do are respected. The word integrity doesnâ€™t get used here but it is a place that actually has oodles of it.
Every now and again I hear the sentence â€śyou are the expert so what do you suggest we do?â€ť or â€śdo what you think is right, you are the expertâ€ť â€¦.and they mean it exactly like that. It makes for a completely different working environment. It motivates, it invigorates and it makes working fun. It puts heaps more pressure and responsibility on me but I am happy as taking that on board because I am convinced that I can do it (even if I still donâ€™t know how right at this moment).
Although this shop is not agile (but more agile than a lot of the shops out there that call themselves agile!) they do something that is one of the main success factors for agile: They re-introduce back the idea of responsibility, professionalism and craftsmanship into (IT) work. And that motivates. I feel like I can call bull**** if it is appropriate to do so or get traction on subjects I think are important.
So although it meant I made a career change away from my original trajectory I made it consciously towards a more ethical work life, where integrity and being the best you can actually counts for something.
Thank you for sharing that with us, Oliver. It goes to show that there are good managers out there who understand craftsmanship and leadership.
A few months ago, I’ve met the founders of a mobile game development studio based in San Francisco, CA (they also have teams in Paris and Argentina): PixowlÂ (backed by a lot of tech entrepreneurs)Â Â Not only are they the nicest people I’ve met in a long time but they’re alsoÂ extremely talented! A great combo to start a friendship! Adrien is one of the best mobile developer I know and Laurel is a great artist doing all the illustration for their games (and has a very cool blog showcasing her work)
Today, they have a very successful Minecraft type of game which is doing really good on the Apple Store and they’re working on the second version of their other flagship game: Greedy Grub. Some of you might remember Snake from the early 80′s. This new version of Grub is Snake on mobile steroid!
My kids played the first version of the game and you can bet that when they’ve heard that the second version was on the horizon, they wanted to be involved! They’re the perfect beta testers and provide tons of feedback (Pixowl uses HockeyApp to manage the beta of Grub which has a very nifty feature to add feedback directly in the game. Very handy!)
There is today only one playable level but Grub is already very addictive. I had a glimpse of what’s coming and I think I will soon get rid of candy crush!
If you like games and want to be involved in the feedback process, join the Grub beta today!
Note:Â I have no affiliation with pixowl. I just like games, testing and Adrien makes really good crepes!Get Shareaholic
Testing is a performance, not an artifact.
Artifacts may be produced before, during, or after the act of testing. Whatever they are, they are not tests. They may be test instructions, test results, or test tools. They cannot be tests.
Note: I am speaking a) authoritatively about how we use terms in Rapid Testing Methodology, b) non-authoritatively of my best knowledge of how testing is thought of more broadly within the Context-Driven school, and c) of my belief about how anyone, anywhere should think of testing if they want a clean and powerful way to talk about it.
I may informally say “I created a test.” What I mean by that is that I designed an experience, or I made a plan for a testing event. That plan itself is not the test, anymore than a picture of a car is a car. Therefore, strictly speaking, the only way to create a test is to perform a test. As Michael Bolton likes to say, there’s a world of difference between sheet music and a musical performance, even though we might commonly refer to either one as “music.” Consider these sentences: “The music at the symphony last night was amazing.” vs. “Oh no, I left the music on my desk at home.”
We don’t always have to speak strictly, but we should know how and know why we might want to.
Why can’t a test be an artifact?
Because artifacts don’t think or learn in the full human sense of that word, that’s why, and thinking is central to the test process. So to claim that an artifact is a test is like wearing a sock puppet on your hand and claiming that it’s a little creature talking to you. That would be no more than you talking to yourself, obviously, and if you removed yourself from that equation the puppet wouldn’t be a little creature, would it? It would be a decorated sock lying on the floor. The testing value of an artifact can be delivered only in concert with an appropriately skilled and motivated tester.
With procedures or code you can create a check. See here for a detailed look at the difference between checking and testing. Checking is part of testing, of course. Anyone who runs checks that fail knows that the next step is figuring out what the failures mean. A tester must also evaluate whether the checks are working properly and whether there are enough of them, or too many, or the wrong kind. All of that is part of the performance of testing.
When a “check engine” light goes on in your car, or any strange alert, you can’t know until you go to a mechanic whether that represents a big problem or a little problem. The check is not testing. The testing is more than the check itself.
But I’ve seen people follow test scripts and only do what the test document tells them to do!
Have you really witnessed that? I think the most you could possibly have witnessed is…
a tester who appeared to do “only” what the test document tells him, while constantly and perhaps unconsciously adjusting and reacting to what’s happening with the system under test. (Such a tester may find bugs, but does so by contributing interpretation, judgment, and analysis; by performing.)
a tester who necessarily missed a lot of bugs that he could have found, either because the test instructions were far too complex, or far too vague, or there was far too little of it (because that documentation is darn expensive) and the tester failed to perform as a tester to compensate.
In either case, the explicitly written or coded “test” artifact can only be an inanimate sock, or a sock puppet animated by the tester. You can choose to suffer without a tester, or to cover up the presence of the tester. Reality will assert itself either way.
What danger could there be in speaking informally about writing “tests?”
It’s not necessarily dangerous to speak informally. However, a possible danger is that non-testing managers and clients of our work will think of testers as “test case writers” instead of as people who perform the skilled process of testing. This may cause them to treat testers as fungible commodities producing “tests” that are comprised solely of explicit rules. Such a theory of testing– which is what we call the Factory school of testing thought– leads to expensive artifacts that uncover few bugs. Their value is mainly in that they look impressive to ignorant people.
If you are talking to people who fully understand that testing is a performance, it is fine to speak informally. Just be on your guard when you hear people say “Where are your tests?” “Have you written any tests?” or “Should you automate those tests?” (I would rather hear “How do you test this?” “Where are you focusing you testing?” or “Are you using tools to help your testing?”)
Thanks to Michael Bolton and Aleksander Simic for reviewing and improving this post.
This is from the New York Times:
Its other hallmark is that Mr. Langella never does the part the same way twice. This is partly because heâ€™s still in the process of discovering the character and partly because itâ€™s almost a point of honor. â€śThe Brit approach is very different from mine,â€ť he said. â€śThereâ€™s a tendency to value consistency over creativity. You get it, you nail it, you repeat it. Iâ€™d rather hang myself. To me, every night within a certain framework â€” the framework of integrity â€” you must forget what you did the night before and create it anew every single time you walk out on the stage.â€ť
I love that phrase the framework of integrity. It ties in to what I’ve been saying about integrity and also what is true about informal testing: if you are well prepared, and you are true to yourself, then whatever you do is going to be spontaneously and rather effortlessly okay, even if it changes over time.
I often hear anxiety in testers and managers about how terrible it is to do something once, some particular way, and then to forget it. What a waste, they say. Let’s write it all down and not deviate from our past behavior, they say. Well I don’t think it’s waste, I think it’s mental hygiene. Testing is a performance, and I want to be free to perform better. So, I make notes, sure. But I am properly reluctant about formalizing what I do.
Doing your best work includes having the courage to let go of pretty good work that you’ve done before.
I have taken down the original text of this post at the request of my colleague who had the courage and audacity to let me post his detailed comment about how he works “under the radar” to change things in his company.
I had posted his comment originally with his permission, of course. But, apparently, in his country, “it’s illegal to harm [one's] employer’s business” and it can reasonably be considered doing harm to express a low opinion of your own company’s behavior, even if you are dedicated to improving that behavior. Dirty laundry in public is arguably bad for business, if your business involves telling people that you’re a trustworthy expert, and your laundry says otherwise.
Of course this is understandable. Working under the radar generally means not being public about what you are doing. Therefore, as much as I prefer the clean feeling of working ON the radar, I wish him good luck with his mode of influencing a big, commercial, ceremonial system.
Here’s a comment from a Context-Driven tester with more patience than me. (His message is italicized and indented. My reply is interjected in normal text…)
Sami SĂ¶derblom writes:
I skimmed your book “Secrets of a Buccaneer-Scholar” and found the same spirit from this post. It was highly inspiring, but it also bothered me a bit. I started to write about this to Twitter (under @pr0mille), but my point was lost quite quickly. Therefore I CHOSE to take the mental effort and explain myself here…
I work in Sogeti Finland. For many it’s a company that doesn’t actually promote integrity. We sell such atrocities as TMap and TPI, and many of our consultants enforce them blindly. Vast majority is unable to tell that they are promoting bad things, because they don’t see them as such. They don’t feel that their integrity is jeopardized. Some have awakened from this dream, but still choose to promote bad things, because their bonuses, career development and ultimately reputation within the company depend on this. They have chosen to lose their integrity.
Yes, it is not necessarily poor integrity to embrace something that some people (like me) think is bad. My model of integrity is based on wholeness of self, and congruence between self as presented to others and self as experienced personally. Therefore: IF I fancy myself an excellent tester, AND I advocate something that I believe is inconsistent with excellent testing, THEN that is an integrity problem. If someone honestly thinks that a “bad” thing is good, it’s a knowledge or wisdom problem, but not challenge to integrity.
I suspect there are people in Sogeti who believe in the value of TMap and TPI. I question their judgment, of course. I wonder if they have ever seriously studied testing. And I know that any of them who decided that TMap should be improved would face stiff resistance from most other people nearby, to the degree that improvement of the model is quite unlikely to occur. But none of that– in itself– is an ethics problem. Of course being intentionally, determinedly ignorant over a period of time is a different matter. I don’t understand how anyone who pays attention to developments in testing in the last 20 years could continue to hold on to such outmoded thinking.
At first I was blissfully ignorant. After gaining experience and trying to apply TMap and TPI to actual work, study them and really understand I started to see the flaws and the fact that they just don’t work. Worse yet, they can seriously harm the organizations they are implemented in. This weighed more on my scale than any bonuses or internal reputation. So I started to rebel.
By exposing that my integrity was more important than questionable company goals, I became open to attacks. I had to spend all of my time making my case, explaining myself to people that could not be convinced. It was a head on collision. Force against force. I didn’t make any progress in changing things for the better.
I don’t generally think of my integrity-related actions as moves in a game but rather as expressions of what I am. I’m not thinking “what do I need to say or do in order to get Sami not to fight me” but rather a three-step process FIRST: “What is true, relevant, helpful, and reasonably non-abusive?” THEN: “What is an effective way to live by that and communicate it?” If I proceed that way, sometimes opposition melts away. If it doesn’t melt away, then perhaps it’s right to fight. Ah, and here is the third step: “IF there is opposition, try to understand it and learn from that.”
Three steps, but none of them is about the ends justifying whatever means that get me what I want.
Sometimes, when integrity seems not to be at stake, I use a strategy of thinking “what does this person want and how can I support him?” Management skills training and years of experience as a husband and father help me have the patience to do that. Then, in a roundabout way without me being in charge, we might end up doing great work. It’s sort of a Taoist strategy, but it takes a lot of patience and time (more than I usually have, in my old age).
Then I remembered my jujutsu teachings. Jujutsu is all about manipulating the opponent via indirect attacks. In jujutsu you never go with force against force. So I changed my approach. I started to play ball. I accepted assignments that were less than optimal integrity-wise, but in them I used grassroot tactics to change things. For example I started to remove all the constraints that prevented people from thinking. I removed expected results from test cases. I simplified test planning. I incorporated roles and made people to talk with each other rather than with systems. I never talked about exploratory or context-driven testing, I never mentioned heuristics, I never bothered people with anything but the actual work.
I flew under the radar.
This sounds like the Taoist thing. Of course, that’s a viable strategy. But are you doing it honestly? “…flew under the radar” and “manipulating” sound like you could be attempting to deceive people.
This is what I meant by the fast food metaphor. A little (polite?) deception won’t kill your integrity outright, but it could make you sick. There are degrees of disguising your intent and methods (“blacker lies” and “whiter lies” shall we say) and I hope you are struggling with these ethical questions as you go.
I find it helpful to ask myself what is in the best interests of the company? Am I taking my paycheck and helping the company succeed on its own terms, or am I a parasite living off its life juices merely for my own gain?
This took some time, some hard days and some dents to my integrity too, but now it has started to pay off. Because I’ve introduced success rather than conflict I now have more and more good or even awesome assignments, and I know how to act in the bad ones. I have even applied these tactics internally and good things are happening as we speak. I cannot talk about all of them, but it’s safe to say that TMap won’t be the same…
It’s too easy to be hasty and send wrong messages over Twitter. I hope you see that I’m far from considering myself as a victim or prisoner. Yes, I work in challenging environment, but I’m choosing to do so. Sometime I’ve thought that I’m doing volunteer work in a conflict area. I believe I can matter and that gives purpose to my work. I want to believe that it builds my integrity too.
You’re using active, responsible language, here. You seem to be taking responsibility. The most impressive part is that you have named yourself and your employer. That seems dangerous, to me, but dangerous is a good way that I admire.
Perhaps predictable for a guy in the midst of a Finnish winter, you seem eager for sunlight!
Ethics Under the Radar
I have heard it called “stealth testing.” Doing the right thing secretly, while publicly following a broken process. I am not completely sure that is what Sami is talking about, but the phrase “under the radar” suggests that he might be.
I have certainly used “stealth mode,” but it troubles me. I ask myself: How would I feel if someone who worked for me was “under the radar” while negating or ignoring my corporate strategy? As it happens I have been in this position several times that I know of. My answer is that I would feel great or terrible depending on what I thought the motive of the person was. I call it “constructive insubordination” when I am disobeyed in a way that honors my intent and my prerogatives as a manager. I celebrate that. It’s like having magic elves working for you. At the same time, it calls into question whether I am being over-controlling as a manager (or father or husband… the same principle works in any situation where you establish a contract with someone to do something for you).
I have a belief that I’m not going to justify– I’m simply going to say it and challenge you to look into your own experience and your own heart and see the truth of it for yourself: Your sense of identity, as a human among humans, is the most powerful force that animates and directs your choices. It is more important than sex or food or religion. It lurks behind every neurosis (including those involving sex or food or religion). As I read history and experience life, answers to the questions “Who am I? Am I a good example of what I should be?” are the prime movers of human choice throughout all of history, and the proximal cause of every war.
There are certainly exceptions to this rule: drug addiction, mental illness, or panic over a sudden, surprising, physical threat. Maybe those things have little to do with identity. Granted. I’m talking about normal daily life (and every Shakespeare play).
“I am an American. I am a human. I am a father. I am a husband. I am lovable. I am helpful. I am a tester. I am a skeptic. I am an outsider. I am dangerous. I am safe. I am honorable. I am fallible. I am truthful. I am intellectual…”Â Each of these statements, for me, are reflective shards that tumble in a kaleidoscope of my identity. The models of personhood they represent comprise my moral compass. Although the pattern formed in that kaleidoscope may seem to shift with the situation, the underlying logic of my adult identity changes little with time.
That is the context for integrity.
Integrity means wholeness; the harmony and completeness of one’s identity. Practically speaking, a person with integrity is a person to lives consistently according to their avowed moral code, as opposed to someone who has no moral code, or who changes it as a matter of convenience. A person of integrity therefore creates continuity across the events of his life, and other people feel they know who they are dealing with.
The Challenge of Finding Your Integrity
Recently, in a discussion about what is reasonable for an employer to ask of a tester, a colleague felt I was trying to impose my own values onto potential employers of my students and wrote that as teachers of new testers “employment [for the testers] should be our first priority.” I disagreed sharply, writing that “our first priority is integrity.” My correspondent seemed to take offense to that.
Now, the employment-first position might be construed to imply that we should advocate robbing banks, because it is the quickest way to get money, or perhaps we should train prostitutes, because prostitution is an old and reliable industry with lots of job security for the right people. That would be absurd, but it’s also a straw man argument. I am certain no one intends to argue that any job is better than no job. Safety, legality and morality do enter into the equation.
Conversely, the integrity-first position might be cast as requiring a tester to immediately protest or resign in the face of any ethical dilemma or systemic ethical lapse, no matter how seemingly minor. This would turn most testers into insufferable, dour lawyers on their projects. We would get very little done. Who would hire such people?
These extreme positions are not very interesting, except as tools for meditating on what might be reasonable behavior. Therefore, I’d like to describe a less extreme position that I think is more defensible and workable. It goes like this:
1. Integrity is a vital and important matter. We suffer as people and society suffers when we treat it too lightly.
2. As testers and technical people, our integrity is routinely threatened by well-meaning clients and colleagues who want us to portray ourselves and the world to be a certain way, even if that isn’t strictly the truth.
3. If we never think directly about integrity, and simply trust in the innate goodness of ourselves and others, we are definitely taking this matter too lightly.
4. Integrity is not like a vase that shatters easily, and that once shattered is irretrievable. Integrity is more like an ongoing public artwork, exposed to and buffeted by the elements, sometimes damaged but always ultimately repairable (although our reputation may be another matter). Integrity is a work in progress for all of us.
5. Integrity, like education, is both personal and social. Your society judges you. It is reasonable that it does. But it is also reasonable to negotiate limits on that judgment. We spend our lives negotiating those lines, one way or another.
6. Forgiveness, although perhaps difficult and expensive to obtain, should always be available to us. (I test this by occasionally imagining my most “depraved” enemies in testing, and then imagining what they could do that would allow me to forgive them and even collaborate with them.)
7. Although integrity is our highest priority, in general, it is not the only thing that matters. We must apply wisdom and judgment so that the maintenance of integrity does not unreasonably affect our ability to survive. There is no set formula for how to do that.
8. Therefore, our practical priority must be: to learn how to think through and solve problems of survival while maintaining reasonable integrity. This itself is an ongoing project, requiring temperance and self-forgiveness.
9. New testers need to realize that they are not necessarily responsible for the quality of their work. Sometimes you will be asked to do things you don’t understand the value of, even though there may be value. In those situations, it’s okay to be compliant, as long as you are under supervision and someone competent is taking responsibility for what you do. It’s okay to watch and learn and not necessarily to make trouble. (Although, I usually did, even as a newbie.)
10. Experienced testers? Well, much is expected of you. Your clients (your non-tester colleagues and bosses) don’t know how to test, but you are supposed to. You can’t just do what you are told. That would be like letting a drunk friend drive home. Remember, someday your clients may sober up and wonder why you agreed to their stupid plan when you were supposed to be the expert.
Having laid this hopefully reasonable and workable strategy before you… I actually think the dispute between me and my correspondent, above, was not about the importance of integrity or employment at all, but rather about the specifics of the case we were debating. I should tell you what that was: whether it is reasonable for an employer to expect an entry-level tester to “write test cases.”
From a context-driven testing perspective, no practice can be declared unreasonable outside all contexts. But I do know a lot about the typical contexts of testing. I have seen profound waste, all around the industry, due to reckless and thoughtless documenting and counting of things called “test cases.” So, I don’t think that it is reasonable, generally speaking, to require young testers to write test cases. First, because “writing test cases” is what people who don’t know how to test think testers do– so, it’s usually an indicator of incompetent management. Second, because entry-level testers do not have the skills to write test cases in such a way that won’t result in a near complete waste of their client’s time and money. And third, because there are such obviously better things to do, in most cases, including learning about the product and actually testing the product.
Many people disagree with me. But I believe their attitude on this is the single most direct and vital cause of the perpetual infancy and impotency that we see in the testing industry. In other words, it’s not just a disagreement about style, it’s something that I think threatens our integrity as sincere and thoughtful testers. Casual shrugging about test case writing must be stamped out the way transfats are being outlawed in fast food. Yes, that also took years to accomplish.
Speaking of fast food…
Here’s a metaphor that might help: eating at McDonalds.
Eating at McDonalds will not kill you (well, not outright). But what if you were forced to eat at McDonalds for your work? Every day, breakfast, lunch and dinner. Nothing but McDonalds. What if it were obvious to you that eating at McDonalds was not helping you actually succeed in your work? What if instead it was clear to you that such a diet was harming your ability to get your work done? For instance, perhaps you are a restaurant reviewer, except you are almost always full of McDonalds food so you can’t ever enjoy a proper meal at a restaurant you are supposed to review? And yet your manager, who knows nothing about restaurant reviewing, insists that you maintain a McDonalds-dominated dietary regimen.
Couldn’t someone say, hey, it’s a job and you should do what you are told? Yes, they could say that. And it might be true enough at first. But over time, that diet would hurt you; over time, you would have to cope with how poorly you were doing what you believed to be your real job. You might even be criticized for missing bugs– I mean– failing to review restaurants fully, even though it’s largely due to your employer’s own unreasonable process prescriptions.
At some point you might say “enough!!” You might refuse to eat another Big Mac. From the point of view of your management and colleagues, it might look like you were risking your job just because you didn’t want to eat a hamburger. It might look crazy to them. But from your point of view, the issue isn’t the one burger, but rather the unhealthy system, slowly killing you. This breakdown comes more quickly if you happen to have a gluten allergy.
Ethics and integrity in testing is not just about following prissy little rules that many other people flout– it’s about not making yourself sick even if other people are willing to live a sickly life. This requires that you be able to judge what health and sickness means to you. Integrity is about identity health.
A Story of Quitting Even Though I Needed the Work
In 1998, I was hired by a consulting company outside of Washington D.C. I negotiated for a $30,000 sign-on bonus, and bought a house in Virginia. I was the sole breadwinner in my family, with a wife and son to support. I bought a new car, too. In short, I was “all in.”
Six months later, I quit. I had no other job to go to. I had bills due. It took me seven years to pay back my sign-on bonus, with interest (I forfeited it because I did not stay for two years). But with the help of colleagues and family over the following weeks, I made the transition to running my own business. I am most thankful for my wife’s response when I came home that night and told her I walked out on our only source of income. She shrugged and said it was surely for the best, and that something good would come of it. (I can only recommend, gentlemen, that you marry an optimist if you can.) I am also thankful to Cem Kaner, who bought me a laptop (my only computer was then owned by my employer) and said “times like these are when you discover who your true friends are.” This was fitting because it was partly because of Cem that I had previously decided never to sacrifice my professional integrity.
This illustrates one lesson about ethics: community support helps us find and maintain our integrity.
I quit because my company was insisting that I bill hours on a project that, in my opinion, was absolutely certain not to benefit from my work. The client wanted me to create fake test cases. They didn’t call them fake test cases, of course. They claimed to want real test cases; and good ones. But no product had been designed at that time! All I had access to was a general description of requirements, which in this case were literally statements of problems the product was intended to solve, with no information on how they would be solved. It was a safety-critical factory process control system, and no one could show me what it would look like or provide any examples of what specifically it might do. The only test cases I could possibly design would therefore be vague and imaginary, based on layers of soft, fluffy assumptions. The customer told me they would be happy if I delivered a document that consisted of the text of each requirement preceded by the phrase “verify that…” I told them they didn’t need a tester for that. They needed a macro.
The integrity picture was clouded, in that case, because the client believed they had to follow the “V-Model” process, which they had interpreted as demanding that I submit a test case specification document. It was a clash between the integrity of a heuristic (the V-Model) vs. the integrity of solving the problem for which the heuristic was designed. My client might have said that I was the one violating the integrity of the process. Whereas I would have said that my client was not competent to make that judgment.
I’m not saying I won’t do bad work… I’m just saying I won’t do bad work for money. If I do bad work, I want it to be for fun or for learning, but not to anyone’s expense or detriment. Hence a line I use once in a while “I could do that for you, except that you pay me too much.” This is one reason I like being independent. I control what I bill for, and if I think a portion of my work is not acceptable, I don’t charge for it– like a chef who refuses to serve an overcooked steak.
It wasn’t as sudden as it looked…
I didn’t just lose my temper at the first sign of trouble. Things had been coming to a boil for a while. On my very first day I reviewed the RFP for that project and concluded it was doomed, but management bid on it anyway, telling me I needed to “be practical” and that surely “we could be helpful to them if they hired us.” I needed the job, so I relented against my better judgment.
During my first staff meeting, my first week on the job, I challenged the consulting staff about what they did to study testing on their own time. My challenge was met with an awkward silence, after which one of the consultants, sounding soul-wounded, told me he was offended that I would suggest that they weren’t already good enough as testers, “These are the best people I’ve ever worked with” said the twenty-something tester with little experience and no public reputation. “But how do you know they are good?” I asked, knowing that our company had just issued a press release about having hired me (a “distinguished industry pioneer” to quote it exactly). There were other murmurs of annoyance around the table, and the manager stepped in to change the subject. I could have pushed the issue, but I didn’t. I needed the job, so I relented against my better judgment.
I was later told that despite my company’s public position, the other consultants felt that I was a mere armchair expert, whereas they were practical men. I don’t know what evidence they had for that. They never showed me what they could do that I supposedly could not. Management tolerated this attitude. That means they were lying directly to their customers about me– claiming I was an expert when clearly they did not believe I was one. I could have insisted they behave in accordance with their public statements about me. But… I needed the job, so I relented against my better judgment.
I knew the day had come when I must quit because I found myself fantasizing about throwing chairs through windows. That triggered a sort of circuit-breaker of judgment: change your life now, now, now.
So what happened after that?
I suffered for this decision. First came the panic attack. I felt dizzy and it was hard to breathe for a few hours. This was followed by a few years of patching together a project here and a project there, never more than 8 weeks from completely running out of money and credit. We were twice within a week of filing for bankruptcy in the early days. During that time I walked away from a few more projects. I resigned from a dysfunctional government project, hopefully saving valuable taxpayer dollars by not writing a completely unnecessary Software Configuration Management plan that no one on the team wanted. I got myself fired from a project at Texas Instruments after about 90 minutes, because I told them a few things they didn’t want to hear (but that I felt were both true and important).
It’s not all suffering, of course. I once was fired from a project (along with the rest of the test team) and then was the only one hired back– partly because the client realized that my high standards meant that I billed far fewer hours than other consultants. In other words, saying no and being a troublemaker earned me another 500 hours of work, while the yes-sayers lost their situations. I also got some great gigs, including my very first one as an independent, specifically because I am a rabble-rousing kind of thinker.
These days, I cultivate as many clients as I can, so that I don’t rely too much on any one of them. And I have established a reputation for being honest and blunt that generally prevents the wrong sort of people from trying to hire me. It’s not easy, but it can be done: I have made integrity my top priority.
What about before I was well known?
Well, I’ve always had this attitude. It’s not some luxury to me. It’s fundamental. That’s why I had to leave high school. I’ve never been able to “play the game” at the expense of feeling like a good honest man. Like I said, I suffered for it. I wanted to go try myself at MIT, where my much more pliable best friend from high school eventually graduated. I am born to be an academic, but since I can’t stand the compliance-to-ceremony culture of the academic world, I must be an independent scholar, without access to expensive journals and fantastic libraries.
Before anybody heard of me, I treated getting work like dating: be a slightly exaggerated version of myself so that I will be rejected quickly if the relationship is not a fit (a stress testing strategy, you might say). My big break came at Apple, where I worked for a man of vision and good humor who seemed to relish being the mentor I needed. The environment was open and supportive. There was an element of luck in that my first ten years in testing I worked for people who didn’t ask me to tell lies or do bad work on purpose.
So I know it’s possible to find such people. They are out there. You don’t have to work for bozos, and if you currently do, there is yet hope.
A person who does not live true to himself feels sick and weak inside. My identity as “excellent software tester” demands that I take my craft seriously. I hope you will take this craft seriously, too.
P.S. What if my sense of identity doesn’t require me to be good at my job?
Then, technically, none of this applies to you. Your ethical code can include doing bad work. But… why are you reading my blog? How did you get in? Guards! Seize him!
I decided to write a second part to my “4 reasons why Appium is not ready for prime time” article. There was an interesting comment posted byÂ Satyajit Malugu which I thought required a post by itself because it raises some important questions. Hopefully, some additional comments will keep coming as I’d like to inform as much as possible testers and developers about the current state of Appium.
Here are the 4 comments made by Satyajit
1. No implicit waits â€“ Appium has good support for implicit waits. Ruby_libÂ is using them and my tests rarely have to wait on something. Let me know I can provide my logs of this works.
I’m not a big fan of implicit waits as they don’t offer flexibility for specific elements you want to wait on. You basically define a global wait-for-element for the entire duration of the test. Typically I woud want to use different type of waits wether I’m dealing with screen-flow for my app or if I need to wait for the return of a back-end call. Implicit waits don’t give me this flexibility. Additionally, I can’t specify the element I want to wait on. Finally, I didn’t see any capabilities to wait for other element’s charactertics such as count, color, enable/disable, etc. Explicit waits are much more powerful and flexible. That’s why we spent a lot of time implementing them in our product and we add new ones almost every week based on the feedback we receive from our community.
2. No parallel runs â€“ Appium/sauce labs philosophy is to use emulators in their cloud and the support parallel tests, its in their road map but not there yet
I have 2 problems with this comment:
3. Gestures â€“ there is support for gestures, not in the most easiest of the ways but I am able to flick, swipe, long tap etc. Now tell me which other framework provides you even that?
Mobile apps rely heavily on gestures and support for ALL OF THEM in a mobile test automation product is critical. It shouldn’t require any coding and should be easy to use. Appium is not there yet and somehow I find it perfectly normal. I’ve seen the SOASTA engineering team working endless hours to support ALL OF THEM across native, hybrid and Mobile Web apps on both iOS and Android. It’s a LOT OF WORK but this is the price to pay to offer ease of use and full support to testers and developers on all platforms. We today support 60+ standard gestures on both iOS and Android, so yes it is possible.
4. Limited support for android < 4.1. Now is it appium’s fault that google decided to come up with new framework 17+? Also there are couple of projects underway to merge selendroid into appium, or using espresso. If you are motivated enough, you could create implement a page object design pattern and use selendroid and appium in the same test suite.
When you build an automation framework, you look at your market and you’re trying to support the majority of your users. Today, Appium supports 18.2% of the market. If I’m a developer or a tester, is this sufficient? Can I wait for Appium to come up with support for < Android 4.1? Or should they not start their automation project until carriers and manufacturers decide to upgrade the phones of their users? Should they mess with Selendroid and read pages of support forums to make their solution work for Android < 4.1? Appium decided to invest in the future and that’s fine. But it means that testers will use the solution … in the future.
Keep the comments coming!Get Shareaholic
This post is not about the sort of testing people talk about when nearing a release and deciding whether it’s done. I have another word for that. I call it “testing,” or sometimes final testing or release testing. Many projects perform that testing in such a perfunctory way that it is better described as checking, according to the distinction between testing and checking I have previously written of on this blog. As Michael Bolton points out, that checking may better be described as rejection checking since a “fail” supposedly establishes a basis for saying the product is not done, whereas no amount of “passes” can show that it is done.
Acceptance testing can be defined in various ways. This post is about what I consider real acceptance testing, which I define as testing by a potential acceptor (a customer), performed for the purpose of informing a decision to accept (to purchase or rely upon) a product.
Do we need acceptance testing?
Whenever a business decides to purchase and rely upon a component or service, there is a danger that the product will fail and the business will suffer. One approach to dealing with that problem is to adopt the herd solution: follow the thickest part of the swarm; choose a popular product that is advertised or reputed to do what you want it to do and you will probably be okay. I have done that with smartphones, ruggedized laptops, file-sharing services, etc. with good results, though sometimes I am disappointed.
My business is small. I am nimble compared to almost every other company in the world. My acceptance testing usually takes the form of getting a trial subscription to service, or downloading the “basic” version of a product. Then I do some work with it and see how I feel. In this way I learned to love Dropbox, despite its troubling security situation (I can’t lock up my Dropbox files), or the fact that there is a significant chance it will corrupt very large files. (I no longer trust it with anything over half of a gig).
But what if I were advising a large company about whether to adopt a service or product that it will rely upon across dozens or hundreds or thousands of employees? What if the product has been customized or custom built specifically for them? That’s when acceptance testing becomes important.
Doesn’t the Service Level Agreement guarantee that the product will work?
There are a couple of problems with relying on vendor promises. First, the vendor probably isn’t promising total satisfaction. The service “levels” in the contract are probably narrowly and specifically drawn. That means if you don’t think of everything that matters and put that in the contract, it’s not covered. Testing is a process that helps reveal the dimensions of the service that matter.
Second, there’s an issue with timing. By the time you discover a problem with the vendor’s product, you may be already relying on it. You may already have deployed it widely. It may be too late to back out or switch to a different solution. Perhaps your company negotiated remedies in that case, but there are practical limitations to any remedy. If your vendor is very small, they may not be able to afford to fix their product quickly. If you vendor is very large, they may be able to afford to drag their feet on the fixes.
Acceptance testing protects you and makes the vendor take quality more seriously.
Acceptance testing should never be handled by the vendor. I was once hired by a vendor to do penetration testing on their product in order to appease a customer. But the vendor had no incentive to help me succeed in my assignment, nor to faithfully report the vulnerabilities I discovered. It would have been far better if the customer had hired me.
Only the accepting party has the incentive to test well. Acceptance testing should not be pre-agreed or pre-planned in any detail– otherwise the vendor will be sure that the product passes those specific tests. It should be unpredictable, so that the vendor has an incentive to make the product truly meet its requirements in a general sense. It should be adaptive (exploratory) so that any weakness you find can be examined and exploited.
The vendor wants your money. If your company is large enough, and the vendor is hungry, they will move mountains to make the product work well if they know you are paying attention. Acceptance testing, done creatively by skilled testers on a mission, keeps the vendor on its toes.
By explicitly testing in advance of your decision to accept the product, you have a fighting chance to avoid the disaster of discovering too late that the product is a lemon.
My management doesn’t think acceptance testing matters. What do I do?1. Make the argument, above. 2. Communicate with management, formally, about this. (In writing, so that there is a record.) It is up to management to make decisions about business risk. They may feel the risk is not worth worrying about. In that case, you must wait and watch. People are usually more persuaded by vivid experiences, rather than abstract principles, so: 1. Collect specific examples of the problems you are talking about. What bugs have you experienced in vendor products? 2. Collect news reports about bugs in products of your vendors (or other vendors) that have been disruptive. 3. In the event you get to do even a little acceptance testing, make a record of the problems you find and be ready to remind management of that history.
There is a cool initiative going on from the folks atÂ Tea Time With TestersÂ andÂ QA Intelligence. They’re launching a survey to help us understand the state of testing! What are the main challenges in the profession? Where is theÂ profession headed? I’m definitely supporting the initiative and interested in the results! I’m following the software testing trends via analysts but a survey coming from the community could be quite insightful!
Note: I have no affiliation with the survey.Get Shareaholic
As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. IÂ particularlyÂ like when I see some innovation, somethingÂ that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.
These are the main shortcomings I’ve noted. Some are really showstoppers for me.
#1 – No support for intelligent waits.Â
This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:
Without being able to wait on a UI elementÂ characteristicsÂ (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.
#2 – On iOS, you can only execute one test at a time per Mac.
This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.
#3 – Limited support for gestures.
#4 – Limited support for Android < 4.1
Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.
These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.
Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.Get Shareaholic