Founder and CEO of @TentamenHr. Organizer of Zagreb Software Testing Club meetups.
895 stories

Testing For Me

1 Share

I spoke at UKSTAR 2018 this week, an eight-minute talk in a Storytelling track. This post is a prettied-up version of the notes I made for it along with some of the slides. The full slide deck is in the Media page.

My story is called The Anatomy of a Definition of Testing. It's not a suspense story though, so I'll give you the definition right up front:
Testing is the pursuit of relevant incongruity.
That is, for me, testing is the pursuit of relevant incongruity. But how did I get there?

Well, the journey started with Explore It! by Elisabeth Hendrickson, a great book about exploratory testing which has, near the beginning, this definition:
Tested = Checked + Explored
It's finessed a little by
Neither checking nor exploring is sufficient on its own
and the idea that testing is to
... interact with the software or system, observe its actual behavior, and compare that to your expectations.
Interestingly, the definition doesn't really play a significant part in the rest of the book, but as I was reading I kept on returning to it, asking myself questions like these:

Eventually I thought I'd ask Elisabeth whether she could help me out (she said she would!) and we exchanged a few emails, and that helped me to clarify my thoughts. But during the conversation, I began to wonder was this process, of thinking about testing, itself testing? I mean, I was doing things that were consistent with things I do when testing:

But was I really testing? By Elisabeth's definition, I wasn't sure that I could say I was. But it felt a lot like testing. So I went looking for other definitions and found loads!

And I recognise aspects of all of those in my testing, but none of them capture all that testing is for me. Reflecting further, I remembered a talk that Rikard Edgren gave at EuroSTAR 2015 where he said this beautiful thing:
Testing is simple: you understand what is important and then you test it.
Adam Knight has talked and written about Fractal Exploratory testing, and describes it like this:
as each flaw ... is discovered ... [a] mini exploration will result in a more targeted testing exploration around this feature area
To me, they're both talking about how investigation of one area leads to investigation in a subset of that area, and in a subset of that area. A kind of traversal through the testing space where the actions that are performed at any level are similar. And I recognise that too, but it's not all that testing is. For me.

I tried to draw my testing on a feature. It looked like this.

Sometimes multiple activities feed into another. Sometimes one activity feeds into multiple others. Activities can run in parallel, overlap, be serial. A single activity can have multiple intended or accidental outcomes ...

I tried to draw it again. It looked like this:

A vision of an attempt to somehow keep a lid on the ambiguity, and the unknowns, and the complexity in order to be able to get on and test.

A colleague pointed me at a drawing by John Bach of his testing.

That scribble on the left is not necessarily confusion and chaos, but cycling, cross-checking, confirming, until a course of action that seems propitious can be identified and followed out to the right with occasions when exploration goes deep down. And, naturally, I recognise that in my testing too. But it isn't all that testing is for me.

So, without a definition but with a lot of thoughts about what I wanted from my definition I tried to list factors of testing and then come up with a definition that covers them all.

And I thought about it for a good long time. I mean, really long. And eventually out popped my definition:
Testing is the pursuit of relevant incongruity.
But that's a bit of a mouthful. Let's unpack it a little:

Incongruity: Oxford Dictionaries define this as "not in harmony or keeping with the surroundings". I interpret lack of harmony as a potential problem and lack of keeping as an actual problem, and those two interpretations are interesting and useful in testing.

Pursuit: Again, there are two senses that capture important aspects of testing for me. You can pursue something that you don't know is there and that you may never find, like a dream. Or you can pursue the solution to a problem that you know you have, that's right in front of you.

Why not some other verb? For me, an investigation has identified something to investigate, finding requires something to be found and I want to be able to say I tested even if I found no issues, exploration can work but I don't want my definition to be thought of as a definition of exploratory testing, much as I love it.

Relevant: if this work doesn't matter to anyone, why are we doing it? Whoever that is can help us to understand whether any incongruities we identify are valuable to them, relevant to the project.

So that's my definition:
Testing is the pursuit of relevant incongruity.
Notice that it says nothing about particular techniques or methods, or products, or systems. It exists, deliberately in a space where it must be applied in context, at a time, for a purpose.

But what do I do with it?

Well, any time I'm working I can ask myself whether what I'm doing is contributing the pursuit of relevant incongruity. If it is, I'm testing — I still have the question of whether I'm testing the right things, at the right time but that's a different problem for another story.

If I'm not in pursuit of relevant incongruity I can ask whether I should be, and why. Sometimes it's legit to take off your tester hat and do something else, project housekeeping for example, because it needs doing, because you've got the skills, because you can be most efficient, because it's your turn, or whatever. But sometimes it can provoke me into thinking that I'm not doing what I should be.

Which is great, and I buzzed along very nicely using it. And then I heard Michael Bolton say this on the Quality Remarks podcast:
The goal of testing is identifying problems that matter
And I thought "Hello! That feels pretty familiar!" Although "problems" loses the subtlety of "incongruity", and "finding" I already said I have some reservations about, and note that he's talking about the "goal" of testing, not testing itself. But still, there's a similar sentiment here, and look how much snappier it is than mine!

So I asked him about it, and he said "many things stay in the mind more easily when they can be expressed concisely and snappily."

Which is true to my experience, and also very useful, because it emphasises his different need. He's a teacher, and he wants to share his description and have his pupils remember it. And that's OK, it keeps me humble: I shouldn't impose my view of testing on others, and other views can be just as valid.

And so that's my definition:
Testing is the pursuit of relevant incongruity.
It's my definition to help me do what I need to do in my context. It's for me. But I hope it was of some interest to you...
Image: Jit Gosai (via Twitter)
Read the whole story
6 hours ago
Share this story

Continuous Delivery using Crowdsourced Testing

1 Share

On Wednesday 14. March 2018 I presented my talk “Continuous Delivery using Crowdsourced Testing” at the Swiss Testing Day conference. It was a pleasure to talk about the topic which occuped lots of my time and thoughts in the last one and a half years. Even if presenting on the main stage at the Samsung Hall made me just a bit nervous at the beginning, I really enjoyed it.

I’d like to underscore the most important learnings I’ve made working intensively with crowdsourced testers and incorporating the crowdsourced manual test cycles into our overall agile testing strategy, while working in a fast-paced DevOps environment. 


It is important to understand what continuous testing is and where it fits into the DevOps culture and mindset.

Continuous testing is a process of executing many different types of tests — both manual and automated – continually throughout the delivery process
to obtain immediate feedback
on the business risks
associated with a software release candidate.

  • Definition adapted from Jez Humble. I encourage you to read his Continuous Delivery book.


Testing actually happens on every stage of the DevOps toolchain. Note that this article from Dan Ashby is from October 2016 but lots of the people still don’t know about it and how true it is.


In our efforts to continuously deliver online storage solutions for our private customers we never forget the manual validations. Automated tests run continuously on our development, integration and test environment but how do you scale and get fast feedback on your UI (web, mobile apps) or usability tests? Also, you can’t automate all your customers flows.

Continuous Deployment Pipeline – note the manual validations phase


Ideally you should, but usually you don’t have an experienced tester in every cross-functional development team. Also, lots of the time it can happen that the team can’t finish all manual or automation tasks before the sprint ends (here I’m not talking about the other causes for sprint-spillover). And even if the team tested all sprint artefacts, how much time would they have to run regression, exploratory, usability or user beta tests?

To compensate the activities for which you’re not skilled or just don’t have enough people to get dirty (which IS part of the game) you can hire a testing service provider who maintains the crowd: community of experienced testers and IT specialists. Having that complementary testing service to internal (testing) teams with a focus on the end-user experience, will certainly help you reach your sprint goals and increase the quality of your products.


Main advantages of crowdsourced testing

  • Resources flexibility
  • Recources scalability
  • Fixed price contract normally includes the hours of CT company’s internal stuff (project manager, technical leads) as well as any possible compensations to the testers for the issues they find.
  • Provides user point of view needed for testing user-centric apps
  • Crowd is not confirmation biased
  • User surveys can be included in every test cycle
  • You can also test mocks and wireframes
  • Crowd testers are rated based on their performance on past projects -> you get an access to a large pool of good testers


Even if the crowd test execution cycle and the whole process might look simple there are a lot of challenges to solve on the way.

  • depending on the product your developers might get overwhelmed with the number of found issues. Some of them will start rejecting them or the process in general.
  • lots of the results found might lack the significance or impact, meaning they get approved as somewhat valuable issues. Remember we want to find those exceptionaly valuable issues that involve business risks.
  • internal coordination & communication time needed can develop into another bottleneck, especially if you’re the only one capable or willing to do those tasks. Already after I started, I found myself spending the half of my working time “only” on preparing next cycles and analysing the results of the current or past ones. That didn’t leave me with enough time for other strategic activities or building up on my test-automation skills.
  • even if the crowd is huge you still paid for some particular number of testing hours per month and having only good testers claiming their slots on a first come first served basis is pure luck. Of course you can invite the names from your “top 20” list but nobody guarantees you they will  really claim their slot and test
  • besides having the good testers it is very important to have a great testing team lead on the CT company side. Imagine having and then loosing a person who’s a brain of your test execution cycles who knows and understands your products, the known issues, remembers what was found in the last weeks and why. When such a valuable resource is not available anymore, it will take lots of luck and some weeks to onboard the other one (or you will have even more work).


There are certain strategies to solve those problems but you need to know where to start and that is normally where it hurts the most.

  • Stick to (your) timeboxed time and delegate all different preparation or follow-up tasks to your team -> be persistent in that and don’t give up
  • Educate the team about the importance of “three amigos” principle, meaning that not only Product Owner is syncing with you on the issues found or on the tests planed to be run or updated, but the developers as well
  • Start buddying and pairing with those developers who value the CT process and don’t forget to appreciate and thank them for their help and support
  • Make sure that the issues found from crowd test cycles get into the next bug refinement
  • Track improvements
  • Continuously adjust your strategy and process and make it flexible
  • Never forget to star and reward good testers and even send them a personal thank you note
  • Never stop educating your team
  • Never stop learning 🙂
    • Also learn from other testers findings in terms of improving your testing skills and mind-maps


Further things to keep in mind:

  • preparation is the key ingredient to your success and good input creates the good output
  • information on which the particular test cycle is based need to be clear and precise
    • Test scope / products to test
    • Software version / environment
    • Test plan (if you want them to run regression based on test scripts)
    • Known issues list
    • Internal release notes
    • Sprint review video demo & presentation slides if any
  • be clear about how you’re using the CT report dashboard with a results summary per risk, devices, browsers, technical or functional etc.
  • concentrate on testing with the devices your real users use the most and get those numbers from the analytics
  • beware the internal costs needed for coordinating the whole process 



  • Know your CT goal & purpose and aim for the “highest” added value:
    • What
    • How
    • When do you want to crowdsource your tests (in our project it’s normally towards the end of the sprint, every 2nd week)
  • CT won’t solve all your testing, QA or software development problems, will though help with functional or user tests
  • test internally as well 
  • every cycle should include explorative tests – the results are much better than with regression test scripts
  • setup the two-way integration with your JIRA
  • insist on keeping the good testers in your next cycle and try to attract them by giving new products and features to test, otherwise they might get bored or will stop testing your current product(s) since they know they won’t find any more valuable issues (less money earned for them)


We are using Applause manual testing services (formerly: utest) and have been satisfied especially with the professionality of their internal stuff who really knows about testing. Check out their how-to pages to understand in more detail how bug verification, two-way JIRA integration and test management  process work. If you have any questions where I could certainly help with my experience and tips, don’t hesitate to contact me.


I’ll write about the conference in the next post. Stay tuned. 🙂 

Read the whole story
6 hours ago
Share this story

Using “Six Honest Serving Men” as a problem analysis tool

1 Share

Rudyard Kipling was one of the best-loved poets of all time. His work was celebrated for its versatility and passion. He was the first English-language writer to receive the Nobel Prize for Literature and, to date, he remains its youngest recipient. Despite being written over 100 years ago, Kipling’s short Six Honest Serving Men poemContinue reading →

The post Using “Six Honest Serving Men” as a problem analysis tool appeared first on Quality is King by Peter Gould.

Read the whole story
7 hours ago
Share this story

Real Life TDD – Test Case Analysis, Part I

1 Share
How does TDD look like in real world development of microservices or web interfaces? That's what this series is all about.
1. Introduction 2. The requirements 3. Test case analysis, Part I

It is customary to think that in TDD we run ahead and just write tests. Not today, Spring will have to wait.

Let’s break down the requirement into test cases, so we can understand them better. Buckle your seat belts, it’s going to be a bumpy ride.

We’ll translate the requirements into “when-then” situations. Here’s the first one:

  • When the calculator starts it displays “0”.

That’s easy. No pressing of keys shows zero. We’ll write examples so many times, so let’s use shorthand for the input to the output.

Nothing => “0”

Let’s continue.

  • The calculator has a keypad with digits (0-9), operations (+,-,*,/), a C clear button and an ‘=‘ button to execute the calculation.

We all know calculators (hopefully) and how they work. We instinctively understand that the specified keys are clickable, while all other keys are not. However we can’t translate this requirement into an example. That’s ok, we can for the next one.

  • Any other key pressed doesn’t affect the display

So for example:

“E” => No change

Wait a minute. There’s no “E” on the keypad. What kind of requirement is this?
A confusing one. Check out how it makes sense when we read the requirements the first time, but when we read it again it’s not helpful.

Maybe because “key” is something we press on the computer’s keyboard, not necessarily the keys on the screen.

Or maybe, we’ve already translated it in our mind into “service” language. API services can accept other keys, but don’t necessarily do anything with them. Hmm… Let’s ask the product manager.

PM: Let’s make it easy – we can accept only the keys available in the GUI. No need to translate it into code, error handling or anything. We can throw that example away.


  • Pressing any digit key, the display adds that digit to the existing display. The exception is ‘0’ which is presented only if a number different from ‘0’ wasn’t already pressed.

Ok. That’s easy enough. Let’s start with the obvious.

“1” => “1”
“2” => “2”

Wait. Do we need all nine examples? After all, these examples translate later into tests, so do we need to write nine tests?

We probably don’t. Examples may look different from each other, but the code would probably be the same for all of them. We don’t know it yet (that’s TDD for you), but knowing the domain and our coding skills, it’s ok for now to have one example of the digits.

Also, ours is a very simple calculator, not a nuclear facility management calculator. If there’s a bug in our calculator (for example: “8” => “k”), our calculator may now work, but it also won’t cause a Chernobyl-style meltdown. If there are high risks, I may want to add the test cases up-front to make sure they work.

Ok, that’s one digit. Let’s do two:

“3”,”4″ => “34”

Here’s another thing: All things being equal, I could choose the following:
“1”,”1″ => “11”

Again, while we didn’t program anything yet, we feel that these examples are equivalent in terms of how the code will behave. But by using more digits in my examples, I’m reducing the risks that the code will work for only keypads with, let’s say, one key. Also, consider that examples are a form of documentation: they explain the capabilities of our calculator even better than the code itself. With more diversity we understand it better.

We might want to add a 3 digit example as well:

“4”,”6″,”8″ => “468”

When should we stop? Again a question to the PM: How big is the display?

PM: (Thinking) Our budget is enough for 10 digits.

Ok, we want an example for the edge case:

“1”,”2″,”3″,”4″,”5″,”6″,”7″,”8″,”9″,”0″ => “1234567890”

I know what your thinking. What happens on the next key?

PM: (Thinking again) the display doesn’t change.

So the example would be:
“1”,”2″,”3″,”4″,”5″,”6″,”7″,”8″,”9″,”0″,”1″ => “1234567890”

Ok, maybe the display doesn’t change, but how about the values? is the last case really “1234567890”, or is it “12345678901” in disguise?

PM: Nope, what you see is what you get.

How do we put that into an example? For the time being we haven’t delved into actual calculation, but let’s do that now for this case:

“1”,”2″,”3″,”4″,”5″,”6″,”7″,”8″,”9″,”0″,”1″,”+”,”1″,”=” => “1234567891”

Now we can expect the calculator understand the values correctly.

Did you really think you’ll see Spring, or even some code by now? You need patience young padawan. We’re just starting out, next time: Test Case Analysis – The Sequel.

Read the whole story
3 days ago
Share this story

Pen and Storyteller

1 Share

The latest in my occasional series of experiments in sketchnoting, this time from UKSTAR 2018. The sketchnote/video thing I did as a promo for my own storytelling talk is still available here

Read the whole story
3 days ago
Share this story

Testival Meetup #36, Zagreb, Croatia

1 Share

Regular talks:

  • Davor Banović: Acceptance test driven development. Is it a myth?

The best quote from the talk:

All good tests must come to an end.

Davor mentioned Chinese number gestures. I did not know that Chinese have a way of showing numbers 0-10 using just one hand. He mentioned it as an example of possible misunderstandings. With two extended fingers some people might intend to show number 2, but Chinese might understand it as number 7. I liked it so much, that I found an article about it and had a lightning talk.

Lightning talks:

  • Karlo Smid: What I learned on European Testing Conference 2018
  • Marko Kruljac: Practice your testing skills by playing board games
  • Željko Filipin: Chinese number gestures

We are always looking for better ways of timing lightning talks, and Krešimir Linke sad Google has a good timer. We have tried it and it works great in Chrome in full screen. Full screen does not work well in Firefox 😢 and it is not available in Chrome on Android. 😢

See you in April at Testival Meetup #37!


Read the whole story
3 days ago
Share this story
Next Page of Stories