Combinatorial Testing – The Quadrant of Massive Efficiency Gains

My brother, Justin Hunter, gives a lightning talk on Combinatorial Testing – The Quadrant of Doom and The Quadrant of Massive Efficiency Gains in the video above. The following text is largely directly quoted from the talk – with a bit of editing by me.

When you have a situation that has many many many possible parameters and each time only a few possible choices (a few items you are trying to vary and test – in his example in the video, 2 choices) you wind up with a ridicules number of possible tests. But you can cover all the possibilities in just 30 tests if your coverage target is all possible pairs. When you have situations like that you will see dramatic efficiency gains. What we have found in real world tests is greatly reduced time to create the tests and consistently 2 to 3 times as many defects found compared to the standard methods used for software testing.

You can read more on these ideas on his blog, where he explores software testing and combinatorial testing. The web base software testing application my brother created and shows in the demo is Hexawise. It is free to try out. I recommend it, though I am biased.

Related: Combinatorial Testing for Software – Video Highlight Reel of Hexawise – a pairwise testing tool and combinatorial testing tool – YouTube Uses Multivariate Experiment To Improve Sign-ups 15% – What Else Can Software Development and Testing Learn from Manufacturing? Don’t Forget Design of Experiments (DoE) – Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

Justin posted the presentation slides online at for anyone who is interested in seeing more details about the test plan he reviewed that had 1,746,756,896,558,880,852,541,440 possible tests. The slides are well worth reading.

Lanette Creamer selected a pair of test input values during the lightning talk. As it turns out, like 80% of the other possible pairs of inputs she could have selected, the pair of values she selected were tested together within the first 10 tests of the plan. (The second test case, in fact). The other 20% of possible pairs she could have identified were tested together somewhere between test #11 and test #36.

The significance of testing for every single possible pair of values is addressed in slide number 12 (of the slides Justin posted).

If users wanted more thorough sets of tests (e.g., testing for all possible triples of values, or quadruples, or even and incredibly thorough set that would test for all possible combinations of 6 input values), the Hexawise test generation tool can also automatically generate test sets meeting each of these increasingly thorough objectives.

Quadrant of doom?
If you have very few parameters and very large test design challenges the benefits of combinatorial testing are reduced. Justin calls this state the quadrant of doom. I must admit I can see how the benefits are reduced, I don’t really see how they are eliminated. And I certainly don’t see how using combinatorial testing insight would create actual harm – but this is a lightning talk so he didn’t have long to explain the doom moniker.

I want to thank Selena Delesie, who posted the video on YouTube.