Combinatorial Testing – The Quadrant of Massive Efficiency Gains

Posted on May 17, 2010  Comments (2)

My brother, Justin Hunter, gives a lightning talk on Combinatorial Testing – The Quadrant of Doom and The Quadrant of Massive Efficiency Gains in the video above. The following text is largely directly quoted from the talk – with a bit of editing by me.

When you have a situation that has many many many possible parameters and each time only a few possible choices (a few items you are trying to vary and test – in his example in the video, 2 choices) you wind up with a ridicules number of possible tests. But you can cover all the possibilities in just 30 tests if your coverage target is all possible pairs. When you have situations like that you will see dramatic efficiency gains. What we have found in real world tests is greatly reduced time to create the tests and consistently 2 to 3 times as many defects found compared to the standard methods used for software testing.

You can read more on these ideas on his blog, where he explores software testing and combinatorial testing. The web base software testing application my brother created and shows in the demo is Hexawise. It is free to try out. I recommend it, though I am biased.

Related: Combinatorial Testing for SoftwareVideo Highlight Reel of Hexawise – a pairwise testing tool and combinatorial testing toolYouTube Uses Multivariate Experiment To Improve Sign-ups 15%What Else Can Software Development and Testing Learn from Manufacturing? Don’t Forget Design of Experiments (DoE)Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

Justin posted the presentation slides online at for anyone who is interested in seeing more details about the test plan he reviewed that had 1,746,756,896,558,880,852,541,440 possible tests. The slides are well worth reading.

Lanette Creamer selected a pair of test input values during the lightning talk. As it turns out, like 80% of the other possible pairs of inputs she could have selected, the pair of values she selected were tested together within the first 10 tests of the plan. (The second test case, in fact). The other 20% of possible pairs she could have identified were tested together somewhere between test #11 and test #36.

The significance of testing for every single possible pair of values is addressed in slide number 12 (of the slides Justin posted).

If users wanted more thorough sets of tests (e.g., testing for all possible triples of values, or quadruples, or even and incredibly thorough set that would test for all possible combinations of 6 input values), the Hexawise test generation tool can also automatically generate test sets meeting each of these increasingly thorough objectives.

Quadrant of doom?
If you have very few parameters and very large test design challenges the benefits of combinatorial testing are reduced. Justin calls this state the quadrant of doom. I must admit I can see how the benefits are reduced, I don’t really see how they are eliminated. And I certainly don’t see how using combinatorial testing insight would create actual harm – but this is a lightning talk so he didn’t have long to explain the doom moniker.

I want to thank Selena Delesie, who posted the video on YouTube.

2 Responses to “Combinatorial Testing – The Quadrant of Massive Efficiency Gains”

  1. Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed — Hexawise
    February 3rd, 2012 @ 4:16 am

    [...] Among the many benefits Hexawise provides is creating a test plan that maximizes test coverage with each new scenario tested. The graph above shows that after just 20 test 83.9% of the test combinations have been tested. Read more about this in our case study of a mortgage application software test plan. Just 48 test combinations are needed to test for every valid pair (3.7 million possible tests combinations exist in this case). If you are lost now, this video may help. [...]

  2. Getting Started with a Test Plan When Faced with a Combinatorial Explosion — Hexawise
    December 26th, 2012 @ 4:56 am

    [...] often is daunting. And the problems of creating an unfathomable number of tests face you due to combinatorial explosion. Hexawise is a software as a service that aids in dealing with this dilemma for software testers. [...]

Leave a Reply





  • Recent Trackbacks

  • Comments