Combinatorial Testing for Software

Combinatorial testing of software is very similar to the design of experiments work my father was involved in, and which I have a special interest in. Combinatorial testing looks at binary interaction effects (success or failure), since it is seeking to find bugs in software, while design of experiments captures the magnitude of interaction effects on performance. In the last several years my brother, Justin Hunter, has been working on using combinatorial testing to improve software development practices. He visited me this week and we discussed the potential value of increasing the adoption of combinatorial testing, which is similar to the value of increasing the adoption of the use of design of experiments: both offer great opportunities for large improvements in current practices.

Automated Combinatorial Testing for Software

Software developers frequently encounter failures that occur only as the result of an interaction between two components. Testers often use pairwise testing – all pairs of parameter values – to detect such interactions. Combinatorial testing beyond pairwise is rarely used because good algorithms for higher strength combinations (e.g., 4-way or more) have not been available, but empirical evidence shows that some errors are triggered only by the interaction of three, four, or more parameters

Practical Combinatorial Testing: Beyond Pairwise by Rick Kuhn, US National Institute of Standards and Technology; Yu Lei, University of Texas, Arlington; and Raghu Kacker, US National Institute of Standards and Technology.

the detection rate increased rapidly with interaction strength. Within the NASA database application, for example, 67 percent of the failures were triggered by only a single parameter value, 93 percent by two-way combinations, and 98 percent by three-way combinations.2 The detection-rate curves for the other applications studied are similar, reaching 100 percent detection with four- to six-way interactions.
These results are not conclusive, but they suggest that the degree of interaction involved in faults is relatively low, even though pairwise testing is insufficient. Testing all four- to six-way combinations might therefore provide reasonably high assurance.

Related: Future Directions for Agile ManagementThe Defect Black MarketMetrics and Software DevelopmentFull and Fractional Factorial Test DesignGoogle Website Optimizer

9 thoughts on “Combinatorial Testing for Software

  1. Justin


    Thanks for getting the word out about the efficiency and quality benefits available through pairwise software testing and other forms of combinatorial software testing. It is surprising to me that the benefits are not more broadly known by testing community, given how fast, easy, and unambiguous it is to measure the benefits.

    For any of your readers interested in finding out more about combinatorial testing, I would recommend a couple clear and concise articles posted at:

    and the pairwise testing instructional videos posted at:

    – Justin

  2. Justin

    And also, please check out Hexawise, the new test design tool I have developed that uses these Design of Experiments principles to improve the efficiency and effectiveness of software testing. Proof of concept pilot projects we are currently conducting at multiple companies are showing the following results: (1) Reduction in time it takes to identify and document test cases of 30-40%, (2) Reduction in test execution times of approximately 25-35% (because fewer, more effective test cases are run), and (3) Reduction in cost to fix defects (because more defects are found sooner with the more effective test cases).

    Free trials (and a free version) are available at:

    Thank you.

    – Justin
    Founder and CEO of Hexawise
    “More coverage. Fewer tests.”

  3. Pingback: Curious Cat Management Blog: Three Years of Real-World IT Projects In Ruby

  4. Pingback: Curious Cat Management Blog: Baking in Quality to Software Development

  5. Tim

    I appreciate that you are helping spread the word about Combinatorial Testing software. I really wish this field was expanded on by the experts!

    Thanks again

  6. Pingback: Curious Cat Management Blog » Improving Software Development with Automated Tests

  7. Pingback: Curious Cat Management Improvement Blog » Combinatorial Testing – The Quadrant of Massive Efficiency Gains

  8. Pingback: Curious Cat Management Improvement Blog » Mistake Proofing Deployment of Software Code

  9. Pingback: Getting an Early Appreciation for Deming’s Ideas « The W. Edwards Deming Institute Blog

Leave a Reply

Your email address will not be published. Required fields are marked *