Combinatorial Testing for Software
Posted on March 26, 2009 Comments (8)
Combinatorial testing of software is very similar to the design of experiments work my father was involved in, and which I have a special interest in. Combinatorial testing looks at binary interaction effects (success or failure), since it is seeking to find bugs in software, while design of experiments captures the magnitude of interaction effects on performance. In the last several years my brother, Justin Hunter, has been working on using combinatorial testing to improve software development practices. He visited me this week and we discussed the potential value of increasing the adoption of combinatorial testing, which is similar to the value of increasing the adoption of the use of design of experiments: both offer great opportunities for large improvements in current practices.
Automated Combinatorial Testing for Software
Practical Combinatorial Testing: Beyond Pairwise by Rick Kuhn, US National Institute of Standards and Technology; Yu Lei, University of Texas, Arlington; and Raghu Kacker, US National Institute of Standards and Technology.
These results are not conclusive, but they suggest that the degree of interaction involved in faults is relatively low, even though pairwise testing is insufficient. Testing all four- to six-way combinations might therefore provide reasonably high assurance.
Related: Future Directions for Agile Management – The Defect Black Market – Metrics and Software Development – Full and Fractional Factorial Test Design – Google Website Optimizer
Categories: Design of Experiments, Process improvement, Quality tools, Software Development
Tags: Bill Hunter, Design of Experiments, Process improvement, Public Sector, Software Development
8 Responses to “Combinatorial Testing for Software”
Leave a Reply
RSS Feed
May 28th, 2009 @ 11:34 am
John,
Thanks for getting the word out about the efficiency and quality benefits available through pairwise software testing and other forms of combinatorial software testing. It is surprising to me that the benefits are not more broadly known by testing community, given how fast, easy, and unambiguous it is to measure the benefits.
For any of your readers interested in finding out more about combinatorial testing, I would recommend a couple clear and concise articles posted at:
http://www.combinatorialtesting.com/clear-introductions-1
and the pairwise testing instructional videos posted at:
http://www.combinatorialtesting.com/videos
- Justin
August 7th, 2009 @ 1:15 am
And also, please check out Hexawise, the new test design tool I have developed that uses these Design of Experiments principles to improve the efficiency and effectiveness of software testing. Proof of concept pilot projects we are currently conducting at multiple companies are showing the following results: (1) Reduction in time it takes to identify and document test cases of 30-40%, (2) Reduction in test execution times of approximately 25-35% (because fewer, more effective test cases are run), and (3) Reduction in cost to fix defects (because more defects are found sooner with the more effective test cases).
Free trials (and a free version) are available at: http://hexawise.com/users/new
Thank you.
- Justin
Founder and CEO of Hexawise
http:www.hexawise.com
“More coverage. Fewer tests.”
August 15th, 2009 @ 2:29 pm
The talk provides a good background on their experience using ruby to manage projects; and how they manage ruby application development projects…
August 15th, 2009 @ 2:34 pm
One of the reasons my organizations switched to Ruby on Rails for software development was the great integration with automated testing…
October 10th, 2009 @ 7:01 am
I appreciate that you are helping spread the word about Combinatorial Testing software. I really wish this field was expanded on by the experts!
Thanks again
March 9th, 2010 @ 9:15 am
if a software developer changes the code, the automated tests are all run and if there is an error noted by the automated testing the code cannot be deployed to the production environment… Thus automated testing mistake proofs the process
May 17th, 2010 @ 9:18 am
When you have a situation that has many many many possible parameters and each time only a few possible choices (a few items you are trying to vary and test – in his example in the video, 2 choices) you wind up with a ridicules number of possible tests if you try to vary one variable at a time. But you can cover all the possibilities in just 30 tests if your coverage target is all possible pairs…
August 24th, 2010 @ 1:10 pm
Mistake-proofing a process both improves it (many poka-yoke solutions make the process easier to use) and prevents an error in case you still try something wrong. So I see the automated tests as a way to serve as a backstop, in case the process improvement you made to the software development process failed in some form…