Tag Archives: software testing

Interviews on Software Testing

I haven’t added a new post here recently. One of the things that has been keeping me busy is putting together some interviews on software testing. Here are excerpts from 3 of the interviews:

Testing Smarter with Alan Page

Hexawise: During your 20 years at Microsoft you have been involved in hiring many software testers. What do you look for when choosing software testers? What suggestions do you have for those looking to advance in their in software testing career?

Alan: The biggest thing I look for in testers is a passion and ability to learn. I’ve interviewed hundreds of testers, including many who came from top universities with advanced degrees who just weren’t excited about learning. For them, maybe learning was a means to an end, but not something they were passionate about.

The testers who really impress me are those who love to learn – not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

As far as suggestions go, keep building your tool box. As long as you’re willing to try new things, you’ll always be able to find challenging, fun work. As soon as you think you know it all, you will be stuck in your career.
…
Hexawise: It seems to me that testing games would have significant challenges not found in testing fairly straightforward business applications. Could you share some strategies for coping with those challenges?

Alan: Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That’s 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Beyond that, testing games requires a lot of human eyeballs and critical thinking to ensure gameplay makes sense, objects are in the right places, etc. I’ve never seen a case where automating gameplay, for example, has been successful. I have, however, seen some really innovative tools written by testers to help make game testing much easier, and much more effective.

Testing Smarter with Dorothy Graham

Hexawise: When looking to automate tests one thing people sometimes overlook is that given the new process it may well be wise to add more tests than existed before. If all test cases were manually completed that list of cases would naturally have been limited by the cost of repeatedly manually checking so many test cases in regression testing. If you automate the tests it may well be wise to expand the breadth of variations in order to catch bugs caused by the interaction of various parameter values. What are your thoughts on this idea?

Dot: Nice analogy – I like the term “grapefruit juice bugs”. Using some of the combinatorial techniques is a good way to cover more combinations in a reasonably rigorous way. Automation can help to run more tests (provided that expected results are available for them) and may be a good way to implement the combination tests, using pair-wise and/or orthogonal arrays.

Smarter Software Testing with James Bach

Continue reading →

Software Testing and the Impact on Quality

My response to a question on Reddit.

“Software quality does not come from testing”
Does anybody have any thoughts on the validity of the above statement?

That statement is similar to the idea you can’t inspect in quality. Basically “Inspection is too late. The quality, good or bad, is already in the product.” W. Edwards Deming

I agree with those ideas. Software testing is a bit different (at least some of it is) from the inspection mentioned above. You are testing while the product is being developed and adjustments are being made before the product is released to customers. Also with internet based software you have the ability to update the software and now all users have that update. Where for physical devices they already have the product and the only option is a recall which is very expensive and often ignored.

Software testing however should pay attention to those points in the 2 links above (defects should be understood as evidence of a process that needs to be improved so defects are not built in the first place). What you want is not just to fix the bugs software testers catch but figure out the reasons those bugs were created and improve you process so you create fewer bugs in the future.

No matter what the software quality is based on the code that is written. At the best software testing can tell people about the bugs but unless the code is fixed the software quality didn’t change. But to say that software testing doesn’t have a big influence of software quality when testing is well done and the software development process is good (listens to feedback and improves) is not very accurate.

Combinatorial Testing – The Quadrant of Massive Efficiency Gains

My brother, Justin Hunter, gives a lightning talk on Combinatorial Testing – The Quadrant of Doom and The Quadrant of Massive Efficiency Gains in the video above. The following text is largely directly quoted from the talk – with a bit of editing by me.

When you have a situation that has many many many possible parameters and each time only a few possible choices (a few items you are trying to vary and test – in his example in the video, 2 choices) you wind up with a ridicules number of possible tests. But you can cover all the possibilities in just 30 tests if your coverage target is all possible pairs. When you have situations like that you will see dramatic efficiency gains. What we have found in real world tests is greatly reduced time to create the tests and consistently 2 to 3 times as many defects found compared to the standard methods used for software testing.

You can read more on these ideas on his blog, where he explores software testing and combinatorial testing. The web base software testing application my brother created and shows in the demo is Hexawise. It is free to try out. I recommend it, though I am biased.

Related: Combinatorial Testing for Software – Video Highlight Reel of Hexawise – a pairwise testing tool and combinatorial testing tool – YouTube Uses Multivariate Experiment To Improve Sign-ups 15% – What Else Can Software Development and Testing Learn from Manufacturing? Don’t Forget Design of Experiments (DoE) – Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

Justin posted the presentation slides online at for anyone who is interested in seeing more details about the test plan he reviewed that had 1,746,756,896,558,880,852,541,440 possible tests. The slides are well worth reading.
Continue reading →

Improving Software Development with Automated Tests

Automated software testing is a mistake proofing (poka-yoke) solution for software development.

The way automated testing works is that software code is written that tests the software code of the application. This automated testing code test that business rules are correctly being followed by the code in the application.

So for example, a user should not be able to create a new account without entering password. Then you create code that does not allow an account to be created without a password. And you write a test that passes if this is true and fails if it is false.

The best implementation will then not allow deploying code to your production environment until the code has passed all the automated tests. So if a software developer changes the code, the automated tests are all run and if there is an error noted by the automated testing the code cannot be deployed to the production environment. So, in the example above, if somehow the changes made to the application code somehow now let an account be created without a password the test would fail, and the developer would know to fix the problem before putting the code into production.

Thus automated testing mistake proofs the process. Now the mistake proofing is only as good as the test that are added. Software development is complex and if the code has an error (based on the business rules) that is not tested then the code can be deployed to production and affect customers. Automated software testing tools are a huge help in preventing many errors from affecting customers.

It seems pretty obvious but until the widespread adoption of agile software development techniques and frameworks that make it easy to adopt automated testing (like Ruby on Rails) this sensible process improvement tool was used far less often than you would think.

Baking in Quality to Software Development

One of the reasons my organizations switched to Ruby on Rails for software development was the great integration with automated testing. We always wanted to have good test coverage on our software applications (which are web applications – some used only inside our organization) but didn’t actually find the time to do so. Since we adopted Ruby we have been doing much better in this regard. It isn’t just the switch to Ruby, of course, but the switch to Ruby coincided with the beginning of many improvements to our software development practices that have continually improved over the last couple of years.

Here is a post on How to build quality software by an agile, Ruby, lean software developer

I’ve mentioned previously about baking-in quality and not having developers throw code over a wall to testers.
…
Everyone on the team is concerned with not only assuring quality in what we deliver, but making it visible to ourselves and the business.

We work in an agile manner, iterating through development with extreme programming practices and Behaviour Driven Development. Facilitating our relationship with the business is Scrum and we utilise kanban principles and systems thinking to maintain a speedy throughput of high-quality work. This mixture allows us to communicate effectively, develop the correct features properly and continuously deploy our work when it is complete, thus maximising business value. I should also mention that we are fortunate enough to have our business people/customer sat across from us.
…
Without testers or a QA team there is no wall over which work can be thrown and the responsibility for quality absolved.
…
The inspection typically carried out end-of-cycle only yields bugs that were low severity and of no real impact to the end user.
…
An agile testing must-have, we use TeamCity to continuously run our unit tests on each check-in. We also execute our Cucumber acceptance tests on scheduled runs. The status of the builds are visible on dedicated monitors around the office as well as a nice 6”² projected screen.

via: @benjaminm

Combinatorial Testing for Software

Combinatorial testing of software is very similar to the design of experiments work my father was involved in, and which I have a special interest in. Combinatorial testing looks at binary interaction effects (success or failure), since it is seeking to find bugs in software, while design of experiments captures the magnitude of interaction effects on performance. In the last several years my brother, Justin Hunter, has been working on using combinatorial testing to improve software development practices. He visited me this week and we discussed the potential value of increasing the adoption of combinatorial testing, which is similar to the value of increasing the adoption of the use of design of experiments: both offer great opportunities for large improvements in current practices.

Automated Combinatorial Testing for Software

Software developers frequently encounter failures that occur only as the result of an interaction between two components. Testers often use pairwise testing – all pairs of parameter values – to detect such interactions. Combinatorial testing beyond pairwise is rarely used because good algorithms for higher strength combinations (e.g., 4-way or more) have not been available, but empirical evidence shows that some errors are triggered only by the interaction of three, four, or more parameters

Practical Combinatorial Testing: Beyond Pairwise by Rick Kuhn, US National Institute of Standards and Technology; Yu Lei, University of Texas, Arlington; and Raghu Kacker, US National Institute of Standards and Technology.

the detection rate increased rapidly with interaction strength. Within the NASA database application, for example, 67 percent of the failures were triggered by only a single parameter value, 93 percent by two-way combinations, and 98 percent by three-way combinations.2 The detection-rate curves for the other applications studied are similar, reaching 100 percent detection with four- to six-way interactions.
These results are not conclusive, but they suggest that the degree of interaction involved in faults is relatively low, even though pairwise testing is insufficient. Testing all four- to six-way combinations might therefore provide reasonably high assurance.