Category Archives: Software Development

Interviews on Software Testing

I haven’t added a new post here recently. One of the things that has been keeping me busy is putting together some interviews on software testing. Here are excerpts from 3 of the interviews:

Testing Smarter with Alan Page

Hexawise: During your 20 years at Microsoft you have been involved in hiring many software testers. What do you look for when choosing software testers? What suggestions do you have for those looking to advance in their in software testing career?

Alan: The biggest thing I look for in testers is a passion and ability to learn. I’ve interviewed hundreds of testers, including many who came from top universities with advanced degrees who just weren’t excited about learning. For them, maybe learning was a means to an end, but not something they were passionate about.

The testers who really impress me are those who love to learn – not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

As far as suggestions go, keep building your tool box. As long as you’re willing to try new things, you’ll always be able to find challenging, fun work. As soon as you think you know it all, you will be stuck in your career.
…
Hexawise: It seems to me that testing games would have significant challenges not found in testing fairly straightforward business applications. Could you share some strategies for coping with those challenges?

Alan: Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That’s 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Beyond that, testing games requires a lot of human eyeballs and critical thinking to ensure gameplay makes sense, objects are in the right places, etc. I’ve never seen a case where automating gameplay, for example, has been successful. I have, however, seen some really innovative tools written by testers to help make game testing much easier, and much more effective.

Testing Smarter with Dorothy Graham

Hexawise: When looking to automate tests one thing people sometimes overlook is that given the new process it may well be wise to add more tests than existed before. If all test cases were manually completed that list of cases would naturally have been limited by the cost of repeatedly manually checking so many test cases in regression testing. If you automate the tests it may well be wise to expand the breadth of variations in order to catch bugs caused by the interaction of various parameter values. What are your thoughts on this idea?

Dot: Nice analogy – I like the term “grapefruit juice bugs”. Using some of the combinatorial techniques is a good way to cover more combinations in a reasonably rigorous way. Automation can help to run more tests (provided that expected results are available for them) and may be a good way to implement the combination tests, using pair-wise and/or orthogonal arrays.

Smarter Software Testing with James Bach

Continue reading →

Add Constraints to Processes Carefully

Take great care in adding constraints to processes to avoid doing so needlessly.

Online you will frequently find forms that have required fields that needn’t be. Certainly if you were designing with focus on what is best for customers those requirements rarely make sense. Occasionally a required field is a sensible constraint on an online form but so often they add unnecessary constraints.

I frequently find those forms even requiring a false answer since a response is required and none of the options are true. Often this is because the organization is thinking of the boxes they expect users to fit themselves into rather than thinking how to create the best user experience.

I wrote previously about a company representative that suggested a customer change their name because the computer system didn’t accept names with 2 characters. Constraints on creating a secure password are a frequent failure of web sites for the last 10 years.

Man without arms denied housing loan due to inability to provide fingerprints

because Wu Jianping has no arms, creditors claimed they could not give him a loan since he was unable to be fingerprinted.

After the case was publicized and there was a great deal of negative publicity on social media the banks modified their process and approved the loan. But your organization shouldn’t have as the mistake-proofing (obviously not mistake-proofing at all) that when the process doesn’t quite work well then rely on a massive social media outcry which is a signal to us to straighten out the issue.

Frequently I see unnecessary constraints creating the edge case excuse. By burdening your process with unnecessary constraints you create edge cases that fail and then use the excuse that each of the edge cases is rare and therefore you can’t justify the expense of fixing them.

But if you designed the process sensibly in the first place the edge case never would have failed and you wouldn’t need special work arounds for such “edge cases.” A simple example of this is unnecessarily complex web page code that fails if to submit a button without javascript. Yes, a small number of users won’t have enabled all javascript to run (today anyway) so it is an “edge case” to deal with if you don’t have the form work without javascript. But there is no decent reason to have it fail in most cases.

Continue reading →

Applying Toyota Kata to Agile Retrospectives

HÃ¥kan Forss, King (interactive entertainment games), presentation at the GOTO Copenhagen 2015 conference.

I strongly recommend Mike Rother’s book: Toyota Kata.

Description from Workshop description “The Toyota Kata Experience”

Kata means pattern, routine, habits or way of doing things. Kata is about creating a fast “muscle memory” of how to take action instantaneously in a situation without having to go through a slower logical procedure. A Kata is something that you practice over and over striving for perfection. If the Kata itself is relative static, the content of the Kata, as we execute it is modified based on the situation and context in real-time as it happens. A Kata as different from a routine in that it contains a continuous self-renewal process.

I think the great number of worthwhile conference presentations we can all now get sitting wherever we are provides us a great opportunity (and lets us avoid missing out of good ideas because “How could they know“).

A point made in the presentation that is very simple but still constantly the source of failure is that the current system isn’t supporting improvement. Retrospectives are a good method to help improve but if there is no time to think about the issues raised and come up with experiments to improve and review of whether those experiments worked or not and why failure to improve is the expected result.

Creating a culture where it is expected that any improvement ideas are tested and evaluated is one of the most important changes on the path to a company that will be able to continually improve. If not, what happens is some changes are good, many are not and soon people lose faith that any effort is worth it because they see how poor the results are. By taking care to evaluate what is working and what isn’t we create a process in which we don’t allow ad hoc and unsuccessful changes to demoralize everyone.

Continue reading →

Software Testing and the Impact on Quality

My response to a question on Reddit.

“Software quality does not come from testing”
Does anybody have any thoughts on the validity of the above statement?

That statement is similar to the idea you can’t inspect in quality. Basically “Inspection is too late. The quality, good or bad, is already in the product.” W. Edwards Deming

I agree with those ideas. Software testing is a bit different (at least some of it is) from the inspection mentioned above. You are testing while the product is being developed and adjustments are being made before the product is released to customers. Also with internet based software you have the ability to update the software and now all users have that update. Where for physical devices they already have the product and the only option is a recall which is very expensive and often ignored.

Software testing however should pay attention to those points in the 2 links above (defects should be understood as evidence of a process that needs to be improved so defects are not built in the first place). What you want is not just to fix the bugs software testers catch but figure out the reasons those bugs were created and improve you process so you create fewer bugs in the future.

No matter what the software quality is based on the code that is written. At the best software testing can tell people about the bugs but unless the code is fixed the software quality didn’t change. But to say that software testing doesn’t have a big influence of software quality when testing is well done and the software development process is good (listens to feedback and improves) is not very accurate.

Functional Websites are Normally Far Superior to Apps

An email to I just sent to Uber

I understand the regular Uber app not having a functional website.

Uber Eats not having a functional website is super lame. It strikes me similar to Walmart 15 years ago telling people “we only have stores go to them, we just use the internet for advertising our stores.” Today for Uber: we only have apps, “we only use the web for advertising our apps.” Both you and Walmart want to use a limited function service that you both are comfortable with and want users to just put up with annoyance because neither of you want users using the connivence of the web.

When you bother to create a functional website maybe I’ll use it (I use several food delivery services now).

Using limited apps is rarely wise (unless you are crippled by the lack of a real computer and are stuck having to use just an app). Uber cars is a rare exception where the needs are so simple a limited app is ok. Picking restaurants and food on a tiny screen with a crippled app is just a lousy experience for anyone that uses real websites. The Ux for the app is horrible.

Just like old school businesses were only comfortable with their old business models and didn’t create functional websites (instead using the web just to advertise that you should go to their store, or giving you forms to complete and fax back to them…) new businesses are often stuck on only using apps even though they often provide a lousy user experience compared to a functional website.

There are some apps that are very useful and not having a functional web app can make sense, but it is fairly limited. Getting a ride apps I can see as only apps. Driving instructions and live maps using GPS to locate you is another great app use. Boarding passes can make sense (though I do question some of that whole process conceptually this could be a good example of a app with no functional website).

Map app showing gps location while floating on bamboo raft in Yangshuo, China

Map app showing my GPS location while I floated down a river on bamboo raft in Yangshuo, China.

But most cases not having a functional website is just lousy Ux.

Now there are some times when using technology to provide good service just isn’t worth the effort. Often though businesses just are stuck in their fax-thinking or physical-store-thinking or app-thinking and fail to use a technology that would provide great benefit to their users. I find it odd how often app vendors seem stuck in their app mindset. It wasn’t so surprising old businesses that were not based on technology didn’t take advantage of the incredible opportunities provided by the internet and the web. But it is less understandable when companies that are thought of as technology savvy are as blinded by their history (can’t see out of the app-mindset).

Continue reading →

Use Urls – Don’t Use Click x, Then Click y, Then Click z Instructions

In the 1980s software applications had to use click x, then click y, then click z type instructions to get you to a specific location in a software application (or at least they had a decent excuse to do that). Too many web application development organizations forget that they now have urls to direct people exactly where to go: and that they shouldn’t rely on ancient “click here, then there, then in that other place” type instructions.

Here is an example I wrote up on my recent experience with iTunes and their failure to do this properly: Bad iTunes Ux and How to Submit a Podcast to iTunes. I see it all the time, that is just one example.

It is so sad that Google can’t even offer mildly decent help for their own software nearly a couple decades after they started out with the goal to organize the world’s information. And lots of other software companies also point you to clicking around various gui (graphical user interface) click paths instead of just

showing the url (say in a help email) – instead of the gui click path text
a clickable link to the url in web documents

On top of the waste inherent in click path instructions they often fail because the interface has changed and no one bothered to change the click path or the click path depends on other things being a certain way and they are not so the click path breaks.

I really can’t comprehend how this usability failure is something I run across all the time. Urls are not some secret idea only PhD computer scientists have heard of. This is super basic stuff – click path instructions should never have been acceptable for any web application. It is pitiful they are still common among companies that see themselves as advanced software development organizations.

Using the proper urls also will help make sure you are using human readable urls. Another super basic usability concept that is ignored far too often by some web application developers.

Building a Great Software Development Team

Elliot: I worked with some of the best programmers I’ve ever known at the tiny, obscure ASEE

Adam Solove: Why do you think that happened? They hired for passion, rather than experience?

If I had to pick one thing, passion would likely be it but really it is a complex assortment of things. Passion for the right things, based on what we aimed to be, mattered a great deal. That took the form of being passionate about the user experience, being passionate about good software development practices, being passionate about good software itself, being passionate about treating each other with respect, being passionate about learning and improving.

I think there were several other important factors, such as: the skill to turn a passion for good software into actual good software. This required intelligence, interest and knowledge about software development but didn’t require specific experience (computer science degree, 2 years of Ruby on Rails development, certification or any such thing). Hiring based on experience is a big mistake. In my opinion hiring based on capability and potential (which is based partially on experience) is wise.

Another factor is that we had people (those first few hires were critical) that were really knowledgable about programing good software and that became a self reinforcing process. The gaps one person’s ability and knowledge could be filled by someone else helping them understand and get better.

The expectation was that we found great solutions. If we didn’t we kept looking and asked each other for help (another factor in creating a great team). We didn’t just accept that we were confident the solution wasn’t very good but couldn’t find any better options so I guess this is the best we can do.

We were interested enough in good results that we would push for better options instead of just accepting something that was kind of ok. This shouldn’t be such a big deal; but in practice it is huge. So many places just end up avoiding conflict to the extent that it is a huge detriment to results.

Without confidence, honest debate about ideas is suppressed as people are constantly taking things personally instead of trying to find the best ideas (and if doing so means my idea is criticized that is ok). Our group was great at this. It is something I find it a bit silly to say a workplace was “great” at but in most places I find the fear of someone being concerned stifles discussion to an unbelievable extent.

This is also one of many areas where the culture within the team was self reinforcing. As new people came on they understood this practice. They saw it in practice. They could see it was about finding good ideas and if their idea was attacked they didn’t take it nearly as personally as most people do in most places. I sought to understand if people we looked at hiring would be comfortable in such an environment.

Continue reading →

Deming and Software Development

I am sometimes asked about how use Deming’s ideas on management in a software development context. My belief is Deming’s ideas work extremely well in a software development context. The main issue is often unlearning some assumptions that people might have about what the Deming management system is.

It really is surprising to me how many “knowledge workers” respect Deming ideas but then say his attempts to treat factory workers as thoughtful people who should be respected and involved in improving their processes doesn’t make sense for them because they are “knowledge workers.”

There are many good things being done to improving the software development process. I think many of them are very Deming-like in their approaches (but to me miss out on aspects of the Deming management system that would be helpful). I think Dr. Deming’s approach to software development would focuses on the system of profound knowledge (the 4 inter-related areas below):

Understanding variation – software development has quite a bit of variation, some probably innate [unique work] and some due to not having good procedures, batching work, not fixing problems right when they are seen, quick fixes that leave the system vulnerable in the long term (when you make one simple change to the code it has an unanticipated consequence due to poor practices that could have been eliminated), etc.. Many good coding practices are effective strategies to deal with this issue. And building an understanding of variation for managers (and business process owners/product owners) is very helpful to the software development process. The ideas in agile and kanban of focusing on smaller delivery units of work (one piece flow, just in time, cycle time…), customer value, maintainable code, sustainable work conditions, etc. are directly found in a Deming management system.
Appreciation for the system of software development. Don’t just complain about bugs. Examine the process of development and then put in place mistake proofing efforts (don’t duplicate code, use integrated regression tests, don’t put artificial constraints on that result in system distortions – unrealistic targets…). Use things like kanban, limited work in progress, delivering value to customers quickly, think of success in terms of getting working software to customers (not meeting internal delivery goals), etc. that take into account our experience with systemic software development problems over the decades.
Theory of knowledge – how do we know what we know? Are estimates reliable? Lets look at what users do, not just what they say (A/B testing…). Software developers often appreciate the value of usability testing, even though they rarely work for organizations willing to invest in usability testing. In my experience when software developers object to usability testing it is normally really an objection to overwork, and the usability testing is just going to give them more work or criticize things they were not allowed to spend the time they needed to do great work. That won’t always be the reason but it is the main one in my experience (I suppose their is also fear and just the psychology of not wanting to hear anything negative about what has been created – even if the usability testing shows tons of great results people will often focus on the negative).
psychology and respect for people – This pretty much seems like it is the same for software development as everywhere else.

Continue reading →

What is the Explanation Going to be if This Attempt Fails?

Occasionally during my career I have been surprised by new insights. One of the things I found remarkable was how quickly I thought up a new explanation for what could have caused a problem when the previously expressed explanation was proven wrong. After awhile I stopped finding it remarkable and found it remarkable how long it took me to figure out that this happened.

I discovered this as I programmed software applications. You constantly have code fail to run as you expect and so get plenty of instances to learn the behavior I described above. While I probably added to my opportunities to learn by being a less than stellar coder I also learned that even stellar coders constantly have to iterate through the process of creating code and seeing if it works, figuring out why it didn’t and trying again.

The remarkable thing is how easily I could come up with an new explanation. Often nearly immediately upon what I expected to work failing to do so. And one of the wonderful things about software code is often you can then make the change in 10 minutes and a few minutes later see if it worked (I am guessing my brain kept puzzling over the ideas involved and was ready with a new idea when I was surprised by failure).

When I struggled a bit to find an initial explanation I found myself thinking, “this has to be it” often because of two self reinforcing factors.

First, I couldn’t think of anything else that would explain it. Sometimes you will think right away of 4 possible issues that could cause this problem. But, when I struggled to find any and then finally came up with an idea it feels like if there was another possibility I should have thought of it while struggling to figure out what I finally settled on.

Second, the idea often seems to explain exactly what happened, and it often feels like “of course it didn’t work, what was I thinking I need to do x.” This often turns out to be true, doing x solves the problem and you move on. But a remarkable percentage of the time, say even just 10%, it doesn’t. And then I would find myself almost immediately thinking, of course I need to do y. Even when 10 seconds ago I was convinced there was no other possibility.

Continue reading →

Double Loop Learning Presentation by Benjamin Mitchell

Benjamin Mitchell – Using the Mutual Learning Model to achieve Double Loop Learning [the original video is not online anymore, I replaced it with another presentation by Benjamin on the same topic at a different conference at close to the same time – JH]

Benjamin Mitchell presents ideas using Chris Argyris thinking on double-loop learning. “Double-loop learning occurs when error is detected and corrected in ways that involve the modification of an organization’s underlying norms, policies and objectives.”

Single loop learning is basically to just try again using the same understanding, thinking and tactics. It is understood that the results were not what was desired so we will try again, but the supporting system is not seen as the reason results were not the desired results. Double loop learning is when the result leads to questioning the system and attempting to adjust the system and make changes and experiment to learn to be able to create systems that get better results.

Argyris: people will blame others and the system when their actions seem to differ from their espoused proper actions. (I see this as similar to the idea of revealed preference versus stated preference: revealed actions versus stated actions – John)