Tag Archives: Software Development

Interviews on Software Testing

I haven’t added a new post here recently. One of the things that has been keeping me busy is putting together some interviews on software testing. Here are excerpts from 3 of the interviews:

Testing Smarter with Alan Page

Hexawise: During your 20 years at Microsoft you have been involved in hiring many software testers. What do you look for when choosing software testers? What suggestions do you have for those looking to advance in their in software testing career?

Alan: The biggest thing I look for in testers is a passion and ability to learn. I’ve interviewed hundreds of testers, including many who came from top universities with advanced degrees who just weren’t excited about learning. For them, maybe learning was a means to an end, but not something they were passionate about.

The testers who really impress me are those who love to learn – not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

As far as suggestions go, keep building your tool box. As long as you’re willing to try new things, you’ll always be able to find challenging, fun work. As soon as you think you know it all, you will be stuck in your career.
…
Hexawise: It seems to me that testing games would have significant challenges not found in testing fairly straightforward business applications. Could you share some strategies for coping with those challenges?

Alan: Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That’s 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Beyond that, testing games requires a lot of human eyeballs and critical thinking to ensure gameplay makes sense, objects are in the right places, etc. I’ve never seen a case where automating gameplay, for example, has been successful. I have, however, seen some really innovative tools written by testers to help make game testing much easier, and much more effective.

Testing Smarter with Dorothy Graham

Hexawise: When looking to automate tests one thing people sometimes overlook is that given the new process it may well be wise to add more tests than existed before. If all test cases were manually completed that list of cases would naturally have been limited by the cost of repeatedly manually checking so many test cases in regression testing. If you automate the tests it may well be wise to expand the breadth of variations in order to catch bugs caused by the interaction of various parameter values. What are your thoughts on this idea?

Dot: Nice analogy – I like the term “grapefruit juice bugs”. Using some of the combinatorial techniques is a good way to cover more combinations in a reasonably rigorous way. Automation can help to run more tests (provided that expected results are available for them) and may be a good way to implement the combination tests, using pair-wise and/or orthogonal arrays.

Smarter Software Testing with James Bach

Continue reading →

Applying Toyota Kata to Agile Retrospectives

HÃ¥kan Forss, King (interactive entertainment games), presentation at the GOTO Copenhagen 2015 conference.

I strongly recommend Mike Rother’s book: Toyota Kata.

Description from Workshop description “The Toyota Kata Experience”

Kata means pattern, routine, habits or way of doing things. Kata is about creating a fast “muscle memory” of how to take action instantaneously in a situation without having to go through a slower logical procedure. A Kata is something that you practice over and over striving for perfection. If the Kata itself is relative static, the content of the Kata, as we execute it is modified based on the situation and context in real-time as it happens. A Kata as different from a routine in that it contains a continuous self-renewal process.

I think the great number of worthwhile conference presentations we can all now get sitting wherever we are provides us a great opportunity (and lets us avoid missing out of good ideas because “How could they know“).

A point made in the presentation that is very simple but still constantly the source of failure is that the current system isn’t supporting improvement. Retrospectives are a good method to help improve but if there is no time to think about the issues raised and come up with experiments to improve and review of whether those experiments worked or not and why failure to improve is the expected result.

Creating a culture where it is expected that any improvement ideas are tested and evaluated is one of the most important changes on the path to a company that will be able to continually improve. If not, what happens is some changes are good, many are not and soon people lose faith that any effort is worth it because they see how poor the results are. By taking care to evaluate what is working and what isn’t we create a process in which we don’t allow ad hoc and unsuccessful changes to demoralize everyone.

Continue reading →

Software Testing and the Impact on Quality

My response to a question on Reddit.

“Software quality does not come from testing”
Does anybody have any thoughts on the validity of the above statement?

That statement is similar to the idea you can’t inspect in quality. Basically “Inspection is too late. The quality, good or bad, is already in the product.” W. Edwards Deming

I agree with those ideas. Software testing is a bit different (at least some of it is) from the inspection mentioned above. You are testing while the product is being developed and adjustments are being made before the product is released to customers. Also with internet based software you have the ability to update the software and now all users have that update. Where for physical devices they already have the product and the only option is a recall which is very expensive and often ignored.

Software testing however should pay attention to those points in the 2 links above (defects should be understood as evidence of a process that needs to be improved so defects are not built in the first place). What you want is not just to fix the bugs software testers catch but figure out the reasons those bugs were created and improve you process so you create fewer bugs in the future.

No matter what the software quality is based on the code that is written. At the best software testing can tell people about the bugs but unless the code is fixed the software quality didn’t change. But to say that software testing doesn’t have a big influence of software quality when testing is well done and the software development process is good (listens to feedback and improves) is not very accurate.

Functional Websites are Normally Far Superior to Apps

An email to I just sent to Uber

I understand the regular Uber app not having a functional website.

Uber Eats not having a functional website is super lame. It strikes me similar to Walmart 15 years ago telling people “we only have stores go to them, we just use the internet for advertising our stores.” Today for Uber: we only have apps, “we only use the web for advertising our apps.” Both you and Walmart want to use a limited function service that you both are comfortable with and want users to just put up with annoyance because neither of you want users using the connivence of the web.

When you bother to create a functional website maybe I’ll use it (I use several food delivery services now).

Using limited apps is rarely wise (unless you are crippled by the lack of a real computer and are stuck having to use just an app). Uber cars is a rare exception where the needs are so simple a limited app is ok. Picking restaurants and food on a tiny screen with a crippled app is just a lousy experience for anyone that uses real websites. The Ux for the app is horrible.

Just like old school businesses were only comfortable with their old business models and didn’t create functional websites (instead using the web just to advertise that you should go to their store, or giving you forms to complete and fax back to them…) new businesses are often stuck on only using apps even though they often provide a lousy user experience compared to a functional website.

There are some apps that are very useful and not having a functional web app can make sense, but it is fairly limited. Getting a ride apps I can see as only apps. Driving instructions and live maps using GPS to locate you is another great app use. Boarding passes can make sense (though I do question some of that whole process conceptually this could be a good example of a app with no functional website).

Map app showing gps location while floating on bamboo raft in Yangshuo, China

Map app showing my GPS location while I floated down a river on bamboo raft in Yangshuo, China.

But most cases not having a functional website is just lousy Ux.

Now there are some times when using technology to provide good service just isn’t worth the effort. Often though businesses just are stuck in their fax-thinking or physical-store-thinking or app-thinking and fail to use a technology that would provide great benefit to their users. I find it odd how often app vendors seem stuck in their app mindset. It wasn’t so surprising old businesses that were not based on technology didn’t take advantage of the incredible opportunities provided by the internet and the web. But it is less understandable when companies that are thought of as technology savvy are as blinded by their history (can’t see out of the app-mindset).

Continue reading →

Good Startup Ideas from Startup Weekend JB (Malaysia)

I like all these startup ideas from Startup Weekend JB (Malaysia). I can’t figure out how to comment on their blog (I am guessing Tumbler just eliminates commenting?), so I started this post – and ended up adding much more than I planned on putting in a comment.

All of these ideas are not very technically challenging and pretty much versions of these businesses are already successful around the globe. But creating great user experiences (in apps or on web sites) is often neglected for doing something passable (and local conditions mean the business is a bit different here than it would be somewhere else).

To create a greatly successful startup focusing on great, not adequate, customer experiences is extremely useful. And you can leap ahead of competitors that don’t focus on customer delight.

One of the interesting things from my experience living in Johor Bahru, Malaysia the last few years is that Malaysia has way more entrepreneurial diversity than the USA (in my limited experience). Many of these businesses stay small. But you have entrepreneurs in all sorts of businesses at events in JB. In the USA the events I went to were all software focused (internet businesses etc.).

Here you have people setting up factories, printing companies, beauty entrepreneurs, construction companies, bakers, motel chain (less than a handful of motels so far) etc. going to HackerSpace meetings and Drinks Entrepreneurs, BarCamp etc.. Startup Weekend I do think was very IT focused, even in JB (it is setup to be that way so it isn’t surprising).

There are small business entrepreneurs in the USA, but they don’t go to entrepreneur/ LeanStartup etc. type meetings (in my experience). And they are more limited; so many businesses in the USA really can’t be done easily by some new college graduate. The capital and legal requirements are just so huge you need so much to start that it isn’t something considered in the cool-startup communities (in general – I’m sure there are some things like micro-brew startups etc.). In JB it seems to me fewer than 33% are IT dominated. Though I expect this will increase rapidly. I think there is a real benefit to including non-IT focused people in these communities.

The number of people outside of IT that decide to be entrepreneurs out of school is tiny (it seems to me). In Malaysia it seems much more common. But in general people don’t talk about it as being entrepreneurs they are trying to make a living and setting up their own business to do so is just a natural thing to do.

SmartDining – find local restaurants (mobile app) and order (for take out or dine in).

The app shouldn’t be too hard to do well. The challenges will be working with restaurants, marketing (so often the case) and maybe mapping (finding good suggestion). How to balance efforts well will also be a challenge – you could spend tons of time on many different valuable efforts related to this business. Vote.

Continue reading →

Agile Software Development and Deming

Part two of three of an interview of me with Bill Fox has been published. See part one: John Hunter on PDSA, Deming and Strategy. From part two, Lean, Agile, Deming, Leadership and Management Systems:

I see that agile is very consistent with Deming. Agile has all sorts of variants, so to different extents, they fit in. But one of the things I find really interesting is the agile folks, and of course the lean software folks, they much more than any other group of people I’ve seen traced the agile ideas back to find Deming’s ideas.

So it seems to me many of the leading agile and lean folks have tracked it back to Deming and then incorporated some Deming’s thinking. Now, the majority of people that are doing agile stuff have no idea that so many of the ideas track back to Deming so well. But I think that agile stuff is largely very consistent with Deming.

And it’s even largely very consistent with Deming when the words don’t match up correctly. So, one of the agile tenets is people over process. That’s not at all what Deming would say. But, in my opinion (from when I read a bunch of the agile stuff and was trying to figure out how to fit things together), what they really said was that the work that people are doing should not be prescribed from on high by processes that prohibit them from doing the work effectively.

In the software development world, they were used to processes being driven by heavy handed business ideas that don’t fit very well with how software development should be done. So that they see the word Ã¢â‚¬Ëœprocess’ as tied to heavily prescriptive ideas from people that don’t understand software development imposing process on software development.

Read the full interview with more on how the Deming management system fits with other management strategies.

Use Urls – Don’t Use Click x, Then Click y, Then Click z Instructions

In the 1980s software applications had to use click x, then click y, then click z type instructions to get you to a specific location in a software application (or at least they had a decent excuse to do that). Too many web application development organizations forget that they now have urls to direct people exactly where to go: and that they shouldn’t rely on ancient “click here, then there, then in that other place” type instructions.

Here is an example I wrote up on my recent experience with iTunes and their failure to do this properly: Bad iTunes Ux and How to Submit a Podcast to iTunes. I see it all the time, that is just one example.

It is so sad that Google can’t even offer mildly decent help for their own software nearly a couple decades after they started out with the goal to organize the world’s information. And lots of other software companies also point you to clicking around various gui (graphical user interface) click paths instead of just

showing the url (say in a help email) – instead of the gui click path text
a clickable link to the url in web documents

On top of the waste inherent in click path instructions they often fail because the interface has changed and no one bothered to change the click path or the click path depends on other things being a certain way and they are not so the click path breaks.

I really can’t comprehend how this usability failure is something I run across all the time. Urls are not some secret idea only PhD computer scientists have heard of. This is super basic stuff – click path instructions should never have been acceptable for any web application. It is pitiful they are still common among companies that see themselves as advanced software development organizations.

Using the proper urls also will help make sure you are using human readable urls. Another super basic usability concept that is ignored far too often by some web application developers.

Root Cause, Interactions, Robustness and Design of Experiments

Eric Budd asked on The W. Edwards Deming Institute group (LinkedIn broke the link with a register wall so I removed the link):

If observed performance/behavior in a system is a result of the interactions between components–and variation exists in those components–the best root cause explanation we might hope for is a description of the interactions and variation at a moment in time. How can we make such an explanation useful?

A single root cause is rare. Normally you can look at the question a bit differently see the scope a bit differently and get a different “root cause.” In my opinion “root cause” is more a decision about what is an effective way to improve the system right now rather than finding a scientifically valid “root cause.”

Sometimes it might be obvious combination which is an issue so must be prevented. In such a case I don’t think interaction root cause is hard – just list out the conditions and then design something to prevent that in the future.

Often I think you may find that the results are not very robust and this time we caught the failure because of u = 11, x = 3, y = 4 and z =1. But those knowledge working on the process can tell the results are not reliable unless x = 5 or 6. And if z is under 3 things are likely to go wrong. and if u is above 8 and x is below 5 and y is below 5 things are in trouble…

To me this often amounts to designing systems to be robust and able to perform with the variation that is likely to happen. And for those areas where the system can’t be made robust for some variation then designing things so that variation doesn’t happen to the system (mistake proofing processes, for example).

In order to deal with interaction, learn about interaction and optimize results possible due to interactions I believe the best method is to use design of experiments (DoE) – factorial experiments.

Continue reading →

Building a Great Software Development Team

Elliot: I worked with some of the best programmers I’ve ever known at the tiny, obscure ASEE

Adam Solove: Why do you think that happened? They hired for passion, rather than experience?

If I had to pick one thing, passion would likely be it but really it is a complex assortment of things. Passion for the right things, based on what we aimed to be, mattered a great deal. That took the form of being passionate about the user experience, being passionate about good software development practices, being passionate about good software itself, being passionate about treating each other with respect, being passionate about learning and improving.

I think there were several other important factors, such as: the skill to turn a passion for good software into actual good software. This required intelligence, interest and knowledge about software development but didn’t require specific experience (computer science degree, 2 years of Ruby on Rails development, certification or any such thing). Hiring based on experience is a big mistake. In my opinion hiring based on capability and potential (which is based partially on experience) is wise.

Another factor is that we had people (those first few hires were critical) that were really knowledgable about programing good software and that became a self reinforcing process. The gaps one person’s ability and knowledge could be filled by someone else helping them understand and get better.

The expectation was that we found great solutions. If we didn’t we kept looking and asked each other for help (another factor in creating a great team). We didn’t just accept that we were confident the solution wasn’t very good but couldn’t find any better options so I guess this is the best we can do.

We were interested enough in good results that we would push for better options instead of just accepting something that was kind of ok. This shouldn’t be such a big deal; but in practice it is huge. So many places just end up avoiding conflict to the extent that it is a huge detriment to results.

Without confidence, honest debate about ideas is suppressed as people are constantly taking things personally instead of trying to find the best ideas (and if doing so means my idea is criticized that is ok). Our group was great at this. It is something I find it a bit silly to say a workplace was “great” at but in most places I find the fear of someone being concerned stifles discussion to an unbelievable extent.

This is also one of many areas where the culture within the team was self reinforcing. As new people came on they understood this practice. They saw it in practice. They could see it was about finding good ideas and if their idea was attacked they didn’t take it nearly as personally as most people do in most places. I sought to understand if people we looked at hiring would be comfortable in such an environment.

Continue reading →

A Good Management System is Robust and Continually Improving

imagine various people working within it, somehow swapping out gears and cogs without the clock stopping or slowing down even a little.

This is a fairly good quote on a good management system. Some people might not like the mechanistic model – comparing an organization to a clock, and I agree that isn’t the right model, but even so it is a good quote.

The quote, from a story about the San Antonio Spurs captures what should happen with a good management system. Things just keep running well as inevitable changes take place (and keep getting better in the case of a management system).

A good management system doesn’t rely on heroic efforts to save the day. The organization is designed to success. It is robust. It will succeed with all the variation thrown at it by the outside world. A good management system takes advantage of the contributions people offer, but it is not perform poorly when others are relied on.

A well run organization has graceful degradation (when one component fails or one person is missing the performance doesn’t suffer, bad results are avoided).

With software for example, a decently created web sites may use javascript to enhance the user experience but if javascript is unavailable the site works fine (just missing some neat features that are nice but don’t prevent users from getting what they need). Poorly designed software has critical dependencies on conditions that cannot be guaranteed to be present for end users and the software just fails to work when those conditions are not met. Ungraceful degradation is too common in software. It is also too common in our management systems.

An organization succeeds because of the efforts of many great people. But the management system has to be created for an organization to prosper as what we all know will happen, happens: people will leave and need to be replaced. And the people that stay will need to adjust to new conditions inside the organization and in response to external forces. A good management system is constantly improving performance, innovating, increasing the robustness of systems and increasing the capability of people.