Deming and Software Development

I am sometimes asked about how use Deming’s ideas on management in a software development context. My belief is Deming’s ideas work extremely well in a software development context. The main issue is often unlearning some assumptions that people might have about what the Deming management system is.

It really is surprising to me how many “knowledge workers” respect Deming’s ideas but then say his attempts to treat factory workers as thoughtful people who should be respected and involved in improving their processes doesn’t make sense for them because they are “knowledge workers.”

There are many good things being done to improving the software development process. I think many of them are very Deming-like in their approaches (but to me miss out on aspects of the Deming management system that would be helpful). I think Dr. Deming’s approach to software development would focuses on the system of profound knowledge (the 4 inter-related areas below):

• Understanding variation – software development has quite a bit of variation, some probably innate [unique work] and some due to not having good procedures, batching work, not fixing problems right when they are seen, quick fixes that leave the system vulnerable in the long term (when you make one simple change to the code it has an unanticipated consequence due to poor practices that could have been eliminated), etc.. Many good coding practices are effective strategies to deal with this issue. And building an understanding of variation for managers (and business process owners/product owners) is very helpful to the software development process. The ideas in agile and kanban of focusing on smaller delivery units of work (one piece flow, just in time, cycle time…), customer value, maintainable code, sustainable work conditions, etc. are directly found in a Deming management system.
• Appreciation for the system of software development. Don’t just complain about bugs. Examine the process of development and then put in place mistake proofing efforts (don’t duplicate code, use integrated regression tests, don’t put artificial constraints on that result in system distortions – unrealistic targets…). Use things like kanban, limited work in progress, delivering value to customers quickly, think of success in terms of getting working software to customers (not meeting internal delivery goals), etc. that take into account our experience with systemic software development problems over the decades.
• Theory of knowledge – how do we know what we know? Are estimates reliable? Lets look at what users do, not just what they say (A/B testing…). Software developers often appreciate the value of usability testing, even though they rarely work for organizations willing to invest in usability testing. In my experience when software developers object to usability testing it is normally really an objection to overwork, and the usability testing is just going to give them more work or criticize things they were not allowed to spend the time they needed to do great work. That won’t always be the reason but it is the main one in my experience (I suppose their is also fear and just the psychology of not wanting to hear anything negative about what has been created – even if the usability testing shows tons of great results people will often focus on the negative).
• psychology and respect for people – This pretty much seems like it is the same for software development as everywhere else.

Improving the process of software development is a huge area with lots of good work being done (largely by the agile and kanban folks).

Dr. Deming’s ideas are about finding ways to improve the effectiveness and reliability of delivering value going forward. That idea is natural for many software developers. I find that software developers understand the value of designing the process to achieve repeatable success over the long term more than others (including managers).

Most of the frustrations software developers voice can be seen and addressed in a Deming context. Many of them amount to thinking about the system and implementing fixes to the systemic weaknesses. Which is largely what sensible developers I work with want to do (with or without a Deming context).

Using a Deming context to guide the management system in my opinion supports much of what software developers want. Software developers are better at seeing systemic weaknesses in the processes they work with than any other group I have dealt with. They may well have the least respect for any management system of any group (though there is a high level of disrespect for management systems by quick a few people, and with very good reason, in my opinion).

It seems to me software development has less use for control charts than a factory. That is fine. The Deming system isn’t meant to force tools where they are not appropriate. The idea is to use tools that improve the ability to deliver value. In different context different concepts and tools take on different levels of importance.

One aspect that seem to worry people is the idea of standardizing processes. Software developers think their work is too creative to fit such a concept. But this just isn’t true. The idea of standardizing processes is about doing so in a way that makes the system most productive. You don’t standardize things that can’t be standardized well. But a whole lot of the complaints you will hear from software developers if you sit in their midst are about failures that could be avoided with better processes.

Software developers and system administrators are constantly developing better methods to provide value by standardizing what can be. Frameworks like Ruby on Rails do this. Tools like Chef do this. Concepts like subclasses do this. Using Ruby gems (for useful code that is easily updatable) does this. Naming practices that allow everyone to understand (and avoid mistakes) would be considered a process standardization. As would deploying to a staging server that mirrors the production environment prior to deployment to production to find issues before the customers are impacted. Clear understanding of customer desires is critical to good software development and again is integral to the Deming management system. Using automated tests to check that code changes didn’t break some aspect of the application that the developer didn’t notice prior to committing the code (and prior to deploying to production) again is a standardization to the process that improves performance and in no way stifles creativity.

I think I am more frustrated about software developers claiming Deming’s ideas are meant for unthinking factor workers because the ideas fit software development so well. There challenges to convincing managers to accept Deming’s ideas I can understand much more easily. It is requiring a change of mindset that can unsettle the current thoughts on what they contribute. And I can see managers being worried about how they will fit into a new system, will they lose clout? They likely are doing pretty well in the current system but if it changes could their security be threatened? They are comfortable in the existing system, they have years, perhaps decades of experience and while they may have frustrations they understand the playing field they face. A big change in how the organization works is risky and can be scary to them.

But I see Deming’s ideas as about perfect for software developers. So when they are reluctant I find it more annoying. I do understanding fear that management will implement Deming’s ideas in a way that are counterproductive to software developers. That is a completely reasonable concern and even, unfortunately, a likely outcome. But the chance of an improved software development process is much greater within a Deming context than any other, in my opinion. The best agile and kanban efforts do a really good job inside the software development area. Sometimes that extends into the rest of the business but often it runs into a complete roadblock as you reach into the other parts of the organization.

How we go about making problems visible in software development will be different than doing so in other contexts but the principle is just as valuable in software development as it is elsewhere.

Deming’s ideas on management are not prescriptive. This makes things a bit more challenging as their is no instruction guide telling you what to do step by step. But this also provides the power for the ideas to be applied effectively in a wide variety of settings.

I have written about these ideas a fair amount on my blog: Interruptions Can Severely Damage Performance – Managing to Test Result Instead of Customer Value – Information Technology and Business Process Support – Software Supporting Processes Instead of Processes Being Determined by Rigid Software – Metrics and Software Development