Category Archives: Data

Introductory Videos on Using Design of Experiments to Improve Results

The video shows Stu Hunter discussing design of experiments in 1966. It might be a bit slow going at first but the full set of videos really does give you a quick overview of the many important aspects of design of experiments including factorial designed experiments, fractional factorial design, blocking and response surface design. It really is quite good, if you find the start too slow for you skip down to the second video and watch it.

My guess is, for those unfamiliar with even the most cursory understanding of design of experiments, the discussion may start moving faster than you can absorb the information. One of the great things about video is you can just pause and give yourself a chance to catch up or repeat a part that you didn’t quite understand. You can also take a look at articles on design of experiments.

I believe design of experiments is an extremely powerful methodology of improvement that is greatly underutilized. Six sigma is the only management improvement program that emphasizes factorial designed experiments.

Related: One factor at a time (OFAT) Versus Factorial DesignsThe purpose of Factorial Designed Experiments

Continue reading

Keys to the Effective Use of the PDSA Improvement Cycle

The PDSA improvement cycle was created by Walter Shewhart where Dr. Deming learned about it. An improvement process is now part of many management improvement methods (A3 for lean manufacturing, DMAIC for six sigma and many other modifications). They are fairly similar in many ways. The PDSA cycle (Plan, Do, Study, Act) has a few key pieces that are either absent in most others processes of greatly de-emphasized which is why I prefer it (A3 is my second favorite).

The PDSA cycle is a learning cycle based on experiments. When using the PDSA cycle, it is important to predict the results. This is important for several reasons but most notably due to an understanding of the theory of knowledge. We will learn much more if we write down our prediction. Otherwise we often just think (after the fact); yeah that is pretty much what I expected (even if it wasn’t). Also we often fail to think specifically enough at the start to even have a prediction. Forcing yourself to make a prediction gets you to think more carefully up front and can help you set better experiments.

PDSA Improvement cycle graphic

PDSA Improvement cycle graphic from my book – Management Matters

An organization using PDSA well will turn the PDSA cycle several times on any topic and do so quickly. In a 3 month period turning it 5 times might be good. Often those organizations that struggle will only turn it once (if they are lucky and even reach the study stage). The biggest reason for effective PDSA cycles taking a bit longer is wanting more data than 2 weeks provides. Still it is better to turn it several times will less data – allowing yourself to learn and adjust than taking one long turn.

The plan stage may well take 80% (or even more) of the effort on the first turn of the PDSA cycle in a new series. The do stage may well take 80% of of the time (when you look at the whole process, multiple turns through the PDSA cycle) – it usually doesn’t take much effort (to just collect a bit of extra data) but it may take time for that data to be ready to collect. In the 2nd, 3rd… turns of the PDSA cycle the Plan stage often takes very little time. Basically you are just adjusting a bit from the first time and then moving forward to gather more data. Occasionally you may learn you missed some very important ideas up front; then the plan stage may again take some time (normally if you radically change your plans).

Remember to think of Do as doing-the-experiment. If you are “doing” a bunch of work (not running an experiment and collecting data) that probably isn’t “do” in the PDSA sense.

Continue reading

Agile Story Point Estimation

In agile software development tasks are documented as user stories. Then the level of effort for those stores can be estimated by assigning each story points. The velocity that can be produced in a period (called a sprint, for us 2 weeks) can be estimated. Thus you can predict what can be delivered in the next sprint (which can help business managers make priority decisions).

I have found estimation to be worthwhile. In doing so, we accept there is a great amount of variation but points give a hint to scale. They can help prioritize (if you have 5 things you want but 1 is much harder you may well drop that to the bottom). I have always accepted a great amount of variation in the velocity, worry about the variation I don’t find worthwhile. I do think trying to act as though the velocity is precise can lead to problems. At the same time having a measure of velocity, even accepting understanding variation was present, was useful.

Over time reducing variation (probably largely through better estimation and perhaps a few better tools, reduced technical debt, better documentation, testing…) is helpful and laudable. We made improvement but still lots of variation existed. The biggest help in reducing the measured velocity was breaking down large stories to more manageable sizes. The challenge of estimating user stories, I suspect, has some fairly high variation (even with good system improvements that can help reduce variation).

Large stories just can hide huge variation in what is really required once getting into implementing it.

The way we did estimation (discussing in a sprint planning meeting) did take some time (but not a huge amount). It was agreed by those involved that the time spent was worthwhile. Sometimes we did slip and spend too much time on this, that was an area we had to pay attention to. The discussions were educational and helped provide guidance on how to approach the story. The value of discussions around estimations was probably the biggest surprise I have had in implementing any agile ideas. The value of those discussion was much higher than I imagined (I basically anticipated them just as non-value added time to get the result of an estimate, but they were a source of learning and consensus building).

Related: Assigning Story Points to Bug FixesMistake Proofing the Deployment of Software CodeChecklists in Software Development

These thoughts were prompted by: Story Points Considered Harmful – Or why the future of estimation is really in our past…

Continue reading

Trust But Verify

The following are my comments, which were sparked by question “Trust, but verify. Is this a good example of Profound Knowledge in action?” on the Linked In Deming Institute group.

Trust but verify makes sense to me. I think of verify as process measures to verify the process is producing as it should. By verifying you know when the process is failing and when to look for special causes (when using control chart thinking with an understanding of variation). There are many ways to verify that would be bad. But the idea of trust (respect for people) is not just a feel-good, “be nice to everyone and good things happen”, in Deming’s System of Profound Knowledge.

I see the PDSA improvement cycle as another example of a trust-but-verify idea. You trust the people at the gemba to do the improvement. They predict what will happen. But they verify what does actually happen before they run off standardizing and implementing. I think many of us have seen what happens when the idea of letting those who do the work, improve the process, is adopted without a sensible support system (PDSA, training, systems thinking…). It may actually be better than what was in place, but it isn’t consistent with Deming’s management system to just trust the people without providing methods to improve (and education to help people be most effective). Systems must be in place to provide the best opportunity to succeed. Trusting the people that do the work, is part of it.

I understand there are ways to verify that would be destructive. But I do believe you need process measures to verify systems are working. Just trusting people to do the right thing isn’t wise.

A checklist is another way of “not-trusting.” I think checklists are great. It isn’t that I don’t trust people to try and do the right thing. I just don’t trust people alone, when systems can be designed with verification that improves performance. I hear people complaign that checklists “don’t respect my expertise” or have the attitude that they are “insulting to me as a professional” – you should just trust me.

Sorry, driving out fear (and building trust – one of Deming’s 14 points) is not about catering to every person’s desire. For Deming’s System of Profound Knowledge: respect for people is part of a system that requires understand variation and systems thinking and an understanding of psychology and theory of knowledge. Checklists (and other forms of verification) are not an indication of a lack of trust. They are a a form of process measure (in a way) that has been proven to improve results.

Continue reading

2011 Management Blog Roundup: Stats Made Easy

The 4th Annual Management blog roundup is coming to a close soon. This is my 3rd and final review post looking back at 2001, the previous two posts looked at: Gemba Panta Rei and the Lean Six Sigma Blog.

I have special affinity for the use of statistics to understand and improve. I imaging it is both genetic and psychological. My father was a statistician and I have found memories of applying statistical thinking to understand a result or system. I also am comfortable with numbers, and like most people enjoy working with things I have an affinity for.

photo of Mark Anderson

Mark Anderson

Mark Anderson’s Stats Made Easy blog brings statistical thinking to managers. And this is not an easy thing to do, as one of his posts shows, we have an ability to ignore data we don’t want to know. Wrong more often than right but never in doubt: “Kahneman examined the illusion of skill in a group of investment advisors who competed for annual performance bonuses. He found zero correlation on year-to-year rankings, thus the firm was simply rewarding luck. What I find most interesting is his observation that even when confronted with irrefutable evidence of misplaced confidence in one’s own ability to prognosticate, most people just carry on with the same level of self-assurance.”

That actually practice of experimentation (PDSA…) needs improvement. Too often the iteration component is entirely missing (only one experiment is done). That is likely partially a result another big problem: the experiments are not nearly short enough. Mark offered very wise advice on the Strategy of experimentation: Break it into a series of smaller stages. “The rule-of-thumb I worked from as a process development engineer is not to put more than 25% of your budget into the first experiment, thus allowing the chance to adapt as you work through the project (or abandon it altogether).” And note that, abandon it altogether option. Don’t just proceed with a plan if what you learn makes that option unwise: too often we act based on expectations rather than evidence.

In Why coaches regress to be mean, Mark explained the problem with reacting to common cause variation and “learning” that it helped to do so. “A case in point is the flight instructor who lavishes praise on a training-pilot who makes a lucky landing. Naturally the next result is not so good. Later the pilot bounces in very badly — again purely by chance (a gust of wind). The instructor roars disapproval. That seems to do the trick — the next landing is much smoother.” When you ascribe special causation to common cause variation you often confirm your own biases.

Mark’s blog doesn’t mention six sigma by name in his 2011 posts but the statistical thinking expressed throughout the year make this a must for those working in six sigma programs.

Related: 2009 Curious Cat Management Blog Carnival2010 Management Blog Review: Software, Manufacturing and Leadership

Eliminate the Waste of Waiting in Line with Queuing Theory

One thing that frustrates me is how managers fail to adopt proven strategies for decades. One very obvious example is using queuing theory to setup lines.

Yes it may be even better to adopt strategies to eliminate as much waiting in line as possible, but if there is still waiting in line occurring and you are not having one queue served by multiple representatives shame on you and your company.

Related: Customer Focus and Internet Travel SearchYouTube Uses Multivariate Experiment To Improve Sign-ups 15%Making Life Difficult for Customers

Steve Jobs Discussing Customer Focus at NeXT

Video from 1991 when Steve Jobs was at NeXT. Even with the customer focus however, NeXT failed. But this does show the difficulty in how to truly apply customer focus. You have to be creative. You have examine data. You have to really understand how your customers use your products or services (go to the gemba). You have to speculate about the future. The video is also great evidence of providing insight to all employees of the current thinking of executives.

Related: Sometimes Micro-managing Works (Jobs)Delighting CustomersWhat Job Does Your Product Do?

One factor at a time (OFAT) Versus Factorial Designs

Guest post by Bradley Jones

Almost a hundred years ago R. A. Fisher‘s boss published an article espousing OFAT (one factor at a time). Fisher responded with an article of his own laying out his justification for factorial design. I admire the courage it took to contradict his boss in print!

Fisher’s argument was mainly about efficiency – that you could learn as much about many factors as you learned about one in the same number of trials. Saving money and effort is a powerful and positive motivator.

The most common argument I read against OFAT these days has to do with inability to detect interactions and the possibility of finding suboptimal factor settings at the end of the investigation. I admit to using these arguments myself in print.

I don’t think these arguments are as effective as Fisher’s original argument.

To play the devil’s advocate for a moment consider this thought experiment. You have to climb a hill that runs on a line going from southwest to northeast but you are only allowed to make steps that are due north or south or due east or west. Though you will have to make many zig zags you will eventually make it to the top. If you noted your altitude at each step, you would have enough data to fit a response surface.

Obviously this approach is very inefficient but it is not impossible. Don’t mistake my intent here. I am definitely not an advocate of OFAT. Rather I would like to find more convincing arguments to persuade experimenters to move to multi-factor design.

Related: The Purpose of Factorial Designed ExperimentsUsing Design of Experimentsarticles by R.A. Fisherarticles on using factorial design of experimentsDoes good experimental design require changing only one factor at a time (OFAT)?Statistics for Experimenters

Warren Buffett’s 2010 Letter to Shareholders

Warren Buffett has published his always excellent annual shareholder letter. His letters, provide excellent investing insight and good management ideas.

Yearly figures, it should be noted, are neither to be ignored nor viewed as all-important. The pace of the earth’s movement around the sun is not synchronized with the time required for either investment ideas or operating decisions to bear fruit. At GEICO, for example, we enthusiastically spent $900 million last year on advertising to obtain policyholders who deliver us no immediate profits. If we could spend twice that amount productively, we would happily do so though short-term results would be further penalized. Many large investments at our railroad and utility operations are also made with an eye to payoffs well down the road.

At Berkshire, managers can focus on running their businesses: They are not subjected to meetings at headquarters nor financing worries nor Wall Street harassment. They simply get a letter from me every two years and call me when they wish. And their wishes do differ: There are managers to whom I have not talked in the last year, while there is one with whom I talk almost daily. Our trust is in people rather than process. A “hire well, manage little” code suits both them and me.

Cultures self-propagate. Winston Churchill once said, “You shape your houses and then they shape you.” That wisdom applies to businesses as well. Bureaucratic procedures beget more bureaucracy, and imperial corporate palaces induce imperious behavior. (As one wag put it, “You know you’re no longer CEO when you get in the back seat of your car and it doesn’t move.”) At Berkshire’s “World Headquarters” our annual rent is $270,212. Moreover, the home-office investment in furniture, art, Coke dispenser, lunch room, high-tech equipment – you name it – totals $301,363. As long as Charlie and I treat your money as if it were our own, Berkshire’s managers are likely to be careful with it as well.

At bottom, a sound insurance operation requires four disciplines… (4) The willingness to walk away if the appropriate premium can’t be obtained. Many insurers pass the first three tests and flunk the fourth. The urgings of Wall Street, pressures from the agency force and brokers, or simply a refusal by a testosterone-driven CEO to accept shrinking volumes has led too many insurers to write business at inadequate prices. “The other guy is doing it so we must as well” spells trouble in any business, but none more so than insurance.

I don’t agree with everything he says. And what works at one company, obviously won’t work everywhere. Copying doesn’t work. Learning from others and understanding what makes it work and then determining how to incorporate some of the ideas into your organization can be valuable. I don’t believe in “Our trust is in people rather than process.” I do believe in “hire well, manage little.” Exactly what those phrases mean is not necessarily straight forward. I believe you need to focus on creating a Deming based management system and that will require educating and coaching managers about how to manage such a system. But that the management decisions about day to day operations should be left to those who are working on the processes in question (which will often be workers, that are not managers, sometimes will be supervisors and managers and sometimes will be senior executives).

Related: Too often, executive compensation in the U.S. is ridiculously out of line with performance.Management Advice from Warren BuffetGreat Advice from Warren Buffett to University of Texas – Austin business school students2004 Warren Buffet Report
Continue reading

Actionable Metrics

Metrics are valuable when they are actionable. Think about what will be done if certain results are shown by the data. If you can’t think of actions you would take, it may be that metric is not worth tracking.

Metrics should be operationally defined so that the data is collected properly. Without operationally definitions data collected by more than one person will often include measurement error (in this case, the resulting data showing the results of different people measuring different things but calling the result the same thing).

And without operational definitions those using the resulting data may well mis-interpret what it is saying. Often data is presented without an operational definition and people think the data is saying something that it is not. I find most often when people say statistics lie it is really that they made an incorrect assumption about what the data said – which most often was because they didn’t understand the operational definition of the data. Data can’t lie. People can. And people can intentionally mislead with data. But far more often people unintentionally mislead with data that is misunderstood (often this is due to failure to operationally define the data).

In response to: Metrics Manifesto: Raising the Standard for Metrics

Related: Outcome MeasuresEvidence-based ManagementMetrics and Software DevelopmentDistorting the System (due to misunderstanding metrics)Manage what you can’t measure