Tag Archives: experiments

Interview of Bill Hunter: Statistical Variability and Interactions

Interview of Bill Hunter on Statistical Variability and Interactions by Peter Scholtes, 1986:

In this interview Bill Hunter describes how results are made up of the impact and interactions of many variables. Many of those variables we don’t know about or account for. What we normally do is try to figure out the most important variables for processes and then experiment with those variables to find the best options given what we are trying to achieve.

Often the description of what is going on in such cases is that there is arbitrary error or random variation that influences the final results. What Bill discusses in this interview is that what is seen as arbitrary or random is often identifiably caused by specific variables. But often we don’t know what those variables are or how they are varying while we are getting different results over time.

He discusses how many research efforts seek to find the most important 2 variables and create a model based on those 2 variables to predict results. Even in PhD level research that is often done. He then discusses how to deal with other important variables.

He discusses the real world problems businesses must face in creating solutions that work.

If they are going to sell the product in Mississippi, and they are going to sell in Arizona and North Dakota, they have to have a robust product that will work in all these different conditions… It is not good enough for them to have a model that works sometimes… they’ve got to probe deeper and learn how relative humidity affects things and build that into the whole system in a different kind of way… they have to try and dig out the effects of these other x’s

So the business has to figure out the impact of many more variables in order to create reliable and robust products and services. This example is about variables that impact the use of the product by a customer, but the same concept applies to processes within your business.
Continue reading →

Understanding Design of Experiments (DoE) in Protein Purification

This webcast, from GE Life Sciences, seeks to provide an understanding Design of Experiments (DoE) using an example of protein purification. It begins with a good overview of the reason why multi-factorial experiments must be used while changing multiple factors at the same time in order to see interactions between factors. These interactions are completely missed by one-factor-at-a-time experiments.

While it is a good introduction it might be a bit confusing if you are not familiar with multi-factorial designed experiments. You may want to read some of the links below or take advantage of the ability to pause the video to think about what he says or to replay portions you don’t pick up immediately.

I have discussed the value of design of experiments in multiple posts on this blog in the past, including: Introductory Videos on Using Design of Experiments to Improve Results by Stu Hunter, Design of Experiments: The Process of Discovery is Iterative and Factorial Designed Experiment Aim.

He also provides a good overview of 3 basic aims of multivariate experiment (DoE):

screening (to determine which factors have the largest impact on the results that are most important)
optimization (optimize the results)
robustness testing (determine if there are risks in variations to factors)

Normally an experiment will focus on one of these aims. So you don’t know the most important factors you may choose to do a screening experiment to figure out which factors you want to study in detail in an optimization experiment.

It could be an optimized set of values for factors provides very good results but is not robust. If you don’t have easy way to make sure the factors do not vary it may be worthwhile to choose another option that provides nearly as good results but is much more robust (good results even with more variation within the values of the factors).

Transform the Management System by Experimenting, Iterating and Adopting Standard Work

In this short video, Dr. John Toussaint describes how ThedaCare applied leadership standard work to create a successful management transformation. The changes to the management system were tested by applying standard work for all positions in 2 parts of the organization (including all senior management positions) and learning and adapting and then spreading the new methods to the rest of the organization.

Changes to the management system require the same testing and piloting of changes on a small scale as other process changes. Experiment by going an inch wide and a mile deep, iterate over PDSA cycles, and once we have a solution that works adopt it widely (the A in PDSA).

George Box Webcast on Statistical Design in Quality Improvement

George Box lecture on Statistical Design in Quality Improvement at the Second International Tampere Conference in Statistics, University of Tampere, Finland (1987).

Early on he shows a graph showing the problems with American cars steady over a 10 years period. Then he overlays the results for Japanese cars which show a steady and significant decline of the same period.

Those who didn’t get to see presentations before power point also get a chance to see old school, hand drawn, overhead slides.

He discusses how to improve the pace of improvement. To start with informative events (events we can learn from) have to be brought to the attention of informed observers. Otherwise only when those events happen to catch the attention of the right observer will we capture knowledge we can use to improve. This results in slow improvement.

A control chart is an example of highlighting that something worth studying happened. The chart will indicate when to pay attention. And we can then improve the pace of improvement.

Next we want to encourage directed experimentation. We intentionally induce informative events and pay close attention while doing so in order to learn.

Every process generates information that can be used to improve it.

He emphasis the point that this isn’t about only manufacturing but it true of any process (drafting, invoicing, computer service, checking into a hospital, booking an airline ticket etc.).

He then discussed an example from a class my father taught and where the students all when to a TV plant outside Chicago to visit. The plant had been run by Motorola. It was sold to a Japanese company that found there was a 146% defect rate (which meant most TVs were taken off the line to be fixed at least once and many twice) – this is just the defect rate before then even get off the line. After 5 years the same plant, with the same American workers but a Japanese management system had reduced the defect rate to 2%. Everyone, including managers, were from the USA they were just using quality improvement methods. We may forget now, but one of the many objections managers gave for why quality improvement wouldn’t work in their company was due to their bad workers (it might work in Japan but not here).

He references how Deming’s 14 points will get management to allow quality improvement to be done by the workforce. Because without management support quality improvement processes can’t be used.

With experimentation we are looking to find clues for what to experiment with next. Experimentation is an iterative process. This is very much the mindset of fast iteration and minimal viable product (say minimal viable experimentation as voiced in 1987).

There is great value in creating iterative processes with fast feedback to those attempting to design and improve. Box and Deming (with rapid turns of the PDSA cycle) and others promoted this 20, 30 and 40 years ago and now we get the same ideas tweaked for startups. The lean startup stuff is as closely related to Box’s ideas of experimentation as an iterative process as it is to anything else.

He also provided a bit of history that I was not aware of saying the first application of orthogonal arrays (fractional factorial designs) in industry was by Tippett in 1933. And he then mentioned work by Finney in 1945, Plackett and Burman in 1946 and Rao in 1947.

Interview on PDSA, Deming, Strategy and More

Bill Fox interviewed me and has posted part one of the interview on his web site: Predicting Results in the Planning Stage (sorry, the link has been hijacked to forward to an unrelated page [so obviously I removed the link], I have posted the interview which can now be reached here):

Bill: John, what is your best process improvement strategy or tactic that has worked well for you or your clients?

John: I would say the PDSA improvement cycle and a few key practices in using the PDSA properly like predicting the results in the plan stage—something that a lot of the times people do not do—to determine what would be done based on the results of that prediction.

People discover, especially when they’re new to this stuff, regarding the data that they’re collecting, that maybe even if they got the results they are predicting, they still don’t have enough data to take action. So you figure that even if that number is 30, they would need to know three other things before they make the change. So then, in the plan stage, you can figure that you need to address these other issues, too. At any time that people are collecting data is useful to figure out, for instance: “What do we need to do if the result is 30 or if the result is 3?” And if you don’t have any difference, why are you collecting the data?

Another important piece is the D in Plan, Do, Study, Act. It means “do the experiment”. A lot of times, people get confused into thinking that D means deploy the results or something like that, but thinking of D as ‘doing the experiment’ can be helpful.

A really big key between people that use PDSA successfully and those who don’t is that the ones that do it successfully turn the cycle quickly.

Another response:

Bill: What is the biggest misunderstanding about the Deming Management System you think people have?

John: I would say that there are a couple. The followers that want to pin everything to Deming tend to overlook the complexities and nuances and other things.

The other problem is that some of the critics latch on to a specific quote from Deming, something like a one-sentence long quote, and then they extrapolate from that one sentence-long quote what that means. And the problem is that Deming has lots of these one-sentence quotes that are very memorable and meaningful and useful, but they don’t capture every nuance and they don’t alone capture what it really means (you need to have the background knowledge to understand it completely).

They are sort of trying to oversimplify the message into these sound bites, and I find that frustrating. Because those individual quotes are wonderful, but they are limited to one little quote out of hours of videotape, books, articles, and when you don’t understand the context in which that resides, that’s a problem.

See the full interview for more details and other topics. I think it is worth reading, of course I am a bit biased.

Taking Risks Based on Evidence

My opinion has long been that football teams are too scared to take an action that is smart but opens the coach to criticism. So instead of attempting to make it on 4th down (if you don’t understand American football, just skip this post) they punt because that is the decision that is accepted as reasonable.

So instead of doing what is wise they do what avoids criticism. Fear drives them to take the less advantageous action. Now I have never looked hard at the numbers, but my impression is that it is well worth the risk to go for it on 4th down often. In a quick search I don’t see a paper by a Harvard professor (this article refers to it also – Fourth down: To punt or to go?) on going for it on 4th down but I found on by a University of California, Berkeley economist (David Romer wrote called “Do Firms Maximize? Evidence from Professional Football.”).

On the 1,604 fourth downs in the sample for which the analysis implies that teams are on average better off kicking, they went for it only nine times. But on the 1,068 fourth downs for which the analysis implies that teams are on average better off going for it, they kicked 959 times.

My guess is that the advantages to going for it on 4th down are greater for high school than college which is greater than the advantage for the pros (but I may be wrong). My guess is this difference is greater the more yardage is needed. Basically my feeling is the variation in high school is very high in high school and decreases with greater skill, experience and preparation. Also the kicking ability (punting and field goals) impacts the choices of going for it on 4th down and that dramatically increases in college. So if I am correct, I think pro coaches should be more aggressive on 4th down, but likely less aggressive than high school coaches should be.

But in any event the data should be explored and strategies should be tested.

Continue reading →

Resources for Using the PDSA Cycle to Improve Results

PDSA Improvement cycle graphic from my book – Management Matters

Using the PDSA cycle (plan-do-study-act) well is critical to building a effective management system. This post provides some resources to help use the improvement cycle well.

I have several posts on this blog about using the PDSA cycle to improve results including:

The authors and consultants with Associates for Process Improvement have the greatest collection of useful writing on the topic. They wrote two indispensable books on the process improvement through experimentation: The Improvement Guide and Quality Improvement Through Planned Experimentation. And they have written numerous excellent articles, including:

Clearing Up Myths About the Deming Cycle by Ron Moen and Cliff Norman
The Importance of Concepts in Creativity and Improvement by Lloyd Provost and Gerald Langley
ASQ’s Accelerating Change Collaborative Series: A Challenge for Industry by Kevin Nolan
Evolution of the PDSA Cycle by Ron Moen and Cliff Norman
Understanding Variation by Thomas Nolan, Lloyd Provost

Jeff Bezos: Innovation, Experiments and Long Term Thinking

Jeff Bezos, bought the Washington Post. He has long showed a willingness to take a long term view at Amazon. He is taking that same thinking to the Washington Post:

In my experience, the way invention, innovation and change happen is [through] team effort. There’s no lone genius who figures it all out and sends down the magic formula. You study, you debate, you brainstorm and the answers start to emerge. It takes time. Nothing happens quickly in this mode. You develop theories and hypotheses, but you don’t know if readers will respond. You do as many experiments as rapidly as possible. ‘Quickly’ in my mind would be years.”

The newspaper business is certainly a tough one today – one that doesn’t seem to have a business model that is working well (for large, national papers). I figured the answer might be that a few (of the caliber of Washington Post, New York Times…) would be owed by foundations and supported largely by a few wealthy people that believed in the value of a strong free press and journalism. Maybe Bezos will find a business model that works. Or maybe he will just run it essentially as a foundation without needing a market return on his investment.

The Guardian (where the article with the quote was published) is an example of good journalism by a foundation. ProPublica is another (though I guess it is really a non-profit but most of the funding seems to be via foundations).

Design of Experiments: The Process of Discovery is Iterative

This video is another excerpt on the design of experiments videos by George Box, see previous posts: Introduction to Fractional Factorial Designed Experiments and The Art of Discovery. This video looks at learning about experimental design using paper helicopters (the paper linked there may be of interest to you also).

[the video is no longer available online]

In this example a screening experiment was done first to find those factors that have the largest impact on results. Once the most important factors are determined more care can be put into studying those factors in greater detail.

The video was posted by Wiley (with the permission of George’s family), Wiley is the publisher of George’s recent autobiography, An Accidental Statistician, and many of his other books.

The importance of keeping the scope (in dollars and time) of initial experiments down was emphasized in the video.

George Box: “Always remember the process of discovery is iterative. The results of each stage of investigation generating new questions to answered during the next.”

Soren Bisgaard and Conrad Fung also appear in this except of the video.

The end of the video includes several suggested resources including: Statistics for Experimenters, Out of the Crisis and The Scientific Context of Quality Improvement.

What is the Explanation Going to be if This Attempt Fails?

Occasionally during my career I have been surprised by new insights. One of the things I found remarkable was how quickly I thought up a new explanation for what could have caused a problem when the previously expressed explanation was proven wrong. After awhile I stopped finding it remarkable and found it remarkable how long it took me to figure out that this happened.

I discovered this as I programmed software applications. You constantly have code fail to run as you expect and so get plenty of instances to learn the behavior I described above. While I probably added to my opportunities to learn by being a less than stellar coder I also learned that even stellar coders constantly have to iterate through the process of creating code and seeing if it works, figuring out why it didn’t and trying again.

The remarkable thing is how easily I could come up with an new explanation. Often nearly immediately upon what I expected to work failing to do so. And one of the wonderful things about software code is often you can then make the change in 10 minutes and a few minutes later see if it worked (I am guessing my brain kept puzzling over the ideas involved and was ready with a new idea when I was surprised by failure).

When I struggled a bit to find an initial explanation I found myself thinking, “this has to be it” often because of two self reinforcing factors.

First, I couldn’t think of anything else that would explain it. Sometimes you will think right away of 4 possible issues that could cause this problem. But, when I struggled to find any and then finally came up with an idea it feels like if there was another possibility I should have thought of it while struggling to figure out what I finally settled on.

Second, the idea often seems to explain exactly what happened, and it often feels like “of course it didn’t work, what was I thinking I need to do x.” This often turns out to be true, doing x solves the problem and you move on. But a remarkable percentage of the time, say even just 10%, it doesn’t. And then I would find myself almost immediately thinking, of course I need to do y. Even when 10 seconds ago I was convinced there was no other possibility.

Continue reading →