Curious Cat Management Improvement Blog: Deming, lean thinking, innovation, customer focus, continual improvement, six sigma.
Design of Experiments Category

Posts about design of experiments (DoE) - six sigma, (factorial designs, multivariate experiments) as used by statisticians since RA Fisher and now with six sigma.
We recommend Statistics for Experimenters. Articles on DoE: Teaching Engineers Experimental Design With a Paper Helicopter, Development of the Theory of Experimental Design by Sir R.A Fisher

July 29, 2008

Full and Fractional Factorial Test Design

An Essential Primer on Full and Fractional Factorial Test Design

Since full factorial gathers additional data, it reveals all possible interactions, but as seen by the numbers above, there is a trade-off. More data equals more information but more data also equals a longer test duration. The minimum data requirements for full factorial are very high since you are showing every experiment.

Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments. You might be wondering how fractional factorial can be accurate if interactions are possible?

Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.

I am a fan of design of experiments as long time readers know (see posts on design of experiments).

Some good resources for more on the topics discussed above: What Can You Find Out From 8 and 16 Experimental Runs? by George Box - Statistics for Experimenters - Design of Experiments in Advertising.

Related: Google Website Optimizer - factorial experiment articles - Using Design of Experiments - Marketers Are Embracing Statistical Design of Experiments

June 29, 2008

Printer Product Development Using Design of Experiments

MEMS development in less than half the time by Christopher N. Delametter, Eastman Kodak Company

The traditional approach to optimizing a product or process using computer simulation is to evaluate the effects of one design parameter at a time. The problem with this approach is that interactions between design factors and second-order effects are likely to result in a locally optimized design that will provide far less performance than the global optimum. Kodak researchers use DOE to develop tests that examine first-order, second-order, and multiple factor effects simultaneously with relatively few simulation runs. The result is that the analyst can iterate to a globally optimized design with a far higher level of certainty and in much less time than the traditional approach.

By using DOE to drive CFD, Kodak researchers were able to optimize the design of the printhead in considerably less time than competitors. The advantages of simulation were especially apparent late in the project when researchers discovered a more optimal ink formulation for one of the colors.

Related: Design of Experiments articles - Using Design of Experiments - Statistics for Experimenters - Why Use Designed Factorial Experiments? - Kodak Debuts Printers With Inexpensive Cartridges

February 4, 2008

Improvement Through Designed Experiments

The Rationale of Scientific Experimentation by John Dowd explains the value of designed experiments.

Another difficulty in industrial experimentation is the existence of interactions. As has been stated, manufacturing processes are complex with many factors involved. In many processes these factors interact. This is particularly so for continuous processes such as plating or sputtering. Saying that the factors interact means more than that they are related to each other. It means that the effect of one (or more) factors on the response variable(s) changes when one (or more) other factor(s) changes its value.

In order to detect interactions and understand the nature of their effects it is necessary to combine the interacting factors into the same experimental runs. The problem is not necessarily knowing in advance if the interactions exist. Sometimes they are predictable with theory. Sometimes they are discovered when the process behaves ’strangely’.

In addition to their efficiency, factorial designs also offer the only method of detecting interactions through experimentation. Because numerous factors can be combined in the same series of experimental runs, the interactions can be detected and the nature of their effects can be evaluated when they are present.

The paper also explains analytic and enumerative studies. Dr. Deming stressed the importance of understanding the distinction between the two.

Related: management improvement articles - Design of Experiments articles - Statistics for Experimenters - search statistical management improvement sites - Using Design of Experiments

November 3, 2007

Design of Experiments in Operational Testing

Edwards looks toward future of testing

We are fielding a Design of Experiments concept to ensure we conduct the right amount of testing — not too much or too little, but just right. We will field this approach in phases as we must train our people and put the right tools in place. However, it is already showing great promise.

In a recent Benefield Anechoic Facility test, the 412th Electronic Warfare Group used Design of Experiments methodology to cut a two-month program to three weeks. This schedule reduction translated directly into savings and helped reduce the concept-to-fielding cycle time while still ensuring the system was thoroughly tested. While building these capabilities is critical, the most critical piece of the puzzle is our people. We must continue to develop engineers, pilots, navigators, program managers and maintainers to test these systems and “find stuff so the warfighter doesn’t.”

It is hard to tell if they really are using Design of Experiments or just using the term but it seem possible they are really using it. As I have said a number of times it is a powerful and under-utilized tool for improvement. Related: Using Design of Experiments - design of experiments articles - posts on public sector management - Why Use Designed Factorial Experiments?

October 19, 2007

2007 William G. Hunter Award

T.N. Goh received ASQ Statistics Division’s 2007 William G. Hunter Award. He sent me this email:

You may not realize that I first met Bill 38 year ago, when he was in Singapore helping us set up the first school of engineering in the country. He persuaded me to go to the graduate school at UW-Madison and I daresay that’s the best advice I ever got in my whole career. Now when I come to think of it, what Bill stood for in his lifetime has not been, and never will be, out of date. He had advocated the use of statistical thinking and the systems approach, which if anything is even more critical today in handling issues such as global warming and government effectiveness.

Also, statistical design of experiments has assumed an increasingly important role in performance improvement and optimization in the face of constrained resources, again something always in the minds of engineers, managers and business leaders. From time to time there are others who package statistical tools under labels Bill might not even have seen himself, such as “Design for Six Sigma“, but the underlying idea is still the same: recognize the existence of variation, and the earlier you anticipate it and do something about it, the better off you will be in the end.

Bill’s zeal in spreading the message and sharing his knowledge and expertise with people in other parts of the world is well known; I would even say that he had recognized that “the world is flat” way before the likes of Tom Friedman discovered the reality of globalization!

So that’s to share my thoughts with you, having being honored by the Bill Hunter award. I am copying this to Stu, also to Doug who chairs the committee for this award. I reality enjoy the professional association and friendship with you all.

I had not realized Dad was helping set up the first school of engineering in Singapore. This is the kind of thing I mentioned in, The Importance of Management Improvement, where I mention people telling me the positive impact Dad had on their lives.

Related: Curious Cat Science and Engineering Blog - Statistics for Experimenters - Singapore Research Fellowship

August 13, 2007

Six Sigma in Software Development

Six Sigma makes inroads in software development organizations

“A lot of big companies are developing their own software engineering variance of Six Sigma training,” said Siviy, “putting software-specific examples into the normal Six Sigma curriculum.” However, she said, it’s early in the adoption curve. “In the software world there is a real lack of case studies that show applications of Six Sigma in software engineering,” she said. And those that use Six Sigma in software are often reluctant to share examples because they consider it a competitive advantage.

Still, Siviy said, “At a lot of software conferences now you see a sprinkling of presentations that somehow touch on Six Sigma or Lean, and the quality and depth of questions have evolved tremendously. In general, and not just in Six Sigma, as the [software] industry matures you see a wave of interest in measurement and analytical techniques.”

McKesson is a prime example. “Measurement is key,” Childers said. “What you can’t or don’t measure, you don’t know.”

A couple points. First, you can know what you don’t measure. Do you know your parents? Do you measure them? Manage what you can’t measure.

The Software Engineering Institute at Carnegie Mellon University has great materials. There is a danger in using those materials to become overly bureaucratic but the material was developed out of an excellent understanding of quality management (way back when that was the way this stuff was referred to). David Anderson provides some good insights, see: Stretching Agile to fit CMMI Level 3

Design of experiments is very suited to testing software: Planning Efficient Software Tests - Design Of Experiment For Software Testing.

six sigma does seem to foster a lack of sharing; which is a shame.

Related: six sigma articles and links - six sigma posts - software development posts - design of experiments articles

June 19, 2007

Mistakes in Experimental Design and Interpretation

Mistakes in Experimental Design and Interpretation

Humans are very good at detecting patterns, but rather poor at detecting randomness. We expect random incidents of cancer to be spread homogeneously, when in fact true randomness results in random clusters, not homogeneity. It is a mistake for an experiment to consider a pool of 47,000 possibilities, and then only report on the 7 cases that seem interesting.

A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (”Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.

This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.

Related: Seeing Patterns Where None Exists - Illusions, Optical and Other - Understanding Data - Dangers of Forgetting the Proxy Nature of Data - How to Deal with False Research Findings - descriptive “theory” and normative theory

April 7, 2007

Google Website Optimizer

Google’s Website Optimizer allows for multivariate testing of your website.

Website Optimizer, Google’s free multivariate testing application, helps online marketers increase visitor conversion rates and overall visitor satisfaction by continually testing different combinations of site content (text and images).

Rather than sitting in a room and arguing over what will work better, you can save time and eliminate the guesswork by simply letting your visitors tell you what works best. We’ll guide you through the process of designing and implementing your first experiment. Start optimizing your most important web pages and see detailed reports within hours.

Google provides an online slide show with audio (a good example of one way to share online information sharing in my opinion). This tool seems to have limited experimental options to what is on the page (it does not appear, for example, that one variable could be current customer v. new visitor…). Still it looks like an very easy way to do some simple multi-factorial experiments. Google offers a list of partners for those interested in consulting and more advanced features (and for those experts reading this you can apply to be a partner).

Related: Design of experiments posts - articles on multi-factorial experimentation - Google: Experiment Quickly and Often - Data Based Decision Making at Google
(more…)

March 18, 2007

Experiment and Learn

Experimenting with milkshakes?

I have been on a mission to convince firms to do simple experiments that will give them feedback regarding the decisions that they make. Just as with people (as Anders Ericsson studies), firms cannot learn with feedback. It turns out, however, that it is not easy for people in companies to see the wisdom in experiments.

Experiments are useful and underused. PDSA and design of experiments are two concepts that aid in experimenting successfully.

Related: Google: Experiment Quickly and Often - Why Use Designed Factorial Experiments? - Using Design of Experiments - theory of knowledge

February 14, 2007

Scientific Thinking - the Modern Way

“Scientific thinking” the modern way by Bill Harris:

What does this all mean? It simply means that Fisher’s designed experiments give us better and faster means to extract insight from tests on system dynamics models than the old one-factor-at-a-time approach.

I thank Deb Schenk, then (and perhaps now) statistician at Hewlett-Packard Company, for teaching me and others about the design of experiments using Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building back in 1981-82.

I admit to a bit of bias, in seeing my father’s book (Statistics for Experimenters 2nd edition was published last year by the way), referenced but Bill Harris is exactly right in the power of design of experiments. The most recent post discusses Ackoff’s excellent f-Laws and a previous post discusses Deming (titled, It’s the process) so I couldn’t resist adding a post myself.

Related: design of experiments posts - Ackoff’s New Book: Management f-Laws

November 28, 2006

Why Use Designed Factorial Experiments?

One-Factor-at-a-Time Versus Designed Experiments by Veronica Czitrom:

The advantages of designed experiments over [One Factor at a Time] OFAT experiments are illustrated using three real engineering OFAT experiments, and showing how in each case a designed experiment would have been better. This topic is important because many scientists and engineers continue to perform OFAT experiments.

I still remember, as a child, asking what my father was going to be teaching the company he was going to consult with for a few days. He said he was going to teach them about using designed factorial experiments. I said, but you explained that to me and I am just a kid, how can you be teaching adults that? Didn’t they learn it in school? The article is a good introduction to the idea of why one factor at a time experiments are an ineffective way to learn.

Related: Design of Experiments articles - Statistics for Experimenters (2nd Edition) - Design of Experiments blog posts

April 23, 2006

Using Design of Experiments as a Process Road Map

Using Design of Experiments as a Process Road Map by Davis Balestracci:

The current design of experiments (DOE) renaissance seems to favor factorial designs and/or orthogonal arrays as a panacea. In my 25 years as a statistician, my clients have always found much more value in obtaining a process “road map” by generating the inherent response surface in a situation. It’s hardly an advanced technique, but it leads to much more effective optimization and process control.

DOE is a tool that is very useful. And while the situations in which DOE is the best tool to use is limited the limited use of DOE is used less than it could be. See more articles on the use of design of Experiments (DOE).

March 17, 2006

Using Design of Experiments

How to Institute DOE in Your Company by Davis Balestracci:

DOE works, but I don’t need to sell that to the readers of this newsletter. But as certain as we all are, no one can deny that design of experiments faces resistance even in environments where it is a proven tool. Every research scientist or engineer who has had a major success from DOE can tell you story after story of how management still wanted problems solved one-factor-at-a-time.

Design of Experiments (DoE) was developed by R.A. Fisher in the 1920s (related terms: factorial design, multivariate expertness). Six Sigma was the first general management approach that specifically highlighted the use of Designed Experiments for improvement. Still the use of factorial designed experiments is much less than it could be.
(more…)

October 11, 2005

Box on Quality

Bill Hunter and George Box

Dr. George Box is not as well known in the general management community as his ideas merit (in my biased opinion - photo of Bill Hunter and George Box). He is well know in the statistics field as one of the leading statistical minds. Box on Quality is an excellent book that gathers his essays from his 65th to 80th year. The book has just been issued in paperback (which helps as the hardback was pricey).

While some of the essays are aimed at a reader with an advanced understanding of statistics, many of the articles are aimed at any manager attempting to apply Quality Management principles (SPC, Deming, process improvement, six sigma, etc.). An except from the book provides a table of contents and an introduction.

Some of the articles from the book are available online. I encourage you to take a look at several of the articles and then go ahead and add this book to your prized management resources, if you find them worthwhile.

August 9, 2005

Design of Experiments in Advertising

How Two Guys From the Gold Country Are Changing Advertising Forever by Robert X. Cringely

James Kowalick and Mario Fantoni, two guys who say they can show you how to use science to design ads that cost less while being 10 or more times as effective as doing it the old way.

Their secret is the Taguchi Method, which is a technique for designing experiments that converge on an ideal product solution.

“I taught over 300 courses for industry where we designed cars and electronic devices, but it wasn’t until one day I took over my wife’s kitchen and used Taguchi to perfect my recipe for vanilla wafer cookies that I realized how broadly it could be applied,” Kowalick recalls. “It took 16 batches, but by the end of the afternoon I had those wafers dialed in.”

It is great to see the application of Designed Experiments increasing. I am reminded of an article by my father, William G. Hunter, from 1975: 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments. Examples of the topics of the designed experiments his students performed:
(more…)

July 25, 2005

Design of Experiments Articles

We have added several Design of Experiments articles to the Curious Cat Management Improvement Library recently, including:

  • DOE It Yourself by Mark J. Anderson, A list of Design of Experiments excercizes that can be done in a classroom setting or as home by students with short explanations and links to documents online wtih more details.
May 27, 2005

Marketers Are Embracing Statistical Design of Experiments

Marketers Are Embracing Statistical Design of Experiments (site broke link so I removed it) by Richard Burnham.

Crayola® conducts an e-mail marketing DOE to attract parents and teachers to their new Internet site. The company discovers a combination of factors that makes their new e-mail pitch three-and-a-half times more effective than the control. (Harvard Business Review, October 2001, “Boost Your Marketing ROI with Experimental Design,” Almquist, Wyner.)

Marketers can’t always be certain what triggers buyers to respond. In the past, we were always admonished to test-test-test, but only one factor at a time – relying on our gut feelings and uncertain hopes. With DOE, marketers have replaced voodoo with the science of statistics.

For more on Design of Experiments see:

May 23, 2005

Open Source Management Terms

Statistical Method Helps Boost Bottom Lines, Batting Averages by Jon Van, Chicago Tribune:

Using statistical techniques embraced by quality guru W. Edwards Deming, Holland has worked for a generation guiding enterprises large and small to boost efficiency.

Yet getting management support for an MVT project is vital, Holland said, because experience shows that only 25 percent of ideas intended to improve a process will have a positive effect. The others will either have no effect or will hurt. Managers hate to see experimental results shoot down ideas they were certain would help, he said.

The article also mentions:

While Holland claims MVT as his own, others say it is really just a variation of strategies widely used in business.

“Multifactor experiments have been around for a long time,” said Ajit Tamhane, Northwestern University professor of statistics and industrial engineering and management sciences.

David Coit, a Rutgers University professor of industrial systems engineering, said that Holland’s MVT is very much like a quality-enhancing scheme called design of experiments.

So often we seem to focus on proprietary solutions. Instead it seems to me, most often what is needed is to do a good job of applying the ideas that have been known for decades. Deming ideas, design of experiments, lean thinking, experimentation, etc. are not secrets. There is a long history of how to apply these ideas to improve organizational performance.

QualPro obviously does well marketing itself (see press clippings from their web site) selling the concept of proprietary solutions to press organizations. Raising the question of whether the proprietary solutions really offers unique ideas, as the The Tribune article did was uncommon in my experience.

I like those encouraging the adoption of statistical tools to improvement management but I find the practice of trademarking terms like Six Sigma and MVT a bad way to encourage innovation in the practice of management. While it is nice to see Six Sigma efforts and others use statistical tools (such as design of experiments) I would encourage people to stay with “open source” management terms and remain part of a community looking to improve the practice of management.

Update: Also see - Management Advice Failures

March 8, 2005

Statistics for Experimenters - Second Edition

Buy Statistics for Experimenters

The classic Statistics for Experimenters has been updated by George Box and Stu Hunter, two of the three original authors. Bill Hunter, who was my father, and the other author, died in 1986. Order online: Statistics for Experimenters: Design, Innovation, and Discovery , 2nd Edition by George E. P. Box, J. Stuart Hunter, William G. Hunter.
I happen to agree with those who call this book a classic, however, I am obviously biased.

Google Scholar citations for the first edition of Statistics for Experimenters.
Citations in Cite Seer to the first edition.

The first edition includes the text of Experiment by Cole Porter. In 1978 finding a recording of this song was next to impossible. Now Experiment can be heard on the De-Lovely soundtrack.

Text from the publisher on the 2nd Edition:
Rewritten and updated, this new edition of Statistics for Experimenters adopts the same approaches as the landmark First Edition by teaching with examples, readily understood graphics, and the appropriate use of computers. Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.
(more…)

September 23, 2004

Management Improvement History

Originally posted to the Deming Electronic Network, 22 Sep 1999, in reponse to this message

I would like to say that I think it is good that we have disagreements on the DEN. I think it is a strength of the DEN, not a weakness. However, I think we sometimes get to personal with no real purpose. One example of this, for me, is: “Well, I guess we knew different Demings. Mine was a teacher named Dr. W. Edwards Deming.” I doubt this statement is meant to be taken literally, and if it is not I do not see what it adds to the discussion. I point this out not because I think this is some bad act that should be punished but that I think we need to continue to develop a sense of how we wish to express our disagreements and I think that we should try to do so more constructively.

For the past 60 years we’ve been looking for the magic bullet that will improve the quality of our products, services and lives. In the 1940s, we applied statistics through sampling, SPC and design of experiments to improve our products. In the 1950s, we used quality cost and total quality control to bring about quality improvement. In the 1960s, zero defects and MIL-Q-9858A drove the quality improvement process. In the 1970s, quality circles, process qualification and supplier qualification became key quality issues. In the 1980s, employee training in problem solving, team activities and just-in-time inventory were the things to do.”

I find this statement so far from the truth that it would seriously damage any PDSA with this as an accepted assessment of history. I do not believe Deming had such an inaccurate view (of course I may be wrong). I do believe we need to improve our practice of Quality (and to do that we need to understand what happened in the past and why it was not more successful). The idea that Design of Experiments (DoE) was at the core of some Quality Movement to me is not at all accurate. (more…)

Curious Cat Management Improvement Blog curiouscat.com 2005-2006 powered by WordPress - professor William G. Hunter homepage

Internal Links

Categories

Links

Popular Posts

Other


Search Blog

Web Search

Management Improvement web search


Author

John Hunter

Archives

September 2008
M T W T F S S
« Aug    
1234567
891011121314
15161718192021
22232425262728
2930  

Blogroll