Tag Archives: Design of Experiments

Highlights from Recent George Box Speech

The JMP blog has posted some highlights from George Box’s presentation at Discovery 2009 [the broken link was removed]

Infusing his entire presentation with humor and fascinating tales of his memories, Box focused on sequential design of experiments. He attributed much of what he knows about DOE [design of experiments] to Ronald A. Fisher. Box explained that Fisher couldn’t find the things he was looking for in his data, “and he was right. Even if he had had the fastest available computer, he’d still be right,” said Box. Therefore, Fisher figured out how to study a number of factors at one time. And so, the beginnings of DOE.

Having worked and studied with many other famous statisticians and analytic thinkers, Box did not hesitate to share his characterizations of them. He told a story about Dr. Bill Hunter and how he required his students to run an experiment. Apparently a variety of subjects was studied [see 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments]

According to Box, the difficulty of getting DOE to take root lies in the fact that these mathematicians “can’t really get the fact that it’s not about proving a theorem, it’s about being curious about things. There aren’t enough people who will apply [DOE] as a way of finding things out. But maybe with JMP, things will change that way.”

George Box is a great mind and great person who I have had the privilege of knowing my whole life. My father took his class at Princeton, then followed George to the University of Wisconsin-Madison (where Dr. Box founded the statistics department and Dad received the first PhD). They worked together building the UW statistics department, writing Statistics for Experimenters and founding the Center for Quality and Productivity Improvement among many other things.

Statistics for Experimenters: Design, Innovation, and Discovery shows that the goal of design of experiments is to learn and refine your experiment based on the knowledge you gain and experiment again. It is a process of discovery. If done properly it is very similar to the PDSA cycle with the application of statistical tools to aid in determining the impact of various factors under study.

Related: Box on QualityGeorge Box Quotationsposts on design of experimentsUsing Design of Experiments

YouTube Uses Multivariate Experiment To Improve Sign-ups 15%

Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

via: @hexawiseMy brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).

Related: Combinatorial Testing for SoftwareStatistics for ExperimentersGoogle’s Website Optimizer allows for multivariate testing of your website.Using Design of Experiments

Combinatorial Testing for Software

Combinatorial testing of software is very similar to the design of experiments work my father was involved in, and which I have a special interest in. Combinatorial testing looks at binary interaction effects (success or failure), since it is seeking to find bugs in software, while design of experiments captures the magnitude of interaction effects on performance. In the last several years my brother, Justin Hunter, has been working on using combinatorial testing to improve software development practices. He visited me this week and we discussed the potential value of increasing the adoption of combinatorial testing, which is similar to the value of increasing the adoption of the use of design of experiments: both offer great opportunities for large improvements in current practices.

Automated Combinatorial Testing for Software

Software developers frequently encounter failures that occur only as the result of an interaction between two components. Testers often use pairwise testing – all pairs of parameter values – to detect such interactions. Combinatorial testing beyond pairwise is rarely used because good algorithms for higher strength combinations (e.g., 4-way or more) have not been available, but empirical evidence shows that some errors are triggered only by the interaction of three, four, or more parameters

Practical Combinatorial Testing: Beyond Pairwise by Rick Kuhn, US National Institute of Standards and Technology; Yu Lei, University of Texas, Arlington; and Raghu Kacker, US National Institute of Standards and Technology.

the detection rate increased rapidly with interaction strength. Within the NASA database application, for example, 67 percent of the failures were triggered by only a single parameter value, 93 percent by two-way combinations, and 98 percent by three-way combinations.2 The detection-rate curves for the other applications studied are similar, reaching 100 percent detection with four- to six-way interactions.
These results are not conclusive, but they suggest that the degree of interaction involved in faults is relatively low, even though pairwise testing is insufficient. Testing all four- to six-way combinations might therefore provide reasonably high assurance.

Related: Future Directions for Agile ManagementThe Defect Black MarketMetrics and Software DevelopmentFull and Fractional Factorial Test DesignGoogle Website Optimizer

Statistics for Experimenters in Spanish

book cover of Estadística para Investigadores

Statistics for Experimenters, second edition, by George E. P. Box, J. Stuart Hunter and William G. Hunter (my father) is now available in Spanish.

Read a bit more can find a bit more on the Spanish edition, in Spanish. Estadística para Investigadores Diseño, innovación y descubrimiento Segunda edición.

Statistics for Experimenters – Second Edition:

Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.

* Graphical Analysis of Variance
* Computer Analysis of Complex Designs
* Simplification by transformation
* Hands-on experimentation using Response Service Methods
* Further development of robust product and process design using split plot arrangements and minimization of error transmission
* Introduction to Process Control, Forecasting and Time Series

Book available via Editorial Reverte

Related: Statistics for Experimenters ReviewCorrelation is Not CausationStatistics for Experimenters Dataposts on design of experiments

Full and Fractional Factorial Test Design

An Essential Primer on Full and Fractional Factorial Test Design

Since full factorial gathers additional data, it reveals all possible interactions, but as seen by the numbers above, there is a trade-off. More data equals more information but more data also equals a longer test duration. The minimum data requirements for full factorial are very high since you are showing every experiment.

Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments. You might be wondering how fractional factorial can be accurate if interactions are possible?

Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.

I am a fan of design of experiments as long time readers know (see posts on design of experiments).

Some good resources for more on the topics discussed above: What Can You Find Out From 8 and 16 Experimental Runs? by George Box – Statistics for ExperimentersDesign of Experiments in Advertising.

Related: Google Website Optimizerfactorial experiment articlesUsing Design of ExperimentsMarketers Are Embracing Statistical Design of Experiments

Printer Product Development Using Design of Experiments

MEMS development in less than half the time by Christopher N. Delametter, Eastman Kodak Company

The traditional approach to optimizing a product or process using computer simulation is to evaluate the effects of one design parameter at a time. The problem with this approach is that interactions between design factors and second-order effects are likely to result in a locally optimized design that will provide far less performance than the global optimum. Kodak researchers use DOE to develop tests that examine first-order, second-order, and multiple factor effects simultaneously with relatively few simulation runs. The result is that the analyst can iterate to a globally optimized design with a far higher level of certainty and in much less time than the traditional approach.

By using DOE to drive CFD, Kodak researchers were able to optimize the design of the printhead in considerably less time than competitors. The advantages of simulation were especially apparent late in the project when researchers discovered a more optimal ink formulation for one of the colors.

Related: Design of Experiments articlesUsing Design of ExperimentsStatistics for ExperimentersWhy Use Designed Factorial Experiments?Kodak Debuts Printers With Inexpensive Cartridges

Management Improvement Carnival #34

Please submit your favorite management posts to the carnival. Read the previous management carnivals.

  • Introduction to Factorial Designs by Jonathan Mendez – “I like the idea of velocity in marketing — test, learn, test, learn, test. Instead of one large test I prefer focusing attention on certain areas or elements to achieve deeper understanding.”
  • MIT’s Message about Lean Enterprise Transformation by Mark Edmondson- “1. Market leaders are good at embracing enterprise change; 2. Enterprise change requires a holistic approach that engages all stakeholders. This includes employees, suppliers, customers, unions, and investors/owners”
  • Two Types of Bottleneck by David J. Anderson – “I now teach that there are two types of bottleneck: capacity constrained resources CCRs; and non-instant availability resources”
  • Oranges, Pebbles, and Sand by Ron Pereira – “In this video my daughters and I demonstrate how meeting an objective is just the beginning to improvement.”
  • Why errorproof when you can double-check? – “If you are in the position to prevent the error in the first place, why wouldn’t you? And, I’d argue, if you can write a tool to detect the screw up – ie, it is possible to programmatically figure out that the template is wrong,”
  • Systems and Improvement by John Dowd – “Thus did Deming, over sixty years ago, show a basic model about how to think about quality and improvement.”
  • Continue reading

Google Website Optimizer

Google’s Website Optimizer allows for multivariate testing of your website.

Website Optimizer, Google’s free multivariate testing application, helps online marketers increase visitor conversion rates and overall visitor satisfaction by continually testing different combinations of site content (text and images).

Rather than sitting in a room and arguing over what will work better, you can save time and eliminate the guesswork by simply letting your visitors tell you what works best. We’ll guide you through the process of designing and implementing your first experiment. Start optimizing your most important web pages and see detailed reports within hours.

Google provides an online slide show with audio (a good example of one way to share online information sharing in my opinion). This tool seems to have limited experimental options to what is on the page (it does not appear, for example, that one variable could be current customer v. new visitor…). Still it looks like an very easy way to do some simple multi-factorial experiments. Google offers a list of partners [the link that Google broke was removed] for those interested in consulting and more advanced features (and for those experts reading this you can apply to be a partner).

Related: Design of experiments postsarticles on multi-factorial experimentationGoogle: Experiment Quickly and OftenData Based Decision Making at Google

Continue reading

Using Design of Experiments

How to Institute DOE in Your Company (link broken – removed) by Davis Balestracci:

DOE works, but I don’t need to sell that to the readers of this newsletter. But as certain as we all are, no one can deny that design of experiments faces resistance even in environments where it is a proven tool. Every research scientist or engineer who has had a major success from DOE can tell you story after story of how management still wanted problems solved one-factor-at-a-time.

Design of Experiments (DoE) was developed by R.A. Fisher in the 1920s (related terms: factorial design, multivariate expertness). Six Sigma was the first general management approach that specifically highlighted the use of Designed Experiments for improvement. Still the use of factorial designed experiments is much less than it could be.
Continue reading

Box on Quality

Bill Hunter and George Box

Dr. George Box is not as well known in the general management community as his ideas merit (in my biased opinion – photo of Bill Hunter and George Box). He is well know in the statistics field as one of the leading statistical minds. Box on Quality is an excellent book that gathers his essays from his 65th to 80th year. The book has just been issued in paperback (which helps as the hardback was pricey).

While some of the essays are aimed at a reader with an advanced understanding of statistics, many of the articles are aimed at any manager attempting to apply Quality Management principles (SPC, Deming, process improvement, six sigma, etc.). An except from the book provides a table of contents and an introduction.

Some of the articles from the book are available online. I encourage you to take a look at several of the articles and then go ahead and add this book to your prized management resources, if you find them worthwhile.