|
|
|
|
|
|
Posts about design of experiments (DoE) - six sigma, (factorial designs, multivariate experiments) as used by statisticians since RA Fisher and now with six sigma.
We recommend Statistics for Experimenters. Articles on DoE: Teaching Engineers Experimental Design With a Paper Helicopter, Development of the Theory of Experimental Design by Sir R.A Fisher
The JMP blog has posted some highlights from George Box’s presentation at Discovery 2009
Having worked and studied with many other famous statisticians and analytic thinkers, Box did not hesitate to share his characterizations of them. He told a story about Dr. Bill Hunter and how he required his students to run an experiment. Apparently a variety of subjects was studied [see 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments]
…
According to Box, the difficulty of getting DOE to take root lies in the fact that these mathematicians “can’t really get the fact that it’s not about proving a theorem, it’s about being curious about things. There aren’t enough people who will apply [DOE] as a way of finding things out. But maybe with JMP, things will change that way.”
George Box is a great mind and great person who I have had the privilege of knowing my whole life. My father took his class at Princeton, then followed George to the University of Wisconsin-Madison (where Dr. Box founded the statistics department and Dad received the first PhD). They worked together building the UW statistics department, writing Statistics for Experimenters and founding the Center for Quality and Productivity Improvement among many other things.
Statistics for Experimenters: Design, Innovation, and Discovery shows that the goal of design of experiments is to learn and refine your experiment based on the knowledge you gain and experiment again. It is a process of discovery. If done properly it is very similar to the PDSA cycle with the application of statistical tools to aid in determining the impact of various factors under study.
Related: Box on Quality – George Box Quotations – posts on design of experiments – Using Design of Experiments
Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment
via: @hexawise – My brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).
Related: Combinatorial Testing for Software – Statistics for Experimenters – Google’s Website Optimizer allows for multivariate testing of your website. – Using Design of Experiments
Combinatorial testing of software is very similar to the design of experiments work my father was involved in, and which I have a special interest in. Combinatorial testing looks at binary interaction effects (success or failure), since it is seeking to find bugs in software, while design of experiments captures the magnitude of interaction effects on performance. In the last several years my brother, Justin Hunter, has been working on using combinatorial testing to improve software development practices. He visited me this week and we discussed the potential value of increasing the adoption of combinatorial testing, which is similar to the value of increasing the adoption of the use of design of experiments: both offer great opportunities for large improvements in current practices.
Automated Combinatorial Testing for Software
Practical Combinatorial Testing: Beyond Pairwise by Rick Kuhn, US National Institute of Standards and Technology; Yu Lei, University of Texas, Arlington; and Raghu Kacker, US National Institute of Standards and Technology.
Related: Future Directions for Agile Management – The Defect Black Market – Metrics and Software Development – Full and Fractional Factorial Test Design – Google Website Optimizer
Statistics for Experimenters, second edition, by George E. P. Box, J. Stuart Hunter and William G. Hunter (my father) is now available in Spanish.
Read a bit more can find a bit more on the Spanish edition, in Spanish. EstadÃstica para Investigadores Diseño, innovación y descubrimiento Segunda edición.
Statistics for Experimenters – Second Edition:
Book available via Editorial Reverte
Related: Statistics for Experimenters Review – Correlation is Not Causation – Statistics for Experimenters Data – posts on design of experiments
The recipient of the 2008 William G. Hunter Award is Ronald Does. The Statistics Division of the American Society for Quality (ASQ) uses the attributes that characterize Bill Hunter’s (my father – John Hunter) career – consultant, educator for practitioners, communicator, and integrator of statistical thinking into other disciplines to decide the recipient. In his acceptance speech Ronald Does said:
Related: 2007 William G. Hunter Award – The Importance of Management Improvement – Resources on using statistical thinking to improve management
An Essential Primer on Full and Fractional Factorial Test Design
Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments. You might be wondering how fractional factorial can be accurate if interactions are possible?
Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.
I am a fan of design of experiments as long time readers know (see posts on design of experiments).
Some good resources for more on the topics discussed above: What Can You Find Out From 8 and 16 Experimental Runs? by George Box – Statistics for Experimenters – Design of Experiments in Advertising.
Related: Google Website Optimizer – factorial experiment articles – Using Design of Experiments – Marketers Are Embracing Statistical Design of Experiments
MEMS development in less than half the time by Christopher N. Delametter, Eastman Kodak Company
Related: Design of Experiments articles – Using Design of Experiments – Statistics for Experimenters – Why Use Designed Factorial Experiments? – Kodak Debuts Printers With Inexpensive Cartridges
The Rationale of Scientific Experimentation by John Dowd explains the value of designed experiments.
In addition to their efficiency, factorial designs also offer the only method of detecting interactions through experimentation. Because numerous factors can be combined in the same series of experimental runs, the interactions can be detected and the nature of their effects can be evaluated when they are present.
The paper also explains analytic and enumerative studies. Dr. Deming stressed the importance of understanding the distinction between the two.
Related: management improvement articles – Design of Experiments articles – Statistics for Experimenters – search statistical management improvement sites – Using Design of Experiments
Edwards looks toward future of testing
In a recent Benefield Anechoic Facility test, the 412th Electronic Warfare Group used Design of Experiments methodology to cut a two-month program to three weeks. This schedule reduction translated directly into savings and helped reduce the concept-to-fielding cycle time while still ensuring the system was thoroughly tested. While building these capabilities is critical, the most critical piece of the puzzle is our people. We must continue to develop engineers, pilots, navigators, program managers and maintainers to test these systems and “find stuff so the warfighter doesn’t.”
It is hard to tell if they really are using Design of Experiments or just using the term but it seem possible they are really using it. As I have said a number of times it is a powerful and under-utilized tool for improvement. Related: Using Design of Experiments – design of experiments articles – posts on public sector management – Why Use Designed Factorial Experiments?
T.N. Goh received ASQ Statistics Division’s 2007 William G. Hunter Award. He sent me this email:
Also, statistical design of experiments has assumed an increasingly important role in performance improvement and optimization in the face of constrained resources, again something always in the minds of engineers, managers and business leaders. From time to time there are others who package statistical tools under labels Bill might not even have seen himself, such as “Design for Six Sigma“, but the underlying idea is still the same: recognize the existence of variation, and the earlier you anticipate it and do something about it, the better off you will be in the end.
Bill’s zeal in spreading the message and sharing his knowledge and expertise with people in other parts of the world is well known; I would even say that he had recognized that “the world is flat” way before the likes of Tom Friedman discovered the reality of globalization!
So that’s to share my thoughts with you, having being honored by the Bill Hunter award. I am copying this to Stu, also to Doug who chairs the committee for this award. I reality enjoy the professional association and friendship with you all.
I had not realized Dad was helping set up the first school of engineering in Singapore. This is the kind of thing I mentioned in, The Importance of Management Improvement, where I mention people telling me the positive impact Dad had on their lives.
Related: Curious Cat Science and Engineering Blog – Statistics for Experimenters – Singapore Research Fellowship
Six Sigma makes inroads in software development organizations
Still, Siviy said, “At a lot of software conferences now you see a sprinkling of presentations that somehow touch on Six Sigma or Lean, and the quality and depth of questions have evolved tremendously. In general, and not just in Six Sigma, as the [software] industry matures you see a wave of interest in measurement and analytical techniques.”
McKesson is a prime example. “Measurement is key,” Childers said. “What you can’t or don’t measure, you don’t know.”
A couple points. First, you can know what you don’t measure. Do you know your parents? Do you measure them? Manage what you can’t measure.
The Software Engineering Institute at Carnegie Mellon University has great materials. There is a danger in using those materials to become overly bureaucratic but the material was developed out of an excellent understanding of quality management (way back when that was the way this stuff was referred to). David Anderson provides some good insights, see: Stretching Agile to fit CMMI Level 3
Design of experiments is very suited to testing software: Planning Efficient Software Tests – Design Of Experiment For Software Testing.
six sigma does seem to foster a lack of sharing; which is a shame.
Related: six sigma articles and links – six sigma posts – software development posts – design of experiments articles
Mistakes in Experimental Design and Interpretation
This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.
Related: Seeing Patterns Where None Exists – Illusions, Optical and Other – Understanding Data – Dangers of Forgetting the Proxy Nature of Data – How to Deal with False Research Findings – descriptive “theory” and normative theory
Google’s Website Optimizer allows for multivariate testing of your website.
Rather than sitting in a room and arguing over what will work better, you can save time and eliminate the guesswork by simply letting your visitors tell you what works best. We’ll guide you through the process of designing and implementing your first experiment. Start optimizing your most important web pages and see detailed reports within hours.
Google provides an online slide show with audio (a good example of one way to share online information sharing in my opinion). This tool seems to have limited experimental options to what is on the page (it does not appear, for example, that one variable could be current customer v. new visitor…). Still it looks like an very easy way to do some simple multi-factorial experiments. Google offers a list of partners for those interested in consulting and more advanced features (and for those experts reading this you can apply to be a partner).
Related: Design of experiments posts – articles on multi-factorial experimentation – Google: Experiment Quickly and Often – Data Based Decision Making at Google
(more…)
Experimenting with milkshakes?
Experiments are useful and underused. PDSA and design of experiments are two concepts that aid in experimenting successfully.
Related: Google: Experiment Quickly and Often – Why Use Designed Factorial Experiments? – Using Design of Experiments – theory of knowledge
“Scientific thinking” the modern way by Bill Harris:
I thank Deb Schenk, then (and perhaps now) statistician at Hewlett-Packard Company, for teaching me and others about the design of experiments using Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building back in 1981-82.
I admit to a bit of bias, in seeing my father’s book (Statistics for Experimenters 2nd edition was published last year by the way), referenced but Bill Harris is exactly right in the power of design of experiments. The most recent post discusses Ackoff’s excellent f-Laws and a previous post discusses Deming (titled, It’s the process) so I couldn’t resist adding a post myself.
Related: design of experiments posts – Ackoff’s New Book: Management f-Laws
One-Factor-at-a-Time Versus Designed Experiments by Veronica Czitrom:
I still remember, as a child, asking what my father was going to be teaching the company he was going to consult with for a few days. He said he was going to teach them about using designed factorial experiments. I said, but you explained that to me and I am just a kid, how can you be teaching adults that? Didn’t they learn it in school? The article is a good introduction to the idea of why one factor at a time experiments are an ineffective way to learn.
Related: Design of Experiments articles – Statistics for Experimenters (2nd Edition) – Design of Experiments blog posts
Using Design of Experiments as a Process Road Map by Davis Balestracci:
DOE is a tool that is very useful. And while the situations in which DOE is the best tool to use is limited the limited use of DOE is used less than it could be. See more articles on the use of design of Experiments (DOE).
How to Institute DOE in Your Company by Davis Balestracci:
Design of Experiments (DoE) was developed by R.A. Fisher in the 1920s (related terms: factorial design, multivariate expertness). Six Sigma was the first general management approach that specifically highlighted the use of Designed Experiments for improvement. Still the use of factorial designed experiments is much less than it could be.
(more…)
Dr. George Box is not as well known in the general management community as his ideas merit (in my biased opinion – photo of Bill Hunter and George Box). He is well know in the statistics field as one of the leading statistical minds. Box on Quality is an excellent book that gathers his essays from his 65th to 80th year. The book has just been issued in paperback (which helps as the hardback was pricey).
While some of the essays are aimed at a reader with an advanced understanding of statistics, many of the articles are aimed at any manager attempting to apply Quality Management principles (SPC, Deming, process improvement, six sigma, etc.). An except from the book provides a table of contents and an introduction.
Some of the articles from the book are available online. I encourage you to take a look at several of the articles and then go ahead and add this book to your prized management resources, if you find them worthwhile.
How Two Guys From the Gold Country Are Changing Advertising Forever by Robert X. Cringely
Their secret is the Taguchi Method, which is a technique for designing experiments that converge on an ideal product solution.
…
“I taught over 300 courses for industry where we designed cars and electronic devices, but it wasn’t until one day I took over my wife’s kitchen and used Taguchi to perfect my recipe for vanilla wafer cookies that I realized how broadly it could be applied,” Kowalick recalls. “It took 16 batches, but by the end of the afternoon I had those wafers dialed in.”
It is great to see the application of Designed Experiments increasing. I am reminded of an article by my father, William G. Hunter, from 1975: 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments. Examples of the topics of the designed experiments his students performed:
(more…)
Curious Cat Management Improvement Blog © curiouscat.com 2005-2006 powered by WordPress - professor William G. Hunter homepage
