Tag Archives: experiments

Introductory Videos on Using Design of Experiments to Improve Results

The video shows Stu Hunter discussing design of experiments in 1966. It might be a bit slow going at first but the full set of videos really does give you a quick overview of the many important aspects of design of experiments including factorial designed experiments, fractional factorial design, blocking and response surface design. It really is quite good, if you find the start too slow for you skip down to the second video and watch it.

My guess is, for those unfamiliar with even the most cursory understanding of design of experiments, the discussion may start moving faster than you can absorb the information. One of the great things about video is you can just pause and give yourself a chance to catch up or repeat a part that you didn’t quite understand. You can also take a look at articles on design of experiments.

I believe design of experiments is an extremely powerful methodology of improvement that is greatly underutilized. Six sigma is the only management improvement program that emphasizes factorial designed experiments.

Related: One factor at a time (OFAT) Versus Factorial DesignsThe purpose of Factorial Designed Experiments

Continue reading

Richard Feynman Explains the PDSA Cycle

Ok, really Richard Feynman Explains the scientific method. But his thoughts make the similarity between the PDSA cycle and the scientific method obvious.

1) Plan, hypothesis.
You make a guess about a theory (in using the PDSA cycle this step is often missed, while in the scientific method this is of the highest priority). You make a prediction based on that theory.

2) Do the experiment

3) Study the results

If the results disprove the theory you were wrong. If they results don’t disprove the theory you may have a useful theory (it can also be that your theory is still wrong, but this experiment happened not to provide results that disprove it).

Step 4, Act, only exists for PDSA. In science the aim is to learn and confirm laws. While the PDSA cycle has an aim to learn and adopt methods that achieve the desired results.

Richard Feynman: “If it disagrees with experiment it is wrong, in that simple statement is the key to science, it doesn’t make any difference how beautiful your guess is, it doesn’t make a difference how smart you are (who made the guess), or what his name is, if it disagrees with experiment it is wrong.”

Actually far to often “PDSA” fails to adopt this understanding. Instead it become PA: no study of the results, just implement and we all already agree it is going to work so don’t bother wasting time testing that it actually does. Some organization do remember to study results of the pilot experiments but then forget to study the results when the new ideas are adopted on a broader scale.

Related: Does the Data Deluge Make the Scientific Method Obsolete?Video of Young Richard Feynman Talking About Scientific ThinkingHow to Use of the PDSA Improvement Cycle Most EffectivelyUsing Design of Experiments

Keys to the Effective Use of the PDSA Improvement Cycle

The PDSA improvement cycle was created by Walter Shewhart where Dr. Deming learned about it. An improvement process is now part of many management improvement methods (A3 for lean manufacturing, DMAIC for six sigma and many other modifications). They are fairly similar in many ways. The PDSA cycle (Plan, Do, Study, Act) has a few key pieces that are either absent in most others processes of greatly de-emphasized which is why I prefer it (A3 is my second favorite).

The PDSA cycle is a learning cycle based on experiments. When using the PDSA cycle, it is important to predict the results. This is important for several reasons but most notably due to an understanding of the theory of knowledge. We will learn much more if we write down our prediction. Otherwise we often just think (after the fact); yeah that is pretty much what I expected (even if it wasn’t). Also we often fail to think specifically enough at the start to even have a prediction. Forcing yourself to make a prediction gets you to think more carefully up front and can help you set better experiments.

An organization using PDSA well will turn the PDSA cycle several times on any topic and do so quickly. In a 3 month period turning it 5 times might be good. Often those organizations that struggle will only turn it once (if they are lucky and even reach the study stage). The biggest reason for effective PDSA cycles taking a bit longer is wanting more data than 2 weeks provides. Still it is better to turn it several times will less data – allowing yourself to learn and adjust than taking one long turn.

The plan stage may well take 80% (or even more) of the effort on the first turn of the PDSA cycle in a new series. The Do stage may well take 80% of of the time – it usually doesn’t take much effort (to just collect a bit of extra data) but it may take time for that data to be ready to collect. In the 2nd, 3rd… turns of the PDSA cycle the Plan stage often takes very little time. Basically you are just adjusting a bit from the first time and then moving forward to gather more data. Occasionally you may learn you missed some very important ideas up front; then the plan stage may again take some time (normally if you radically change your plans).

Remember to think of Do as doing-the-experiment. If you are “doing” a bunch of work (not running an experiment and collecting data) that probably isn’t “do” in the PDSA sense.

Study should not take much time. The plan should have already have laid out what data is important and an expectation of what results will be achieved and provide a good idea on next steps. Only if you are surprised (or in the not very common case that you really have no idea what should come next until you experiment) will the study phase take long.

Continue reading

Factorial Designed Experiment Aim

Multivariate experiments are a very powerful management tool to learn and improve performance. Experiments in general, and designed factorial experiments in particular, are dramatically underused by managers. A question on LinkedIn asks?

When doing a DOE we select factors with levels to induce purposely changes in the response variable. Do we want the response variable to move within the specs of the customers? Or it doesn’t matter since we are learning about the process?

The aim needs to consider what you are trying to learn, costs and potential rewards. Weighing the various factors will determine if you want to aim to keep results within specification or can try options that are likely to return results that are outside of specs.

If the effort was looking for breakthrough improvement and costs of running experiments that might produce results outside of spec were low then specs wouldn’t matter much. If the costs of running experiments are very high (compared with expectations of results) then you may well want to try designed experiment values that you anticipate will still produce results within specs.

There are various ways costs come into play. Here I am mainly looking at the costs as (costs – revenue). For example the case where if the results are withing spec and can be used the costs (net costs, including revenue) of the experiment run are substantially lower.
Continue reading

Learn by Seeking Knowledge – Not Just from Mistakes

Being open to new ideas and new knowledge is what is needed to learn. Experimenting, seeking out new knowledge is even better.

You can be successful and see an even better way to do things and learn from it. This seems the best way to learn to me – not to just learn from mistakes. Of course this means your goal has to be improvement not just avoiding more mistakes than before.

Your actions are based on theories (often unconsciously): and learning involves improving those theories. Learning requires updating faulty ideas (or learning new ideas – in which case ignorance rather than a faulty theory may have lead to the mistake). Encouraging people to learn from mistakes is useful when it is about freeing them to make errors and learn from them. But you should be learning all the time – not just when you make mistakes.

You can be also be wrong and not learn (lots of people seem to do this). This is by far the biggest state I see. It isn’t an absence of people making mistakes (including carrying out processes based on faulty theories) that is slowing learning. People are very reluctant to make errors of commission (and errors of commission due to a change is avoided even more). This reluctance obviously makes learning (and improvement) more difficult. And the reluctance is often enhanced by fear created by the management system.

It is best to be open and seek out new knowledge and learn that way as much as possible. Now, you should also not be scared to be wrong. Taking the right risks is important to improving – encouraging creativity and innovation and risk taking is wise.

Experiment and be open to learn from what could be better and improve (PDSA is a great way to try things and evaluate how they work). And the idea is not to be so conservative that every turn of the PDSA cycle has no failures. In order to get significant successes it is likely you will try things that don’t always work.

The desire to improve understanding (and the desire to improve results provides focus to the learning) is what is valuable in learning – not being wrong. Creating a culture where being wrong needs to be avoided harms learning because people avoid risk and seek to distance themselves from failure instead of experimenting and digging into the details when something goes wrong. Instead of learning from mistakes people try to stay as far away from them and hide them from others. That is not helpful. But what is needed is more desire to continually learn – learning from mistakes is wise but hardly the only way to learn.

Related: The Illusion of Knowledgeconfirmation biasManagement is Prediction

Problems With Student Evaluations as Measures of Teacher Performance

Dr. Deming was, among other things a professor. He found the evaluation of professors by students an unimportant (and often counterproductive measure) – used in some places for awards and performance appraisal. He said for such a measure to be useful it should survey students 20 years later to see which professors made a difference to the students. Here is an interesting paper that explored some of these ideas. Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors by Scott E. Carrell, University of California, Davis and National Bureau of Economic Research; and James E. West, U.S. Air Force Academy:

our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value”added but positively correlated with follow”on course value”added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow”on related curriculum.

Student evaluations are positively correlated with contemporaneous professor value”added and negatively correlated with follow”on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value”added in follow”on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy of this practice.

These findings have broad implications for how students should be assessed and teacher quality measured.

Related: Applying Lean Tools to University CoursesK-12 Educational ReformImproving Education with Deming’s IdeasLearning, Systems and ImprovementHow We Know What We Know

Extrinsic Incentives Kill Creativity

If you read this blog, you know I believe extrinsic motivation is a poor strategy. This TED webcast Dan Pink discusses studies showing extrinsic rewards failing. This is a great webcast, definitely worth 20 minutes of your time.

  • “you’ve got an incentive designed to sharpen thinking and accelerate creativity and it does just the opposite. It dulls thinking and blocks creativity… This has been replicated over and over and over again for nearly 40 years. These contingent motivators, if you do this then you get that, work in some circumstances but in a lot of tasks they actually either don’t work or, often, they do harm.”
  • there is a mismatch between what science knows and what business does
  • “This is a fact.”

What does Dan Pink recommend based on the research? Management should focus on providing workplaces where people have autonomy, mastery and purpose to build on intrinsic motivation.

via: Everything You Think about Pay for Performance Could Be Wrong [the broken link was removed]

Related: Righter IncentivizationWhat’s the Value of a Big Bonus?Dangers of Extrinsic MotivationMotivate or Eliminate De-MotivationGreat Marissa Mayer Webcast on Google Innovation

YouTube Uses Multivariate Experiment To Improve Sign-ups 15%

Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

via: @hexawiseMy brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).

Related: Combinatorial Testing for SoftwareStatistics for ExperimentersGoogle’s Website Optimizer allows for multivariate testing of your website.Using Design of Experiments

Statistics for Experimenters in Spanish

book cover of Estadística para Investigadores

Statistics for Experimenters, second edition, by George E. P. Box, J. Stuart Hunter and William G. Hunter (my father) is now available in Spanish.

Read a bit more can find a bit more on the Spanish edition, in Spanish. Estadística para Investigadores Diseño, innovación y descubrimiento Segunda edición.

Statistics for Experimenters – Second Edition:

Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.

* Graphical Analysis of Variance
* Computer Analysis of Complex Designs
* Simplification by transformation
* Hands-on experimentation using Response Service Methods
* Further development of robust product and process design using split plot arrangements and minimization of error transmission
* Introduction to Process Control, Forecasting and Time Series

Book available via Editorial Reverte

Related: Statistics for Experimenters ReviewCorrelation is Not CausationStatistics for Experimenters Dataposts on design of experiments

What’s the Value of a Big Bonus?

What’s the Value of a Big Bonus? by Dan Ariely

To look at this question, three colleagues and I conducted an experiment. We presented 87 participants with an array of tasks that demanded attention, memory, concentration and creativity. We asked them, for instance, to fit pieces of metal puzzle into a plastic frame, to play a memory game that required them to reproduce a string of numbers and to throw tennis balls at a target. We promised them payment if they performed the tasks exceptionally well. About a third of the subjects were told they’d be given a small bonus, another third were promised a medium-level bonus, and the last third could earn a high bonus.

So it turns out that social pressure has the same effect that money has. It motivates people, especially when the tasks at hand require only effort and no skill. But it can provide stress, too, and at some point that stress overwhelms the motivating influence.

When I recently presented these results to a group of banking executives, they assured me that their own work and that of their employees would not follow this pattern. (I pointed out that with the right research budget, and their participation, we could examine this assertion. They weren’t that interested.)

This is an interesting look at an effect of bonuses. We all know monetary bonuses can influence behavior. The problem is the type of behaviors that result. Huge bonuses, for example, create huge incentives to risk the future of the company for the chance at a huge bonus for the executive. Extrinsic motivation leads to many problems.

Problems with bonuses: Losses Covered Up to Protect Bonuses“Pay for Performance” is a Bad IdeaProblems with BonusesBook: Punished By Rewards: The Trouble With Gold Stars, Incentive Plans, A’s, Praise, and Other Bribes by Alfie Kohn – posts on executive pay