Anscombe’s quartet: all four sets are identical when examined statistically, but vary considerably when graphed. Image via Wikipedia.
___________________ Anscombe’s quartet comprises four datasets that have identical simple statistical properties, yet are revealed to be very different when inspected graphically. Each dataset consists of eleven (x,y) points. They were constructed in 1973 by the statistician F.J. Anscombe to demonstrate the importance of graphing data before analyzing it, and of the effect of outliers on the statistical properties of a dataset.
A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.
While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.
Managers need to eliminate de-motivation in the work systems not try and find bonus schemes to motivate behavior. Eliminating de-motivation is often much more work. You can’t just get some money from the bonus pool and start giving it away. You have to manage. But if you are a manager you shouldn’t be afraid to actually manage the system and make it better.
Revealed Preference: the preference consumers display by their action, in contrast to what they may say they prefer. While surveys may be useful people often say they will do one thing and actually when given the choice to do so, don’t.
Normally what matters is not what people say they want but what they actually will choose. For that reason revealed preference is a better measure than stated preference. Stated preference is often used as a proxy for actual preference (which may be fine) but it is important to understand that it is just a proxy for actual preference.
One aspect of managing people is to provide positive feedback and show appreciation. Doing so is important. People benefit from encouragement and reinforcement. In addition to just telling them, take action to show your appreciation.
The Dilbert workplace is alive and well. And even in above average management systems there is plenty of resistance faced by those looking to improve systems. For those employees that are making the attempt to improve the organization go beyond saying thanks: actually demonstrate your appreciation. Do what you can to help them achieve.
A manager should be enabling their employees to perform. That means taking positive steps that help them perform. This is even more appreciated than saying thanks. And has the added benefit of helping the organization by helping along their good idea. It is win, win, win. They win, you win and the organization wins.
Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.
* Graphical Analysis of Variance
* Computer Analysis of Complex Designs
* Simplification by transformation
* Hands-on experimentation using Response Service Methods
* Further development of robust product and process design using split plot arrangements and minimization of error transmission
* Introduction to Process Control, Forecasting and Time Series
The recipient of the 2008 William G. Hunter Award is Ronald Does. The Statistics Division of the American Society for Quality (ASQ) uses the attributes that characterize Bill Hunter’s (my father – John Hunter) career – consultant, educator for practitioners, communicator, and integrator of statistical thinking into other disciplines to decide the recipient. In his acceptance speech Ronald Does said:
The first advice I received from my new colleagues was to read the book by Box, Hunter and Hunter. The reason was clear. Because I was not familiar with industrial statistics I had to learn this from the authors who were really practicing statisticians. It took them years to write this landmark book.
For the past 15 years I have been the managing director of the Institute for Business and Industrial Statistics. This is a consultancy firm owned by the University of Amsterdam. The interaction between scientific research and the application of quality technology via our consultancy work is the core operating principle of the institute. This is reflected in the type of people that work for the institute, all of whom are young professionals having strong ambitions in both the academic world and in business and industry.
The kickoff conference attracted approximately 80 statisticians and statistical practitioners from all over Europe. ENBIS was officially founded in June 2001 as “an autonomous Society having as its objective the development and improvement of statistical methods, and their application, throughout Europe, all this in the widest sense of the words” Since the first meeting membership has grown to about 1300 from nearly all European countries.
Since full factorial gathers additional data, it reveals all possible interactions, but as seen by the numbers above, there is a trade-off. More data equals more information but more data also equals a longer test duration. The minimum data requirements for full factorial are very high since you are showing every experiment.
Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments. You might be wondering how fractional factorial can be accurate if interactions are possible?
Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.
Another difficulty in industrial experimentation is the existence of interactions. As has been stated, manufacturing processes are complex with many factors involved. In many processes these factors interact. This is particularly so for continuous processes such as plating or sputtering. Saying that the factors interact means more than that they are related to each other. It means that the effect of one (or more) factors on the response variable(s) changes when one (or more) other factor(s) changes its value.
In order to detect interactions and understand the nature of their effects it is necessary to combine the interacting factors into the same experimental runs. The problem is not necessarily knowing in advance if the interactions exist. Sometimes they are predictable with theory. Sometimes they are discovered when the process behaves ‘strangely’.
In addition to their efficiency, factorial designs also offer the only method of detecting interactions through experimentation. Because numerous factors can be combined in the same series of experimental runs, the interactions can be detected and the nature of their effects can be evaluated when they are present.
The paper also explains analytic and enumerative studies. Dr. Deming stressed the importance of understanding the distinction between the two.
In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2‐5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security.
Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an
overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes.
Interesting paper. I would guess most readers of this blog won’t be able to apply prediction markets to there workplace in the short term but never-the-less I find the paper interesting.
At New York Hospital, Eichenwald and infectious disease specialist Henry Shinefield conceived and developed a controversial program that entailed deliberately inoculating a newborn’s nostrils and umbilical stump with a comparatively harmless strain of staph before 80/81 could move in. Shinefield had found the protective strain – dubbed 502A – in the nostrils of a New York Hospital baby nurse. Like a benign Typhoid Mary, Nurse Lasky had been spreading her staph to many of the newborns in her care. Here babies remained remarkably healthy, while those under the care of other nurses were falling ill.
This is a great example of a positive special cause. How would you identify this? First you would have to stratify the data. It also shows that sometimes looking at the who is important (the problem is just that we far too often look at who instead of the system so at times some get the idea that it is not ok to stratify data based on who – it is just be careful because we often do that when it is not the right approach and we can get fooled by random variation into thinking there is a cause – see the red bead experiment for an example); that it is possible to stratify the data by person to good effect.
The following 20 pages in the book are littered with very interesting details many of which tie to thinking systemically and the perils of optimizing part of the system (both when considering the system to be one person and also when viewing it as society).
This is a pretty counter-intuitive statement, I believe:
You save more fuel switching from a 15 to 18 mpg car than switching from a 50 to 100 mpg car.
But some simple math shows it is true. If you drive 10,000 miles you would use: 667 gallons, 556 gallons, 200 gallons and 100 gallons. Amazing. I must admit, when I first read the quote I thought that it must be an wrong. But there is the math. You save 111 gallons improving from 15 mpg to 18 mpg and just 100 improving from 50 to 100 mpg. Other than those of you who automatically guess that whatever seems wrong must be the answer when you see a title like this I can’t believe anyone thinks 15 to 18 mpg is the change that has the bigger impact. It is great how a little understanding of math can help you see the errors in your initial beliefs. Via: 18 Is Enough.
It also illustrates that the way the data is presented makes a difference. You can also view 100 mpg as 1/100 gallon per mile, 2/100 gallons per mile, 5.6/100 gpm and 6.7 gpm. That way most everyone sees that the 6.7 to 5.6 gpm saves more fuel than 2 to 1 gpm does. Mathematics and scientific thinking are great – if you are willing to think you can learn to better understand the world we live in every day.
Yes, that’s more or less the point. If the system is behaving statistically, it will show apparent sequential trends that in reality are mirages. The dice experiment demonstrates that – and if you look at statistical and sequential temperature data, you see the exact same behavior!
When people are asked to explain random variations in data they will make up special causes (that they often even believe are special causes even when they are not) but you can improve management a great deal by just stopping the requirement to “explain” common cause variation (which in practices mean to claim a special cause for the common cause variation). Use that time instead to standardize processes. Create control charts for critical processes. Run experiments using PDSA cycle…
You may not realize that I first met Bill 38 year ago, when he was in Singapore helping us set up the first school of engineering in the country. He persuaded me to go to the graduate school at UW-Madison and I daresay that’s the best advice I ever got in my whole career. Now when I come to think of it, what Bill stood for in his lifetime has not been, and never will be, out of date. He had advocated the use of statistical thinking and the systems approach, which if anything is even more critical today in handling issues such as global warming and government effectiveness.
Also, statistical design of experiments has assumed an increasingly important role in performance improvement and optimization in the face of constrained resources, again something always in the minds of engineers, managers and business leaders. From time to time there are others who package statistical tools under labels Bill might not even have seen himself, such as “Design for Six Sigma“, but the underlying idea is still the same: recognize the existence of variation, and the earlier you anticipate it and do something about it, the better off you will be in the end.
Bill’s zeal in spreading the message and sharing his knowledge and expertise with people in other parts of the world is well known; I would even say that he had recognized that “the world is flat” way before the likes of Tom Friedman discovered the reality of globalization!
So that’s to share my thoughts with you, having being honored by the Bill Hunter award. I am copying this to Stu, also to Doug who chairs the committee for this award. I reality enjoy the professional association and friendship with you all.
I had not realized Dad was helping set up the first school of engineering in Singapore. This is the kind of thing I mentioned in, The Importance of Management Improvement, where I mention people telling me the positive impact Dad had on their lives.
Performance measurement systems are used to establish specific goals, align employee behavior, and increase accountability. Organizations often use these systems to set targets for component units (e.g., individuals, profit centers, divisions, plants). Each unit is expected to develop its own goals consistent with overall targets. This process, sometimes called a “roll-up,” reflects the premise that if all units achieve their targets then the overall goals will be met. The methods used by most companies to establish these numerical targets often involve the use of stretch targets or benchmarking best practices.
Once the targets are established, most organizations measure the performance of component units by comparing targets to actual performance for certain time periods. Variances from expected results are noted and explanations are required. The popular business press trumpets the efficacy of the above approach, but this methodology has serious flaws. In fact, the design and use of performance measurement systems in most organizations suffer from a number of fatal flaws that can undermine an organization’s ability to use its measurement system to improve processes and make better decisions.
The proper role of measurements should be seen in the context of helping employees connect with the overall aim of the organization. Management must gather and analyze information that will help employees become better contributors to the firm’s purpose.
The articles does an excellent job of explaining the flaws in how performance measurement is applied (both in Management by Objective (MBO) and performance appraisals).
Last month, in a long post criticizing the ACSI I took issue with, among other things, the implications being drawn from an ACSI rating. The ACSI rating of Yahoo was higher than that of Google (though statistically insignificantly so). Anyway, here is some new data on search volumes of the leading providers:
Top 5 Search Providers for August 2007, Ranked by Searches (U.S.)
So Google grew 39.8% year over year and Yahoo grew 8.9% year over year. Google now has 53.6% of the total searches. Granted this is limited data but it seems to confirm that Google is in fact continuing to increase their lead in search volume. Practically all evidence seems to support this belief – the ACSI seems to be the exception. Which might indicate great insight provided by the ACSI that everyone else is missing. Or it might show ACSI results are doing a poor job of providing a useful measure of customer satisfaction with search engines. I go with the second.
If organizations just adopt management improvement practices I firmly believe customer service, financial performance and employee satisfaction could be improved. This was a big part of the reason I started to use the internet to share management improvement ideas back in 1996 (plus I find management improvement interesting).
On the note of making a difference in people’s lives. I have had far more people tell me how my father (Bill Hunter) made a huge difference in their lives (far more than ever tell me anything like that). Now there is the sensible explanation, that he actually had a big impact on people’s lives (but you also have to figure most of those people never saw me so the chance for them to say anything didn’t exist…). I believe far more people told me (after he died) than ever told him, which says something about psychology in the USA, I think. But I don’t really know what people told him – so I could be wrong about that.
Anyway the point of this is that many people have told me their life was significantly changed by working with him on management improvement initiatives (mechanics talking about how he changed the workplace they had been in for years, people who saw that they could contribute more and changed careers, managers that realized how much damage they had done but now were on the right track…). There was obviously a great deal of emotion for many people. And it was largely about applying concepts like Deming’s management system, Toyota Management practices, statistics (yes even that)… and his ability to talk to everyone and make them comfortable (tons of people mentioned this – that this university professor would ask me questions and talk to me like a person, not talk down to me and be interested in my answers and…). As I continue through life I realize that this management improvement stuff really can matter if done right.
I have grown to enjoy maintaining the management improvement resources and other Curious Cat web sites but this is the reason I started and continued these efforts over the years. Today there is a great amount of useful management information online – but for years the pickings were quite slim.
Humans are very good at detecting patterns, but rather poor at detecting randomness. We expect random incidents of cancer to be spread homogeneously, when in fact true randomness results in random clusters, not homogeneity. It is a mistake for an experiment to consider a pool of 47,000 possibilities, and then only report on the 7 cases that seem interesting.
A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (“Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.
This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.