Currently browsing the Statistics Category

YouTube Uses Multivariate Experiment To Improve Sign-ups 15%

Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

via: @hexawiseMy brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).

Related: Combinatorial Testing for SoftwareStatistics for ExperimentersGoogle’s Website Optimizer allows for multivariate testing of your website.Using Design of Experiments

When Performance-related Pay Backfires

When Economic Incentives Backfire by Samuel Bowles, Sante Fe Institute

Dozens of recent experiments show that rewarding self-interest with Economic incentives can backfire when they undermine what Adam Smith called “the moral sentiments.”

Punished by Rewards, by Alfie Kohn, is a great book on this topic. The area of “motivating” employees is one it is often hard for managers to learn. Even managers that have been studying Deming, Ackoff, Ohno… for years still have trouble with the idea that trying to find the right incentive scheme to motivate the right behavior is the wrong approach. Read the The Human Side Of Enterprise by Douglas Mcgregor (in 1960) to re-enforce the understanding of human motivation provided by Toyota’s respect for people principles.

Managers need to eliminate de-motivation in the work systems not try and find bonus schemes to motivate behavior. Eliminating de-motivation is often much more work. You can’t just get some money from the bonus pool and start giving it away. You have to manage. But if you are a manager you shouldn’t be afraid to actually manage the system and make it better.

Related: “Pay for Performance” is a Bad IdeaReward and Incentive Programs are Ineffective — Even Harmful by Peter Scholtes – The Defect Black MarketWhat’s the Value of a Big Bonus?Problems with BonusesLosses Covered Up to Protect BonusesStop Demotivating Employees

When performance-related pay backfires:
Continue reading

Revealed Preference

Revealed Preference: the preference consumers display by their action, in contrast to what they may say they prefer. While surveys may be useful people often say they will do one thing and actually when given the choice to do so, don’t.

Normally what matters is not what people say they want but what they actually will choose. For that reason revealed preference is a better measure than stated preference. Stated preference is often used as a proxy for actual preference (which may be fine) but it is important to understand that it is just a proxy for actual preference.

See more explanations from the Curious Cat Management Dictionary.

Related: Packaging ImprovementAll Models Are Wrong But Some Are UsefulDangers of Forgetting the Proxy Nature of DataConfirmation BiasBe Careful What You Measure

How to Create a Control Chart for Seasonal or Trending Data

Lynda Finn, President of Statistical Insight, has written an article on how to create a control chart for seasonal or trending data (where there is an underlying structural variation in the data). Essentially you need to account for the structural variation to create the control limits for the control chart. She also provides a Minitab project file. Both are available for download from the Curious Cat Management Improvement Library.

Related: Control Charts in Health CareCommon Cause VariationManaging with Control ChartsMeasurement and Data CollectionFourth Generation Management

Friday Fun: Correlation

Correlation doesn't imply causation

From the excellent xkcd comic.

Related: Correlation is Not CausationDoes the Data Deluge Make the Scientific Method Obsolete?Understanding DataTheory of KnowledgeWhat Makes Scientists Different :-)Dangers of Forgetting the Proxy Nature of DataSeeing Patterns Where None Exists

Helping Employees Improve

One aspect of managing people is to provide positive feedback and show appreciation. Doing so is important. People benefit from encouragement and reinforcement. In addition to just telling them, take action to show your appreciation.

The Dilbert workplace is alive and well. And even in above average management systems there is plenty of resistance faced by those looking to improve systems. For those employees that are making the attempt to improve the organization go beyond saying thanks: actually demonstrate your appreciation. Do what you can to help them achieve.

A manager should be enabling their employees to perform. That means taking positive steps that help them perform. This is even more appreciated than saying thanks. And has the added benefit of helping the organization by helping along their good idea. It is win, win, win. They win, you win and the organization wins.

Thoughts on: Rewards and Recognition

Related: Keeping Good EmployeesRespect for People Requires Understanding Psychology- People are Our Most Important AssetMotivationIncentive Programs are Ineffective

Statistics for Experimenters in Spanish

book cover of Estadística para Investigadores

Statistics for Experimenters, second edition, by George E. P. Box, J. Stuart Hunter and William G. Hunter (my father) is now available in Spanish.

Read a bit more can find a bit more on the Spanish edition, in Spanish. Estadística para Investigadores Diseño, innovación y descubrimiento Segunda edición.

Statistics for Experimenters – Second Edition:

Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.

* Graphical Analysis of Variance
* Computer Analysis of Complex Designs
* Simplification by transformation
* Hands-on experimentation using Response Service Methods
* Further development of robust product and process design using split plot arrangements and minimization of error transmission
* Introduction to Process Control, Forecasting and Time Series

Book available via Editorial Reverte

Related: Statistics for Experimenters ReviewCorrelation is Not CausationStatistics for Experimenters Dataposts on design of experiments

ASQ William Hunter Award 2008: Ronald Does

The recipient of the 2008 William G. Hunter Award is Ronald Does. The Statistics Division of the American Society for Quality (ASQ) uses the attributes that characterize Bill Hunter’s (my father – John Hunter) career – consultant, educator for practitioners, communicator, and integrator of statistical thinking into other disciplines to decide the recipient. In his acceptance speech Ronald Does said:

The first advice I received from my new colleagues was to read the book by Box, Hunter and Hunter. The reason was clear. Because I was not familiar with industrial statistics I had to learn this from the authors who were really practicing statisticians. It took them years to write this landmark book.

For the past 15 years I have been the managing director of the Institute for Business and Industrial Statistics. This is a consultancy firm owned by the University of Amsterdam. The interaction between scientific research and the application of quality technology via our consultancy work is the core operating principle of the institute. This is reflected in the type of people that work for the institute, all of whom are young professionals having strong ambitions in both the academic world and in business and industry.

The kickoff conference attracted approximately 80 statisticians and statistical practitioners from all over Europe. ENBIS was officially founded in June 2001 as “an autonomous Society having as its objective the development and improvement of statistical methods, and their application, throughout Europe, all this in the widest sense of the words” Since the first meeting membership has grown to about 1300 from nearly all European countries.

Related: 2007 William G. Hunter AwardThe Importance of Management ImprovementResources on using statistical thinking to improve management

Full and Fractional Factorial Test Design

An Essential Primer on Full and Fractional Factorial Test Design

Since full factorial gathers additional data, it reveals all possible interactions, but as seen by the numbers above, there is a trade-off. More data equals more information but more data also equals a longer test duration. The minimum data requirements for full factorial are very high since you are showing every experiment.

Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments. You might be wondering how fractional factorial can be accurate if interactions are possible?

Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.

I am a fan of design of experiments as long time readers know (see posts on design of experiments).

Some good resources for more on the topics discussed above: What Can You Find Out From 8 and 16 Experimental Runs? by George Box – Statistics for ExperimentersDesign of Experiments in Advertising.

Related: Google Website Optimizerfactorial experiment articlesUsing Design of ExperimentsMarketers Are Embracing Statistical Design of Experiments

Improvement Through Designed Experiments

The Rationale of Scientific Experimentation by John Dowd explains the value of designed experiments.

Another difficulty in industrial experimentation is the existence of interactions. As has been stated, manufacturing processes are complex with many factors involved. In many processes these factors interact. This is particularly so for continuous processes such as plating or sputtering. Saying that the factors interact means more than that they are related to each other. It means that the effect of one (or more) factors on the response variable(s) changes when one (or more) other factor(s) changes its value.

In order to detect interactions and understand the nature of their effects it is necessary to combine the interacting factors into the same experimental runs. The problem is not necessarily knowing in advance if the interactions exist. Sometimes they are predictable with theory. Sometimes they are discovered when the process behaves ‘strangely’.

In addition to their efficiency, factorial designs also offer the only method of detecting interactions through experimentation. Because numerous factors can be combined in the same series of experimental runs, the interactions can be detected and the nature of their effects can be evaluated when they are present.

The paper also explains analytic and enumerative studies. Dr. Deming stressed the importance of understanding the distinction between the two.

Related: management improvement articlesDesign of Experiments articlesStatistics for Experimenterssearch statistical management improvement sitesUsing Design of Experiments

Prediction Markets with Google Employees

Another interesting experiment from Google: Using Prediction Markets to Track Information Flows: Evidence from Google

In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2‐5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security.

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an
overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes.

Interesting paper. I would guess most readers of this blog won’t be able to apply prediction markets to there workplace in the short term but never-the-less I find the paper interesting.

Related: Management is PredictionGoogle Experiments Quickly and OftenSecrets of the World’s Best Companies

Stratification and Systemic Thinking

I am reading a fascinating book by Jessica Snyder Sachs: Good Germs, Bad Germs. From page 108:

At New York Hospital, Eichenwald and infectious disease specialist Henry Shinefield conceived and developed a controversial program that entailed deliberately inoculating a newborn’s nostrils and umbilical stump with a comparatively harmless strain of staph before 80/81 could move in. Shinefield had found the protective strain – dubbed 502A – in the nostrils of a New York Hospital baby nurse. Like a benign Typhoid Mary, Nurse Lasky had been spreading her staph to many of the newborns in her care. Here babies remained remarkably healthy, while those under the care of other nurses were falling ill.

This is a great example of a positive special cause. How would you identify this? First you would have to stratify the data. It also shows that sometimes looking at the who is important (the problem is just that we far too often look at who instead of the system so at times some get the idea that it is not ok to stratify data based on who – it is just be careful because we often do that when it is not the right approach and we can get fooled by random variation into thinking there is a cause – see the red bead experiment for an example); that it is possible to stratify the data by person to good effect.

The following 20 pages in the book are littered with very interesting details many of which tie to thinking systemically and the perils of optimizing part of the system (both when considering the system to be one person and also when viewing it as society).

I have recently taken to reading more and more about viruses, bacteria, cells, microbiology etc.: it is fascinating stuff.

Related: Science Books by topicData Can’t LieUnderstanding Data

Bigger Impact: 15 to 18 mpg or 50 to 100 mpg?

This is a pretty counter-intuitive statement, I believe:

You save more fuel switching from a 15 to 18 mpg car than switching from a 50 to 100 mpg car.

But some simple math shows it is true. If you drive 10,000 miles you would use: 667 gallons, 556 gallons, 200 gallons and 100 gallons. Amazing. I must admit, when I first read the quote I thought that it must be an wrong. But there is the math. You save 111 gallons improving from 15 mpg to 18 mpg and just 100 improving from 50 to 100 mpg. Other than those of you who automatically guess that whatever seems wrong must be the answer when you see a title like this I can’t believe anyone thinks 15 to 18 mpg is the change that has the bigger impact. It is great how a little understanding of math can help you see the errors in your initial beliefs. Via: 18 Is Enough.

It also illustrates that the way the data is presented makes a difference. You can also view 100 mpg as 1/100 gallon per mile, 2/100 gallons per mile, 5.6/100 gpm and 6.7 gpm. That way most everyone sees that the 6.7 to 5.6 gpm saves more fuel than 2 to 1 gpm does. Mathematics and scientific thinking are great – if you are willing to think you can learn to better understand the world we live in every day.

Related: Statistics Don’t Lie, But People Can be FooledUnderstanding DataSeeing Patterns Where None ExistsOptical Illusions and Other Illusions1=2: A Proof

Fooled by Randomness

This is a nice article discussing how people are often fooled by thinking there must be special causes for patterns in random data. I still remember my father showing my classes these lessons when I was in grade school. Playing At Dice – What That “Weekend Exercise” Was All About:

Yes, that’s more or less the point. If the system is behaving statistically, it will show apparent sequential trends that in reality are mirages. The dice experiment demonstrates that – and if you look at statistical and sequential temperature data, you see the exact same behavior!

When people are asked to explain random variations in data they will make up special causes (that they often even believe are special causes even when they are not) but you can improve management a great deal by just stopping the requirement to “explain” common cause variation (which in practices mean to claim a special cause for the common cause variation). Use that time instead to standardize processes. Create control charts for critical processes. Run experiments using PDSA cycle

Related: Seeing Patterns Where None ExistsUnderstanding DataOperational Definitions and Data CollectionRed Bead Experiment

2007 William G. Hunter Award

T.N. Goh received ASQ Statistics Division’s 2007 William G. Hunter Award. He sent me this email:

You may not realize that I first met Bill 38 year ago, when he was in Singapore helping us set up the first school of engineering in the country. He persuaded me to go to the graduate school at UW-Madison and I daresay that’s the best advice I ever got in my whole career. Now when I come to think of it, what Bill stood for in his lifetime has not been, and never will be, out of date. He had advocated the use of statistical thinking and the systems approach, which if anything is even more critical today in handling issues such as global warming and government effectiveness.

Also, statistical design of experiments has assumed an increasingly important role in performance improvement and optimization in the face of constrained resources, again something always in the minds of engineers, managers and business leaders. From time to time there are others who package statistical tools under labels Bill might not even have seen himself, such as “Design for Six Sigma“, but the underlying idea is still the same: recognize the existence of variation, and the earlier you anticipate it and do something about it, the better off you will be in the end.

Bill’s zeal in spreading the message and sharing his knowledge and expertise with people in other parts of the world is well known; I would even say that he had recognized that “the world is flat” way before the likes of Tom Friedman discovered the reality of globalization!

So that’s to share my thoughts with you, having being honored by the Bill Hunter award. I am copying this to Stu, also to Doug who chairs the committee for this award. I reality enjoy the professional association and friendship with you all.

I had not realized Dad was helping set up the first school of engineering in Singapore. This is the kind of thing I mentioned in, The Importance of Management Improvement, where I mention people telling me the positive impact Dad had on their lives.

Related: Curious Cat Science and Engineering BlogStatistics for ExperimentersSingapore Research Fellowship

Seven Fatal Flaws of Performance Measurement

The Seven Fatal Flaws of Performance Measurement by Joseph F. Castellano, Saul Young, and Harper A. Roehm

Performance measurement systems are used to establish specific goals, align employee behavior, and increase accountability. Organizations often use these systems to set targets for component units (e.g., individuals, profit centers, divisions, plants). Each unit is expected to develop its own goals consistent with overall targets. This process, sometimes called a “roll-up,” reflects the premise that if all units achieve their targets then the overall goals will be met. The methods used by most companies to establish these numerical targets often involve the use of stretch targets or benchmarking best practices.

Once the targets are established, most organizations measure the performance of component units by comparing targets to actual performance for certain time periods. Variances from expected results are noted and explanations are required. The popular business press trumpets the efficacy of the above approach, but this methodology has serious flaws. In fact, the design and use of performance measurement systems in most organizations suffer from a number of fatal flaws that can undermine an organization’s ability to use its measurement system to improve processes and make better decisions.

The proper role of measurements should be seen in the context of helping employees connect with the overall aim of the organization. Management must gather and analyze information that will help employees become better contributors to the firm’s purpose.

The articles does an excellent job of explaining the flaws in how performance measurement is applied (both in Management by Objective (MBO) and performance appraisals).

Related: Performance Without AppraisalJeffrey Pfeffer on Evidence-Based PracticesProblems Caused by Performance AppraisalThe Danger of Forgetting the Proxy Nature of Data

Search Share Data – Checking the ACSI

Last month, in a long post criticizing the ACSI I took issue with, among other things, the implications being drawn from an ACSI rating. The ACSI rating of Yahoo was higher than that of Google (though statistically insignificantly so). Anyway, here is some new data on search volumes of the leading providers:

Top 5 Search Providers for August 2007, Ranked by Searches (U.S.)

Provider Searches (000) Year over Year Growth Share of Searches
1. Google Search 4,199,495 39.8% 53.6%
2. Yahoo! Search 1,561,903 8.9% 19.9%
3. MSN/Windows Live Search 1,011,398 69.8% 12.9%
4. AOL Search 435,088 32.4% 5.6%
5. Ask.com Search 136,853 0.0% 1.7%

Source: Nielsen//NetRatings MegaView Search

So Google grew 39.8% year over year and Yahoo grew 8.9% year over year. Google now has 53.6% of the total searches. Granted this is limited data but it seems to confirm that Google is in fact continuing to increase their lead in search volume. Practically all evidence seems to support this belief – the ACSI seems to be the exception. Which might indicate great insight provided by the ACSI that everyone else is missing. Or it might show ACSI results are doing a poor job of providing a useful measure of customer satisfaction with search engines. I go with the second.

Related: posts on using data effectivelyWebsite DataUnderstanding Dataposts on Google managementCurious Cat Management Improvement Search

The Importance of Management Improvement

John and Bill Hunter

If organizations just adopt management improvement practices I firmly believe customer service, financial performance and employee satisfaction could be improved. This was a big part of the reason I started to use the internet to share management improvement ideas back in 1996 (plus I find management improvement interesting).

On the note of making a difference in people’s lives. I have had far more people tell me how my father (Bill Hunter) made a huge difference in their lives (far more than ever tell me anything like that). Now there is the sensible explanation, that he actually had a big impact on people’s lives (but you also have to figure most of those people never saw me so the chance for them to say anything didn’t exist…). I believe far more people told me (after he died) than ever told him, which says something about psychology in the USA, I think. But I don’t really know what people told him – so I could be wrong about that.

Anyway the point of this is that many people have told me their life was significantly changed by working with him on management improvement initiatives (mechanics talking about how he changed the workplace they had been in for years, people who saw that they could contribute more and changed careers, managers that realized how much damage they had done but now were on the right track…). There was obviously a great deal of emotion for many people. And it was largely about applying concepts like Deming’s management system, Toyota Management practices, statistics (yes even that)… and his ability to talk to everyone and make them comfortable (tons of people mentioned this – that this university professor would ask me questions and talk to me like a person, not talk down to me and be interested in my answers and…). As I continue through life I realize that this management improvement stuff really can matter if done right.

I have grown to enjoy maintaining the management improvement resources and other Curious Cat web sites but this is the reason I started and continued these efforts over the years. Today there is a great amount of useful management information online – but for years the pickings were quite slim.

Photo is of Dad and me a few years ago. Related: Quality in the Community: Madison, WIStatistics for ExperimentersDoing More With Less in the Public Sector: A Progress Report from Madison, WisconsinManaging Our Way to Economic Success: Two Untapped ResourcesInvest in new management methods not a failing company

Mistakes in Experimental Design and Interpretation

Mistakes in Experimental Design and Interpretation

Humans are very good at detecting patterns, but rather poor at detecting randomness. We expect random incidents of cancer to be spread homogeneously, when in fact true randomness results in random clusters, not homogeneity. It is a mistake for an experiment to consider a pool of 47,000 possibilities, and then only report on the 7 cases that seem interesting.

A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (“Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.

This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.

Related: Seeing Patterns Where None ExistsIllusions, Optical and OtherUnderstanding DataDangers of Forgetting the Proxy Nature of DataHow to Deal with False Research Findingsdescriptive “theory” and normative theory

Jeffrey Pfeffer on Evidence-Based Practices

Jeffrey Pfeffer Testifies to Congress About Evidence-Based Practices:

In this short statement, I want to make five points as succinctly as possible, providing references for background and documentation for my arguments. First, organizations in both the public and private sector ought to base policies not on casual benchmarking, on ideology or belief, on what they have done in the past or what they are comfortable with doing, but instead should implement evidence-based management. Second, the mere prevalence or persistence of some management practice is not evidence that it works — there are numerous examples of widely diffused and quite persistent management practices, strongly advocated by practicing executives and consultants, where the systematic empirical evidence for their ineffectiveness is just overwhelming. Third, the idea that individual pay for performance will enhance organizational operations rests on a set of assumptions. Once those assumptions are spelled out and confronted with the evidence, it is clear that many — maybe all — do not hold in most organizations. Fourth, the evidence for the effectiveness of individual pay for performance is mixed, at best — not because pay systems don’t motivate behavior, but more frequently, because such systems effectively motivate the wrong behavior. And finally, the best way to encourage performance is to build a high performance culture. We know the components of such a system, and we ought to pay attention to this research and implement its findings.

Great stuff. Read the entire document. via: Bob Sutton’s Work Matters

Related: Evidence-based ManagementIllusions – Optical and Other

Books: The Knowing-Doing Gap by Jeffrey Pfeffer and Robert Sutton – Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management by Jeffrey Pfeffer and Robert Sutton

  • Recent Trackbacks

  • Comments