Tag Archives: Statistics

How to Manage What You Can’t Measure

In Out of the Crisis, page 121, Dr. Deming wrote:

the most important figures that one needs for management are unknown or unknowable (Lloyd S. Nelson, director of statistical methods for the Nashua corporation), but successful management must nevertheless take account of them.

So what do you do then? I am a strong advocate of Deming’s ideas on management. I see understanding system thinking, psychology, the theory of knowledge and variation as the tools to use when you can’t get precise measures (or when you can).

Even if you can’t measure exactly what you want, you can learn about the area with related data. You are not able to measure the exact benefit of a happy customer but you can get measures that give you evidence of the value and even magnitude. And you can get measures of the costs of dis-satisfied customers. I just mention this to be clear getting data is very useful and most organizations need to focus on gathering sensible data and using it well.

Without precise measures though you must use judgment. Judgment will often be better with an understanding of theory and repeated attempts to test those theories and learn. Understanding variation can be used even if you don’t have control charts and data. Over-reaction to special causes is very common. Even without data, this idea can be used to guide your thoughts.

The danger is that we mistake measures for the thing itself. Measures are a proxy and we need to understand the limitation of the data we use. The main point Deming was making was we can’t just pretend the data we have tells us everything we need to know. We need to think. We need to understand that the data is useful but the limitations need to be remembered.

Human systems involve people. To manage human systems you need to learn about psychology. Paying attention to what research can show about motivation, fear, trust, etc. is important and valuable. It aids management decisions when you can’t get the exact data that you would like. If people are unhappy you can see it. You may also be able to measure aspects of this (increased sick leave, increased turnover…). If people are unhappy they often will not be as pleasant to interact with as people who are happy. You can make judgments about the problems created by internal systems that rob people of joy in work and prevent them from helping customers.

For me the key is to use the Deming’s management system to guide action when you can’t get clear data. We should keep trying to find measures that will help. In my experience even though there are many instances where we can get definite data on exactly what we want we fail to get data that would help guide actions a great deal). Then we need to understand the limitations of the data we can gather. And then we need to continually improve and continually learn.

When you have clear data, Deming’s ideas are also valuable. But when the data is lacking it is even more important to take a systemic approach to making management decisions. Falling back into using the numbers you can get to drive decision making is a recipe for trouble.

Related: Manage what you can’t measureStatistical Engineering Links Statistical Thinking, Methods and Toolsoutcome measures

Actionable Metrics

Metrics are valuable when they are actionable. Think about what will be done if certain results are shown by the data. If you can’t think of actions you would take, it may be that metric is not worth tracking.

Metrics should be operationally defined so that the data is collected properly. Without operationally definitions data collected by more than one person will often include measurement error (in this case, the resulting data showing the results of different people measuring different things but calling the result the same thing).

And without operational definitions those using the resulting data may well mis-interpret what it is saying. Often data is presented without an operational definition and people think the data is saying something that it is not. I find most often when people say statistics lie it is really that they made an incorrect assumption about what the data said – which most often was because they didn’t understand the operational definition of the data. Data can’t lie. People can. And people can intentionally mislead with data. But far more often people unintentionally mislead with data that is misunderstood (often this is due to failure to operationally define the data).

In response to: Metrics Manifesto: Raising the Standard for Metrics

Related: Outcome MeasuresEvidence-based ManagementMetrics and Software DevelopmentDistorting the System (due to misunderstanding metrics)Manage what you can’t measure

Statistical Engineering Links Statistical Thinking, Methods and Tools

In Closing the Gap [broken link was removed, it is really a shame how many sites have no long term vision or capability] Roger W. Hoerl and Ronald D. Snee lay out a sensible case for focusing on statistical engineering.

We’re not suggesting that society no longer needs research in new statistical techniques for improvement; it does. The balance needed at this time, however, is perhaps 80% for statistics as an engineering discipline and 20% for statistics as a pure science.

True, though I would put the balance more like 95% engineering, 5% science.

There is a good discussion on LinkedIn [the link was broken by LinkedIn, so it has been removed]:

Davis Balestracci [link updated]: Unfortunately, we snubbed our noses at the Six Sigma movement…and got our lunch eaten. Ron Snee has been developing this message for the last 20 years (I developed it in four years’ worth of monthly columns for Quality Digest from 2005-2008). BUT…as long as people have a computer, color printer, and a package that does trend lines, academic arguments won’t “convert” anybody.

Recently, we’ve lost our way and evolved into developing “better jackhammers to drive tacks”…and pining for the “good ol’ days” when people listened to us (which they were forced to do because they didn’t have computers, and statistical packages were clunky). Folks, we’d better watch it…or we’re moribund

Was there really a good old days when business listened to statisticians? Of course occasionally they did, but “good old days”? Here is a report from 1986 the theme of which seems to me to be basically how to get statisticians listened to by the people that make the important decisions: The Next 25 Years in Statistics, by Bill Hunter and William Hill. Maybe I do the report a disservice with my understanding of the basic message, but it seems to me to be how to make sure the important contributions of applied statisticians actually get applied in organizations. And it discusses how statisticians need to take action to drive adoption of the ideas because currently (1986) they are too marginalized (not listened to when they should be contributing) in most organizations.
Continue reading

Taxes per Person by Country

I think that the idea that data lies is false, and that such a notion is commonly held a sign of lazy intellect. You can present data in different ways to focus on different aspects of a system. And you can make faulty assumptions based on data you look at.

It is true someone can just provide false data, that is an issue you have to consider when drawing conclusions from data. But often people just don’t think about what the data is really saying. Most often when people say data lies they just were misled because they didn’t think about what the data actually showed. When you examine data provided by someone else you need to make sure you understand what it is actually saying and if they are trying to support their position you may be wise to be clear they are not misleading you with their presentation of the data.

Here is some data from Greg Mankiw’s Blog. He wants to make his point that the USA is taxed more on par with Europe than some believe because he want to reduce current taxes. So he shows that while taxes as a percent of economic activity is low in the USA taxes per person is comparable to Europe.

Taxes/GDP x GDP/Person = Taxes/Person

France .461 x 33,744 = $15,556

Germany .406 x 34,219 = $13,893

UK .390 x 35,165 = $13,714

US .282 x 46,443 = $13,097

Canada .334 x 38,290 = $12,789

Italy .426 x 29,290 = $12,478

Spain .373 x 29,527 = $11,014

Japan .274 x 32,817 = $8,992

The USA is the 2nd lowest for percent of GDP taxes 28.2% v 27.4% for Japan. But in taxes per person toward the middle of the pack. France which has 46% taxes/GDP totals $15,556 in tax per person compared to $13,097 for the USA. Both measures of taxes are useful to know, in my opinion. Neither lies. Both have merit in providing a understanding of the system (the economies of countries).

Related: Fooled by RandomnessSimpson’s ParadoxMistakes in Experimental Design and InterpretationGovernment Debt as Percentage of GDP 1990-2008 by CountryCommunicating with the Visual Display of DataIllusion of Explanatory Depth

Soren Bisgaard

photo of Soren Bisgaard

Soren Bisgaard died earlier this month of cancer. Soren was a student of my father’s who shared the commitment to making a difference in people’s lives by using applied statistics properly. I know this seem odd to many (I tried to describe this idea previously, also read his acceptance of the 2002 William G. Hunter award). Soren served as the director of the director of the Center for Quality and Productivity Improvement at the University of Wisconsin-Madison (founded by William Hunter and George Box) for several years.

Most recently Soren Bisgaard, Ph.D. was Professor of technology management at Eugene M. Isenberg School of Management at the University of Massachusetts – Amherst. He was an ASQ Fellow; recipient of Shewart Medal, Hunter Award, George Box Medal, among many others awards.

I will remember the passion he brought to his work. He reminded me of my father in his desire to improve how things are done and allow people to have better lives. Those that bring passion to their work in management improvement are unsung heroes. It seems odd, to many, to see that you can bring improvement to people’s lives through work. But we spend huge amounts of our time at work. And by improving the systems we work in we can improve people’s lives. Soren will be missed, by those who knew him and those who didn’t (even if they never realize it).

Contributions in honor of Søren may be made to The International Mesothelioma Program or to the European Network for Business and Industrial Statistics. Read more articles by Søren Bisgaard.

The Future of Quality Technology: From a Manufacturing to a Knowledge Economy and From Defects to Innovations (pdf) by Soren Bisgaard

Related: The Work of Peter ScholtesManagement Improvement LeadersThe Scientific Context of Quality Improvement by George Box and Soren Bisgaard, 1987 – Obituary Søren Bisgaard at ENBISObituary: Soren Bisgaard, Isenberg Professor in Integrative Studies

Highlights from Recent George Box Speech

The JMP blog has posted some highlights from George Box’s presentation at Discovery 2009 [the broken link was removed]

Infusing his entire presentation with humor and fascinating tales of his memories, Box focused on sequential design of experiments. He attributed much of what he knows about DOE [design of experiments] to Ronald A. Fisher. Box explained that Fisher couldn’t find the things he was looking for in his data, “and he was right. Even if he had had the fastest available computer, he’d still be right,” said Box. Therefore, Fisher figured out how to study a number of factors at one time. And so, the beginnings of DOE.

Having worked and studied with many other famous statisticians and analytic thinkers, Box did not hesitate to share his characterizations of them. He told a story about Dr. Bill Hunter and how he required his students to run an experiment. Apparently a variety of subjects was studied [see 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments]

According to Box, the difficulty of getting DOE to take root lies in the fact that these mathematicians “can’t really get the fact that it’s not about proving a theorem, it’s about being curious about things. There aren’t enough people who will apply [DOE] as a way of finding things out. But maybe with JMP, things will change that way.”

George Box is a great mind and great person who I have had the privilege of knowing my whole life. My father took his class at Princeton, then followed George to the University of Wisconsin-Madison (where Dr. Box founded the statistics department and Dad received the first PhD). They worked together building the UW statistics department, writing Statistics for Experimenters and founding the Center for Quality and Productivity Improvement among many other things.

Statistics for Experimenters: Design, Innovation, and Discovery shows that the goal of design of experiments is to learn and refine your experiment based on the knowledge you gain and experiment again. It is a process of discovery. If done properly it is very similar to the PDSA cycle with the application of statistical tools to aid in determining the impact of various factors under study.

Related: Box on QualityGeorge Box Quotationsposts on design of experimentsUsing Design of Experiments

Statistical Learning as the Ultimate Agile Development Tool by Peter Norvig

Interesting lecture on Statistical Learning as the Ultimate Agile Development Tool by Peter Norvig. The webcast is likely to be of interest to a fairly small segment of readers of this blog. But for geeks it may be interesting. He looks at the advantages of machine learning versus hand programming every case (for example spelling correction).

Google translate does a very good job (for computer based translation) based on machine learning. You can translate any of the pages on this blog into over 30 languages using Google translate (using the widget in the right column).

Via: @seanstickle

Related: Mistakes in Experimental Design and InterpretationDoes the Data Deluge Make the Scientific Method Obsolete?Website DataAn Introduction to Deming’s Management Ideas by Peter Scholtes (webcast)

Communicating with the Visual Display of Data

graphs showing data sets with different looks even though some statistical characteristics are the same
Anscombe’s quartet: all four sets are identical when examined statistically, but vary considerably when graphed. Image via Wikipedia.

Anscombe’s quartet comprises four datasets that have identical simple statistical properties, yet are revealed to be very different when inspected graphically. Each dataset consists of eleven (x,y) points. They were constructed in 1973 by the statistician F.J. Anscombe to demonstrate the importance of graphing data before analyzing it, and of the effect of outliers on the statistical properties of a dataset.

Of course we also have to be careful of drawing incorrect conclusions from visual displays.

For all four datasets:

Property Value
Mean of each x variable 9.0
Variance of each x variable 10.0
Mean of each y variable 7.5
Variance of each y variable 3.75
Correlation between each x and y variable 0.816
Linear regression line y = 3 + 0.5x

Edward Tufte uses the quartet to emphasize the importance of looking at one’s data before analyzing it in the first page of the first chapter of his book, The Visual Display of Quantitative Information.

Related: Great ChartsSimpson’s ParadoxSeeing Patterns Where None ExistsVisible DataControl ChartsEdward Tufte’s: Beautiful Evidence

YouTube Uses Multivariate Experiment To Improve Sign-ups 15%

Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

via: @hexawiseMy brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).

Related: Combinatorial Testing for SoftwareStatistics for ExperimentersGoogle’s Website Optimizer allows for multivariate testing of your website.Using Design of Experiments

Revealed Preference

Revealed Preference: the preference consumers display by their action, in contrast to what they may say they prefer. While surveys may be useful people often say they will do one thing and actually when given the choice to do so, don’t.

Normally what matters is not what people say they want but what they actually will choose. For that reason revealed preference is a better measure than stated preference. Stated preference is often used as a proxy for actual preference (which may be fine) but it is important to understand that it is just a proxy for actual preference.

See more explanations from the Curious Cat Management Dictionary.

Related: Packaging ImprovementAll Models Are Wrong But Some Are UsefulDangers of Forgetting the Proxy Nature of DataConfirmation BiasBe Careful What You Measure