Getting organization to think of data as critical to making effective decisions is often a challenge. But the very next problem is that while data is used it is actually more misused than used.
What is important is not just having numbers mentioned when decisions are being made. Or even having numbers mentioned when those decisions are evaluated after they have been implemented (or course many organizations don’t even evaluate the results of many changes they adopt, but that is a different problem). What is important for “evidence based decision making” is that what those numbers actually mean must be understood. It is easy to be mislead if you don’t think critically about what the numbers tell you and what they do not.
As I ran the addresses through a precise parcel-level geocoding process and visually inspected individual blood lead levels, I was immediately struck by the disparity in the spatial pattern. It was obvious Flint children had become far more likely than out-county children to experience elevated blood lead when compared to two years prior.
How had the state so blatantly and callously disregarded such information? To me – a geographer trained extensively in geographic information science, or computer mapping – the answer was obvious upon hearing their unit of analysis: the ZIP code.
Their ZIP code data included people who appeared to live in Flint and receive Flint water but actually didn’t, making the data much less accurate than it appeared [emphasis added].
This type of assumption about data leading to mistakes in analysis is common. The act of using data doesn’t provide benefits is the data isn’t used properly. The more I see of the misuse of data to more importance I place on the skill of thinking critically. We must challenge assumptions and challenge what the data we look at actually means.
When people try to use a short quote as an accurate encapsulation of a management concept they will often be disappointed.
It is obvious that Dr. Deming believed that organizations failed to use data effectively to improve needed to change and use data effectively in order to thrive over the long term. He believed that greatly increasing the use of data in decision making would be useful. He also believe there were specific problems with how data was used, when it is was used. Failing to understand variation leads to misinterpreting what conclusions can appropriately be drawn from data.
I believe Dr. Deming would have said something like “In God we trust, all others bring data” (I haven’t been able to find a source verifying he did say it). Others don’t believe he would referencing the Lloyd Nelson quote and all Deming’s other work showing that Dr. Deming’s opinion that data isn’t all that matters. I believe they are correct that Dr. Deming wouldn’t mean for the quote to be taken literally as a summation of everything he ever said. That doesn’t mean he wouldn’t use a funny line that emphasized an important message – we need to stop relying so much on unsubstantiated opinion and instead back up opinion with data (including experiments).
Quotes can help crystallize a concept and drive home a point. They are very rarely a decent way to pass on the whole of what the author meant, this is why context is so important. But, most often quotes are shared without context and that of course, leads to misunderstandings.
A funny example of this is the Deming quote that you often see: “if you can’t measure it, you can’t manage it.” Deming did actually say that. But without the context you get 100% the wrong understanding of what he said. Deming’s full statement is “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” Now normally much more context is required to truly understand the author’s point. But this is a funny example of how a quote can be even be accurate when passed on to you and yet completely misleading because it is taken out of context.
A Case Study Madison, Wisconsin (1981-1993)
Step 1: Educate and inform everyone in the organization about the vision, the goals, and Quality Leadership. This step must be passionately led by the top leader.
Begin discussion with top management team and train them.
Discuss and ask employees; get feedback from them.
Share feedback with the chief and his management team.
Get buy-in from top department managers.
Survey external customers—citizens; those who live and work in the community.
Create an employee’s advisory council; ask, listen, inform, and keep them up to date on what’s going on.
The chief keeps on message; tells, sells, and persuades, newsletters, meetings and all available media.
Step 2: Prepare for the transformation. Before police services to the community can be improved, it is essential to prepare the inside first — to cast a bold vision and to have leaders that would “walk the talk.”
Appoint a top-level, full-time coordinator to train, coach, and assist in the transformation.
Form another employee council to work through problems and barriers encountered during implementation of the transformation and Quality Leadership.
Require anyone who seeks to be a leader to have the knowledge and ability to practice Quality Leadership.
Step 3: Teach Quality Leadership. This begins at the top with the chief and the chief’s management team.
Train all organizational leaders in Quality Leadership.
Train all employees as to what Quality Leadership is, why the transformation is necessary, and what it means for them.
Step 4: Start practicing Quality Leadership. If top managers within the organization are not authentically practicing Quality Leadership neither will anyone else.
George Box lecture on Statistical Design in Quality Improvement at the Second International Tampere Conference in Statistics, University of Tampere, Finland (1987).
Early on he shows a graph showing the problems with American cars steady over a 10 years period. Then he overlays the results for Japanese cars which show a steady and significant decline of the same period.
Those who didn’t get to see presentations before power point also get a chance to see old school, hand drawn, overhead slides.
He discusses how to improve the pace of improvement. To start with informative events (events we can learn from) have to be brought to the attention of informed observers. Otherwise only when those events happen to catch the attention of the right observer will we capture knowledge we can use to improve. This results in slow improvement.
A control chart is an example of highlighting that something worth studying happened. The chart will indicate when to pay attention. And we can then improve the pace of improvement.
Next we want to encourage directed experimentation. We intentionally induce informative events and pay close attention while doing so in order to learn.
Every process generates information that can be used to improve it.
He emphasis the point that this isn’t about only manufacturing but it true of any process (drafting, invoicing, computer service, checking into a hospital, booking an airline ticket etc.).
He then discussed an example from a class my father taught and where the students all when to a TV plant outside Chicago to visit. The plant had been run by Motorola. It was sold to a Japanese company that found there was a 146% defect rate (which meant most TVs were taken off the line to be fixed at least once and many twice) – this is just the defect rate before then even get off the line. After 5 years the same plant, with the same American workers but a Japanese management system had reduced the defect rate to 2%. Everyone, including managers, were from the USA they were just using quality improvement methods. We may forget now, but one of the many objections managers gave for why quality improvement wouldn’t work in their company was due to their bad workers (it might work in Japan but not here).
He references how Deming’s 14 points will get management to allow quality improvement to be done by the workforce. Because without management support quality improvement processes can’t be used.
With experimentation we are looking to find clues for what to experiment with next. Experimentation is an iterative process. This is very much the mindset of fast iteration and minimal viable product (say minimal viable experimentation as voiced in 1987).
There is great value in creating iterative processes with fast feedback to those attempting to design and improve. Box and Deming (with rapid turns of the PDSA cycle) and others promoted this 20, 30 and 40 years ago and now we get the same ideas tweaked for startups. The lean startup stuff is as closely related to Box’s ideas of experimentation as an iterative process as it is to anything else.
He also provided a bit of history that I was not aware of saying the first application of orthogonal arrays (fractional factorial designs) in industry was by Tippett in 1933. And he then mentioned work by Finney in 1945, Plackett and Burman in 1946 and Rao in 1947.
There was also a great deal of work done with the Police department, as the police chief, David Couper, saw great value in Deming’s ideas. The Police department did some great work and David’s blog shares wonderful ideas on improving policing. I don’t think Dad was that directly involved in what happened there, but it is one of the nice benefits of seeding new ideas: as they take root and grow wonderful things happen without any effort on your part.
As to why Dad got involved with the city, he returned from a summer teaching design of experiments and quality improvement methods in China (this is just before China was really open, a few outsiders were let in to teach). We had also lived overseas several other times, always returning to Madison. He decided he wanted to contribute to the city he loved, Madison, and so he talked to the Mayor about helping improve performance of the city.
The mayor listened and they started with a pilot project which Dad work on with Peter Scholtes. Dad talked to Peter, who he had know for years, and who worked for the city, before talking to the mayor. Read more about the efforts in Madison via the links at the end of this post.
In this example a screening experiment was done first to find those factors that have the largest impact on results. Once the most important factors are determined more care can be put into studying those factors in greater detail.
Quality and The Art of Discovery by Professor George Box (1990):
Quotes by George Box in the video:
“I think of statistical methods as the use of science to make sense of numbers”
“The scientific method is how we increase the rate at which we find things out.”
“I think the quality revolution is nothing more, or less, than the dramatic expansion of the of scientific problem solving using informed observation and directed experimentation to find out more about the process, the product and the customer.”
“It really amounts to this, if you know more about what it is you are doing then you can do it better and you can do it cheaper.”
“We are talking about involving the whole workforce in the use of the scientific method and retraining our engineers and scientists in a more efficient way to run experiments.”
“Tapping into resources:
Every operating system generates information that can be used to improve it.
Everyone has creativity.
Designed experiments can greatly increase the efficiency of experimentation.
An informed observer and directed experimentation are necessary for the scientific method to be applied. He notes that the control chart is used to notify an informed observer to explain what is special about the conditions when a result falls outside the control limits. When the chart indicates a special cause is likely present (something not part of the normal system) an informed observer should think about what special cause could lead to the result that was measured. And it is important this is done quickly as the ability of the knowledgable observer to determine what is special is much greater the closer in time to the result was created.
photo of (from right to left) Peter Scholtes, John Hunter and George Box in Madison, Wisconsin at the 2008 Deming Conference
George Box (1919 to 2013) by John Hunter – George Box was a very kind, smart, caring and fun person. He was a gifted storyteller and writer. He was also one of the most important statisticians of the last 100 years. He had the ability to present ideas so they were easy to comprehend and appreciate…
George Box: A remembrance by Bradley Jones – “His greatest contribution to my life was the wonderful book, Statistics for Experimenters, which he wrote with William G. Hunter and Stu Hunter and published in 1978, the same year he served as president of the American Statistical Association. I remember the excitement I felt on reading the description of how the attainment of knowledge is an endless spiral proceeding alternately from deduction to induction and back. Even now, I recall with pleasure the discussion of the randomization distribution early in the book.”
Getting Started with Factorial Design of Experiments by Eston Martz – “When I talk to quality professionals about how they use statistics, one tool they mention again and again is design of experiments, or DOE. I’d never even heard the term before I started getting involved in quality improvement efforts, but now that I’ve learned how it works, I wonder why I didn’t learn about it sooner. If you need to find out how several factors are affecting a process outcome, DOE is the way to go.”
I would most likely not exist if it were not for George Box. My father took a course from George while my father was a student at Princeton. George agreed to start the Statistics Department at the University of Wisconsin – Madison, and my father followed him to Madison, to be the first PhD student. Dad graduated, and the next year was a professor there, where he and George remained for the rest of their careers.
George Box was a fantastic statistician. I am not the person to judge, but from what I have read one of the handful of most important applied statisticians of the last 100 years. His contributions are enormous. Several well know statistical methods are known by his name, including:
George was elected a member of the American Academy of Arts and Sciences in 1974 and a Fellow of the Royal Society in 1979. He also served as president of the American Statistics Association in 1978. George is also an honorary member of ASQ.
George was a very kind, caring and fun person. He was a gifted storyteller and writer. He had the ability to present ideas so they were easy to comprehend and appreciate. While his writing was great, seeing him in person added so much more. Growing up I was able to enjoy his stories often, at our house or his. The last time I was in Madison, my brother and I visited with him and again listened to his marvelous stories about Carl Pearson, Ronald Fisher and so much more. He was one those special people that made you very happy whenever you were near him.
George Box, Stuart Hunter and Bill Hunter (my father) wrote what has become a classic text for experimenters in scientific and business circles, Statistics for Experimenters. I am biased but I think this is acknowledged as one of (if not the) most important books on design of experiments.
George also wrote other classic books: Time series analysis: Forecasting and control (1979, with Gwilym Jenkins) and Bayesian inference in statistical analysis. (1973, with George C. Tiao).
George Box and Bill Hunter co-founded the Center for Quality and Productivity Improvement at the University of Wisconsin-Madison in 1984. The Center develops, advances and communicates quality improvement methods and ideas.
One of my pet peeves is when people say that a point outside the control limits is a special cause. It is not. It is an indication that it likely a special cause exists, and that special cause thinking is the correct strategy to use to seek improvement. But that doesn’t mean there definitely was a special cause – it could be a false signal.
Similarly, a result that doesn’t signal a special cause (inside the control limits without raising some other flag, say a run of continually increasing points) does not mean a special cause is not present.
The reason control charts are useful is to help us maximize our effectiveness. We are biased toward using special cause thinking when it is not the most effective approach. So the control chart is a good way to keep us focused on common cause thinking for improvement. It is also very useful in flagging when it is time to immediately start using special cause thinking (since timing is key to effective special cause thinking).
However, if there is result that is close to the control limit (but inside – so no special cause is indicated) and the person that works on the process everyday thinks, I noticed x (some special cause) earlier, they should not just ignore that. It very well could be a special cause that, because of other common cause variation, resulted in a data point that didn’t quite reach the special cause signal. Where the dot happened to land (just above or just below the control limit – does not determine if a special cause existed).
The signal is just to help us systemically make the best choice of common cause or special cause thinking. The signal does not define whether a special cause (an assignable cause) exists of not. The control chart tool helps guide us to use the correct type of improvement strategy (common cause or special cause). But it is just a signaling device, it isn’t some arbiter of whether a special cause actually exists.