Getting organization to think of data as critical to making effective decisions is often a challenge. But the very next problem is that while data is used it is actually more misused than used.
What is important is not just having numbers mentioned when decisions are being made. Or even having numbers mentioned when those decisions are evaluated after they have been implemented (or course many organizations don’t even evaluate the results of many changes they adopt, but that is a different problem). What is important for “evidence based decision making” is that what those numbers actually mean must be understood. It is easy to be mislead if you don’t think critically about what the numbers tell you and what they do not.
As I ran the addresses through a precise parcel-level geocoding process and visually inspected individual blood lead levels, I was immediately struck by the disparity in the spatial pattern. It was obvious Flint children had become far more likely than out-county children to experience elevated blood lead when compared to two years prior.
How had the state so blatantly and callously disregarded such information? To me – a geographer trained extensively in geographic information science, or computer mapping – the answer was obvious upon hearing their unit of analysis: the ZIP code.
Their ZIP code data included people who appeared to live in Flint and receive Flint water but actually didn’t, making the data much less accurate than it appeared [emphasis added].
This type of assumption about data leading to mistakes in analysis is common. The act of using data doesn’t provide benefits is the data isn’t used properly. The more I see of the misuse of data to more importance I place on the skill of thinking critically. We must challenge assumptions and challenge what the data we look at actually means.
When people try to use a short quote as an accurate encapsulation of a management concept they will often be disappointed.
It is obvious that Dr. Deming believed that organizations failed to use data effectively to improve needed to change and use data effectively in order to thrive over the long term. He believed that greatly increasing the use of data in decision making would be useful. He also believe there were specific problems with how data was used, when it is was used. Failing to understand variation leads to misinterpreting what conclusions can appropriately be drawn from data.
I believe Dr. Deming would have said something like “In God we trust, all others bring data” (I haven’t been able to find a source verifying he did say it). Others don’t believe he would referencing the Lloyd Nelson quote and all Deming’s other work showing that Dr. Deming’s opinion that data isn’t all that matters. I believe they are correct that Dr. Deming wouldn’t mean for the quote to be taken literally as a summation of everything he ever said. That doesn’t mean he wouldn’t use a funny line that emphasized an important message – we need to stop relying so much on unsubstantiated opinion and instead back up opinion with data (including experiments).
Quotes can help crystallize a concept and drive home a point. They are very rarely a decent way to pass on the whole of what the author meant, this is why context is so important. But, most often quotes are shared without context and that of course, leads to misunderstandings.
A funny example of this is the Deming quote that you often see: “if you can’t measure it, you can’t manage it.” Deming did actually say that. But without the context you get 100% the wrong understanding of what he said. Deming’s full statement is “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” Now normally much more context is required to truly understand the author’s point. But this is a funny example of how a quote can be even be accurate when passed on to you and yet completely misleading because it is taken out of context.
There are many factors that are important to effectively practice the management improvement ideas I have discussed in this blog for over a decade. One of the most important is a culture that encourages critical thinking as well as challenging claims, decisions and assumptions.
I discussed this idea some in: Customers Are Often Irrational. There is a difference between saying people wish to have their desires met and people act in the manner to maximize the benefits they wish to receive.
It is important to study customer’s choice and learn from them. But being deceived by what their choice mean is easier than is usually appreciated. Often the decision made is contrary to the ideal choice based on their beliefs. It is often poor decision making not an indication that really they want a different result than they express (as revealed versus stated preference can show). People that ignore the evidence behind climate change and condemn coastal areas to severe consequences don’t necessarily prefer the consequences that their decision leads to. It may well be that decision to ignore the evidence is not based on a desire to suffer long term consequences in order to get short term benefits. It may well be just an inability to evaluate evidence in an effective way (fear of challenging ourselves to learn about matters we find difficult often provides a strong incentive to avoid doing so).
Photo of me and my artwork in my father’s office by Bill Hunter
It is important to clearly articulate the details of the decision making process. We need to note the actual criticism (faulty logic, incorrect beliefs/assumptions…) that results in what some feel is a poor conclusion. But we seem to find shy away from questioning faulty claims (beliefs that are factually incorrect – that vaccines don’t save people from harm, for example) or lack of evidence (no data) or poor reasoning (drawing unsupported conclusions from a well defined set of facts).
Critical thinking is important to applying management improvement methods effectively. It is important to know when decisions are based on evidence and when decisions are not based on evidence. It can be fine to base some decisions on principles that are not subject to rational criticism. But it is important to understand the thought process that is taken to make each decision. If we are not clear on the basis (evidence or opinion regardless of evidence) we cannot be as effective in targeting our efforts to evaluate the results and continually improve the processes in our organizations.
Describing the decision as “irrational” is so imprecise that it isn’t easy to evaluate how much merit the criticism has. If specific facts are called into question or logical fallacies within the decision making process are explained it is much more effective at providing specific items to explore to evaluate whether the criticism has merit.
When specific criticisms are made clear then those supporting such a decision can respond to the specific issues raised. And in cases where the merits of one course of action cannot be agreed to then such critical thought can often be used to create measures to be used to evaluate the effectiveness of the decision based on the results. Far too often the results are not examined to determine if they actually achieved what was intended. And even less often is care taken to examine the unintended consequences of the actions that were taken.
Technological innovation brings great opportunity for improving results and our quality of life. But transforming potential benefits into real results comes with many challenges.
ASQ has asked their Influential Voices to explore the idea of the fourth industrial revolution: “this new era is founded on the practical use of technological innovations like artificial intelligence, big data, robotics, and the Internet of Things (IoT).”
In this decade we are finally reaching the point where robotics is really making incredible strides. Robotics has provided huge benefits for decades, when used appropriately, but the ease of use and benefits from robotics have greatly increased recently.
I think robotics is going to be an incredibly powerful source of benefits to society in the next 20 years. Amazon is very well placed to profit in this area. Several other companies (Toyota, Boston Dynamics*, Honda, SoftBank…) are likely to join them (though which will be the biggest winners and which will stumble is not obvious)
Photo by John Hunter of Cliff Palace (built in the 1190s), Mesa Verde National Park.
I am less confident in the Internet of Things. It seems to me that much of the IoT effort currently is flailing around in ways similar to GMs approach to robotics in the 1980s and 1990s. There is huge potential for IoT but the architecture of those solutions and the impact of that architecture on security (and fragile software that creates many more problems than it solves) is not being approached wisely in my opinion. IoT efforts should focus on delivering robust solutions in the areas where there is a clear benefit to adopting IoT solutions. And that needs to be done with an understanding of security and the lifecycle of the devices and businesses.
I think it will be much wiser to have an internet hub in the business or home that has all IoT traffic route through it in a very clear and visible way. Users need clear ways to know what the IoT is trying to do and to have control to determine what is and what is not sent out from their system. Having devices that share information in a non-transparent way is not wise. This is especially when those devices have cameras or microphones.
we often fail to explore whether changes in the numbers (which we call results) are representative of the “true results” of the system or if the data is misleading.
Data is meant to provide us insight into a more complex reality. We need to understand the limitations when we look at “results” and understand data isn’t really the results but a representation we hope is close to reality so we can successfully use the data to make decisions.
But we need to apply thought to how we use data. Lab results are not the same are what happens in the field. It is cheaper and faster to examine results in a lab. But relying on lab results involves risk. That doesn’t mean relying on lab results is bad, we have to balance the costs and benefits of getting more accurate data.
But relying on lab results and not understanding the risk is dangerous. This is the same idea of going to the gemba to get an accurate understanding instead of relying on your ability to imagine reality based upon some data and ideas of what it is probably like.
VW Beetle (in Bangkok, Thailand) has some sort of modification along the back bumper but I don’t know what it is meant to do. Any ideas? More of my photos from Bangkok.
Volkswagen AG lost almost a quarter of its market value after it admitted to cheating on U.S. air pollution tests for years
During normal driving, the cars with the software — known as a “defeat device” — would pollute 10 times to 40 times the legal limits, the EPA estimated. The discrepancy emerged after the International Council on Clean Transportation commissioned real-world emissions tests of diesel vehicles including a Jetta and Passat, then compared them to lab results.
Obviously VW was managing-to-test-result instead of real world value. It seems they were doing so intentionally to provide misleading data. Obviously one of the risks with lab test results (medical trials etc.) is that those with an interest in showing better results could manipulate the data and lab procedures (or systems) to have the data show their product in the most favorable light.
A Case Study Madison, Wisconsin (1981-1993)
Step 1: Educate and inform everyone in the organization about the vision, the goals, and Quality Leadership. This step must be passionately led by the top leader.
Begin discussion with top management team and train them.
Discuss and ask employees; get feedback from them.
Share feedback with the chief and his management team.
Get buy-in from top department managers.
Survey external customers—citizens; those who live and work in the community.
Create an employee’s advisory council; ask, listen, inform, and keep them up to date on what’s going on.
The chief keeps on message; tells, sells, and persuades, newsletters, meetings and all available media.
Step 2: Prepare for the transformation. Before police services to the community can be improved, it is essential to prepare the inside first — to cast a bold vision and to have leaders that would “walk the talk.”
Appoint a top-level, full-time coordinator to train, coach, and assist in the transformation.
Form another employee council to work through problems and barriers encountered during implementation of the transformation and Quality Leadership.
Require anyone who seeks to be a leader to have the knowledge and ability to practice Quality Leadership.
Step 3: Teach Quality Leadership. This begins at the top with the chief and the chief’s management team.
Train all organizational leaders in Quality Leadership.
Train all employees as to what Quality Leadership is, why the transformation is necessary, and what it means for them.
Step 4: Start practicing Quality Leadership. If top managers within the organization are not authentically practicing Quality Leadership neither will anyone else.
When your customer service organization is universally recognized as horrible adding sales requirements to customer service representatives jobs is a really bad practice. Sadly it isn’t at all surprising to learn of management doing just that at our largest companies. Within a system where cash and corruption buys freedom from market forces (see below for more details) such practices can continue.
Such customer hostile practices shouldn’t continue. They shouldn’t be allowed to continue. And even though the company’s cash has bought politically corrupt parties to allow such a system to survive it isn’t even in the selfish interest of the business. They could use the cover provided by bought-and-paid-for-politicians-and-parties to maintain monopolistic pricing (which is wrong ethically and economically but could be seen as in the self interest of a business). But still provide good service (even while you take monopolistic profits allowed with corrupt, though legal, cash payments).
Comcast executives have to know they are running a company either rated the worst company in the country or close to it year after year. They, along with several others in their industry, as well as the cell phone service providers and too-big-to-fail-banks routinely are the leaders of companies most reviled by customers. Airlines are also up their for treating customer horribly but they are a bit different than the others (political corruption is much less of the reason for their ability to abuse customers for decades than is for the others listed above).
The company’s choice to transform what is traditionally a non-revenue-generating area—customer service—into a revenue-generating one is playing out with almost hilariously Kafkaesque consequences. It is the nature of large corporations like Comcast to have dozens of layers of management through which leadership instructions and directives are filtered. The bigger the company, the more likely that members of senior leadership (like Tom Karinshak) typically make broad policy and leave specific implementations to lower levels.
Here, what was likely praised in the boardroom as an “innovative” strategy to raise revenue is instead doing much to alienate customers and employees alike. Karinshak’s assurances that he doesn’t want employees to feel pressured to sell in spite of hard evidence that Comcast demands just that are hard to square with the content of the document.
So what is going on here? Most people can easily see this is likely a horrible practice. It is a practice that a well run company theoretically could pull off without harming customers too much. But for a company like Comcast to do this it is obviously going to be horrible for customers (same for all those too-big to fail banks, cell phone service providers and other ISPs and cable TV providers).
Lets just pretend Comcast’s current leadership executives were all replaced with readers of the Curious Cat Management Improvement blog. And lets say that for now you are suppose to focus on improving the policies in place (while thinking about policy changes for later but not making them yet).
George Box lecture on Statistical Design in Quality Improvement at the Second International Tampere Conference in Statistics, University of Tampere, Finland (1987).
Early on he shows a graph showing the problems with American cars steady over a 10 years period. Then he overlays the results for Japanese cars which show a steady and significant decline of the same period.
Those who didn’t get to see presentations before power point also get a chance to see old school, hand drawn, overhead slides.
He discusses how to improve the pace of improvement. To start with informative events (events we can learn from) have to be brought to the attention of informed observers. Otherwise only when those events happen to catch the attention of the right observer will we capture knowledge we can use to improve. This results in slow improvement.
A control chart is an example of highlighting that something worth studying happened. The chart will indicate when to pay attention. And we can then improve the pace of improvement.
Next we want to encourage directed experimentation. We intentionally induce informative events and pay close attention while doing so in order to learn.
Every process generates information that can be used to improve it.
He emphasis the point that this isn’t about only manufacturing but it true of any process (drafting, invoicing, computer service, checking into a hospital, booking an airline ticket etc.).
He then discussed an example from a class my father taught and where the students all when to a TV plant outside Chicago to visit. The plant had been run by Motorola. It was sold to a Japanese company that found there was a 146% defect rate (which meant most TVs were taken off the line to be fixed at least once and many twice) – this is just the defect rate before then even get off the line. After 5 years the same plant, with the same American workers but a Japanese management system had reduced the defect rate to 2%. Everyone, including managers, were from the USA they were just using quality improvement methods. We may forget now, but one of the many objections managers gave for why quality improvement wouldn’t work in their company was due to their bad workers (it might work in Japan but not here).
He references how Deming’s 14 points will get management to allow quality improvement to be done by the workforce. Because without management support quality improvement processes can’t be used.
With experimentation we are looking to find clues for what to experiment with next. Experimentation is an iterative process. This is very much the mindset of fast iteration and minimal viable product (say minimal viable experimentation as voiced in 1987).
There is great value in creating iterative processes with fast feedback to those attempting to design and improve. Box and Deming (with rapid turns of the PDSA cycle) and others promoted this 20, 30 and 40 years ago and now we get the same ideas tweaked for startups. The lean startup stuff is as closely related to Box’s ideas of experimentation as an iterative process as it is to anything else.
He also provided a bit of history that I was not aware of saying the first application of orthogonal arrays (fractional factorial designs) in industry was by Tippett in 1933. And he then mentioned work by Finney in 1945, Plackett and Burman in 1946 and Rao in 1947.
My opinion has long been that football teams are too scared to take an action that is smart but opens the coach to criticism. So instead of attempting to make it on 4th down (if you don’t understand American football, just skip this post) they punt because that is the decision that is accepted as reasonable.
So instead of doing what is wise they do what avoids criticism. Fear drives them to take the less advantageous action. Now I have never looked hard at the numbers, but my impression is that it is well worth the risk to go for it on 4th down often. In a quick search I don’t see a paper by a Harvard professor (this article refers to it also – Fourth down: To punt or to go?) on going for it on 4th down but I found on by a University of California, Berkeley economist (David Romer wrote called “Do Firms Maximize? Evidence from Professional Football.”).
On the 1,604 fourth downs in the sample for which the analysis implies that teams are on average better off kicking, they went for it only nine times. But on the 1,068 fourth downs for which the analysis implies that teams are on average better off going for it, they kicked 959 times.
My guess is that the advantages to going for it on 4th down are greater for high school than college which is greater than the advantage for the pros (but I may be wrong). My guess is this difference is greater the more yardage is needed. Basically my feeling is the variation in high school is very high in high school and decreases with greater skill, experience and preparation. Also the kicking ability (punting and field goals) impacts the choices of going for it on 4th down and that dramatically increases in college. So if I am correct, I think pro coaches should be more aggressive on 4th down, but likely less aggressive than high school coaches should be.
But in any event the data should be explored and strategies should be tested.
There was also a great deal of work done with the Police department, as the police chief, David Couper, saw great value in Deming’s ideas. The Police department did some great work and David’s blog shares wonderful ideas on improving policing. I don’t think Dad was that directly involved in what happened there, but it is one of the nice benefits of seeding new ideas: as they take root and grow wonderful things happen without any effort on your part.
As to why Dad got involved with the city, he returned from a summer teaching design of experiments and quality improvement methods in China (this is just before China was really open, a few outsiders were let in to teach). We had also lived overseas several other times, always returning to Madison. He decided he wanted to contribute to the city he loved, Madison, and so he talked to the Mayor about helping improve performance of the city.
The mayor listened and they started with a pilot project which Dad work on with Peter Scholtes. Dad talked to Peter, who he had know for years, and who worked for the city, before talking to the mayor. Read more about the efforts in Madison via the links at the end of this post.