All Data is Wrong, Some is Useful

From my first blog post on this blog – Dangers of Forgetting the Proxy Nature of Data

we often fail to explore whether changes in the numbers (which we call results) are representative of the “true results” of the system or if the data is misleading.

Data is meant to provide us insight into a more complex reality. We need to understand the limitations when we look at “results” and understand data isn’t really the results but a representation we hope is close to reality so we can successfully use the data to make decisions.

But we need to apply thought to how we use data. Lab results are not the same are what happens in the field. It is cheaper and faster to examine results in a lab. But relying on lab results involves risk. That doesn’t mean relying on lab results is bad, we have to balance the costs and benefits of getting more accurate data.

But relying on lab results and not understanding the risk is dangerous. This is the same idea of going to the gemba to get an accurate understanding instead of relying on your ability to imagine reality based upon some data and ideas of what it is probably like.

VW Beetle (in Bangkok, Thailand) has some sort of modification along the back bumper but I don’t know what it is meant to do. Any ideas? More of my photos from Bangkok.

Volkswagen Drops 23% After Admitting Diesel Emissions Cheat

Volkswagen AG lost almost a quarter of its market value after it admitted to cheating on U.S. air pollution tests for years
…
During normal driving, the cars with the software — known as a “defeat device” — would pollute 10 times to 40 times the legal limits, the EPA estimated. The discrepancy emerged after the International Council on Clean Transportation commissioned real-world emissions tests of diesel vehicles including a Jetta and Passat, then compared them to lab results.

Obviously VW was managing-to-test-result instead of real world value. It seems they were doing so intentionally to provide misleading data. Obviously one of the risks with lab test results (medical trials etc.) is that those with an interest in showing better results could manipulate the data and lab procedures (or systems) to have the data show their product in the most favorable light.

VW certainly behaved badly. When it happens on such a critical area that is not that difficult to verify I think it would be wise to take this risk seriously elsewhere. Studies of medical studies are constantly finding misleading results being reported (this is even with many measures in place attempting to counter the dangerous of misleading results being reported by those with an incentive to do so).

Getting real world data is critical. It is even more critical when those creating the lab results have a good reason to distort the system or distort the data. And if they know the test conditions to collect the data they can also create solutions that perform well in that context but not in the conditions that data is meant to be a proxy for.

Standardized testing in education is one example. Most educators agree standardized tests are a lousy measure of education. If a school system is evaluated on the basis of standardized test scores then a the school system has an incentive to focus on getting good results on standardized tests, even if that means harming education.

But even if they don’t intentionally design a system to make the data look as good for them as possible, the laboratory is a model for the real world and will not be the same as the real world conditions.

Those working on the process producing the data should have good insight into when the results are most likely to be questionable. But they can also be fooled. You have to rely on data to make decisions. You need to understand the risks of the data you have and judge if that wrong-data (it is just a proxy) is useful enough or if you need to get better data (often by going to the user gemba) to make a better decision.

Related: User Gemba: learn how customers actually use the product or service – Studies in Quality Improvement: Designing Environmental Regulations – Numeracy: The Educational Gift That Keeps on Giving

*Title is based on George Box’s quote – “All Models Are Wrong But Some Are Useful.”

Here is a question about data that hasn’t been manipulated – What Reduces the use of gasoline more, going from 15 to 18 mpg or going from 50 to 100 mpg?