Written in relation to a thread (Staff Attitude) on the Deming Electronic Network (link removed because the network no longer is online).
We use data to act as a proxy for some results of the system. Often people forget that the desired end result is not for the number to be improved but for the situation to be improved. We hope, if the measure improves the situation will have improved. But there are many reasons this may not be the case (one number improving at the expense of other parts of the system, the failure of the number to accurately serve as a proxy, distorting numbers, etc.).
I find something I learned from Brian Joiner an excellent summary – which I remember as:
Data (measuring a system) can be improved by
1) distorting the system
2) distorting the data
or 3) improving the system (which tends to be more difficult though likely what is desired)
Brian Joiner’s book, 4th Generation Management is a great book for managers.
It seems to me another danger to examine when looking at “results” is whether the proxy nature of the measure is deminishing. It seems to me the situation where turnover increases because people are “growing” (and therefore the assumed value as measure of employee dissatisfaction is diminished) is a situation where the system changes and reliance on the proxy of turnover to measure dissatisfaction breaks down.
This, to me, relates to the concept that all models are wrong but some are useful (which I learned from George Box) and leads me to believe that measures such as turnover and sick leave use can be valuable. And can be helpful in measuring the results of attempting to improve the system.
Another version of that concept states that all models are wrong but in a given world a model may be useful. In a certain state of equilibrium it may be that you could track increases in turnover (and use it as a proxy for employee satisfaction) even if that was not perfect measure for why all turnover happened. But then once the system shifted to a new state the turnover rate no longer served as an effective proxy. Reality is always more complex than the measures we use to try and understand it so even while the model is useful it is not a completely accurate representation of reality.
It seems to me that once we begin collecting data we often put very little thought into whether the data continue to be valuable (assuming we did so to begin with) and continue to serve the purpose they once did. And we often fail to explore whether changes in the numbers (which we call results) are representative of the “true results” of the system or the data are misleading.
This phenomenon appears in many different ways. Companies rise and fall based on “numbers.” It’s not uncommon for companies to hire contractors over regular employees in order to increase their profit to employee ratio, even if contractors cost more money. A contractor is not technically an employee, it makes the statistic look better, and that’s what’s seen as important.
Theoretically you could establish well thought out metrics for a company’s performance which resist that kind of tampering, but nobody within the financial community seems terribly interested in doing so.
Pingback: Curious Cat Management Improvement Blog » Blog Archive » Understanding Data
Pingback: Curious Cat Management Improvement Blog » Blog Archive » Dangers of Extrinsic Motivation
Pingback: Curious Cat Science and Engineering Blog » USA Under-counting Engineering Graduates
Pingback: Curious Cat Management Improvement Blog » Information Quality
Pingback: Transforming With Lean
This reminds both of Goldratt’s Theory of Constraints (managers often optimize to local goals like machine efficiencies while driving larger measures like plant profit down, because local measures are easier to see even if they’re irrelevant) and my experience in complex experiments (only when we took carefully balanced overdetermined sensor data did we get warning when our conceptual response models were wrong, a surprising 30% of the time with very sophisticated modellers).