|
|
|
|
|
|
Written in relation to a thread (Staff Attitude) on the Deming Electronic Network (but the DEN archive seems to indicate I never posted this message).
We use data to act as a proxy for some results of the system. Often people forget that the desired end result is not for the number to be improved but for the situation to be improved. We hope, if the measure improves the situation will have improved. But there are many reasons this may not be the case (one number improving at the expense of other parts of the system, the failure of the number to accurately serve as a proxy, distorting numbers, etc.).
I find something I learned from Brian Joiner an excellent summary – which I remember as:
Data (measuring a system) can be improved by
1) distorting the system
2) distorting the data
or 3) improving the system (which tends to be more difficult though likely what is desired)
Brian Joiner’s book, 4th Generation Management is a great book for managers.
It seems to me another danger to examine when looking at “results” is whether the proxy nature of the measure is deminishing. It seems to me the situation where turnover increases because people are “growing” (and therefore the assumed value as measure of employee dissatisfaction is diminished) is a situation where the system changes and reliance on the proxy of turnover to measure dissatisfaction breaks down.
This, to me, relates to the concept that all models are wrong but some are useful (which I learned from George Box) and leads me to believe that measures such as turnover and sick leave use can be valuable. And can be helpful in measuring the results of attempting to improve the system.
Another version of that concept states that all models are wrong but in a given world a model may be useful. In a certain state of equilibrium it may be that you could track increases in turnover (and use it as a proxy for employee satisfaction) even if that was not perfect measure for why all turnover happened. But then once the system shifted to a new state the turnover rate no longer served as an effective proxy. Reality is always more complex than the measures we use to try and understand it so even while the model is useful it is not a completely accurate representation of reality.
It seems to me that once we begin collecting data we often put very little thought into whether the data continue to be valuable (assuming we did so to begin with) and continue to serve the purpose they once did. And we often fail to explore whether changes in the numbers (which we call results) are representative of the “true results” of the system or they fail to do so.
| M | T | W | T | F | S | S |
|---|---|---|---|---|---|---|
| Sep » | ||||||
| 1 | ||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| 23 | 24 | 25 | 26 | 27 | 28 | 29 |
| 30 | 31 | |||||
August 30th, 2004 at 3:35 pm
This phenomenon appears in many different ways. Companies rise and fall based on “numbers.” It’s not uncommon for companies to hire contractors over regular employees in order to increase their profit to employee ratio, even if contractors cost more money. A contractor is not technically an employee, it makes the statistic look better, and that’s what’s seen as important.
Theoretically you could establish well thought out metrics for a company’s performance which resist that kind of tampering, but nobody within the financial community seems terribly interested in doing so.
June 18th, 2006 at 12:14 pm
[...] To be effective you need to learn to think about not what is printed on the page but what lies behind the numbers you see. The numbers are just proxies for the real situation. Look beyond the numbers you see to what they mean and understand how the numbers presented may not fully capture the important details you need to consider. [...]
August 9th, 2006 at 11:40 pm
I’ve identified three major styles: two easy, dysfunctional styles and one hard, functional style…
September 4th, 2006 at 3:09 pm
[...] How accurately the data reflects the situation is something that must always be considered: data is a proxy for something. All models are wrong, some are useful – George Box. [...]
November 1st, 2006 at 1:28 pm
Fear often can lead to obfuscated, hidden and even faked data…
January 31st, 2007 at 2:15 pm
[...] but improving the numbers is not the aim, the numbers are merely proxies for that aim. [...]
August 11th, 2008 at 4:43 pm
This reminds both of Goldratt’s Theory of Constraints (managers often optimize to local goals like machine efficiencies while driving larger measures like plant profit down, because local measures are easier to see even if they’re irrelevant) and my experience in complex experiments (only when we took carefully balanced overdetermined sensor data did we get warning when our conceptual response models were wrong, a surprising 30% of the time with very sophisticated modellers).