Be Careful What You Measure

Posted on May 9, 2007 by John Hunter

Be Careful What You Measure by Mike Wroblewski:

Although this recalculation of productivity had a positive affect, it is not what I would consider a triumph. Ongoing efforts are still required to truly increase productivity, so it’s back to gemba. However, I am modifying the lesson to “Be careful what and how you measure, measurements drive action and behavior”

Excellent points. Behavior can be changed by what is measured. The problem with arbitrary numerical targets (to take one measurement related example) is not that attempts to achieve those targets won’t have an affect. They very well may have an affect. However they may not lead to the desired result. When focused on improving a number (which can happen when focused on measures – especially as the focus on those measures is tied to bonuses, favorable treatment…) the focus is not necessarily on on improving the system. Often distorting the system is the result.

Measures need to be used with a conscience effort to remember the data is merely a proxy to quantify the results (not the end themselves). Taking care in choosing the measures is one necessary step to assure the best improvement results. One strategy is to include some measures that are outcome measures. Often those measures are difficult to pin to specific process improvements tightly so you will also want to include specific process measures. The outcome measures help make sure you maintain a focus on the important system level results. Process measures will help you test and improve processes (as well as monitor and react, when necessary to ongoing processes).

Often improving the process measures can be mistaken for the aim. Care needs to be taken to underscore the role of process measures (process management). Also measures should be re-examined periodically to determine if they are still the correct measures. Systems with people are heavily influenced by what is measured. People will often react to what is measured and make adjustments to how the work is done to make the numbers better. The danger is that those attempts to make the measures look better can actually harm the overall system (when poor measures are used).

Related: Targets Distorting the System – Understanding Data – Operational Definitions and Data Collection – Dangers of Forgetting the Proxy Nature of Data

This entry was posted in Data, Deming, Management, Process improvement, Psychology and tagged , , . Bookmark the permalink.

6 Responses to Be Careful What You Measure

  1. Pingback: CuriousCat: The Problem with Targets

  2. Pingback: Curious Cat Management Improvement Blog » Better Meetings

  3. Pingback: Curious Cat Science and Engineering Blog » Cardiac Cath Lab: Innovation on Site

  4. Pingback: Curious Cat Management Improvement Blog » Managing to Test Result Instead of Customer Value

  5. Anonymous says:
    July 8, 2012 at 1:09 am

    What you describe in the management setting reminds me that this problem is widely encountered in bio-medical research too, namely what to measure, and how to measure it. Measuring outcomes (e.g. disease, mortality) is too slow, so we measure early or intermediate markers associated with disease. Then we have endless problems with conflicting results due to the study population, method of measurement, and the meaning of the measurement. One can only begin to imagine the economic and social cost of measuring the wrong things, and the wrong way, in all walks of life. As you rightly say, “Be careful what you measure”.

  6. Pingback: Distorting the System, Distorting the Data or Improving the System « The W. Edwards Deming Institute Blog

Comments are closed.