Distort the System

From our post: Targets Distorting the System, Dr. Brian Joiner:

Another example of this in practice: Recount helps one university rise in the rankings [the broken link was removed]:

I am sure “classes under 20” is a proxy for an intimate learning environment and interaction with knowledgeable professors that can teach well. You can’t directly measure the benefit of interaction with a professor in a small group on learning to create data to be used in ranking schools (Deming on unknown and unknowable figures). So classes with under 20 students and % of faculty with PhDs… are used as proxies for this idea.

If the proxy is the focus (as in school rankings) then distorting the system to create better looking data is a likely result. The purpose behind the action has great significance. If an institution desired to create a better learning environment and they used say a cause and effect diagram to find a group of problems and then determined one appropriate improvement step was to reduce class size (and perhaps another was to reduce the importance of tests and perhaps another was to provide professors training on effective teaching strategies) that a sensible path to improving the system.

When people mistake the data proxy for the thing to improve (in this example the system is actually being managed to improve a piece of data though in this case the person taking the action seems to know they are just distorting the system to get a better number, they don’t seem to even believe that they are improving) they focus on improvement of how the data looks not of the system. That is the wrong strategy. The correct strategy is to focus on improving the system and as a way of verifying results you then look at measures. But you must always remember those measures are not the end they are an attempt to measure the end you are trying to achieve.

The article also discusses another of Joiner’s methods for getting better looking data: distorting the figures. This is the easiest thing to do (you just change a number how hard is that?), though since it is dishonest people have an aversion to doing so. But under enough pressure this option becomes inviting (see posts on management and psychology and respect for people).

History shows over and over again that numbers will be distorted (outright lies, or numbers people justify but only do so because of the pressure to meet some value). To focus on improvement the distortion in the numbers needs to be reduced as much as possible which is difficult if you have things like forced ranking and huge bonuses, etc..

Related:

• Dangers of Forgetting the Proxy Nature of Data
• Fourth Generation Management by Brian Joiner