The recent drastic reductions again emphasize (once again) that changes in the federal funds rate are not correlated with changes in the 30 year fixed mortgage rate. In the last 4 months the discount rate has been reduced nearly 200 basis points, while 30 year fixed mortgage rates have fallen 18 basis points.

I have update my article showing the historical comparison of 30 year fixed mortgage rates and the federal funds rate. The chart shows the federal funds rate and the 30 year fixed rate mortgage rate from January 2000 through April 2008 (for more details see the article).

It is surprising to me how incorrect beliefs about what predicted federal funds rate cuts will mean for mortgage rates are so common when the data makes it clear that such beliefs (I should wait because the fed is going to cut the fed funds rate and so if I wait my mortgage rate will be lower) are unwarranted. But I should not be this type of behavior goes on all the time, inside organization and outside them.
(more…)

The Defect Black Market

Well, this doesn’t sound very well thought out. Bonuses often distort behavior. Dr. Deming was not against such targets and bonuses because he thought they would not result in bugs being fixed: Dr. Deming on the problems with targets or goals. It is a question of how that will happen. The system being distorted is the most likely result of any such system.

(more…)

Rental Car IT

This is a good post on the systemic drivers of complex processes, take the time to read the whole post. I have a bias is against off the shelf software as it often ends up forcing the process to be designed around the software. And with the amazing power and relative ease of web based applications creating solutions that are specifically designed to the organization are often relatively easy. And yet, as indicated in this article there is often a strong bias in the other direction for buying off the shelf software because it is cheaper and/or faster.

Of course, the decision in each case must be weighed to determine the benefits and cost of the various alternatives. Just remember, if you decide you want simple and flexible, to have your decisions reflect that. I enjoy a telling quote from a software vendor on Toyota’s IT expectations: “it demands that the software or technology be flexible and adapt, often by customizing the code, to its business processes, and not the other way around.” They are right.

Related: Agile Software Development - Complicating Simplicity - Joy in Software Development

The Rationale of Scientific Experimentation by John Dowd explains the value of designed experiments.

The paper also explains analytic and enumerative studies. Dr. Deming stressed the importance of understanding the distinction between the two.

Related: management improvement articles - Design of Experiments articles - Statistics for Experimenters - search statistical management improvement sites - Using Design of Experiments

Another interesting experiment from Google: Using Prediction Markets to Track Information Flows: Evidence from Google

Interesting paper. I would guess most readers of this blog won’t be able to apply prediction markets to there workplace in the short term but never-the-less I find the paper interesting.

Related: Management is Prediction - Google Experiments Quickly and Often - Secrets of the World’s Best Companies

I am reading a fascinating book by Jessica Snyder Sachs: Good Germs, Bad Germs. From page 108:

This is a great example of a positive special cause. How would you identify this? First you would have to stratify the data. It also shows that sometimes looking at the who is important (the problem is just that we far too often look at who instead of the system so at times some get the idea that it is not ok to stratify data based on who - it is just be careful because we often do that when it is not the right approach and we can get fooled by random variation into thinking there is a cause - see the red bead experiment for an example); that it is possible to stratify the data by person to good effect.

The following 20 pages in the book are littered with very interesting details many of which tie to thinking systemically and the perils of optimizing part of the system (both when considering the system to be one person and also when viewing it as society).

I have recently taken to reading more and more about viruses, bacteria, cells, microbiology etc.: it is fascinating stuff.

Related: Science Books by topic - Data Can’t Lie - Understanding Data

This is a pretty counter-intuitive statement, I believe:

But some simple math shows it is true. If you drive 10,000 miles you would use: 667 gallons, 556 gallons, 200 gallons and 100 gallons. Amazing. I must admit, when I first read the quote I thought that it must be an wrong. But there is the math. You save 111 gallons improving from 15 mpg to 18 mpg and just 100 improving from 50 to 100 mpg. Other than those of you who automatically guess that whatever seems wrong must be the answer when you see a title like this I can’t believe anyone thinks 15 to 18 mpg is the change that has the bigger impact. It is great how a little understanding of math can help you see the errors in your initial beliefs. Via: 18 Is Enough.

It also illustrates that the way the data is presented makes a difference. You can also view 100 mpg as 1/100 gallon per mile, 2/100 gallons per mile, 5.6/100 gpm and 6.7 gpm. That way most everyone sees that the 6.7 to 5.6 gpm saves more fuel than 2 to 1 gpm does. Mathematics and scientific thinking are great - if you are willing to think you can learn to better understand the world we live in every day.

Related: Statistics Don’t Lie, But People Can be Fooled - Understanding Data - Seeing Patterns Where None Exists - Optical Illusions and Other Illusions - 1=2: A Proof

This is a nice article discussing how people are often fooled by thinking there must be special causes for patterns in random data. I still remember my father showing my classes these lessons when I was in grade school. Playing At Dice - What That “Weekend Exercise” Was All About:

When people are asked to explain random variations in data they will make up special causes (that they often even believe are special causes even when they are not) but you can improve management a great deal by just stopping the requirement to “explain” common cause variation (which in practices mean to claim a special cause for the common cause variation). Use that time instead to standardize processes. Create control charts for critical processes. Run experiments using PDSA cycle…

Related: Seeing Patterns Where None Exists - Understanding Data - Operational Definitions and Data Collection - Red Bead Experiment

Last month, in a long post criticizing the ACSI I took issue with, among other things, the implications being drawn from an ACSI rating. The ACSI rating of Yahoo was higher than that of Google (though statistically insignificantly so). Anyway, here is some new data on search volumes of the leading providers:

Top 5 Search Providers for August 2007, Ranked by Searches (U.S.)

Source: Nielsen//NetRatings MegaView Search

So Google grew 39.8% year over year and Yahoo grew 8.9% year over year. Google now has 53.6% of the total searches. Granted this is limited data but it seems to confirm that Google is in fact continuing to increase their lead in search volume. Practically all evidence seems to support this belief - the ACSI seems to be the exception. Which might indicate great insight provided by the ACSI that everyone else is missing. Or it might show ACSI results are doing a poor job of providing a useful measure of customer satisfaction with search engines. I go with the second.

Related: posts on using data effectively - Website Data - Understanding Data - posts on Google management - Curious Cat Management Improvement Search

Another essay by Paul Graham packed with great thoughts - this one on hiring, colleges, measuring performance of people, etc..

Related: Hiring the Right Workers - Malcolm Gladwell, Synchronicity, College Admissions… - Google and Paul Graham’s Latest Essay - Interviewing and Hiring Programmers - What Business Can Learn from Open Source - Google’s Answer to Filling Jobs Is an Algorithm - Hiring: Silicon Valley Style - Curious Cat Management Improvement Career Connections

Data is often displayed poorly, making it difficult to see what is important. When data is displayed well the important facts should leap off the page and into the viewers mind. Edward Tufte is an expert on this topic with great books. If you have not read them, you should: Beautiful Evidence, The Visual Display of Quantitative Information, Envisioning Information and Visual Explanations.

Smashing magazine has some nice examples of good display techniques in Data Visualization: Modern Approaches. I don’t like all the examples they show but it does provide some help by showing some creative ways to display data.

Related: Edward Tufte’s new book: Beautiful Evidence - Great Charts - Data Visualization Example

Ok, this is one of those posts you might want to ignore or you might enjoy. Before blogs there is little chance this would reach you. But I am tired of seeing the American Customer Satisfaction Index (ACSI) promoted as if it were some encouragement for better management when all it seems to do to me is encourage superficial, non data based claims. And since it my blog I can rant if I feel like it.

ACSI: Flat Customer Satisfaction Suggests Continued Weak Consumer Spending

That is the title of the news release. Are they kidding!! They think a flat American Customer Satisfaction Index (ACSI) reading is going to lead to weak consumer spending? I doubt it. I really doubt it. What data, or theory is that based on? Jeez this whole thing just makes me crazy. Trying to use a index to promote the “importance of quality principles” (ASQ is one of the “sponsors” of this effort) and customer focus in this way - ARGH. It does the opposite - showing people how to misuse numbers. How to overreact to variation. How to compare one dot to another dot and make claims from those 2 dots. I am sure I will make mistakes in my statements but the ACSI has bugged me since it was started with the way it ignores sound quality practices and promotes the opposite of what people like Dr. Deming taught.

I suppose it it possible their was a statistically significant change in the actual consumer satisfaction in favor of the Big Three versus Asian Manufacturers, though I doubt it. But fine, lets say it isn’t just random variation. And heck for a sentence or two lets even accept this measure of “satisfaction” is even meaningful. Why would making more cars mean your quality is going to slip? This seems like trying to say something about numbers when you don’t really have anything to say. Toyota will make more cars next year, most likely (unless there is a large recession), so is your prediction that their ASCI is likely to slip? Please read Practice and Malpractice in Management Research v 6.0 by Paul R. Carlile and Clayton M. Christensen.

Making a prediction and testing it out would at least be applying some semblance of the PDSA cycle (granted I probably shouldn’t even bring that up as it is such a stretch from a what PDSA really is) - but the concept of PDSA is that it is a learning cycle. You make a prediction based on your theory and then test out your theory. The claim is making more cars means your quality is going to slip (which in the context I take them to mean is equivalent to the ASCI number slipping - otherwise the quote is basically a non-sequitur)?
(more…)

Many people state that data can lie. Obviously data can’t lie.

There are three kinds of lies: Lies, damn lies and statistics - Mark Twain

Many people don’t understand the difference between being manipulated because they can’t understand what the data really says and data itself “lying” (which, of course, doesn’t even make sense). The same confusion can come in when someone just draws the wrong conclusion from the data that exists (and them blames the data for “lying” instead of themselves for drawing a faulty conclusion). The data can be wrong (and the data can even be made faulty intentionally by someone). Or someone can draw the wrong conclusion from data that is correct. But in neither case is the data lying. It is also common to believe the data means something other than what it does (therefore leading to a faulty conclusion).

For a very simple example, believing if the average height for adults in the USA is 5 feet 9 inches that half the people must be taller and half the people must be shorter. You could then draw the conclusion that half the adults must be shorter than 5 feet 9 inches. But that is not what an average height means (it is basically what median means, though if you want to get technical, it doesn’t mean exactly that). You might draw the conclusion that the average height of an adult in California is 5 feet 9 inches but that is not supported by only the data that says what the height of an average adult in the country is. The same hold for drawing the conclusion that 5 feet 9 inches is the average height of a women. Now in this simple examples, hopefully people can see the faulty reasoning but such reasoning often goes on without consideration.

In a great speech by Marisa Meyer she speaks of Google makes decisions using data and that data is apolitical. One benefit of this, she says, is that Google makes decisions on what the data supports not political considerations. The belief that basing decision on what the data supports leads to better decisions can seem false for those that accept the quote about 3 types of lies (or those that see there is some weakness to this point if those supposedly basis decisions on data don’t really understand how to do so).
(more…)

Mistakes in Experimental Design and Interpretation

This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.

Related: Seeing Patterns Where None Exists - Illusions, Optical and Other - Understanding Data - Dangers of Forgetting the Proxy Nature of Data - How to Deal with False Research Findings - descriptive “theory” and normative theory

Be Careful What You Measure by Mike Wroblewski:

Excellent points. Behavior can be changed by what is measured. The problems with arbitrary numerical targets (to take one measurement related example) is not that attempts to achieve those targets won’t have an affect. They very well may have an affect. However they may not have the desired result. When focused on improving a number (which can happen when focused on measures - especially as the focus on those measures is tied to bonuses, favorable treatment…) the focus is not necessarily on on improving the system. Often distorting the system is the result.

Measures need to be used with a conscience effort to remember the data is merely a proxy to quantify the results (not the end themselves). Taking care in choosing the measures is one necessary step to assure the best improvement results. One strategy is to include some measures that are outcome measures. Often those measures are difficult to pin to specific process improvements tightly so you will also want to include specific process measures. The outcome measures help make sure you maintain a focus on the important system level results. Process measures will help you test and improve processes (as well as monitor and react, when necessary to ongoing processes).

Often improving the process measures can be mistaken for the aim. Care needs to be taken to underscore the role of process measures (process management). Also measures should be re-examined periodically to determine if they are still the correct measures. Systems with people are heavily influenced by what is measured. People will often react to what is measured and make adjustments to how the work is done to make the numbers better. The danger is that those attempts to make the measures look better can actually harm the overall system (when poor measures are used).

Related: Targets Distorting the System - Understanding Data - Operational Definitions and Data Collection - Dangers of Forgetting the Proxy Nature of Data

In Myths About the Developing World, Hans Rosling shows some great graphics to display data on health care outcomes. This is one of the talks from the great TED conference that we have mentioned before. They really have some great webcasts available on their site.

The presentation also gives a concrete example of faulty knowledge (people thinking things which are not so - related to theory of knowledge). He also makes good points on stratifying data at the 14 minute mark. See gapminder.org for good additional material.

Related: Great Charts - Open Access Education Materials

Lean-based Metrics for Agile CM Environments by by Brad Appleton, Robert Cowham and Steve Berczuk:

I agree measuring individuals is normally not an effective way improve. And “measuring up” can often be valuable. Often a fixation on small process measures can result in improvements that don’t actually improve the end result. But rather than the measure up view, I find looking at outcome measures (to measure overall effectiveness) and process measures (for viewing specific parts of the system “big picture”) the most useful strategy.

The reason for process measures is not to improve those results alone. But those process measures can be selected to measure key processes within the system. Say finding 3 process measures that if we can improve these then this important outcome measure will improve (using PDSA to make sure your prediction is accurate - don’t fall into the trap of focusing on improving that measure even after the data shows it does not result in the desired improvement to the overall results that was predicted).

Also, process measures are helpful in serving as indicators that something is going wrong (or potentially going better than normal). Process measures will change quickly (good ones can be close to real time) thus facilitate immediate remedies and immediate examination of what lead to the problem to aid in avoiding that condition in the future.
(more…)

Jeffrey Pfeffer Testifies to Congress About Evidence-Based Practices:

Great stuff. Read the entire document. via: Bob Sutton’s Work Matters

Related: Evidence-based Management - Illusions - Optical and Other

Books: The Knowing-Doing Gap by Jeffrey Pfeffer and Robert Sutton - Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management by Jeffrey Pfeffer and Robert Sutton

The Alexa toolbar, from amazon.com (and by the guy, Brewster Kahle, who created the internet archive too, by the way) is one way to get some idea of how popular a web site is (the toolbar shows the web site rank for each site you visit). It is very inaccurate, but is free, and easy, so it is used. Alexa Toolbar and the Problem of Experiment Design shows why data based decision making is not the solution to all your problems:

As Dr. Deming said best efforts are not enough, you need to know what to do first. The same holds true with data, first you need to have useful data.

Related: Manage what you can’t measure - podcast interview: Brewster Kahle - Gladwell and more IT Conversations

If the output for working for the year is a square. And the job is to produce dark squares who do you pay more A or B? Of course it is a trick question, the squares are the same color. But it doesn’t look that way at first does it? Optical illusions provide evidence that you cannot always trust what seems obvious.

Dr. Deming’s red bead experiment provides some additional insight into the idea that our management systems often use “evidence” to support our believes when in fact the “evidence” does not mean what we think it does. Dr. Deming included the theory of knowledge (how do we know what we know) as one of the four areas of his management system. It is the areas of his work that is least appreciated and understood by managers today. Optical illusions provide a simple reminder of how easily we can think we know things that are not so.

Just as Toyota is always dissatisfied and looking for how to improve, it is important to question what you believe. Even when it is as obvious as the A square being darker than the B square. Understanding the ease with which we can reach false conclusions can be a powerful aid in improving management decision making.

Related: The Illusion of Understanding - Change is not Improvement - Performance Appraisal Problems - Dr. Deming on Performance Appraisal: “The fact is that the system that people work in and the interaction with people may account for 90 or 95 percent of performance” (from the introduction to the Team Handbook) - It is a mistake to think improving the figures is the goal

Optical illusion by Edward H. Adelson