Data Based Blathering

Ok, this is one of those posts you might want to ignore or you might enjoy. Before blogs there is little chance this would reach you. But I am tired of seeing the American Customer Satisfaction Index (ACSI) promoted as if it were some encouragement for better management when all it seems to do to me is encourage superficial, non data based claims. And since it my blog I can rant if I feel like it.

ACSI: Flat Customer Satisfaction Suggests Continued Weak Consumer Spending

That is the title of the news release. Are they kidding!! They think a flat American Customer Satisfaction Index (ACSI) reading is going to lead to weak consumer spending? I doubt it. I really doubt it. What data, or theory is that based on? Jeez this whole thing just makes me crazy. Trying to use a index to promote the “importance of quality principles” (ASQ is one of the “sponsors” of this effort) and customer focus in this way – ARGH. It does the opposite – showing people how to misuse numbers. How to overreact to variation. How to compare one dot to another dot and make claims from those 2 dots. I am sure I will make mistakes in my statements but the ACSI has bugged me since it was started with the way it ignores sound quality practices and promotes the opposite of what people like Dr. Deming taught.

I suppose it it possible their was a statistically significant change in the actual consumer satisfaction in favor of the Big Three versus Asian Manufacturers, though I doubt it. But fine, lets say it isn’t just random variation. And heck for a sentence or two lets even accept this measure of “satisfaction” is even meaningful. Why would making more cars mean your quality is going to slip? This seems like trying to say something about numbers when you don’t really have anything to say. Toyota will make more cars next year, most likely (unless there is a large recession), so is your prediction that their ASCI is likely to slip? Please read Practice and Malpractice in Management Research v 6.0 by Paul R. Carlile and Clayton M. Christensen.

Making a prediction and testing it out would at least be applying some semblance of the PDSA cycle (granted I probably shouldn’t even bring that up as it is such a stretch from a what PDSA really is) – but the concept of PDSA is that it is a learning cycle. You make a prediction based on your theory and then test out your theory. The claim is making more cars means your quality is going to slip (which in the context I take them to mean is equivalent to the ASCI number slipping – otherwise the quote is basically a non-sequitur)?

Now, for such a simple prediction we could also look back in history to see what the results of increased production were. I doubt there is a correlation to more cars by Toyota is correlated to lowering ACSI score (this wouldn’t prove cause and effect but might show such a theory is worth thinking about possibly considering). Who knows, maybe the data has been looked at – I just believe, without any data from ACSI, that the theory that the Toyota Production System, as practiced by Toyota, would have that result is a bad theory). Did Toyota have increased recalls recently. I think so. Do I think Toyota needs to improve? Yes. Do I think the number of cars they make next year will be correlated with their quality results (however that is to be measured)? Nope. They may well have made some mistakes recently – maybe even related to trying to expand to fast (though that is a huge leap in my opinion – I think they will be much more specific failures that Toyota will find and address that might have been facilitated by increased production, but might very well not have been).

If they tried to increase production next year by 35% I could see a decent case being made that Toyota is attempting to expand faster than Toyota can while maintaining the quality levels they have now. But that is not nearly the same thing as saying “When you make more cars, chances are quality is going to slip.” I do not overlook the possibility of quotes being taken out of context etc.. But this type of thinking is the feel I get from the ASCI over the years – vague claims without seeming to acknowledge what management improvements we want to see. Like looking at all the data, not one or two data points. Like testing out theories, not just making claims and forgetting the claims we made last time… Like promoting the use of operational definitions Like not tampering (reacting not to sensible understanding of the system but silly superstitions).

What? So do these people believe Yahoo is providing higher customer satisfaction than Google? Oh well, believe what they say if you want.

But lets say they are claiming that those that use Yahoo like it better than those who use Google. The claim that those that use Yahoo like it better than those that use Google would lead me to question your methodology more than accept your claim (especially given the others claims in the press release). Even just reading your press release I would guess your own data shows no statistical difference between the two scores.

What is the method for quantifying the “satisfaction” of those customers that used Yahoo, didn’t like it and left to use Google? If that is not incorporated doesn’t that make the quoted figure pretty lame? Yeah 80% of the market decided they couldn’t stand our product even though it is free but of those that actually still use it the ACSI survey gives us a score of 79. While all those people that left us and decided to use Google instead are included with the rest of people that use Google and they get a ASCI score of 78. We outscored Google so we must be giving better service to our customers than Google gives to theirs (as each separate customer base rates it). This is a pretty lame piece of data, and I don’t even believe that number is an accurate reflection of those customers.

Is that measure anything close to the measure you think is most important to measuring customer satisfaction? What about huge customer growth at Google compared to Yahoo? Isn’t a much more telling figure than some ACSI number? What about a huge increase in revenue and profit? Granted those figures are not directly measures of customer satisfaction. It is possible to have figured out a way to generate income in the short term but not pleasing your customers and having measures that tell you this would be important. That could give you earlier warnings to react to than waiting to see your customers leave in droves as soon as someone provides better service, better value…

What about looking at some Google number and a Yahoo number and comparing them? Is that useful? Possibly looking at search results comparisons – maybe. Overall company figures I highly doubt has much value at all.

How much trouble is it for any Google customer to use any other search engine? How about, practically none and no cost. If these other companies provide a better service Google will lose customers very fast. Or at least stop gaining increasing market share practically every quarter. Oh, I wish I could just make a short simple post to explain what is so wrong with promoting such thinking but I just seem to be ranting so I will stop.