Currently browsing the Science Category

One factor at a time (OFAT) Versus Factorial Designs

Guest post by Bradley Jones

Almost a hundred years ago R. A. Fisher‘s boss published an article espousing OFAT (one factor at a time). Fisher responded with an article of his own laying out his justification for factorial design. I admire the courage it took to contradict his boss in print!

Fisher’s argument was mainly about efficiency – that you could learn as much about many factors as you learned about one in the same number of trials. Saving money and effort is a powerful and positive motivator.

The most common argument I read against OFAT these days has to do with inability to detect interactions and the possibility of finding suboptimal factor settings at the end of the investigation. I admit to using these arguments myself in print.

I don’t think these arguments are as effective as Fisher’s original argument.

To play the devil’s advocate for a moment consider this thought experiment. You have to climb a hill that runs on a line going from southwest to northeast but you are only allowed to make steps that are due north or south or due east or west. Though you will have to make many zig zags you will eventually make it to the top. If you noted your altitude at each step, you would have enough data to fit a response surface.

Obviously this approach is very inefficient but it is not impossible. Don’t mistake my intent here. I am definitely not an advocate of OFAT. Rather I would like to find more convincing arguments to persuade experimenters to move to multi-factor design.

Related: The Purpose of Factorial Designed ExperimentsUsing Design of Experimentsarticles by R.A. Fisherarticles on using factorial design of experimentsDoes good experimental design require changing only one factor at a time (OFAT)?Statistics for Experimenters

Factorial Designed Experiment Aim

Multivariate experiments are a very powerful management tool to learn and improve performance. Experiments in general, and designed factorial experiments in particular, are dramatically underused by managers. A question on LinkedIn asks?

When doing a DOE we select factors with levels to induce purposely changes in the response variable. Do we want the response variable to move within the specs of the customers? Or it doesn’t matter since we are learning about the process?

The aim needs to consider what you are trying to learn, costs and potential rewards. Weighing the various factors will determine if you want to aim to keep results within specification or can try options that are likely to return results that are outside of specs.

If the effort was looking for breakthrough improvement and costs of running experiments that might produce results outside of spec were low then specs wouldn’t matter much. If the costs of running experiments are very high (compared with expectations of results) then you may well want to try designed experiment values that you anticipate will still produce results within specs.

There are various ways costs come into play. Here I am mainly looking at the costs as (costs – revenue). For example the case where if the results are withing spec and can be used the costs (net costs, including revenue) of the experiment run are substantially lower.
Continue reading

Problems With Student Evaluations as Measures of Teacher Performance

Dr. Deming was, among other things a professor. He found the evaluation of professors by students an unimportant (and often counterproductive measure) – used in some places for awards and performance appraisal. He said for such a measure to be useful it should survey students 20 years later to see which professors made a difference to the students. Here is an interesting paper that explored some of these ideas. Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors by Scott E. Carrell, University of California, Davis and National Bureau of Economic Research; and James E. West, U.S. Air Force Academy:

our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value‐added but positively correlated with follow‐on course value‐added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow‐on related curriculum.

Student evaluations are positively correlated with contemporaneous professor value‐added and negatively correlated with follow‐on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value‐added in follow‐on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy of this practice.

These findings have broad implications for how students should be assessed and teacher quality measured.

Related: Applying Lean Tools to University CoursesK-12 Educational ReformImproving Education with Deming’s IdeasLearning, Systems and ImprovementHow We Know What We Know

Statistical Engineering Links Statistical Thinking, Methods and Tools

In Closing the Gap Roger W. Hoerl and Ronald D. Snee lay out a sensible case for focusing on statistical engineering.

We’re not suggesting that society no longer needs research in new statistical techniques for improvement; it does. The balance needed at this time, however, is perhaps 80% for statistics as an engineering discipline and 20% for statistics as a pure science.

True, though I would put the balance more like 95% engineering, 5% science.

There is a good discussion on LinkedIn:

Davis Balestracci: Unfortunately, we snubbed our noses at the Six Sigma movement…and got our lunch eaten. Ron Snee has been developing this message for the last 20 years (I developed it in four years’ worth of monthly columns for Quality Digest from 2005-2008). BUT…as long as people have a computer, color printer, and a package that does trend lines, academic arguments won’t “convert” anybody.

Recently, we’ve lost our way and evolved into developing “better jackhammers to drive tacks”…and pining for the “good ol’ days” when people listened to us (which they were forced to do because they didn’t have computers, and statistical packages were clunky). Folks, we’d better watch it…or we’re moribund

Was there really a good old days when business listened to statisticians? Of course occasionally they did, but “good old days”? Here is a report from 1986 the theme of which seems to me to be basically how to get statisticians listened to by the people that make the important decisions: The Next 25 Years in Statistics, by Bill Hunter and William Hill. Maybe I do the report a disservice with my understanding of the basic message, but it seems to me to be how to make sure the important contributions of applied statisticians actually get applied in organizations. And it discusses how statisticians need to take action to drive adoption of the ideas because currently (1986) they are too marginalized (not listened to when they should be contributing) in most organizations.
Continue reading

Incentivizing Behavior Doesn’t Improve Results

In the webcast Dan Pink’s shares research results exploring human motivation and ideas on how to manage organization given the scientific research on motivation.

  • “once a task called for even rudimentary cognitive skill a larger reward led to poorer performance”
  • “Pay people enough to take the issue of money off the table. Pay people enough so they are not thinking about money they are thinking about the work.”
  • “3 factors lead to better performance: autonomy, mastery and purpose” [not additional cash rewards]
  • Open source software is created by highly skilled people contributing their time to collaborative projects that are then given away (such as Linux, Ruby, Apache). For large efforts their are often people paid by companies to contribute to the open source software but many people contribute 20-30, and more hours a week for free to such efforts, why? “Challenge, mastery and making a contribution”
  • “When the profit motive becomes unmoored from the purpose motive, bad thing happen. Bad things ethically sometimes, but also bad things like not good stuff, like crappy products, like lame services, like uninspiring places to work… People don’t do great things”
  • “If we start treating people like people… get past this ideology of idea of carrots and sticks and look at the science we can actually build organization and work life that make us better off, but I also think they have the promise to make our world a just a little bit better.”

The ideas presented emphasize respect for people, an understanding of psychology and validating beliefs with data. All of it fits very well with Deming’s ideas on management and the idea I try to explore in this blog. It isn’t easy to adjust your ideas. But the evidence continues to pile up against some outdated management practices. And good managers have to learn and adapt their practices to what is actually effective.

Related: Extrinsic Incentives Kill CreativityThe Trouble with Incentives: They WorkRighter IncentivizationIndividual Bonuses Are Bad Management

Video Overview of the PDSA Cycle

Robert Lloyd, PhD From the IHI Open School‘s, presents a nice overview of the PDSA Cycle (plan-do-study-act). The webcast includes an example of using PDSA to improve the discharge process for a hospital.

As I have said many times the keys to success are to turn the PDSA cycle rapidly, predict the results in advance, and analyze the results to continually improve. the Improvement Handbook is an excellent resource.

The IHI Open School is a great resource and exactly the type of thing organizations with a mission to improve performance should be doing. Provide resources online that are easy for people to access and then apply in their organization. See more management webcasts.

Related: Tom Nolan on PDSASaving Lives: US Health Care Improvement5 Million Lives Campaign

Extrinsic Incentives Kill Creativity

If you read this blog, you know I believe extrinsic motivation is a poor strategy. This TED webcast Dan Pink discusses studies showing extrinsic rewards failing. This is a great webcast, definitely worth 20 minutes of your time.

  • “you’ve got an incentive designed to sharpen thinking and accelerate creativity and it does just the opposite. It dulls thinking and blocks creativity… This has been replicated over and over and over again for nearly 40 years. These contingent motivators, if you do this then you get that, work in some circumstances but in a lot of tasks they actually either don’t work or, often, they do harm.”
  • there is a mismatch between what science knows and what business does
  • “This is a fact.”

What does Dan Pink recommend based on the research? Management should focus on providing workplaces where people have autonomy, mastery and purpose to build on intrinsic motivation.

via: Everything You Think about Pay for Performance Could Be Wrong

Related: Righter IncentivizationWhat’s the Value of a Big Bonus?Dangers of Extrinsic MotivationMotivate or Eliminate De-MotivationGreat Marissa Mayer Webcast on Google Innovation

An Introduction to Deming’s Management Ideas by Peter Scholtes (webcast)

An Introduction to Deming’s Management Teaching and Philosophy by Peter Scholtes – webcast from the Annual W. Edwards Deming Institute conference in Madison, Wisconsin, November 9th, 2008. My previous post on this speech: 6 Leadership Competencies.

Next month, the Annual Deming Institute conference will be held at Purdue on Oct 10th, 2009.

Related: Peter Scholtes’ LifeCurious Cat’s Deming on ManagementThe Leader’s HandbookPerformance without Appraisal

Unfortunately I cannot actually use the website to watch more than 5 minutes because the site fails to support linux operating system with their solution for longer videos. Google will only allow 10 minute videos without special permission – YouTube has not replied to my request for over 6 months. Update: Twitvid let me upload the whole video.

YouTube Uses Multivariate Experiment To Improve Sign-ups 15%

Google does a great job of using statistical and engineering principles to improve. It is amazing how slow we are to adopt new ideas but because we are it provides big advantages to companies like Google that use concepts like design of experiments, experimenting quickly and often… while others don’t. Look Inside a 1,024 Recipe Multivariate Experiment

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

via: @hexawiseMy brother has created a software application to provide much better test coverage with far fewer tests using the same factorial designed experiments ideas my father worked with decades ago (and yet still far to few people use).

Related: Combinatorial Testing for SoftwareStatistics for ExperimentersGoogle’s Website Optimizer allows for multivariate testing of your website.Using Design of Experiments

Bogus Theories, Bad for Business

The Wall Street Journal has a book review of The Management Myth by Matthew Stewart. The book flushes out the ideas Matthew Stewert explored in a previous article in the Atlantic about the failure of management to mature as a discipline.

Mr. Stewart quotes Bruce Henderson, the founder of the ­Boston Consulting Group, who describes consulting as “the most improbable business on earth” and who goes on to ask: “Can you think of anything less ­improbable [sic] than taking the world’s most ­successful firms, leaders in their businesses, and ­hiring people just fresh out of school and telling them how to run their ­businesses, and they are willing to pay ­millions of dollars for their ­advice?”

I’m not sure about the book, I have not read it but that is a great statement. And I firmly believe managers need to become experts at managing and by and large they have quite a long way to go. Dr. Deming talked about how we “know” what we know in the aspect of his management called the theory of knowledge (which is not included in any other management philosophy I have seen). That area (with interactions in other areas) explores why people often believe what is not so. And management seems to have a surplus of beliefs that are not based on sound theories.

Read this good article I have mentioned before on this topic by Carlie and Christensen: The Cycles of Theory Building in Management Research.

Related: Righter IncentivizationAnother Quota Failure ExampleManagement Advice FailuresWhy Extrinsic Motivation FailsInnovation StrategyDoes the Data Deluge Make the Scientific Method Obsolete?Data Based BlatheringDoing the Wrong Things RighterHarvard’s Masters of the Apocalypse
Continue reading

Toyota Develops Thought-controlled Wheelchair

Toyota has developed a thought-controlled wheelchair (along with Japanese government research institute, RIKEN, and Genesis Research Institute). Honda has also developed a system that allows a person to control a robot through thoughts. Both companies continue to invest in innovation and science and engineering. The story of a bad economy and bad sales for a year or two is what you read in most newspapers. In my opinion the more important story is why Toyota and Honda will be dominant companies 20 years from now. And that story is based on their superior management and focus on long term success instead of short term quarterly results.

Yes Toyota can improve their performance, based on the last few years. Does management understand what they need to do? I think so. Does management understand that the system needs to be improved rather than the numbers on the spreadsheets of various managers have to be made better? I think so. Do I think most companies today, with bad results, understand the difference between bad numbers on spreadsheets that are used to judge various managers and a system that needs to be improved? No.

I do not believe the bad earnings for the last year for Toyota are indicative of a failed system. The results do show a weakness in the Toyota system that allowed them to perform this poorly during this credit crisis. The risk to Toyota’s future is that they become too focused on short term results, mistakenly thinking the problem to be fixed in the bad quarterly results recently. They need to focus on improving the system for the long term. And the recent experience likely shows some areas that need to be improved. But in no way do the fundamental tenants of the management system need to be changed. For many other companies today, changing fundamental aspects of their management is what is needed.

Related: Toyota as HomebuilderHonda’s Robolegs Help People WalkHonda has Never had Layoffs and has been Profitable Every YearToyota’s Partner RobotNUMMI, and GM’s Failure to Manage EffectivelyToyota iUnitInvest in New Management Methods Not a Failing Company by William Hunter, 1986
Continue reading

Dr. Russell Ackoff Webcast on Systems Thinking

Dr. Ackoff is one of two management thinkers that any manager, that is serious about improving management results in their organization, should study (the other is Dr. Deming). There are plenty of others that are also great resources. From part 2 of his talk: “Why-questions, about objects called systems, cannot be answered by the use of analysis… The product of explanation is understanding… The product of analysis is how things work, never why they work the way they do. Explanations always lie outside the system, never inside it.”

Synthesis (thinking about systems) involves 3 steps: 1) what is this system of which this is a part of; 2) understanding the behavior of the containing whole; 3) identify the role of function of the system in question within the containing system. Every system is defined by its role in the larger system.

Related: posts on Russ Ackoff’s ideasAckoff’s New Book: Management f-LawsWrite Down PredictionsKnowledge Management – Management is Prediction

How to Create a Control Chart for Seasonal or Trending Data

Lynda Finn, President of Statistical Insight, has written an article on how to create a control chart for seasonal or trending data (where there is an underlying structural variation in the data). Essentially you need to account for the structural variation to create the control limits for the control chart. She also provides a Minitab project file. Both are available for download from the Curious Cat Management Improvement Library.

Related: Control Charts in Health CareCommon Cause VariationManaging with Control ChartsMeasurement and Data CollectionFourth Generation Management

Friday Fun: Correlation

Correlation doesn't imply causation

From the excellent xkcd comic.

Related: Correlation is Not CausationDoes the Data Deluge Make the Scientific Method Obsolete?Understanding DataTheory of KnowledgeWhat Makes Scientists Different :-) Dangers of Forgetting the Proxy Nature of DataSeeing Patterns Where None Exists

Embrace Diversity, Erase Uniformity

Guest Post by Jurgen Appelo, author of the Managing Software Development blog.

Five years ago, when I started working for my current employer, the entire organization (about 30 people) consisted only of 20-something white straight single males. The atmosphere was what you would expect from such an environment: conversations on football/soccer, lewd jokes, the smell of beer, and trash in every corner. In short, the perfect place to work, if you were a 20-something white straight single male.

Then the organization started changing. The subculture of 20-something white straight single males in our region could not keep up with the rapid growth of our company. And so the women arrived. And the married guys. And people with kids. And people older than 40. And people from all sorts of ethnic, religious, sexual, and disabled minorities. Before we knew it, the organization had grown to 200 people, and the group of 20-something white straight single males had dwindled to just another minority. And it’s still a great place to work, particularly for the large majority of people representing one or two minorities.

Diversity is Important
In biological ecosystems, genetic diversity is one of the most important principles. Biodiversity (the variation of species) is the most obvious form of it, but there’s also diversity within species themselves. Did you know that honey bees are slightly different from each other? That’s how they regulate the temperature in their beehives. When a hive gets too cold, the bees start huddling together, buzzing their wings. And when it gets too hot, the bees spread out and they start fanning their wings. Now, when the bees would respond to the same specific temperatures, they would all start buzzing or fanning their wings at the same time, resulting in a wildly oscillating temperature in the hive. Therefore, to improve stability, nature has made sure that the bees respond to different temperature levels. When the temperature rises, one by one the bees will start fanning their wings. And the more bees join in, the slower the temperature will rise, until it stops completely. Diversity among bees smoothes and stabilizes the temperature in the beehive.

Continue reading

Does the Data Deluge Make the Scientific Method Obsolete?

The End of Theory: The Data Deluge Makes the Scientific Method Obsolete by Chris Anderson

“All models are wrong, but some are useful.”

So proclaimed statistician George Box 30 years ago, and he was right. But what choice did we have? Only models, from cosmological equations to theories of human behavior, seemed to be able to consistently, if imperfectly, explain the world around us. Until now. Today companies like Google, which have grown up in an era of massively abundant data, don’t have to settle for wrong models. Indeed, they don’t have to settle for models at all.

Speaking at the O’Reilly Emerging Technology Conference this past March, Peter Norvig, Google’s research director, offered an update to George Box’s maxim: “All models are wrong, and increasingly you can succeed without them.”

There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

see update, below. Norvig was misquoted, he agrees with Box’s maxim

I must say I am not at all convinced that a new method without theory ready to supplant the existing scientific method. Now I can’t find peter Norvig’s exact words online (come on Google – organize all the world’s information for me please). If he said that using massive stores of data to make discoveries in new ways radically changing how we can learn and create useful systems, that I believe. I do enjoy the idea of trying radical new ways of viewing what is possible.

Practice Makes Perfect: How Billions of Examples Lead to Better Models (summary of his talk on the conference web site):

In this talk we will see that a computer might not learn in the same way that a person does, but it can use massive amounts of data to perform selected tasks very well. We will see that a computer can correct spelling mistakes, translate from Arabic to English, and recognize celebrity faces about as well as an average human—and can do it all by learning from examples rather than by relying on programming.

Related: Will the Data Deluge Makes the Scientific Method Obsolete?Pragmatism and Management KnowledgeData Based Decision Making at GoogleSeeing Patterns Where None ExistsManage what you can’t measureData Based BlatheringUnderstanding DataWebcast on Google Innovation
Continue reading

Multitasking Decreases Productivity

The problems with multitasking are becoming more and more well know, thankfully. Here is another article on the lower productivity multitasking produces – Multitasking Madness Decreases Productivity by Barbara Bartlein:

In a recent study by Eric Horvitz and the University of Illinois, a group of Microsoft workers took, on average, 15 minutes to return to serious mental tasks, like writing reports or computer code, after responding to incoming e-mail or instant messages. They often strayed off to reply to other messages or browse news, sports or entertainment web sites.

These findings are similar to those of David E. Meyer, a cognitive scientist and director of the Brain, Cognition and Action Laboratory at the University of Michigan. “Multitasking is going to slow you down, increasing the chances of mistakes,” said Meyer. “Disruptions and interruptions are a bad deal from the standpoint of our ability to process information.”

“Many people delusionally believe they’re good at this,” he says. “The problem is that we only have one brain and it doesn’t work that way. In reality, nobody can effectively do more than one remotely complicated thing at a time.”

Related: The Siren Song of MultitaskingMulti-Tasking: Why Projects Take so LongFlow (the opposite of multitasking)

Fairness Matters

Sense of Fairness Affects Outlook, Decisions

Burnout has been long associated with being overworked and underpaid, but psychologists Christina Maslach and Michael Leiter found that these were not the crucial factors. The single biggest difference between employees who suffered burnout and those who did not was the whether they thought that they were being treated unfairly or fairly.

Their research on fairness dovetails with work by other researchers showing that humans care a great deal about how they are being treated relative to others. In many ways, fairness seems to matter more than absolute measures of how well they are faring — people seem willing to endure tough times if they have the sense the burden is being shared equally, but they quickly become resentful if they feel they are being singled out for poor treatment.

If the sum is $100, for example, the first person might offer to give away $25 and keep $75 for himself. If the second person agrees, the money is divided accordingly. But if the second person rejects the deal, neither one gets anything.

If people cared only about absolute rewards, then Person B ought to accept whatever Person A offers, because getting even $1 is better than nothing. But experiments show that many people will reject the deal if they feel the first person is dividing the money unfairly.

Related: Obscene CEO PayRespect for People and Understanding PsychologyWhy Pay Taxes or be HonestThe Illusion of UnderstandingThe Psychology of Too Much Choice

Packaging Improvement

McDonald’s Branding Makes Food Tastier for Tots

Researchers at Stanford University have found that children tend to rate food that is wrapped up in McDonald’s-branded paper as tasting better than the same food wrapped in plain paper — a finding that suggests that even the youngest consumers are heavily influenced by advertising. The new study was released Monday in the Archives of Pediatric and Adolescent Medicine.

The study had 63 children, aged 3 to 5 years old, tasting five pairs of identical foods and beverages — one in McDonald’s wrapping and the other in unbranded packaging. The researchers then asked them a simple question: “Which one tastes better?” An overwhelming number of the children said the food in the McDonald’s wrapping was tastier.

Oddly enough, this applied even to vegetables and milk. Sixty-one percent of the children in the study preferred the taste of carrots and 54 percent preferred the taste of milk if they were reminded by the packaging that it came from McDonald’s.

This is another reminder that tackling problems directly is not always the best strategy. The packaging doesn’t actually change the taste, but really it is not the taste that is likely a concern but rather the perception of taste. To me this is very similar to the studies on people preferring wine they are told costs more.

Ignore psychology at your peril: in marketing and in management. Deming’s management system include 4 interdependent areas: understanding variation, systems thinking, theory of knowledge and understanding psychology.

Effects of Fast Food Branding on Young Children’s Taste Preferences (I think this is the study referenced in the article though it was published in August 2007 – John).

Related: Indian researcher shows most people do judge a drink by its containerMarketing in a Lean CompanyThe Psychology of Too Much ChoiceBe Careful What You Measure

Drug Price Crisis

In 2005 I posted about some of the problems with drug pricing. It is nice to find at least a couple of people at MIT that want to have MIT focus research on the public good instead of private profit. As I have mentioned too many universities now act like they are for-profit drug or research companies. That is wrong. Drug companies can do so, institutions with purported higher purposes should not be driven to place advancing science below profiting the institution.

Solving the drug price crisis

The mounting U.S. drug price crisis can be contained and eventually reversed by separating drug discovery from drug marketing and by establishing a non-profit company to oversee funding for new medicines, according to two MIT experts on the pharmaceutical industry.

Following the utility model, Finkelstein and Temin propose establishing an independent, public, non-profit Drug Development Corporation (DDC), which would act as an intermediary between the two new industry segments — just as the electric grid acts as an intermediary between energy generators and distributors.

The DDC also would serve as a mechanism for prioritizing drugs for development, noted Finkelstein. “It is a two-level program in which scientists and other experts would recommend to decision-makers which kinds of drugs to fund the most. This would insulate development decisions from the political winds,” he said.

I see their idea as one worth trying. Lets see how it works. Their book: Reasonable Rx – Solving the Drug Price Crisis by Stan Finkelstein and Peter Temin

Related: USA Spent $2.1 Trillion on Health Care in 2006Measuring the Health of NationsAntibiotics Too Often Prescribed for Sinus Woes$600 Million for Basic Biomedical Researcharticles on improving the health care system

  • Recent Trackbacks

  • Comments