Curious Cat Management Improvement Blog

Computer Network Operations Center Failures

John Hunter — Fri, 03 Jul 2009 19:06:57 +0000

Obviously many businesses are now dependent on computer Network Operations Centers (NOC). Some of these data centers can cause millions of dollars in lost sales each minute if they fail. So sound engineering, including off-site redundancy is critical. Authorize.net is a recent example of such a failure, Authorize.net Goes Down, E-Commerce Vendors Left Hanging

Payment gateway service provider Authorize.net has been down and out for several hours… That has big implications: since the service is used by tens of thousands of e-commerce vendors to accept credit card and electronic checks payments on their websites, it likely means millions are being lost during its downtime. PayPal and Google Checkout are still up and running.
…
A fire in Fisher Plaza, Seattle has cause a massive power outage causing leading IP-based payment gateway solution Authorize.Net to go down around approximately 11:15pm PST (last night). A traffic reporter for KOMO News that operates out of Fisher Plaza tweeted that a fire set off the sprinkler system which fried the generators.

From what I can piece together it seems within about 5 hours services were back up, at least partially. NOC failures are not uncommon (either due to fire, power failure [including backup systems], government raids, software glitches [not exactly the same as a NOC failure but some can have the affect of essentially knocking off a NOC from providing the specific service desired]). Evaluating these risks must be part of management systems with significant NOC dependencies.

Authorize.net set up a Twitter account and within hours has 2,500 followers. I am not a huge fan of Twitter, it is nice but seems pretty limited to me. But this is an example of using it effectively. You can follow me on Twitter @curiouscat_com.

Management Improvement Carnival #68

John Hunter — Thu, 02 Jul 2009 03:08:23 +0000

The Curious Cat Management Improvement Carnival began in 2006 with the goal to provide links to interesting blog posts for those interesting in improving the practice of management.

Reward: Creativity’s Forbidden Fruit by Matt May - “Kaizen does not attempt to light a fire under people. It lights the fire within them.”
Elegance and Encapsulation by Pete Abilla - “Encapsulation is an elegant and simple principle to ease the burden on your customer by subtracting or covering the unnecessary and adding the meaningful.”
Deming’s Theory of Knowledge by Marc Hersch - “Systems thinking comes down to developing methods and instincts for hearing the voice of the process, or if you will, the voice of the system. This is the opposite of the reduction that has become the common sense of by-the-numbers and just-the-facts thinking in Western enterprise.”
Virginia Mason’s CEO on Health Reform by Mark Graban - “The path to better quality and safety is the same as the path to reduced cost… Our system is so full of waste (non-value-added activities), need to systematically reduce and eliminate that waste”
Does good experimental design require changing only one factor at a time (OFAT)? by Mark J. Anderson - “Multifactor testing is far more effective for statistical power, screening efficiency and detection of interactions. Industrial experimenters are well-advised to forget their indoctrination in OFAT and make use of multifactorial designs.”
Getting More People Involved in Improvement by Lee Fried - “make sure that all leaders are getting out of the conference room and into the gemba to make sure that the appropriate checking and coaching activities are taking place.”
How clean is clean enough? by Ron Pereira - “In other words, the true purpose of this step is to clean to inspect.”
Seeking: Checklist for a Sense of Urgency by Jon Miller - “This is a delicate balance. We need to think long-term, but act each day with urgency.”
Toyota Develops Thought-controlled Wheelchair by John Hunter - “the more important story is why Toyota and Honda will be dominant companies 20 years from now. And that story is based on their superior management and focus on long term success instead of short term quarterly results.”

Toyota Develops Thought-controlled Wheelchair

John Hunter — Wed, 01 Jul 2009 02:23:05 +0000

Toyota has developed a thought-controlled wheelchair (along with Japanese government research institute, RIKEN, and Genesis Research Institute). Honda has also developed a system that allows a person to control a robot through thoughts. Both companies continue to invest in innovation and science and engineering. The story of a bad economy and bad sales for a year or two is what you read in most newspapers. In my opinion the more important story is why Toyota and Honda will be dominant companies 20 years from now. And that story is based on their superior management and focus on long term success instead of short term quarterly results.

Yes Toyota can improve their performance, based on the last few years. Does management understand what they need to do? I think so. Does management understand that the system needs to be improved rather than the numbers on the spreadsheets of various managers have to be made better? I think so. Do I think most companies today, with bad results, understand the difference between bad numbers on spreadsheets that are used to judge various managers and a system that needs to be improved? No.

I do not believe the bad earnings for the last year for Toyota are indicative of a failed system. The results do show a weakness in the Toyota system that allowed them to perform this poorly during this credit crisis. The risk to Toyota’s future is that they become too focused on short term results, mistakenly thinking the problem to be fixed in the bad quarterly results recently. They need to focus on improving the system for the long term. And the recent experience likely shows some areas that need to be improved. But in no way do the fundamental tenants of the management system need to be changed. For many other companies today, changing fundamental aspects of their management is what is needed.

Related: Toyota as Homebuilder - Honda’s Robolegs Help People Walk - Honda has Never had Layoffs and has been Profitable Every Year - Toyota’s Partner Robot - NUMMI, and GM’s Failure to Manage Effectively - Toyota iUnit - Invest in New Management Methods Not a Failing Company by William Hunter, 1986

If Toyota really wants to take a on a big improvement project they should deal with the dealer problem. The biggest weakness now (in my opinion) is not the quality of their cars, or the overcapacity, or pricing, or recent management decisions, or the model lineup, but that the dealers are not nearly as focused on customer value as Toyota is. Many still have the perverse incentive that the worse deal they provide customers the better financial performance the dealer gains. This short term success is harmful: it results in behavior that teaches customers Toyota (because they think of the dealer as Toyota) is trying to rip them off as much as they can instead of think of Toyota as trying to provide them the best value. Yes I realize it is a hard problem to deal with. Guess what, you should work on important problems even if they are hard.

“There are two kinds of companies, those that work to try to charge more and those that work to charge less” - Jeff Bezos, Amazon Founder and CEO

With the dealer network now, Toyota’s system include both aspects. Toyota Motor Company works to charge less the dealers work to charge more. That is not the best system.

Read more on the technology of the wheelchair on the Curious Cat Science and Engineering Blog.

The Myth of the Genius Programmer

John Hunter — Mon, 29 Jun 2009 12:36:35 +0000

Nice talk on fear of looking foolish. The speakers discuss the idea that visibility is good. Don’t hide. Make everything visible and the benefit from many people’s ideas. The talk focuses on software development but is true for any work.

“criticism is not evil” - Very true. “At Google we are not allowed to submit code until there is code review.” At the bottom line they are repeating Deming’s ideas: improve the system - people are not the problem, bad systems are the problem. Iterate quickly.

Google Innovates Again with Google Wave

John Hunter — Thu, 25 Jun 2009 16:47:10 +0000

Google Wave is a new tool for communication and collaboration on the web, coming later this year. They are developing this as an open access project. The creative team is lead by the creators for Google Maps (brothers Lars and Jens Rasmussen). A wave is equal parts conversation and document. People can communicate and work together with richly formatted text, photos, videos, maps, and more. You really have to watch to understand what it is.

This is a long webcast (1 hour and 20 minutes) and likely will be best only for those interested in internet technology solutions. But it also provides useful insight into how Google is managing the creation of new tools. But the ideas are not explicit (the demo was meant to present the new product Google Wave, not explain the thought behind producing useful technology solutions), so you have to think about how what they are doing can apply in other situations.

For software developer readers they also highly recommended the Google Web Development Kit, which they used heavily on this project. They also have a very cool context sensitive spell checker that can highlight misspelled words that are another dictionary word but not right in the context used (about 44:30 in the webcast). And they discuss using Wave to manage bug tracking and manage information about dealing with bugs (@ 1 hour 4 min point).

Very cool stuff. The super easy blog interaction is great. And the user experience with notification and collaborative editing seems excellent. The playback feature to view changes seems good though that is still an area I worry about on heavily collaborative work. Hopefully they let you see like all change x person made, search changes…

Top 10 Reasons Why Employees Leave in IT

John Hunter — Tue, 23 Jun 2009 14:52:14 +0000

Some of the problems expressed in the post linked to are specific to IT, and some are more important in software development (where as I have said before employees have higher expectations of management than most employees do), but many have truth for many employees. A good manager can create an environment where these problems are eliminated or reduced.

Top 10 Reasons Why Employees Leave in IT

No prioritization on items therefore constant interruptions in projects are the norm leaving projects unfinished due to a shift to “yet another project or task unexpectedly”
…
Boss doesn’t communicate things that affect the team or you as an individual and makes all decisions without your knowledge only you finding about it later through another source
…
Managers who fail to promote the very people who deserve it rather than who is popular or who they like
…
Bad co-workers who do not get stomped out (let go) and hurt the culture
…
Teams work best when they collaborate and are allowed to question what the proposed process or standard is, not just following and doing what is told 100% of the time. If the process suggested or currently ongoing sucks, question it and expect your team to question it!
…
Employee comes up with an idea and manager disregards it because “no I’ve always done it my way” even if it’s a 1999 way of doing things

Management Improvement Carnival #67

John Hunter — Sun, 21 Jun 2009 18:09:22 +0000

Not Invented Here by Nicole Radziwill - “when a NIH culture is observed, perhaps the resources and opportunities that are available to a group or an organization that could use them are truly invisible.”
Re-th!nk[ing IT strategy] by David Anderson - “if something is not strategic and we are not good at it then we should outsource it and buy the service instead. If we are good or world class at something but it is not strategic then we should spin it out and sell that service to our competitors.”
Let’s Teach Our Kids about Standard Work by Bryan Lund - “If we are going to nip this standard work thing in the bud, we need to go back to formula and start teaching our kids two things, 1) how to think about processes and 2) how to instruct.”
It Might Be Bad Design If Instructions Are Required by Mark Graban - “Sure enough, there was a full page sheet of laminated instructions on the desk for operating the lights”
HR and Innovation “HR must work diligently to make sure that right systems are in place to attract, identify, and capture the best talent to drive innovation.”
Learning from Community Medical Care Successes by John hunter - “The only way to get as bad results as we do for the huge cost is to keep doing what we are doing”

Visit the Curious Cat Management Improvement Carnival home

Community Medical Care Successes

John Hunter — Thu, 18 Jun 2009 16:42:54 +0000

The Cost Conundrum by Atul Gawande, New Yorker (The Power of a Checklist was published there in 2007 by the same author)

For example, Rochester, Minnesota, where the Mayo Clinic dominates the scene, has fantastically high levels of technological capability and quality, but its Medicare spending is in the lowest fifteen per cent of the country—$6,688 per enrollee in 2006, which is eight thousand dollars less than the figure for McAllen. Two economists working at Dartmouth, Katherine Baicker and Amitabh Chandra, found that the more money Medicare spent per person in a given state the lower that state’s quality ranking tended to be. In fact, the four states with the highest levels of spending—Louisiana, Texas, California, and Florida—were near the bottom of the national rankings on the quality of patient care.
…
I talked to Denis Cortese, the C.E.O. of the Mayo Clinic, which is among the highest-quality, lowest-cost health-care systems in the country. A couple of years ago, I spent several days there as a visiting surgeon. Among the things that stand out from that visit was how much time the doctors spent with patients. There was no churn—no shuttling patients in and out of rooms while the doctor bounces from one to the other. I accompanied a colleague while he saw patients. Most of the patients, like those in my clinic, required about twenty minutes. But one patient had colon cancer and a number of other complex issues, including heart disease. The physician spent an hour with her, sorting things out. He phoned a cardiologist with a question.

“I’ll be there,” the cardiologist said. Fifteen minutes later, he was. They mulled over everything together. The cardiologist adjusted a medication, and said that no further testing was needed. He cleared the patient for surgery, and the operating room gave her a slot the next day.

The whole interaction was astonishing to me. Just having the cardiologist pop down to see the patient with the surgeon would be unimaginable at my hospital. The time required wouldn’t pay. The time required just to organize the system wouldn’t pay.

The core tenet of the Mayo Clinic is “The needs of the patient come first”—not the convenience of the doctors, not their revenues. The doctors and nurses, and even the janitors, sat in meetings almost weekly, working on ideas to make the service and the care better, not to get more money out of patients. I asked Cortese how the Mayo Clinic made this possible.

“It’s not easy,” he said. But decades ago Mayo recognized that the first thing it needed to do was eliminate the financial barriers. It pooled all the money the doctors and the hospital system received and began paying everyone a salary, so that the doctors’ goal in patient care couldn’t be increasing their income. Mayo promoted leaders who focussed first on what was best for patients, and then on how to make this financially possible.

No one there actually intends to do fewer expensive scans and procedures than is done elsewhere in the country. The aim is to raise quality and to help doctors and other staff members work as a team. But, almost by happenstance, the result has been lower costs. [actually the Deming Chain Reaction]
…
The Mayo Clinic is not an aberration. One of the lowest-cost markets in the country is Grand Junction, Colorado, a community of a hundred and twenty thousand that nonetheless has achieved some of Medicare’s highest quality-of-care scores. Michael Pramenko is a family physician and a local medical leader there. Unlike doctors at the Mayo Clinic, he told me, those in Grand Junction get piecework fees from insurers. But years ago the doctors agreed among themselves to a system that paid them a similar fee whether they saw Medicare, Medicaid, or private-insurance patients, so that there would be little incentive to cherry-pick patients. They also agreed, at the behest of the main health plan in town, an H.M.O., to meet regularly on small peer-review committees to go over their patient charts together. They focussed on rooting out problems like poor prevention practices, unnecessary back operations, and unusual hospital-complication rates. Problems went down. Quality went up. Then, in 2004, the doctors’ group and the local H.M.O. jointly created a regional information network—a community-wide electronic-record system that shared office notes, test results, and hospital data for patients across the area. Again, problems went down. Quality went up. And costs ended up lower than just about anywhere else in the United States.
…
This approach has been adopted in other places, too: the Geisinger Health System, in Danville, Pennsylvania; the Marshfield Clinic, in Marshfield, Wisconsin; Intermountain Healthcare, in Salt Lake City; Kaiser Permanente, in Northern California. All of them function on similar principles. All are not-for-profit institutions. And all have produced enviably higher quality and lower costs than the average American town enjoys.

Hopefully those that have taken decades to get around to dealing with the health care crisis will listen to these success stories. I can’t understand how people keep saying we can’t afford to create a better health care system. We have the most costly system by far and we have worse result than most other rich economies. It seems we should expect better results for less money. The only way to get as bad results as we do for the huge cost is to keep doing what we are doing (or I suppose choose some option that does even worse than are very bad current system - though that seems a bit unlikely).

If Your Staff Doesn’t Bring You Problems That is a Bad Sign

John Hunter — Mon, 15 Jun 2009 14:44:14 +0000

The day soldiers stop bringing you their problems is the day you have stopped leading them. They have either lost confidence that you can help them or concluded that you do not care. Either case is a failure of leadership.

- Colin Powell

I discussed my feelings on this in a previous post, Bring Me Problems:

If an employee never learns how to find possible solutions themselves that is not a good sign. But it is much, much better to bring problems to managements attention than to fail to do so because they know the manager thinks that doing so is weak. It is the attitude that problems are not to be shared that is weak, in my opinion.

“Having no problems is the biggest problem of all.” - Taiichi Ohno

Management Improvement Carnival #66

John Hunter — Fri, 12 Jun 2009 14:54:57 +0000

The Curious Cat Management Improvement Carnival began in 2006 with the goal to provide links to interesting blog posts for those interesting in improving the practice of management.

Jidoka is not just “built-in-quality” by Jason Yip - “Jidoka is not just about stopping and notifying of problems immediately. It also includes the concept of separating human and machine work. Effectively the idea of using machines to free humans.”
How to Engage People in Kaizen by Jon Miller - “Frame all actions as experiments and not permanent or irreversible changes This allows people to think that they are not really changing something, only “trying it”. In fact if the method is demonstrably better, it may become the new way.”
To Estimate or Not, That is the Question - “Create your estimates quickly and don’t get paralyzed by precision. Being quick may be the middle ground between estimating and not estimating.”
Generation Y Deserves No Special Treatment by Chris Young - “Generation Y should not be treated any differently than any other generation. The very same employee performance standards applies to Generation Y as they apply to any other generation.”
Rapid prototyping in NY’s Times Square by Karen Wilhelm - Experience and observation can determine optimum flow of people (and vehicles now diverted from their usual routes). Just like simulating flow in a full-size cell mockup can allow teams to adjust reality to assumptions. No long meetings, proposals, and computer models can substitute for a simple and quick prototype of a new system.”
Stop Making Excuses by Nick McCormick - “Refuse to accept any reasons for why things that need to be done can’t be. It’s OK to identify the problems. The important thing is to develop solutions to those problems.”
Cool Email Mistake Proofing - My main desktop computer runs Ubuntu Linux. The default email client is called Evolution. A recent upgrade introduced a very cool feature. When I hit ‘Send’ it looks for language in the email that might indicate I meant to include an attachment. If there is no attachment, it pops up [a] handy reminder
Waste and SAP? by Kevin Meyer - “Moral of the story? Be careful of product demos.”

Curious Cat management web search.