P
parallax-scroll
http://venturebeat.com/2008/09/05/x...istory-of-microsofts-video-game-console-woes/
1.)
Xbox 360 defects: an inside history of Microsoft’s video game console
woes
Dean Takahashi | September 5th, 2008
When his fourth Xbox 360 video game console died in April, Chris
Szarek wasn’t surprised.
The Chicopee, Mass. gamer was accustomed to the hardware failures that
became known throughout the Internet as RROD, or the “red rings of
death” which flash when the console becomes inoperable.
A 40-year-old photographer, Szarek was a hardcore Microsoft fan who
spent more than $1,000 on his games. But each time one of his Xbox 360
consoles failed, he had to spend time convincing Microsoft’s tech
support that they should send him a new console. Each time he got a
refurbished console as a replacement (a machine that had been returned
to a repair center in Texas, fixed as much as possible, and then
shipped back out). When he complained on the Internet and to the media
about the shoddy product and poor customer service, people branded him
a cry baby and wrote him off as a statistical anomaly. But by the
spring of 2008, Szarek was vindicated. There were at least a million
or two other people like him.
Szarek’s fourth machine lasted almost two years, experiencing the same
short life that many other Xbox 360s suffered. Microsoft replaced
these machines for free under the warranty that it announced on July
5, 2007, for defective Xbox 360s exhibiting what it more politely
called the “three flashing red lights.” That warranty program cost
Microsoft up to $1.15 billion, but the loss of face and loyalty among
gamers in the fierce console war with Nintendo and Sony has been
immeasurable. Szarek, who became a spokesman for dispossessed
defective Xbox 360 owners, played a part in making Microsoft
acknowledge its console quality problem.
This is the unauthorized tale of how Microsoft lost its chance to
become the leader in the biggest market it has attacked beyond its
twin monopolies in Office and Windows software. Rival game console
maker Nintendo out-thought the larger players Microsoft and Sony by
designing the Wii game console with a clever, intuitive game
controller. Even so, Microsoft could have captured more gamers during
this product generation, yet the RROD problem held it back. The Xbox
360’s defect problem will go down as one of the worst snafus in
consumer electronics history.
Its own worst enemy
Microsoft knew it had flawed machines, but it did not delay its launch
because it believed the quality problems would subside over time. With
each new machine, the company figured it would ride the “learning
curve,” or continuously improve its production. Even though
Microsoft’s leaders knew their quality wasn’t top notch, they did not
ensure that resources were in place to handle returns and quickly
debug bad consoles. There were plenty of warning signs, but the
company chose to ignore them. The different parts of the business
weren’t aligned.
It reminds me of the German war machine just before World War I, as
chronicled by Barbara Tuchman in the classic history book, “The Guns
of August.” The German generals were intent on keeping their trains on
time; but the leaders overlooked their chances for stopping the war
altogether. The Schlieffen plan called for them to strike first. Once
the Russians and French mobilized, the Germans had to move into
action. They marched off blindly into tragedy.
Likewise, Microsoft’s strategy depended on beating its rivals to
market. It couldn’t afford to stop and delay the launch in order to
solve its quality problems, or so upper management believed. What
Microsoft’s leaders didn’t realize was that getting to market first
with a flawed machine would only win them a battle; and it risked the
loss of the war.
“They got enamored with the idea of the Microsoft army rolling
everything out at the same time,” said one knowledgeable source who
asked not to be identified.
The quality problem negated much of the advantage of going first, and
it has delayed the company’s plan to aggressively market the console
and slash its prices. (Microsoft disputes this point; it cut the price
of all three versions of its Xbox consoles by $50 to $79 on Wednesday.
And the company believes it will sell more boxes than Sony will. But
prices ought to be lower still during this stage of the console life
cycle). That has stopped the company from reaching the broader market
of consumers that Nintendo has won over. It has lowered its ambitions,
hoping instead just to get a clear edge on third-placed Sony. The
future profits that the company once hoped for are now likely to wind
up in Nintendo’s pockets.
Microsoft’s top game executive, Robbie Bach, president of the
Entertainment & Devices group, said at a dinner in July that
Microsoft’s own research shows that gamers have largely forgiven the
company for defective Xbox 360s. Microsoft has still sold more Xbox
360 consoles than Sony to date. But there is no doubt that the company
has lost considerable good will among gamers. Before Microsoft offered
free replacements, connsumers grumbled that they had to turn to
forums, such as those on Ars Technica, to vent and to find solutions
to problems that the company didn’t openly discuss. And for a couple
of months now, Sony’s PlayStation 3 has been outselling the Xbox 360
in the U.S. for the first time.
“Fundamentally, their thinking shows that they are a software company
at heart,” said one veteran manufacturing executive. “They put
something out and figure they can fix it with the next patch or come
up with a bug fix.”
The terrifying part of the story is that this kind of problem — where
technology fails and no one knows what to do about it — can happen to
any company.
About this story
I asked Microsoft to confirm or deny 35 different facts contained in
this story. Instead, I received a formal statement from a Microsoft
spokesperson, saying the company had already acknowledged an
“unacceptable number of repairs” to Xbox 360 consoles and responded to
the hardware failures with a free replacement program. The statement
also said, “This topic has already been covered extensively in the
media. This new story repeats old information, and contains rumors and
innuendo from anonymous sources, attempting to create a new
sensational angle, and is highly irresponsible.”
I don’t think this story is sensational. I have tried to verify the
facts over several years. I view this story as the last chapter for my
book on the making of the Xbox 360, “The Xbox 360 Uncloaked: The Real
Story Behind Microsoft’s Next-Generation Video Game Console.”
The facts revealed themselves slowly, emerging from the day-to-day
stories that I wrote about the game business. Some people might
consider this post mortem to be ancient history. But the
reverberations are still playing out today. They help explain why
Microsoft isn’t being aggressive with its price cuts and why gamers
aren’t getting bargains on hardware as they did the last generation.
While I talked to many people for this story, few were willing to let
me use their names. As you will see, not every source is anonymous,
and we have included the viewpoint of Microsoft executives from past
interviews.
The details are interesting because they offer a deeper look into how
the console business runs than is otherwise available. Microsoft, for
instance, still hasn’t perfected its Xbox 360 manufacturing process.
In the absence of a precise chronology from Microsoft, some anonymous
sources have tried to describe what happened. But the history of the
decision making and inside story of what happened on the RROD has
never been told, until now.
2.)
Microsoft’s mea culpa
In many ways, the Xbox 360 was a big achievement. Microsoft had beaten
its rivals Sony and Nintendo to market by a year. Sony had outsold
Microsoft five-to-one in the previous generation of consoles, but
Microsoft has outsold Sony’s PlayStation 3 with the Xbox 360.
Microsoft’s to game executive Bach (left) had repeatedly promised that
Microsoft would be profitable in his Entertainment & Devices group in
the fiscal year ending June 30, 2008. The company met that target. Big
losses have turned into profits.
But when the company launched its Xbox 360 video game console in
November, 2005, it didn’t have a handle on product quality and it was
not prepared to systematically analyze its product returns and debug
bad consoles.
Microsoft has admitted those mistakes. On July 5, 2007, the company
said it would take a billion-dollar write-off to pay for free
replacement of Xbox 360s by up to three years from the date of
purchase. Peter Moore, who at the time was head of the games business
at Microsoft, said in an interview the decision to take the write-off
was due to an “unacceptable number” of returns. The company made
improvements to the console’s quality, but Moore said it was unclear
how many units would fail in the field. Microsoft then followed up its
new policy with numerous public apologies, including a statement from
Bach, (pictured above).
Microsoft never disclosed its actual return rates. But according to
data obtained by VentureBeat, the total number climbed above 1.2
million consoles in early 2007. That is a huge amount, considering
Microsoft had only shipped 11.6 million into stores by the time of the
announcement in mid-2007.
The company was dragged kicking and screaming to its admission of
widespread defects — with gamers and the press doing the dragging. In
interviews in April and May of 2007, Microsoft executives denied that
quality problems were haunting them. Todd Holmdahl, (pictured left)
general manager in charge of the Xbox platform for Microsoft in
Redmond, Wash., said in an interview with me on May 9, 2007, while I
was at the San Jose Mercury News, that the “vast majority” of
Microsoft’s Xbox 360 customers were having a great experience with
their consoles. He declined, however, to say what percentage of
consoles were being returned for defect reasons. In doing so, he was
as coy as any of the console makers are about such sensitive data. But
everyone wanted an answer to that question.
It’s not clear why it took Microsoft so long to announce its warranty
extension last year, almost 19 months from the time when the first
complaints arose at the outset of the launch. As it was investigating
the cause of the problems, the company stayed quiet, acknowledging
problems only little by little. The company delayed its free-
replacement announcement until it understood the scope of the problem.
In July, game executive Bach said that the number of returns was not
measurably big until a year after the launch. It was a problem that
became evident only over time, he said.
“It wasn’t related to things we were seeing in testing or some
judgment call we had to make about whether the product was ready,”
Bach said at the dinner. “We were confident the product was ready. We
did a lot of testing. The problem that shows up with the three red
lights on the console is a complex interaction with some very complex
parts.”
Ignoring warnings
But the evidence for the quality debacle was there to see even before
Microsoft shipped any machines. In August, 2005, as Microsoft was
gearing up production, an engineer raised a hand and said, “Stop. You
have to shut down the line.” This wasn’t just a brief moment. The
engineer spoke up repeatedly.
That engineer, who asked not to be identified, had deep experience in
manufacturing. When production results were really off kilter,
stopping a line and tracing a problem back to its roots was the
answer. But the higher-ranking engineers, managers and executives
chose to risk going forward. There wasn’t a universal backlash from
the engineering ranks, according to one engineering source.
Nobody listened to that engineer — who spoke on condition of anonymity
— apparently because console launches are always hurried affairs.
Yields — the percentage of working products in a given batch of total
products produced — generally start low. As the manufacturers conduct
statistical analysis and tight controls on every step in assembly,
they learn how to drive the yields up.
Still, the picture wasn’t pretty. The defect rate for the machines was
an abysmal 68 percent at that point, according to several sources.
That meant for every 100 machines that Microsoft’s contract
manufacturers, Flextronics and Wistron, made at their factories in
China, 68 didn’t work. At the recent dinner, Bach denied that there
was a big concern about defects at that point in time.
At that point, it is likely true that the engineers weren’t raising
enough red flags for the executives to pay attention. Early yields on
electronic goods are almost always lousy. Those veteran engineers
figured that they would be able to debug the problems and bring the
yields up quickly. But the expected rapid improvement in quality just
didn’t happen. The communication between upper management and the
engineers wasn’t clear. Nor was the strategy aligned between marketing
and customer support.
There were plenty of warning signs. Early reports on the problems were
myriad. In an Aug. 30, 2005 memo, the team reported overheating
graphics chip, cracking heat sinks, cosmetic issues with the hard disk
drive and the front of the box, under-performing graphics memory chips
from Infineon (now Qimonda), a problem with the DVD drive, and other
things. At that point, the contract manufacturers were behind schedule
and had only built hundreds of units. They were supposed to have been
in high gear, making thousands every week.
The illusion of a well-oiled machine
Yet around that same time, in mid-August 2005, Microsoft gave me a
tour of the hardware test center in Mountain View, Calif. Hundreds of
boxes were being tested in heat chambers and test labs. Leslie Leland,
an engineering manager, said all was on schedule. Microsoft wanted to
convey its own confidence to its partners, game developers, and
gamers. To my untrained eye, it looked like a smooth operation.
But on the inside, it was a frenetic time. The initial yield on the
most critical chip, the three-core microprocessor designed and
manufactured by IBM, was only 16 percent. For every 100 produced, only
16 worked. This low yield was surprising because a plaque on the wall
in Microsoft’s campus in Mountain View proclaimed that IBM had “taped
out,” or completed the design, of its first microprocessor prototype
on Dec. 8, 2004. The timing of the tape out was right on schedule,
which was good because it is normal for six months to a year of
debugging to follow a tape out. IBM had an easier time than the maker
of the graphics chip.
ATI Technologies (a graphics chip maker that has since been acquired
by Advanced Micro Devices) was late to deliver working graphics chips,
or GPUs. Like IBM, it had promised to deliver one of the most complex
chips it had ever made in about half the time it usually took to
create such chips. But ATI only had half the number of the 400
engineers IBM had on the project.
The fact that both companies had designs done at all was the result of
a Herculean effort. Microsoft’s engineers started working on the Xbox
360 at least a year after Sony’s engineers began work on the
PlayStation 3, yet Microsoft wound up shipping a year ahead of Sony.
Everything Microsoft did was under time pressure, including the
creation of the IBM microprocessor and the ATI graphics chip. The only
way to pull it all off was through a virtual organization, where
Microsoft outsourced many of the tasks to companies that specialized
in particular tasks. Microsoft’s hardware engineers in Redmond, Wash.
and Mountain View, Calif. were the master integrators of all of the
suppliers. The Microsoft chip designers in Mountain View also designed
an important video processing chip themselves.
Considering all of the work, Microsoft had too few hardware people.
Some of the designers of the Xbox 360, including engineering chief
Greg Gibson, were stretched thin. Gibson and J Allard (pictured left),
who led the console design effort, had begun work on Zune, Microsoft’s
portable media player. Top brass had approved the project to dethrone
Apple’s iPod, but Microsoft kept Zune secret from the outside world
until much later. Some engineers were pulled of the Xbox 360 at a
critical moment to join Allard’s effort to create a music player.
Those who were left to work on the test team worked around the clock,
traveling to China to work in the factory.
3.)
Seasoned hardware team executed well on the original Xbox
Outsiders frequently slam Microsoft’s hardware engineers as having a
software (fix it later) mentality. But Microsoft’s engineers had been
trained to put quality first. Most of them had years of training.
Holmdahl, for instance, had been with Microsoft in the early days of
its hardware division. By the time he joined the Xbox project, he was
a veteran, having worked with contractors such as Flextronics to make
20 million mice per year. Making consoles was an altogether different
task, but the team had expert help through its Web TV acquisition,
which brought veteran console hardware engineers aboard in 1998. One
of the best at debugging was Nick Baker, one of the architects of the
Xbox 360’s chips.
In 2001, the team went through the paces making sure that the original
Xbox had good quality. On that launch, Microsoft was 20 months behind
Sony’s launch of the PlayStation2. Microsoft’s original Xbox team
moved as fast as they could, but they took a long time defining the
box and picking vendors.
By the time they hired Nvidia in March 2000 to make the graphics chip,
they were hopelessly behind. Nvidia combined two different chips into
one in half the time it took to design a chip, but it fell behind
schedule in part because of a bug. Someone else had specified a sub-
par power supply for the Xbox, and that made the graphics chip hiccup
now and then. Once the team fixed the problem, they raced ahead.
But Flextronics, the contract manufacturer, wasn’t ready. It had to
build a bigger manufacturing database to accommodate the design.
Microsoft had to postpone its launch by a week. It also decided to
launch only in the U.S. market and would wait until the next spring to
launch in Europe and Japan. No console maker had ever pulled off a
worldwide launch before, and now Microsoft understood why. The delays
in other territories put it even further behind Sony. Sony eventually
outsold Microsoft by more than five to one — scarring into memory a
tough lesson about market timing.
But the Microsoft hardware team had come out shining. Aside from a
problem with a supplier’s DVD drive, the number of defective Xboxes
was low. After the DVD drive issue was fixed, yields rose above 90
percent and stayed there for the duration of the manufacturing.
Microsoft thus had a seasoned team to work on the Xbox 360.
Lessons of round one drove grand strategy for the Xbox 360 business
Despite the good quality manufacturing, Microsoft’s game business was
deeply flawed in the Xbox generation. In the second round, Microsoft
vowed to learn from its mistakes.
In the first generation, the company lost $3.7 billion over four
years. Most of that was because the costs of the box — particularly
its hard disk drive — were too high. Microsoft’s planners thought the
hard drive would give them an edge over Sony’s PS 2. It didn’t.
Bill Gates didn’t really care about the losses in the first
generation. That was simply the ante for getting into an exciting new
business. Microsoft has always viewed its ambitions in the game
business as strategic. It isn’t in the business just to dominate video
games, but to own the living room. The game console is a kind of
gateway to the Internet and all of the future entertainment services
that can be piped into the living room.
But Steve Ballmer, who took over from Gates as CEO during the first
generation, really wanted the Xbox business to be profitable second
time around. He wanted to keep the foothold in the living room, but
not at the expense of draining Microsoft’s cash, which was needed to
fight Google and other rivals.
This time, Gates said he felt that Microsoft would have a fundamental
cost advantage over Sony. One of the key ways Bach hoped to outwit his
rivals was with speed: getting a machine to market in four years
instead of five. Secretly, Microsoft had planned on selling 50 million
Xbox 360s at that point.
The Xbox team met for a series of retreats. One such retreat took
place in early 2004 at the elegant Salish Lodge beside the Snoqualmie
Falls made famous in the Twin Peaks TV show. After that meeting, Bach,
who had headed the Xbox business as its senior executive from the time
of its inception, decided that company would aim for profitability
with the Xbox 360. As much as possible, the machines would not be sold
for a loss.
Bach believed the hardware itself should be profitable over the life
cycle of the system. His planners counted on good revenues from
accessories and games to ensure profits the second time around. And,
in the three-page paper that outlined the grand strategy, he said that
Microsoft wanted to hit the market at the same time as its rivals.
Microsoft’s best guess was that Sony would launch the PS 3 in 2005.
That turned out to be a year off, since Sony ran into its own
development problems related to its decision to put Blu-ray drives
instead of DVD drives into the PS 3.
Microsoft had more preparation for Xbox 360
There was more time to design the box than the first-generation team
had, but it was still a race. Different teams sped into planning. The
different groups included chip designers, hardware planners, game
designers, software operating system engineers, marketers and
industrial designers.
With the Xbox 360, Microsoft was guilty of packing a little too much
into its box. The company sent out its scouts in late 2002 to ask
gamers what they wanted. The first Xbox came off as “The Incredible
Hulk.” The second Xbox, Microsoft had mandated, had to be smaller, if
only to fit in the smaller homes in Japan, a key target market. The
designers thought it had to be more like Bruce Lee. Microsoft’s own
user-experience team ran the show, while outside industrial design
firms competed in a bake-off to define the soul of the machine.
Hers Experimental Laboratory of Japan and Astro Studios in San
Francisco won the bake-off. They wanted to create something that
looked elegant and iconic, qualities that were sometimes at odds with
the computer that it was. Those firms gave the box its signature look
— curved sides that looked as if the box were inhaling air and was
about to explode with energy (that was the marketing image, anyway).
There was a constant tug of war between the industrial designers, who
wanted something small, and the engineers, who wanted to pack a lot of
performance into the box. The more powerful the system was, the bigger
the box had to be to provide air flow for heat-sensitive components.
Microsoft decided late to add a hard disk drive to most of the
machines. It also came up late with a plan to add wireless
controllers; all of the previous consoles shipped with wired
controllers. The hard drive blocked a lot of the air flow on one side
of the machine. And the wireless modules had to have enough of their
own space to ensure that there was no electrical interference. In the
end, the machine was a series of compromises. The gigantic power
supply needed to run the machine sat outside in the power cord, while
the console shell was poked full of holes on the sides to ensure air
flow.
“It turned out in the end that this was all going too far, too fast,”
said one source. “They were adding too many features after things were
locked down. That incremental feature adding just made it fragile.”
The marketers also decided the box had to be backward compatible with
the older games, including “Halo” and “Halo 2.” And they also felt the
box had to have enough horsepower to make games look beautiful on
high-
definition TVs.
Those parameters, in turn, gave direction to the chip and hardware
designers. This time, hardware chief Todd Holmdahl wanted Microsoft to
have much more control over its fate. And he believed he had built up
a team that could act as the “system integrator.”
Microsoft had a team of 100 chip engineers and 100 more hardware
engineers between Redmond and Mountain View. They were the overseers
of the integration of the parts into a working console. It was a
relatively small team for such a large project compared to a hardware
company such as Sony, but Microsoft had plenty of help from its
partners.
In contracts with IBM and ATI, Microsoft set early deadlines for
delivery of the first working chips. It also decided that it wanted to
own the chip designs. That way, if it wanted to, it could take the
designs and have them made by a factory of its choosing and wasn’t
locked into using IBM and ATI (or ATI’s chosen partner) chip
factories. The reason for this change was that Microsoft wanted to
avoid dependence on its chip partners. It had learned the hard way
after getting into a bad dispute with Nvidia over the prices charged
for the graphics chip in the original Xbox. The case had to go to
arbitration and Microsoft ultimately lost that battle. However, this
was a mistake: it was now responsible for failures related to the
chip.
In Microsoft’s favor this time was the fact that it had credibility
among game developers thanks to hits such as “Halo,” (pictured right)
which sold over 10 million units. Microsoft believed it could charge
more for each machine and more for its games; and not every machine
would have an expensive hard drive. In the second generation,
Microsoft planned on reducing the cost of the hardware over time.
Contract manufacturing had matured since the last generation.
Flextronics was willing to make each console for just $10 each, while
another manufacturer, Wistron, would do the same for $7, according to
sources. They hoped to make profits on large volumes of machines sold.
They were also practiced in the art of reducing the costs of the
components they put into their machines. Overall, Microsoft had a much
more formidable hardware operation, when you consider all of its
virtual resources, on the second time around. But it still wasn’t
enough to do a perfect job.
4.)
Go or no go?
Leading up to the launch in the fall of 2005, the number of defective
units would soon grow to tens of thousands. Any other consumer
electronics company would likely have postponed a launch with such low
yields. But Microsoft had more money in the bank than anyone else. The
decision this time would fall to Bach and Moore (pictured left). The
costs of launching with low yields — where you take big losses on
every product sold — could bankrupt other companies. But Microsoft
could afford to do so. Microsoft did delay the launch date from
October until November. But some inside the company still believed
returns would be out of control.
Some of the defects were “latent.” That means they may not show up for
some time after the consumers starts playing with the machine. Latent
defects can be as high as 50 percent of all defects. That means if you
have a low yield to start with, you expect to have a lot of returns
when the latent defects kick in later on. Bach said that the real
problems with the Xbox 360s didn’t really show up until a year after
the launch. That’s when returns were mounting, when they should have
been declining.
The yield problem was only discussed internally, and so the public at
large was left wondering whether Microsoft was intentionally creating
a shortage of consoles by making just a small number of machines. The
truth was that Microsoft had to produce a lot of units — many of which
failed — to get working consoles that it could ship. It was trying to
get as many machines to the market as it could.
A bad decision on testing machines?
The problems began before the Xbox 360s got to the market. The testing
machines were not ready, and the battery of tests that they ran were
not fully developed. That meant that the testing machines would
inspect the Xbox 360s coming off the line and approve them for
shipment, even though there were likely flaws.
The test machines were not properly debugged, due to an ill-advised
cost-cutting initiative that shaved $2 million from $25 million paid
to Cimtek, a test machine maker in Canada. The Microsoft team decided
not to pay the consulting fee to Cimtek to build, manage and debug the
test machines. Sources familiar with the matter said there were only
about 500 test machines at the time of launch, a third of the 1,500
needed.
“There were so many problems, you didn’t know what was wrong,” said
one source of the machines. “The [test engineers] didn’t have enough
time to get up and running.”
The Xbox 360’s yield climbed somewhat throughout the fall of 2005, as
Microsoft readied the worldwide launch in November. But the “first
pass yield” (before machines were taken off the line to be reworked
inside the factory) was never over 70 percent. Microsoft was counting
on its virtual organization of chip designers, assembly houses, and
Chinese factory workers to solve the problems. It considered this low-
cost infrastructure to be one of its main advantages over Sony, which
had high-cost operations under its own roof. But the Chinese
manufacturing machine wasn’t as honed as it is today.
If it had shut down the line until it properly debugged everything,
Microsoft might have had to delay the launch in Europe or Japan. The
Japanese market was considered important to crack, but the games being
developed for that market were late anyway. But Robbie Bach had
repeatedly said that it was strategically important for Microsoft to
fulfill its promises.
If Microsoft had an outstanding test process, then the problems with
the console might have been more glaring and Bach might have had more
reason to delay.
A tough but successful launch
On Nov. 22, 2005, Microsoft started selling the consoles. It invited
3,000 gamers to come to a hangar in the middle of the Mojave Desert.
Gamers rushed into the hangar doors like Greeks rushing into Troy.
There, they played the first games for 30 hours and were rewarded with
the chance to be the first to buy the machines. Across the country,
gamers lined up at stores and waited for midnight. They took them home
and obsessively played games like Activision’s World War II shooting
game, “Call of Duty 2.”
The hype around the launch was so strong that gamers snapped up every
machine. Microsoft had hoped to sell millions, but it was running
short in Europe and the U.S., while machines in Japan were barely
moving off the shelves. By the end of 2005, Microsoft had only shipped
1.5 million consoles worldwide. That compared to 1.4 million shipped
in 2001 with the original Xbox, when it launched in only the U.S.
There was now much more demand. Consumers were understandably upset
with the relative shortage, since an equal number of units was now
shared across three major territories. In Japan, the lack of home-
grown Japanese games to go with the Xbox 360 consoles meant that
demand was tepid; machines sat unsold in stores even as consumers in
the U.S. were paying hundreds more beyond retail on auction sites such
as eBay.
The consoles started failing almost immediately. Consumers were
excited about being the first on their block to own the new system.
They could run graphically beautiful games on high-definition flat-
screen TVs, which were just becoming affordable. But their
disappointment upon encountering the red rings of death was evident in
posts on the Internet. Some customers swore off Microsoft. At the
time, Microsoft said that it had received “isolated reports” of
console failures and that returns were within the normal range.
Many consumers complained that Microsoft’s customer support people
often tried to talk them out of doing a return. (The support people
could come across as condescending. They would caution users to avoid
putting the console on a plush carpet, or inside a home entertainment
center with no air flow). It would be many months before the company
set up an easy process where customers could track their returns and
replacement units on the web.
According to the Consumer Electronics Association, the average return
rate for products where the consumer gets their money back is about 2
percent. That is based on a study conducted in 2005 which showed the
return rate for any consumer electronic product is fairly constant,
said Steve Koenig, a senior analyst at the CEA.
According to Microsoft’s internal data, the assumption for the long
term was that 6 percent or 7 percent of the consoles would be
defective before shipping into the market, meaning the yield would be
about 94 percent or 93 percent. The rate of return was expected to be
low as well. But even after the holidays, Microsoft continued to
struggle to ship enough units.
Good quality is taken for granted as a requirement for doing business
these days, but it is still difficult to assess what is considered a
“normal rate of product returns or yield rates,” said Shukri Souri, a
manufacturing expert at Exponent, a Menlo Park, Calif. company that
figures out why products fail.
“A significant factor that determines rate of return is how complex a
product is to build,” Souri said.
The Xbox 360 was in fact more difficult to build than anything the
team had built before. It had 1,700 parts, many of them hard to make.
There were more than 200 suppliers. The chips and the box (insides
pictured left) were designed to balance performance, aesthetics and
power dissipation. As Holmdahl pointed out, a problem with any one of
the parts or any one of the vendors could result in a problem that
affected the whole box.
In 2006, it only got worse
Despite the problems, some results were turning out better than
expected, such as how quickly Microsoft would sign gamers up for its
Xbox Live online games service. (That figure has now hit 12 million
subscriptions for both paid and unpaid subscriptions, more than
originally forecast).
But some things, Microsoft clearly didn’t plan for. The complaints of
shortages and problems with consoles continued into the spring of
2006. By the end of March 31, Microsoft said it had shipped more than
3.3 million consoles to retailers. There was a growing “bone pile” in
a warehouse at Wistron and at a repair center in Texas.
Microsoft had more than 500,000 defective consoles that sat in
warehouses. They were either duds coming out of the factory or they
were returned boxes, according to inside sources. The yield was
climbing, but far too slowly. The company stood by its statement that
returns were within “normal rates for consumer electronics products.”
In the spring of 2006, Moore said with some relief that Microsoft was
finally readying a surge of new consoles for the market. Celestica, a
third manufacturing partner, was bringing its factory online. It
seemed that Microsoft was finally turning a corner on production.
Because Sony and Nintendo didn’t have their new machines ready yet,
Microsoft had a wide-open field in 2006. It was racing to snare the
most customers before the rivals could compete. But the cheaper priced
and older PlayStation 2 continued to outsell the Xbox 360 throughout
2006.
Customer complaints gather steam
Szarek, the previously mentioned resident of Chicopee, Mass. gained
some notoriety when he said that he suffered three failed consoles by
May, 2006. His first console had lasted only 28 days after he picked
it up. Then he became even more frustrated with the returns process.
Under the glare of publicity, Moore personally apologized to Szarek
and made sure he got a free game and wasn’t charged for the repairs.
Moore’s comments were conciliatory and he managed to calm some of the
critics. One of the problems: though the return rate was predictably
high, Microsoft had not made big investments in its return
infrastructure to enable a smooth return process. One PR woman said
she had to do shifts doing support calls.
“When you have tens of millions of people who own something, you’ll
have people using the Internet as a forum,” Moore said at the time.
“I’m not hiding behind the fact that people have had problems with the
console and at times we have been deficient with the way we’ve handled
them. We are improving our customer service. There were times when we
did not handle complaints the way we should have. I take
accountability for that. I have not seen since that any further
issues.”
The denial of widespread problems infuriated Szarek and others who
felt Microsoft was stonewalling. As the inventory surge materialized,
Microsoft was able to send replacement units out more quickly and deal
with those who were unhappy about the shortage. Sales topped 5 million
by June 30, 2006. Moore had personally set a goal of selling 10
million machines before his rivals sold their first machines. That was
going to be tough, since Sony and Nintendo had set their launches for
the fall of 2006.
5.)
The internal response
During the spring of 2006, the engineers were working on a transition
to a new motherboard, code-named Zephyr. But they postponed that
transition to work on fixing the “bone pile” issues and “maximize the
yield,” according to an email circulated by engineering manager Harjit
Singh to the hardware team on March 10, 2006. The yield at that point
was an abysmal 50 percent on the first pass. When the bad machines
were reworked within the factory, the yield went up to 75 percent –-
hardly acceptable. Singh had said the process in the factories was
“not repeatable” at a time when they were scheduled to triple
production in the coming months. That meant that something could go
slightly wrong and Microsoft would have no idea how to fix the
problem.
Microsoft finally started throwing more engineers at the problem. It
took engineers off projects such as reducing the costs of its wireless
controllers because, Singh wrote, “if we don’t have the consoles, we
don’t need the peripherals.” Marc Whitten was temporarily assigned to
work on evaluations for adding new heat sinks to address thermal
issues on the big chips. It was essentially “all hands on deck” for
engineers, who were expected to devote 75 percent of their time to the
yield/bone pile issues. But, again, those engineers didn’t achieve
magical fixes. And it was late in the game.
The company pushed to stir up sales. Because Sony appeared to be
falling off schedule a second time, the Microsoft executive team
challenged the engineers to improve shipment targets by 25 percent
through June, 2006. The team decided to postpone the launch of Zephyr
(a board with HDMI connector for better video quality) and concentrate
on shipping Xenon boards. The target was to hit 80 percent first-pass
yield, but that wasn’t reaslistic.
During the fall, Microsoft offered bundles, or packages of games and
consoles together, which effectively amounted to a price cut. Some
analysts accused Microsoft of “stuffing the channel,” or shipping more
consoles to retailers than they wanted or needed for the holidays.
Microsoft denied those claims.
“We’re a very responsible company,” Moore said to me in the May, 2007
interview. “How do you determine who stuffed the channel? What is the
optimal number of units? We’re not in a position to force the product
upon the retailers. It wouldn’t be the console wars without conspiracy
theories.”
But problems with manufacturing continued. The yield on the consoles
was only 85 percent at the beginning of 2008, not at the 90-percent
plus level that was expected at this point in the console cycle. That
meant there was an extra billion dollars in cost that Microsoft hadn’t
anticipated. But, again, Moore at the time said that the manufacturing
problems Microsoft had were not enough to ruin the business model.
Dead Rising
Robert Delaware, a former game and hardware tester at one of
Microsoft’s contractors, Redmond, Wash.-based VMC, saw the problems
popping up in the spring of 2006 as he was testing consoles and new
games. Like many testers, he was in the dark about the overall
problems and only saw a small slice of the efforts to debug and fix
consoles. He found a bug that could cause a reproducible crash on
every game he tested.
“If you coordinated the music player with the dashboard, you could get
almost every 360 to lock up,” he said in an interview. “I did it first
on a combo DVD/audio disk. With NBA 2K6, you would select the music.
The screen went black.”
The NBA 2K6 flaw resulted in the Red Rings of Death. Delaware, who as
of this writing works at Microsoft as a game tester, agreed to go on
the record for this story because he said he believes passionately in
his work, which involved painstakingly playing games over and over
again in order to uncover bugs. But he also believed passionately in
the Xbox 360 gamer community. He is one of those thousands of
employees who was good at his job, loved the games, and sought to push
Microsoft to excellence. But he said he found it difficult to stay
silent on problems that caused a number of his friends a great deal of
consternation. He said he realized that speaking out could cost him
his job and he plans on asking Microsoft to forgive him for it. He was
courageous in coming forward, and he was not the source of emails
cited in this story.
When the Capcom game “Dead Rising” (pictured left) shipped in the
summer of 2006, another flaw emerged. Console owners reported putting
the game disks into their machines and then seeing a breakdown. One of
the reasons was the process by which Microsoft updated the Xbox Live
online service. Many of the system owners were not connected online so
they didn’t get live fixes downloaded to their machines. Thus,
Microsoft had to ship Xbox console fixes with some of the consoles. In
the case of Dead Rising, the game came with a bug fix that actually
crippled some consoles. Microsoft often did a couple of firmware
updates a year, and sometimes they led to broken machines.
Delaware was convinced that at least some of the problems reported
about Dead Rising were related to the “2858 dashboard update” for Xbox
Live, which he believed was embedded in the game. Another explanation
was that games such as Dead Rising demanded more from the consoles,
which gave out under stress. Delaware believed that the practice of
embedding console and dashboard updates in game disks is responsible
for some hardware failures.
In October, 2006 Delaware was concerned about how much information
Microsoft was sharing with its testing partner and offered a warning
about the Xbox Live updates. “With the upcoming Wii and PS 3 launches,
MS cannot afford the bad PR that might result from Update related
issues,” he said. “Asking these questions from the start might help to
prevent possible criticism of our testing process, in the event of
update-related problems. The last thing VMC needs is to take the blame
for problems that Microsoft has (known) about from the start.”
Shooting too high, moving too fast, stretched too thin?
So what exactly was wrong with the machines? As time would reveal,
there was no single reason for the failures, though many of the
problems could be blamed on the ATI graphics chip, which could
overheat so much it warped the motherboard. This put stress on bad
solder joints, causing them to fail early in the machine’s life.
Sometimes the heat sinks on top of the GPU were put on the wrong way,
resulting in heat problems. Finally, games would sometimes crash
because of sub-par memory. Infineon had been brought aboard as the
second supplier behind Samsung for the GDDR3 memory used in the Xbox
360. This new kind of memory chip was specified for 700 megahertz, but
the Infineon parts were falling short of that target. Microsoft had to
set up a line for sorting through the good parts and the bad parts,
contributing to a shortage of consoles.
Problems with the DVD drive also lasted longer than expected. And the
console was also one of the first products that had to meet new
environmental standards in Europe, which prohibited the use of lead in
solder (which, when melted, fuses electronic components together).
Paul Wang, a Microsoft test engineer on the Xbox 360, said in a speech
in 2007 before a Silicon Valley engineering group that the lead-free
solder created a lot of problems.
Perhaps Microsoft had too many balls in the air. As it launched the
Xbox 360, it was already planning other big related projects. Cobalt
was a next-generation HD-DVD movie player (pictured above) that
Microsoft would launch in the holidays of 2006 as a $199 add-on. A
project code-named Zephyr (later renamed the Xbox 360 Elite) would add
a more expensive hard disk drive with 120 gigabytes of storage and a
new HDMI connector. And the company’s engineers were hard at work with
suppliers on a project code-named Falcon, which was an effort to
create a motherboard, or main circuit board inside the box, to reduce
the costs of the console itself during 2007.
“That was really annoying, that they had us working on so many
things,” an engineering source said.
As the engineers wound down one task, they had to move to the next.
There was no team set aside to deal with the low yields, in part
because the poor yields hadn’t been anticipated.
Fessing up
In September, 2006 that the company made a rare admission. It said in
a statement that the quality of the consoles it made during 2005
wasn’t as high as it expected and therefore it would extend the policy
of free replacement for consoles made during 2005, even though the
warranties had expired.
While some industry observers expected Microsoft to cut the costs on
the console and offer consumers a price cut during the second holiday
season, that was never in the plans. Instead, the company was ramping
up to offer a new accessory, an HD-DVD drive that could play next-
generation DVD movies for the low price of $199. That was meant to
parry Sony’s PlayStation 3, which had a Blu-ray next-generation DVD
drive built into the system.
Fortunately, Sony was stumbling. Besides being a year late, the costs
on the Blu-ray drive were so high that Sony had to price its consoles
at $499 and $599, far above Microsoft’s prices of $299 and $399.
Market analysts estimated the cost of Sony’s box was around $800 or
more. Microsoft didn’t need to offer a price cut after all,
particularly since Sony was struggling to build enough consoles for
its fall launch.
It was looking like, even with the manufacturing problems, Microsoft
was going to do just fine. In mid-November 2006, about 645,079 units,
or 8.8 percent of the total units shipped, had been returned for
repair. Sony had been more conscientious about delaying its launch
when the product wasn’t ready. JackTretton, the president of Sony’s
U.S. games division, said he was proud that Sony waited. But it was
going to pay for that cautiousness.
The three Microsoft contract manufacturers had now ramped up to full
production. They produced more than 12 million consoles. In early
2007, Peter Moore announced that Microsoft had hit its revised target
and had shipped more than 10.4 million consoles into the market before
the end of 2006. It only sold through 9 million consoles. There were a
lot of leftover consoles sitting on shelves.
At about the same time, in December, 2006, Microsoft took another step
to appease consumers who had failed machines. It said that it was
extending the warranty of free repair for all consoles made for
customers in the U.S. and Canada. Now the machines would carry a one-
year warranty, not just 90 days.
A drastic measure
Microsoft decided to shut down manufacturing of the Xbox 360 in
January, 2007. Between January and June, it didn’t build any new
machines. The reason was partly because it made too many machines
earlier, but the other reason was to track down the source of its
quality problems. It did that even though it was launching a new
version of the console, dubbed the Xbox 360 Elite (pictured left),
with a bigger hard drive and a new HDMI connector for $480. It had a
new board, dubbed Zephyr, inside. That new black version of the
console was made in limited quantities for hardcore fans who might
otherwise be attracted to buying the most expensive PS 3.
Since it had an oversupply, Microsoft had a cushion. It could afford
to suspend manufacturing as it sold off its inventory of consoles. The
company stopped promoting the console heavily and shaved a million
units off the goal for the fiscal year ended June 30, 2007. The
problem was that Nintendo’s Wii was starting to catch fire. Nintendo
had come up with a cheap and innoviative console, which was proving
extremely popular at $250, about $50 to $150 cheaper than Microsoft’s
Xbox 360 models. During the six-month period ending June 30, 2007,
Microsoft would sell only 1.2 million consoles. It was a disastrous
slowdown, compared to the Wii’s sell-out success.
In January, 2007 at the Consumer Electronics Show, Bill Gates
announced that the Xbox 360 would be capable of serving as an IPTV
set-
top box, meaning that telephone companies could use it to deliver
video programming and other services that could compete with cable TV
providers. Gates also said that because the Xbox 360 had beaten the PS
3 to market and didn’t include a hard drive on every machine,
Microsoft would be able to ride a silicon cost-reduction curve faster
than Sony, which saddled every unit with Blu-ray and hard drive costs.
At any given point, the Xbox 360’s costs will be lower and more easily
reduced, giving Microsoft a fundamental advantage over Sony in
pricing, Gates said.
The Cassinghams stir discontent
The topic of defects flared up again in February, 2007, when Rob and
Mindy Cassingham (pictured left) of Moab, Utah, revealed that they had
the frustrating experience of returning seven defective Xbox 360s.
They were die-hard fans who had driven across the country to be
present at the launch of the Xbox 360.
They used several in their retail gaming center, where kids came in to
play against each other. That prompted critics to blame the
Cassinghams for over-using the machines. But several three of the
machines that broke down were actually personal machines that the
Cassinghams used in their own homes. Consumer outrage flared, and
Moore again offered apologies to customers. One game magazine, 360
Gamer, conducted a poll that found many console owners reported more
than one console failure.
Fixing bugs in the trenches
Microsoft quietly acknowledged that it was no longer using Wistron as
a manufacturing partner. In April, 2007, Larry Yang, the general
manager of Xbox division hardware, said in a group email that he was
resigning that job and would take another position within Microsoft.
He would still stay and help Microsoft get out its most important
redesign, dubbed Falcon.
Falcon was the code name for the cost-reduced Xbox 360 that included a
major redesign of its CPU. Scheduled for once every couple of years,
the transition to new chips was a big deal. Microsoft’s partners had
been making their chips with 90-nanometer technology since 2005. Now
they would make the CPUs with 65-nanometer technology, meaning the
widths between the circuits were smaller. With smaller circuits, the
same chip design can fit on a smaller amount of space. With less
space, the chip can be manufactured with less material, resulting in
lower costs. Quality also gets better since it’s easier to make
smaller chips than big chips. Shifting to a new generation of chips
was thus the best way to reduce the cost of a game console and improve
its quality.
An email sent out by S. Srini, director of Xbox manufacturing, on May
27, 2007, said that the overall yield of units in production was 85
percent. But the email said that the IBM microprocessors were still
exhibiting “excessive failures” of 4 percent to 8 percent because of
heat problems. There was still no “fresh start” manufacturing going
on.
The hardware engineers worked mainly to address the overheating of the
graphics chip from ATI. The chip was overheating and causing both the
chip and the circuit board to warp, cracking the joints that were held
together by the lead-free solder. Given enough time, many units will
suffer this problem. To fix the problem, Microsoft put epoxy on both
the IBM microprocessor and the ATI graphics chip. They also took out a
50 cent extruded aluminum heat sink (which sits on top of a chip and
dissipates heat away from it) from the graphics chip and replaced it
with a $5 heat sink that could do the job more efficiently. They also
used a pipe to move heat in front of a fan.
Beyond the temporary solution of the heat sink change, Microsoft made
more changes. Microsoft tried to get Falcon out as fast as it could.
While the system box would remain the same size, there would be fewer
components inside. That, in turn, would leave more room for air flow
and to reduce the use of fans in the system, which meant it wouldn’t
be as noisy as before. When rivals such as Sony made this kind of
switch before, they had big shortages because it was always difficult
to make the transition.
Engineers were under pressure from the top executives to describe the
nature of the problem. When the engineers finally had the situation
under control, Microsoft finally announced its free replacement
program on July 5, 2007.
“Hopefully consumers will recognize we’re trying to do the right
thing,” Moore said in an interview that day. “It’s a courageous step
because it is not an inconsequential step. We are not burying our
heads in the sand.”
The announcement put a dent in the cottage industry for repairing Xbox
360s (such as the book pictured at left). Even so, in early 2008,
companies such as game retailer GameStop were making a fortune
repairing thousands of Xbox 360s a week and then returning them as
refurbished units to store shelves at a discounted price. Most of
GameStop’s repairs were successful and involved re-soldering the
graphics chip to the main board; unsuccessful repairs were passed on
to Microsoft.
Peer Schneider, vice president of content publishing at gamer web site
IGN, said, “The extended warranty and the reimbursement policies are
important steps to win back consumer trust. The only question from
Xbox fans is: what took Microsoft so long?”
6.)
After the write-off
There has been a lot of executive turnover in Microsoft’s game
division, but none of it has been laid directly at the feet of the
defect problem. Moore left Microsoft a year ago to become head of the
sports label at Electronic Arts, while Don Mattrick, the former head
of EA’s worldwide game studios, took the top games job under Bach at
Microsoft. Bach said that Moore’s departure had nothing to do with the
RROD problem, and Moore said the reason for his departure was that he
and his family wanted to return to the San Francisco Bay Area. Bach
remains one of the top executives at Microsoft, running the
Entertainment & Devices group, (including the IPTV, Games for Windows,
Xbox, and Zune businesses) which reported its first-ever profit in the
fiscal year that ended June 30, 2008. Allard is chief technology
officer of the E&D group, while Todd Holmdahl is head of games
hardware.
Oddly enough, demand for Microsoft’s console continued, partly because
gamers loved playing games so much that they kept on buying
replacement consoles. By the fall of 2007, Microsoft had its Falcon-
based machines ready. It was able to cut the price of the console, but
not so much that it hurt its bottom line. The launch of “Halo 3” in
the fall helped spur console sales, helping Microsoft stay ahead of
Sony and at least stay competitive with the Wii in terms of game
sales. In a development that was reminiscent of the “Dead Rising”
failures, a number of Halo 3 buyers said the game appeared to fry
their consoles.
Microsoft never announced how many machines had been returns. Based on
an estimated $60 repair price, the company’s reserve can cover lots of
machines. In early 2008, SquareTrade, which sells warranties for
electronics, reported that there was a 16.4 percent failure rate for
Xbox 360 systems. Michael Pachter, an analyst at Wedbush Morgan,
estimated that the number is 3 million, or about 15 percent of the
total. It isn’t clear if the numbers should count consoles that fail
and aren’t repaired for free (only those with the three red lights are
replaced for free); also retailers such as GameStop have begun
repairing many of the units themselves. Bach said that Microsoft’s own
research shows that fans continued to love the console and buy games,
in spite of the problems with the defective units. Pachter believes
that is true.
“In the ordinary course of something like this, you would expect it to
show up in the customer reaction data,” Bach said. “We just haven’t
seen that. It speaks to the fact that they love their games and Xbox
Live. Does it frustrate them? Yes. On the other hand, they know we’re
taking care of them. People have a certain amount of respect for that.
If it had happened on a product that had less baseline customer
satisfaction, it would have had a bigger impact. We really haven’t
seen that.”
At this point, the Wii has begun to far outsell its rivals on a
worldwide basis. It’s not inconceivable that it will sell as many
units as its two rivals combined. It isn’t clear how much the RROD
problem hurt Microsoft, which remains ahead of Sony. The scorecard so
far: the Wii, 29 million; Xbox 360, 20 million; PS 3 (pictured left),
14.4 million.
Was it worth it to go first, despite the concerns of the engineers and
the inevitable consumer return problems?
“If you take the question of whether it was the right thing to try to
be first, the answer to that is definitely yes,” Bach said in July
2008. “It has given us a leg up in a number of places that are super
important. It has given us a leg up with game developers. It has given
us a leg up from an economics perspective. It helped us expand Xbox
Live quickly. At a strategy level, if you asked if we wanted to be
first again, I would say yes. It’s easy for me to go back and say, if
I knew what would transpire over the next two years, would I go back
and do something different, I think that’s an obvious answer. But the
fact is, based on the data we had at the time and all the hard work we
put into it, there was no way to see what actually happened.”
The problem for Microsoft was that it sacrificed something to rush
into the market and establish the Xbox 360 as the premiere platform of
the living room. What it sacrificed was the good will of consumers,
who are actually critical in terms of establishing a lasting platform.
As for profits, Microsoft has finally turned a fiscal-year profit in
the Entertainment & Devices group. It did so because it wrote off the
costs of the defective units and warranty program in the previous
fiscal year. (The $1.15 billion write-off covered several years of
repair costs, some of which would have otherwise been incurred during
the profitable fiscal year.) But Bach said that the business model
for the Xbox 360 isn’t permanently ruined.
“It doesn’t make it good,” he said. “Would I like the billion dollars
back? Of course, the answer is yes. But there is still plenty of
economics in the business for us to be successful.”
Microsoft hasn’t been as aggressive at cutting prices this generation,
but Sony has a more expensive box and so it hasn’t been putting price
pressure on Microsoft. Had Sony been more aggressive, the consequences
could have been much worse in terms of customer defections due to the
RROD. Even with the $50 to $79 price cuts announced by Microsoft on
Wednesday, it is taking a lot longer for prices to fall in this
generation.
Jerry Knoben, a former Intel manufacturing chief, joined Microsoft in
January this year as head of Xbox 360 manufacturing. He recently sent
out a memo to the staff saying that the yield on Xbox 360 production
had improved from the mid-80-percent-range to 92 percent. But he said
that there were still some surprises in manufacturing that Microsoft
had to fix.
The improved consoles use a new motherboard, dubbed Jasper. This one
has both a 65nm IBM microprocessor as well as a 65nm graphics chip
from AMD. The improved graphics chip may finally run at acceptable
thermal limits. But it’s very late in coming, compared to the rest of
the chip industry, in part because Taiwan Semiconductor Manufacturing
Co., the chip contract manufacturer making the graphics chips, was
slow to shift to 65nm factories. Late or not, Jasper is what enabled
Microsoft to cut its prices this week.
By next year at the earliest, Microsoft is scheduled to launch a new
iteration of the Xbox 360 with a board dubbed Valhalla. That board
will have a single chip that combines both the graphics chip and
microprocessor on a single device. This is the kind of technical tour
de force that Sony achieved with the chips for the PlayStation 2, a
move that allowed the company to significantly reduce manufacturing
costs. Microsoft still has an advantage over the PS 3 in manufacturing
costs, since it doesn’t have an expensive Blu-ray player.
Shane Kim, (pictured left) until recently head of Microsoft Game
Studios and now head of the game division’s business development
efforts, recently said that it was too early to call the console war
in favor of Nintendo. We won’t know the winner, he says, until
somebody sells 100 million units. Microsoft’s plan is to sell 75
million Xbox 360s. The only way they could ever do that is to be more
aggressive on their pricing, which means driving down the costs of the
console as low as possible. And by doing the right thing for their
customers from now on.
Sony and Nintendo aren’t safe just yet. Bach has always talked about
how Microsoft has a 10-year to 15-year plan for success. Maybe Xbox
3.0, just like Windows 3.1, will do the trick.
To conclude, the video game industry has never seen a consumer problem
as bad as the “red rings of death” and the size of the $1.15 billion
charge stands as one of the biggest liability glitches in consumer
electronics history. How Microsoft handled the flaw may provide a
lesson for all modern electronics companies; that is, if you are going
to promote the hell out of something, it better work the way you say
it does and you better have a strong customer support and engineering
debugging team to back it up.
Amazon and eBay have their periodic outages. Sony had its problems
with batteries, Intel had flawed Pentium chips that were bad at math,
Nvidia took a $200 million charge to deal with graphics failures on
laptops, and Apple has had its share of glitches on the iPhone. As
electronics become and more complicated, it becomes a practical
impossibility to test for every single flaw. But with the force of the
Internet behind them, angry customers can exact a revenge that can be
extremely damaging to a company’s brand image.
One thing is clear. Microsoft has to move beyond its mentality of
being a software company that can launch fast and fix later. With
global markets and global launches, the consequences of such a
cavalier approach to hardware quality can start to pile up. The
company clearly has the cash and the size to compete among the best
hardware companies out there. It cannot afford to tarnish its brand
when competitors on various fronts — such as Google on the web side
and Apple, Sony, and Nintendo on the hardware side — are there to
exploit its stumbles.
At this writing, it seems clear Microsoft’s hegemony in operating
systems and productivity software won’t last forever. The company has
limited time to get the rest of its house in order. It has to
diversify into fields such as video games and make them pay off with
profits. Microsoft has demonstrated an incredible tenacity in a lot of
markets. It took almost a decade for the company to march to victory
with its Office productivity software. I have no doubt the company can
execute. It just has to decide that quality is something that matters.
If Bach is correct and Xbox 360 gamers are loyal, then Microsoft has
been lucky. In the future, I suspect consumers won’t be as forgiving
as fans such as Chris Szarek, who doesn’t own any other console and
has bought three dozen Xbox 360 games.
Szarek said, “They got me. There was a time when I was furious with
them.”
So why did he stay with the Xbox 360?
“What can I say?,” he said. “I like the games.”
1.)
Xbox 360 defects: an inside history of Microsoft’s video game console
woes
Dean Takahashi | September 5th, 2008
When his fourth Xbox 360 video game console died in April, Chris
Szarek wasn’t surprised.
The Chicopee, Mass. gamer was accustomed to the hardware failures that
became known throughout the Internet as RROD, or the “red rings of
death” which flash when the console becomes inoperable.
A 40-year-old photographer, Szarek was a hardcore Microsoft fan who
spent more than $1,000 on his games. But each time one of his Xbox 360
consoles failed, he had to spend time convincing Microsoft’s tech
support that they should send him a new console. Each time he got a
refurbished console as a replacement (a machine that had been returned
to a repair center in Texas, fixed as much as possible, and then
shipped back out). When he complained on the Internet and to the media
about the shoddy product and poor customer service, people branded him
a cry baby and wrote him off as a statistical anomaly. But by the
spring of 2008, Szarek was vindicated. There were at least a million
or two other people like him.
Szarek’s fourth machine lasted almost two years, experiencing the same
short life that many other Xbox 360s suffered. Microsoft replaced
these machines for free under the warranty that it announced on July
5, 2007, for defective Xbox 360s exhibiting what it more politely
called the “three flashing red lights.” That warranty program cost
Microsoft up to $1.15 billion, but the loss of face and loyalty among
gamers in the fierce console war with Nintendo and Sony has been
immeasurable. Szarek, who became a spokesman for dispossessed
defective Xbox 360 owners, played a part in making Microsoft
acknowledge its console quality problem.
This is the unauthorized tale of how Microsoft lost its chance to
become the leader in the biggest market it has attacked beyond its
twin monopolies in Office and Windows software. Rival game console
maker Nintendo out-thought the larger players Microsoft and Sony by
designing the Wii game console with a clever, intuitive game
controller. Even so, Microsoft could have captured more gamers during
this product generation, yet the RROD problem held it back. The Xbox
360’s defect problem will go down as one of the worst snafus in
consumer electronics history.
Its own worst enemy
Microsoft knew it had flawed machines, but it did not delay its launch
because it believed the quality problems would subside over time. With
each new machine, the company figured it would ride the “learning
curve,” or continuously improve its production. Even though
Microsoft’s leaders knew their quality wasn’t top notch, they did not
ensure that resources were in place to handle returns and quickly
debug bad consoles. There were plenty of warning signs, but the
company chose to ignore them. The different parts of the business
weren’t aligned.
It reminds me of the German war machine just before World War I, as
chronicled by Barbara Tuchman in the classic history book, “The Guns
of August.” The German generals were intent on keeping their trains on
time; but the leaders overlooked their chances for stopping the war
altogether. The Schlieffen plan called for them to strike first. Once
the Russians and French mobilized, the Germans had to move into
action. They marched off blindly into tragedy.
Likewise, Microsoft’s strategy depended on beating its rivals to
market. It couldn’t afford to stop and delay the launch in order to
solve its quality problems, or so upper management believed. What
Microsoft’s leaders didn’t realize was that getting to market first
with a flawed machine would only win them a battle; and it risked the
loss of the war.
“They got enamored with the idea of the Microsoft army rolling
everything out at the same time,” said one knowledgeable source who
asked not to be identified.
The quality problem negated much of the advantage of going first, and
it has delayed the company’s plan to aggressively market the console
and slash its prices. (Microsoft disputes this point; it cut the price
of all three versions of its Xbox consoles by $50 to $79 on Wednesday.
And the company believes it will sell more boxes than Sony will. But
prices ought to be lower still during this stage of the console life
cycle). That has stopped the company from reaching the broader market
of consumers that Nintendo has won over. It has lowered its ambitions,
hoping instead just to get a clear edge on third-placed Sony. The
future profits that the company once hoped for are now likely to wind
up in Nintendo’s pockets.
Microsoft’s top game executive, Robbie Bach, president of the
Entertainment & Devices group, said at a dinner in July that
Microsoft’s own research shows that gamers have largely forgiven the
company for defective Xbox 360s. Microsoft has still sold more Xbox
360 consoles than Sony to date. But there is no doubt that the company
has lost considerable good will among gamers. Before Microsoft offered
free replacements, connsumers grumbled that they had to turn to
forums, such as those on Ars Technica, to vent and to find solutions
to problems that the company didn’t openly discuss. And for a couple
of months now, Sony’s PlayStation 3 has been outselling the Xbox 360
in the U.S. for the first time.
“Fundamentally, their thinking shows that they are a software company
at heart,” said one veteran manufacturing executive. “They put
something out and figure they can fix it with the next patch or come
up with a bug fix.”
The terrifying part of the story is that this kind of problem — where
technology fails and no one knows what to do about it — can happen to
any company.
About this story
I asked Microsoft to confirm or deny 35 different facts contained in
this story. Instead, I received a formal statement from a Microsoft
spokesperson, saying the company had already acknowledged an
“unacceptable number of repairs” to Xbox 360 consoles and responded to
the hardware failures with a free replacement program. The statement
also said, “This topic has already been covered extensively in the
media. This new story repeats old information, and contains rumors and
innuendo from anonymous sources, attempting to create a new
sensational angle, and is highly irresponsible.”
I don’t think this story is sensational. I have tried to verify the
facts over several years. I view this story as the last chapter for my
book on the making of the Xbox 360, “The Xbox 360 Uncloaked: The Real
Story Behind Microsoft’s Next-Generation Video Game Console.”
The facts revealed themselves slowly, emerging from the day-to-day
stories that I wrote about the game business. Some people might
consider this post mortem to be ancient history. But the
reverberations are still playing out today. They help explain why
Microsoft isn’t being aggressive with its price cuts and why gamers
aren’t getting bargains on hardware as they did the last generation.
While I talked to many people for this story, few were willing to let
me use their names. As you will see, not every source is anonymous,
and we have included the viewpoint of Microsoft executives from past
interviews.
The details are interesting because they offer a deeper look into how
the console business runs than is otherwise available. Microsoft, for
instance, still hasn’t perfected its Xbox 360 manufacturing process.
In the absence of a precise chronology from Microsoft, some anonymous
sources have tried to describe what happened. But the history of the
decision making and inside story of what happened on the RROD has
never been told, until now.
2.)
Microsoft’s mea culpa
In many ways, the Xbox 360 was a big achievement. Microsoft had beaten
its rivals Sony and Nintendo to market by a year. Sony had outsold
Microsoft five-to-one in the previous generation of consoles, but
Microsoft has outsold Sony’s PlayStation 3 with the Xbox 360.
Microsoft’s to game executive Bach (left) had repeatedly promised that
Microsoft would be profitable in his Entertainment & Devices group in
the fiscal year ending June 30, 2008. The company met that target. Big
losses have turned into profits.
But when the company launched its Xbox 360 video game console in
November, 2005, it didn’t have a handle on product quality and it was
not prepared to systematically analyze its product returns and debug
bad consoles.
Microsoft has admitted those mistakes. On July 5, 2007, the company
said it would take a billion-dollar write-off to pay for free
replacement of Xbox 360s by up to three years from the date of
purchase. Peter Moore, who at the time was head of the games business
at Microsoft, said in an interview the decision to take the write-off
was due to an “unacceptable number” of returns. The company made
improvements to the console’s quality, but Moore said it was unclear
how many units would fail in the field. Microsoft then followed up its
new policy with numerous public apologies, including a statement from
Bach, (pictured above).
Microsoft never disclosed its actual return rates. But according to
data obtained by VentureBeat, the total number climbed above 1.2
million consoles in early 2007. That is a huge amount, considering
Microsoft had only shipped 11.6 million into stores by the time of the
announcement in mid-2007.
The company was dragged kicking and screaming to its admission of
widespread defects — with gamers and the press doing the dragging. In
interviews in April and May of 2007, Microsoft executives denied that
quality problems were haunting them. Todd Holmdahl, (pictured left)
general manager in charge of the Xbox platform for Microsoft in
Redmond, Wash., said in an interview with me on May 9, 2007, while I
was at the San Jose Mercury News, that the “vast majority” of
Microsoft’s Xbox 360 customers were having a great experience with
their consoles. He declined, however, to say what percentage of
consoles were being returned for defect reasons. In doing so, he was
as coy as any of the console makers are about such sensitive data. But
everyone wanted an answer to that question.
It’s not clear why it took Microsoft so long to announce its warranty
extension last year, almost 19 months from the time when the first
complaints arose at the outset of the launch. As it was investigating
the cause of the problems, the company stayed quiet, acknowledging
problems only little by little. The company delayed its free-
replacement announcement until it understood the scope of the problem.
In July, game executive Bach said that the number of returns was not
measurably big until a year after the launch. It was a problem that
became evident only over time, he said.
“It wasn’t related to things we were seeing in testing or some
judgment call we had to make about whether the product was ready,”
Bach said at the dinner. “We were confident the product was ready. We
did a lot of testing. The problem that shows up with the three red
lights on the console is a complex interaction with some very complex
parts.”
Ignoring warnings
But the evidence for the quality debacle was there to see even before
Microsoft shipped any machines. In August, 2005, as Microsoft was
gearing up production, an engineer raised a hand and said, “Stop. You
have to shut down the line.” This wasn’t just a brief moment. The
engineer spoke up repeatedly.
That engineer, who asked not to be identified, had deep experience in
manufacturing. When production results were really off kilter,
stopping a line and tracing a problem back to its roots was the
answer. But the higher-ranking engineers, managers and executives
chose to risk going forward. There wasn’t a universal backlash from
the engineering ranks, according to one engineering source.
Nobody listened to that engineer — who spoke on condition of anonymity
— apparently because console launches are always hurried affairs.
Yields — the percentage of working products in a given batch of total
products produced — generally start low. As the manufacturers conduct
statistical analysis and tight controls on every step in assembly,
they learn how to drive the yields up.
Still, the picture wasn’t pretty. The defect rate for the machines was
an abysmal 68 percent at that point, according to several sources.
That meant for every 100 machines that Microsoft’s contract
manufacturers, Flextronics and Wistron, made at their factories in
China, 68 didn’t work. At the recent dinner, Bach denied that there
was a big concern about defects at that point in time.
At that point, it is likely true that the engineers weren’t raising
enough red flags for the executives to pay attention. Early yields on
electronic goods are almost always lousy. Those veteran engineers
figured that they would be able to debug the problems and bring the
yields up quickly. But the expected rapid improvement in quality just
didn’t happen. The communication between upper management and the
engineers wasn’t clear. Nor was the strategy aligned between marketing
and customer support.
There were plenty of warning signs. Early reports on the problems were
myriad. In an Aug. 30, 2005 memo, the team reported overheating
graphics chip, cracking heat sinks, cosmetic issues with the hard disk
drive and the front of the box, under-performing graphics memory chips
from Infineon (now Qimonda), a problem with the DVD drive, and other
things. At that point, the contract manufacturers were behind schedule
and had only built hundreds of units. They were supposed to have been
in high gear, making thousands every week.
The illusion of a well-oiled machine
Yet around that same time, in mid-August 2005, Microsoft gave me a
tour of the hardware test center in Mountain View, Calif. Hundreds of
boxes were being tested in heat chambers and test labs. Leslie Leland,
an engineering manager, said all was on schedule. Microsoft wanted to
convey its own confidence to its partners, game developers, and
gamers. To my untrained eye, it looked like a smooth operation.
But on the inside, it was a frenetic time. The initial yield on the
most critical chip, the three-core microprocessor designed and
manufactured by IBM, was only 16 percent. For every 100 produced, only
16 worked. This low yield was surprising because a plaque on the wall
in Microsoft’s campus in Mountain View proclaimed that IBM had “taped
out,” or completed the design, of its first microprocessor prototype
on Dec. 8, 2004. The timing of the tape out was right on schedule,
which was good because it is normal for six months to a year of
debugging to follow a tape out. IBM had an easier time than the maker
of the graphics chip.
ATI Technologies (a graphics chip maker that has since been acquired
by Advanced Micro Devices) was late to deliver working graphics chips,
or GPUs. Like IBM, it had promised to deliver one of the most complex
chips it had ever made in about half the time it usually took to
create such chips. But ATI only had half the number of the 400
engineers IBM had on the project.
The fact that both companies had designs done at all was the result of
a Herculean effort. Microsoft’s engineers started working on the Xbox
360 at least a year after Sony’s engineers began work on the
PlayStation 3, yet Microsoft wound up shipping a year ahead of Sony.
Everything Microsoft did was under time pressure, including the
creation of the IBM microprocessor and the ATI graphics chip. The only
way to pull it all off was through a virtual organization, where
Microsoft outsourced many of the tasks to companies that specialized
in particular tasks. Microsoft’s hardware engineers in Redmond, Wash.
and Mountain View, Calif. were the master integrators of all of the
suppliers. The Microsoft chip designers in Mountain View also designed
an important video processing chip themselves.
Considering all of the work, Microsoft had too few hardware people.
Some of the designers of the Xbox 360, including engineering chief
Greg Gibson, were stretched thin. Gibson and J Allard (pictured left),
who led the console design effort, had begun work on Zune, Microsoft’s
portable media player. Top brass had approved the project to dethrone
Apple’s iPod, but Microsoft kept Zune secret from the outside world
until much later. Some engineers were pulled of the Xbox 360 at a
critical moment to join Allard’s effort to create a music player.
Those who were left to work on the test team worked around the clock,
traveling to China to work in the factory.
3.)
Seasoned hardware team executed well on the original Xbox
Outsiders frequently slam Microsoft’s hardware engineers as having a
software (fix it later) mentality. But Microsoft’s engineers had been
trained to put quality first. Most of them had years of training.
Holmdahl, for instance, had been with Microsoft in the early days of
its hardware division. By the time he joined the Xbox project, he was
a veteran, having worked with contractors such as Flextronics to make
20 million mice per year. Making consoles was an altogether different
task, but the team had expert help through its Web TV acquisition,
which brought veteran console hardware engineers aboard in 1998. One
of the best at debugging was Nick Baker, one of the architects of the
Xbox 360’s chips.
In 2001, the team went through the paces making sure that the original
Xbox had good quality. On that launch, Microsoft was 20 months behind
Sony’s launch of the PlayStation2. Microsoft’s original Xbox team
moved as fast as they could, but they took a long time defining the
box and picking vendors.
By the time they hired Nvidia in March 2000 to make the graphics chip,
they were hopelessly behind. Nvidia combined two different chips into
one in half the time it took to design a chip, but it fell behind
schedule in part because of a bug. Someone else had specified a sub-
par power supply for the Xbox, and that made the graphics chip hiccup
now and then. Once the team fixed the problem, they raced ahead.
But Flextronics, the contract manufacturer, wasn’t ready. It had to
build a bigger manufacturing database to accommodate the design.
Microsoft had to postpone its launch by a week. It also decided to
launch only in the U.S. market and would wait until the next spring to
launch in Europe and Japan. No console maker had ever pulled off a
worldwide launch before, and now Microsoft understood why. The delays
in other territories put it even further behind Sony. Sony eventually
outsold Microsoft by more than five to one — scarring into memory a
tough lesson about market timing.
But the Microsoft hardware team had come out shining. Aside from a
problem with a supplier’s DVD drive, the number of defective Xboxes
was low. After the DVD drive issue was fixed, yields rose above 90
percent and stayed there for the duration of the manufacturing.
Microsoft thus had a seasoned team to work on the Xbox 360.
Lessons of round one drove grand strategy for the Xbox 360 business
Despite the good quality manufacturing, Microsoft’s game business was
deeply flawed in the Xbox generation. In the second round, Microsoft
vowed to learn from its mistakes.
In the first generation, the company lost $3.7 billion over four
years. Most of that was because the costs of the box — particularly
its hard disk drive — were too high. Microsoft’s planners thought the
hard drive would give them an edge over Sony’s PS 2. It didn’t.
Bill Gates didn’t really care about the losses in the first
generation. That was simply the ante for getting into an exciting new
business. Microsoft has always viewed its ambitions in the game
business as strategic. It isn’t in the business just to dominate video
games, but to own the living room. The game console is a kind of
gateway to the Internet and all of the future entertainment services
that can be piped into the living room.
But Steve Ballmer, who took over from Gates as CEO during the first
generation, really wanted the Xbox business to be profitable second
time around. He wanted to keep the foothold in the living room, but
not at the expense of draining Microsoft’s cash, which was needed to
fight Google and other rivals.
This time, Gates said he felt that Microsoft would have a fundamental
cost advantage over Sony. One of the key ways Bach hoped to outwit his
rivals was with speed: getting a machine to market in four years
instead of five. Secretly, Microsoft had planned on selling 50 million
Xbox 360s at that point.
The Xbox team met for a series of retreats. One such retreat took
place in early 2004 at the elegant Salish Lodge beside the Snoqualmie
Falls made famous in the Twin Peaks TV show. After that meeting, Bach,
who had headed the Xbox business as its senior executive from the time
of its inception, decided that company would aim for profitability
with the Xbox 360. As much as possible, the machines would not be sold
for a loss.
Bach believed the hardware itself should be profitable over the life
cycle of the system. His planners counted on good revenues from
accessories and games to ensure profits the second time around. And,
in the three-page paper that outlined the grand strategy, he said that
Microsoft wanted to hit the market at the same time as its rivals.
Microsoft’s best guess was that Sony would launch the PS 3 in 2005.
That turned out to be a year off, since Sony ran into its own
development problems related to its decision to put Blu-ray drives
instead of DVD drives into the PS 3.
Microsoft had more preparation for Xbox 360
There was more time to design the box than the first-generation team
had, but it was still a race. Different teams sped into planning. The
different groups included chip designers, hardware planners, game
designers, software operating system engineers, marketers and
industrial designers.
With the Xbox 360, Microsoft was guilty of packing a little too much
into its box. The company sent out its scouts in late 2002 to ask
gamers what they wanted. The first Xbox came off as “The Incredible
Hulk.” The second Xbox, Microsoft had mandated, had to be smaller, if
only to fit in the smaller homes in Japan, a key target market. The
designers thought it had to be more like Bruce Lee. Microsoft’s own
user-experience team ran the show, while outside industrial design
firms competed in a bake-off to define the soul of the machine.
Hers Experimental Laboratory of Japan and Astro Studios in San
Francisco won the bake-off. They wanted to create something that
looked elegant and iconic, qualities that were sometimes at odds with
the computer that it was. Those firms gave the box its signature look
— curved sides that looked as if the box were inhaling air and was
about to explode with energy (that was the marketing image, anyway).
There was a constant tug of war between the industrial designers, who
wanted something small, and the engineers, who wanted to pack a lot of
performance into the box. The more powerful the system was, the bigger
the box had to be to provide air flow for heat-sensitive components.
Microsoft decided late to add a hard disk drive to most of the
machines. It also came up late with a plan to add wireless
controllers; all of the previous consoles shipped with wired
controllers. The hard drive blocked a lot of the air flow on one side
of the machine. And the wireless modules had to have enough of their
own space to ensure that there was no electrical interference. In the
end, the machine was a series of compromises. The gigantic power
supply needed to run the machine sat outside in the power cord, while
the console shell was poked full of holes on the sides to ensure air
flow.
“It turned out in the end that this was all going too far, too fast,”
said one source. “They were adding too many features after things were
locked down. That incremental feature adding just made it fragile.”
The marketers also decided the box had to be backward compatible with
the older games, including “Halo” and “Halo 2.” And they also felt the
box had to have enough horsepower to make games look beautiful on
high-
definition TVs.
Those parameters, in turn, gave direction to the chip and hardware
designers. This time, hardware chief Todd Holmdahl wanted Microsoft to
have much more control over its fate. And he believed he had built up
a team that could act as the “system integrator.”
Microsoft had a team of 100 chip engineers and 100 more hardware
engineers between Redmond and Mountain View. They were the overseers
of the integration of the parts into a working console. It was a
relatively small team for such a large project compared to a hardware
company such as Sony, but Microsoft had plenty of help from its
partners.
In contracts with IBM and ATI, Microsoft set early deadlines for
delivery of the first working chips. It also decided that it wanted to
own the chip designs. That way, if it wanted to, it could take the
designs and have them made by a factory of its choosing and wasn’t
locked into using IBM and ATI (or ATI’s chosen partner) chip
factories. The reason for this change was that Microsoft wanted to
avoid dependence on its chip partners. It had learned the hard way
after getting into a bad dispute with Nvidia over the prices charged
for the graphics chip in the original Xbox. The case had to go to
arbitration and Microsoft ultimately lost that battle. However, this
was a mistake: it was now responsible for failures related to the
chip.
In Microsoft’s favor this time was the fact that it had credibility
among game developers thanks to hits such as “Halo,” (pictured right)
which sold over 10 million units. Microsoft believed it could charge
more for each machine and more for its games; and not every machine
would have an expensive hard drive. In the second generation,
Microsoft planned on reducing the cost of the hardware over time.
Contract manufacturing had matured since the last generation.
Flextronics was willing to make each console for just $10 each, while
another manufacturer, Wistron, would do the same for $7, according to
sources. They hoped to make profits on large volumes of machines sold.
They were also practiced in the art of reducing the costs of the
components they put into their machines. Overall, Microsoft had a much
more formidable hardware operation, when you consider all of its
virtual resources, on the second time around. But it still wasn’t
enough to do a perfect job.
4.)
Go or no go?
Leading up to the launch in the fall of 2005, the number of defective
units would soon grow to tens of thousands. Any other consumer
electronics company would likely have postponed a launch with such low
yields. But Microsoft had more money in the bank than anyone else. The
decision this time would fall to Bach and Moore (pictured left). The
costs of launching with low yields — where you take big losses on
every product sold — could bankrupt other companies. But Microsoft
could afford to do so. Microsoft did delay the launch date from
October until November. But some inside the company still believed
returns would be out of control.
Some of the defects were “latent.” That means they may not show up for
some time after the consumers starts playing with the machine. Latent
defects can be as high as 50 percent of all defects. That means if you
have a low yield to start with, you expect to have a lot of returns
when the latent defects kick in later on. Bach said that the real
problems with the Xbox 360s didn’t really show up until a year after
the launch. That’s when returns were mounting, when they should have
been declining.
The yield problem was only discussed internally, and so the public at
large was left wondering whether Microsoft was intentionally creating
a shortage of consoles by making just a small number of machines. The
truth was that Microsoft had to produce a lot of units — many of which
failed — to get working consoles that it could ship. It was trying to
get as many machines to the market as it could.
A bad decision on testing machines?
The problems began before the Xbox 360s got to the market. The testing
machines were not ready, and the battery of tests that they ran were
not fully developed. That meant that the testing machines would
inspect the Xbox 360s coming off the line and approve them for
shipment, even though there were likely flaws.
The test machines were not properly debugged, due to an ill-advised
cost-cutting initiative that shaved $2 million from $25 million paid
to Cimtek, a test machine maker in Canada. The Microsoft team decided
not to pay the consulting fee to Cimtek to build, manage and debug the
test machines. Sources familiar with the matter said there were only
about 500 test machines at the time of launch, a third of the 1,500
needed.
“There were so many problems, you didn’t know what was wrong,” said
one source of the machines. “The [test engineers] didn’t have enough
time to get up and running.”
The Xbox 360’s yield climbed somewhat throughout the fall of 2005, as
Microsoft readied the worldwide launch in November. But the “first
pass yield” (before machines were taken off the line to be reworked
inside the factory) was never over 70 percent. Microsoft was counting
on its virtual organization of chip designers, assembly houses, and
Chinese factory workers to solve the problems. It considered this low-
cost infrastructure to be one of its main advantages over Sony, which
had high-cost operations under its own roof. But the Chinese
manufacturing machine wasn’t as honed as it is today.
If it had shut down the line until it properly debugged everything,
Microsoft might have had to delay the launch in Europe or Japan. The
Japanese market was considered important to crack, but the games being
developed for that market were late anyway. But Robbie Bach had
repeatedly said that it was strategically important for Microsoft to
fulfill its promises.
If Microsoft had an outstanding test process, then the problems with
the console might have been more glaring and Bach might have had more
reason to delay.
A tough but successful launch
On Nov. 22, 2005, Microsoft started selling the consoles. It invited
3,000 gamers to come to a hangar in the middle of the Mojave Desert.
Gamers rushed into the hangar doors like Greeks rushing into Troy.
There, they played the first games for 30 hours and were rewarded with
the chance to be the first to buy the machines. Across the country,
gamers lined up at stores and waited for midnight. They took them home
and obsessively played games like Activision’s World War II shooting
game, “Call of Duty 2.”
The hype around the launch was so strong that gamers snapped up every
machine. Microsoft had hoped to sell millions, but it was running
short in Europe and the U.S., while machines in Japan were barely
moving off the shelves. By the end of 2005, Microsoft had only shipped
1.5 million consoles worldwide. That compared to 1.4 million shipped
in 2001 with the original Xbox, when it launched in only the U.S.
There was now much more demand. Consumers were understandably upset
with the relative shortage, since an equal number of units was now
shared across three major territories. In Japan, the lack of home-
grown Japanese games to go with the Xbox 360 consoles meant that
demand was tepid; machines sat unsold in stores even as consumers in
the U.S. were paying hundreds more beyond retail on auction sites such
as eBay.
The consoles started failing almost immediately. Consumers were
excited about being the first on their block to own the new system.
They could run graphically beautiful games on high-definition flat-
screen TVs, which were just becoming affordable. But their
disappointment upon encountering the red rings of death was evident in
posts on the Internet. Some customers swore off Microsoft. At the
time, Microsoft said that it had received “isolated reports” of
console failures and that returns were within the normal range.
Many consumers complained that Microsoft’s customer support people
often tried to talk them out of doing a return. (The support people
could come across as condescending. They would caution users to avoid
putting the console on a plush carpet, or inside a home entertainment
center with no air flow). It would be many months before the company
set up an easy process where customers could track their returns and
replacement units on the web.
According to the Consumer Electronics Association, the average return
rate for products where the consumer gets their money back is about 2
percent. That is based on a study conducted in 2005 which showed the
return rate for any consumer electronic product is fairly constant,
said Steve Koenig, a senior analyst at the CEA.
According to Microsoft’s internal data, the assumption for the long
term was that 6 percent or 7 percent of the consoles would be
defective before shipping into the market, meaning the yield would be
about 94 percent or 93 percent. The rate of return was expected to be
low as well. But even after the holidays, Microsoft continued to
struggle to ship enough units.
Good quality is taken for granted as a requirement for doing business
these days, but it is still difficult to assess what is considered a
“normal rate of product returns or yield rates,” said Shukri Souri, a
manufacturing expert at Exponent, a Menlo Park, Calif. company that
figures out why products fail.
“A significant factor that determines rate of return is how complex a
product is to build,” Souri said.
The Xbox 360 was in fact more difficult to build than anything the
team had built before. It had 1,700 parts, many of them hard to make.
There were more than 200 suppliers. The chips and the box (insides
pictured left) were designed to balance performance, aesthetics and
power dissipation. As Holmdahl pointed out, a problem with any one of
the parts or any one of the vendors could result in a problem that
affected the whole box.
In 2006, it only got worse
Despite the problems, some results were turning out better than
expected, such as how quickly Microsoft would sign gamers up for its
Xbox Live online games service. (That figure has now hit 12 million
subscriptions for both paid and unpaid subscriptions, more than
originally forecast).
But some things, Microsoft clearly didn’t plan for. The complaints of
shortages and problems with consoles continued into the spring of
2006. By the end of March 31, Microsoft said it had shipped more than
3.3 million consoles to retailers. There was a growing “bone pile” in
a warehouse at Wistron and at a repair center in Texas.
Microsoft had more than 500,000 defective consoles that sat in
warehouses. They were either duds coming out of the factory or they
were returned boxes, according to inside sources. The yield was
climbing, but far too slowly. The company stood by its statement that
returns were within “normal rates for consumer electronics products.”
In the spring of 2006, Moore said with some relief that Microsoft was
finally readying a surge of new consoles for the market. Celestica, a
third manufacturing partner, was bringing its factory online. It
seemed that Microsoft was finally turning a corner on production.
Because Sony and Nintendo didn’t have their new machines ready yet,
Microsoft had a wide-open field in 2006. It was racing to snare the
most customers before the rivals could compete. But the cheaper priced
and older PlayStation 2 continued to outsell the Xbox 360 throughout
2006.
Customer complaints gather steam
Szarek, the previously mentioned resident of Chicopee, Mass. gained
some notoriety when he said that he suffered three failed consoles by
May, 2006. His first console had lasted only 28 days after he picked
it up. Then he became even more frustrated with the returns process.
Under the glare of publicity, Moore personally apologized to Szarek
and made sure he got a free game and wasn’t charged for the repairs.
Moore’s comments were conciliatory and he managed to calm some of the
critics. One of the problems: though the return rate was predictably
high, Microsoft had not made big investments in its return
infrastructure to enable a smooth return process. One PR woman said
she had to do shifts doing support calls.
“When you have tens of millions of people who own something, you’ll
have people using the Internet as a forum,” Moore said at the time.
“I’m not hiding behind the fact that people have had problems with the
console and at times we have been deficient with the way we’ve handled
them. We are improving our customer service. There were times when we
did not handle complaints the way we should have. I take
accountability for that. I have not seen since that any further
issues.”
The denial of widespread problems infuriated Szarek and others who
felt Microsoft was stonewalling. As the inventory surge materialized,
Microsoft was able to send replacement units out more quickly and deal
with those who were unhappy about the shortage. Sales topped 5 million
by June 30, 2006. Moore had personally set a goal of selling 10
million machines before his rivals sold their first machines. That was
going to be tough, since Sony and Nintendo had set their launches for
the fall of 2006.
5.)
The internal response
During the spring of 2006, the engineers were working on a transition
to a new motherboard, code-named Zephyr. But they postponed that
transition to work on fixing the “bone pile” issues and “maximize the
yield,” according to an email circulated by engineering manager Harjit
Singh to the hardware team on March 10, 2006. The yield at that point
was an abysmal 50 percent on the first pass. When the bad machines
were reworked within the factory, the yield went up to 75 percent –-
hardly acceptable. Singh had said the process in the factories was
“not repeatable” at a time when they were scheduled to triple
production in the coming months. That meant that something could go
slightly wrong and Microsoft would have no idea how to fix the
problem.
Microsoft finally started throwing more engineers at the problem. It
took engineers off projects such as reducing the costs of its wireless
controllers because, Singh wrote, “if we don’t have the consoles, we
don’t need the peripherals.” Marc Whitten was temporarily assigned to
work on evaluations for adding new heat sinks to address thermal
issues on the big chips. It was essentially “all hands on deck” for
engineers, who were expected to devote 75 percent of their time to the
yield/bone pile issues. But, again, those engineers didn’t achieve
magical fixes. And it was late in the game.
The company pushed to stir up sales. Because Sony appeared to be
falling off schedule a second time, the Microsoft executive team
challenged the engineers to improve shipment targets by 25 percent
through June, 2006. The team decided to postpone the launch of Zephyr
(a board with HDMI connector for better video quality) and concentrate
on shipping Xenon boards. The target was to hit 80 percent first-pass
yield, but that wasn’t reaslistic.
During the fall, Microsoft offered bundles, or packages of games and
consoles together, which effectively amounted to a price cut. Some
analysts accused Microsoft of “stuffing the channel,” or shipping more
consoles to retailers than they wanted or needed for the holidays.
Microsoft denied those claims.
“We’re a very responsible company,” Moore said to me in the May, 2007
interview. “How do you determine who stuffed the channel? What is the
optimal number of units? We’re not in a position to force the product
upon the retailers. It wouldn’t be the console wars without conspiracy
theories.”
But problems with manufacturing continued. The yield on the consoles
was only 85 percent at the beginning of 2008, not at the 90-percent
plus level that was expected at this point in the console cycle. That
meant there was an extra billion dollars in cost that Microsoft hadn’t
anticipated. But, again, Moore at the time said that the manufacturing
problems Microsoft had were not enough to ruin the business model.
Dead Rising
Robert Delaware, a former game and hardware tester at one of
Microsoft’s contractors, Redmond, Wash.-based VMC, saw the problems
popping up in the spring of 2006 as he was testing consoles and new
games. Like many testers, he was in the dark about the overall
problems and only saw a small slice of the efforts to debug and fix
consoles. He found a bug that could cause a reproducible crash on
every game he tested.
“If you coordinated the music player with the dashboard, you could get
almost every 360 to lock up,” he said in an interview. “I did it first
on a combo DVD/audio disk. With NBA 2K6, you would select the music.
The screen went black.”
The NBA 2K6 flaw resulted in the Red Rings of Death. Delaware, who as
of this writing works at Microsoft as a game tester, agreed to go on
the record for this story because he said he believes passionately in
his work, which involved painstakingly playing games over and over
again in order to uncover bugs. But he also believed passionately in
the Xbox 360 gamer community. He is one of those thousands of
employees who was good at his job, loved the games, and sought to push
Microsoft to excellence. But he said he found it difficult to stay
silent on problems that caused a number of his friends a great deal of
consternation. He said he realized that speaking out could cost him
his job and he plans on asking Microsoft to forgive him for it. He was
courageous in coming forward, and he was not the source of emails
cited in this story.
When the Capcom game “Dead Rising” (pictured left) shipped in the
summer of 2006, another flaw emerged. Console owners reported putting
the game disks into their machines and then seeing a breakdown. One of
the reasons was the process by which Microsoft updated the Xbox Live
online service. Many of the system owners were not connected online so
they didn’t get live fixes downloaded to their machines. Thus,
Microsoft had to ship Xbox console fixes with some of the consoles. In
the case of Dead Rising, the game came with a bug fix that actually
crippled some consoles. Microsoft often did a couple of firmware
updates a year, and sometimes they led to broken machines.
Delaware was convinced that at least some of the problems reported
about Dead Rising were related to the “2858 dashboard update” for Xbox
Live, which he believed was embedded in the game. Another explanation
was that games such as Dead Rising demanded more from the consoles,
which gave out under stress. Delaware believed that the practice of
embedding console and dashboard updates in game disks is responsible
for some hardware failures.
In October, 2006 Delaware was concerned about how much information
Microsoft was sharing with its testing partner and offered a warning
about the Xbox Live updates. “With the upcoming Wii and PS 3 launches,
MS cannot afford the bad PR that might result from Update related
issues,” he said. “Asking these questions from the start might help to
prevent possible criticism of our testing process, in the event of
update-related problems. The last thing VMC needs is to take the blame
for problems that Microsoft has (known) about from the start.”
Shooting too high, moving too fast, stretched too thin?
So what exactly was wrong with the machines? As time would reveal,
there was no single reason for the failures, though many of the
problems could be blamed on the ATI graphics chip, which could
overheat so much it warped the motherboard. This put stress on bad
solder joints, causing them to fail early in the machine’s life.
Sometimes the heat sinks on top of the GPU were put on the wrong way,
resulting in heat problems. Finally, games would sometimes crash
because of sub-par memory. Infineon had been brought aboard as the
second supplier behind Samsung for the GDDR3 memory used in the Xbox
360. This new kind of memory chip was specified for 700 megahertz, but
the Infineon parts were falling short of that target. Microsoft had to
set up a line for sorting through the good parts and the bad parts,
contributing to a shortage of consoles.
Problems with the DVD drive also lasted longer than expected. And the
console was also one of the first products that had to meet new
environmental standards in Europe, which prohibited the use of lead in
solder (which, when melted, fuses electronic components together).
Paul Wang, a Microsoft test engineer on the Xbox 360, said in a speech
in 2007 before a Silicon Valley engineering group that the lead-free
solder created a lot of problems.
Perhaps Microsoft had too many balls in the air. As it launched the
Xbox 360, it was already planning other big related projects. Cobalt
was a next-generation HD-DVD movie player (pictured above) that
Microsoft would launch in the holidays of 2006 as a $199 add-on. A
project code-named Zephyr (later renamed the Xbox 360 Elite) would add
a more expensive hard disk drive with 120 gigabytes of storage and a
new HDMI connector. And the company’s engineers were hard at work with
suppliers on a project code-named Falcon, which was an effort to
create a motherboard, or main circuit board inside the box, to reduce
the costs of the console itself during 2007.
“That was really annoying, that they had us working on so many
things,” an engineering source said.
As the engineers wound down one task, they had to move to the next.
There was no team set aside to deal with the low yields, in part
because the poor yields hadn’t been anticipated.
Fessing up
In September, 2006 that the company made a rare admission. It said in
a statement that the quality of the consoles it made during 2005
wasn’t as high as it expected and therefore it would extend the policy
of free replacement for consoles made during 2005, even though the
warranties had expired.
While some industry observers expected Microsoft to cut the costs on
the console and offer consumers a price cut during the second holiday
season, that was never in the plans. Instead, the company was ramping
up to offer a new accessory, an HD-DVD drive that could play next-
generation DVD movies for the low price of $199. That was meant to
parry Sony’s PlayStation 3, which had a Blu-ray next-generation DVD
drive built into the system.
Fortunately, Sony was stumbling. Besides being a year late, the costs
on the Blu-ray drive were so high that Sony had to price its consoles
at $499 and $599, far above Microsoft’s prices of $299 and $399.
Market analysts estimated the cost of Sony’s box was around $800 or
more. Microsoft didn’t need to offer a price cut after all,
particularly since Sony was struggling to build enough consoles for
its fall launch.
It was looking like, even with the manufacturing problems, Microsoft
was going to do just fine. In mid-November 2006, about 645,079 units,
or 8.8 percent of the total units shipped, had been returned for
repair. Sony had been more conscientious about delaying its launch
when the product wasn’t ready. JackTretton, the president of Sony’s
U.S. games division, said he was proud that Sony waited. But it was
going to pay for that cautiousness.
The three Microsoft contract manufacturers had now ramped up to full
production. They produced more than 12 million consoles. In early
2007, Peter Moore announced that Microsoft had hit its revised target
and had shipped more than 10.4 million consoles into the market before
the end of 2006. It only sold through 9 million consoles. There were a
lot of leftover consoles sitting on shelves.
At about the same time, in December, 2006, Microsoft took another step
to appease consumers who had failed machines. It said that it was
extending the warranty of free repair for all consoles made for
customers in the U.S. and Canada. Now the machines would carry a one-
year warranty, not just 90 days.
A drastic measure
Microsoft decided to shut down manufacturing of the Xbox 360 in
January, 2007. Between January and June, it didn’t build any new
machines. The reason was partly because it made too many machines
earlier, but the other reason was to track down the source of its
quality problems. It did that even though it was launching a new
version of the console, dubbed the Xbox 360 Elite (pictured left),
with a bigger hard drive and a new HDMI connector for $480. It had a
new board, dubbed Zephyr, inside. That new black version of the
console was made in limited quantities for hardcore fans who might
otherwise be attracted to buying the most expensive PS 3.
Since it had an oversupply, Microsoft had a cushion. It could afford
to suspend manufacturing as it sold off its inventory of consoles. The
company stopped promoting the console heavily and shaved a million
units off the goal for the fiscal year ended June 30, 2007. The
problem was that Nintendo’s Wii was starting to catch fire. Nintendo
had come up with a cheap and innoviative console, which was proving
extremely popular at $250, about $50 to $150 cheaper than Microsoft’s
Xbox 360 models. During the six-month period ending June 30, 2007,
Microsoft would sell only 1.2 million consoles. It was a disastrous
slowdown, compared to the Wii’s sell-out success.
In January, 2007 at the Consumer Electronics Show, Bill Gates
announced that the Xbox 360 would be capable of serving as an IPTV
set-
top box, meaning that telephone companies could use it to deliver
video programming and other services that could compete with cable TV
providers. Gates also said that because the Xbox 360 had beaten the PS
3 to market and didn’t include a hard drive on every machine,
Microsoft would be able to ride a silicon cost-reduction curve faster
than Sony, which saddled every unit with Blu-ray and hard drive costs.
At any given point, the Xbox 360’s costs will be lower and more easily
reduced, giving Microsoft a fundamental advantage over Sony in
pricing, Gates said.
The Cassinghams stir discontent
The topic of defects flared up again in February, 2007, when Rob and
Mindy Cassingham (pictured left) of Moab, Utah, revealed that they had
the frustrating experience of returning seven defective Xbox 360s.
They were die-hard fans who had driven across the country to be
present at the launch of the Xbox 360.
They used several in their retail gaming center, where kids came in to
play against each other. That prompted critics to blame the
Cassinghams for over-using the machines. But several three of the
machines that broke down were actually personal machines that the
Cassinghams used in their own homes. Consumer outrage flared, and
Moore again offered apologies to customers. One game magazine, 360
Gamer, conducted a poll that found many console owners reported more
than one console failure.
Fixing bugs in the trenches
Microsoft quietly acknowledged that it was no longer using Wistron as
a manufacturing partner. In April, 2007, Larry Yang, the general
manager of Xbox division hardware, said in a group email that he was
resigning that job and would take another position within Microsoft.
He would still stay and help Microsoft get out its most important
redesign, dubbed Falcon.
Falcon was the code name for the cost-reduced Xbox 360 that included a
major redesign of its CPU. Scheduled for once every couple of years,
the transition to new chips was a big deal. Microsoft’s partners had
been making their chips with 90-nanometer technology since 2005. Now
they would make the CPUs with 65-nanometer technology, meaning the
widths between the circuits were smaller. With smaller circuits, the
same chip design can fit on a smaller amount of space. With less
space, the chip can be manufactured with less material, resulting in
lower costs. Quality also gets better since it’s easier to make
smaller chips than big chips. Shifting to a new generation of chips
was thus the best way to reduce the cost of a game console and improve
its quality.
An email sent out by S. Srini, director of Xbox manufacturing, on May
27, 2007, said that the overall yield of units in production was 85
percent. But the email said that the IBM microprocessors were still
exhibiting “excessive failures” of 4 percent to 8 percent because of
heat problems. There was still no “fresh start” manufacturing going
on.
The hardware engineers worked mainly to address the overheating of the
graphics chip from ATI. The chip was overheating and causing both the
chip and the circuit board to warp, cracking the joints that were held
together by the lead-free solder. Given enough time, many units will
suffer this problem. To fix the problem, Microsoft put epoxy on both
the IBM microprocessor and the ATI graphics chip. They also took out a
50 cent extruded aluminum heat sink (which sits on top of a chip and
dissipates heat away from it) from the graphics chip and replaced it
with a $5 heat sink that could do the job more efficiently. They also
used a pipe to move heat in front of a fan.
Beyond the temporary solution of the heat sink change, Microsoft made
more changes. Microsoft tried to get Falcon out as fast as it could.
While the system box would remain the same size, there would be fewer
components inside. That, in turn, would leave more room for air flow
and to reduce the use of fans in the system, which meant it wouldn’t
be as noisy as before. When rivals such as Sony made this kind of
switch before, they had big shortages because it was always difficult
to make the transition.
Engineers were under pressure from the top executives to describe the
nature of the problem. When the engineers finally had the situation
under control, Microsoft finally announced its free replacement
program on July 5, 2007.
“Hopefully consumers will recognize we’re trying to do the right
thing,” Moore said in an interview that day. “It’s a courageous step
because it is not an inconsequential step. We are not burying our
heads in the sand.”
The announcement put a dent in the cottage industry for repairing Xbox
360s (such as the book pictured at left). Even so, in early 2008,
companies such as game retailer GameStop were making a fortune
repairing thousands of Xbox 360s a week and then returning them as
refurbished units to store shelves at a discounted price. Most of
GameStop’s repairs were successful and involved re-soldering the
graphics chip to the main board; unsuccessful repairs were passed on
to Microsoft.
Peer Schneider, vice president of content publishing at gamer web site
IGN, said, “The extended warranty and the reimbursement policies are
important steps to win back consumer trust. The only question from
Xbox fans is: what took Microsoft so long?”
6.)
After the write-off
There has been a lot of executive turnover in Microsoft’s game
division, but none of it has been laid directly at the feet of the
defect problem. Moore left Microsoft a year ago to become head of the
sports label at Electronic Arts, while Don Mattrick, the former head
of EA’s worldwide game studios, took the top games job under Bach at
Microsoft. Bach said that Moore’s departure had nothing to do with the
RROD problem, and Moore said the reason for his departure was that he
and his family wanted to return to the San Francisco Bay Area. Bach
remains one of the top executives at Microsoft, running the
Entertainment & Devices group, (including the IPTV, Games for Windows,
Xbox, and Zune businesses) which reported its first-ever profit in the
fiscal year that ended June 30, 2008. Allard is chief technology
officer of the E&D group, while Todd Holmdahl is head of games
hardware.
Oddly enough, demand for Microsoft’s console continued, partly because
gamers loved playing games so much that they kept on buying
replacement consoles. By the fall of 2007, Microsoft had its Falcon-
based machines ready. It was able to cut the price of the console, but
not so much that it hurt its bottom line. The launch of “Halo 3” in
the fall helped spur console sales, helping Microsoft stay ahead of
Sony and at least stay competitive with the Wii in terms of game
sales. In a development that was reminiscent of the “Dead Rising”
failures, a number of Halo 3 buyers said the game appeared to fry
their consoles.
Microsoft never announced how many machines had been returns. Based on
an estimated $60 repair price, the company’s reserve can cover lots of
machines. In early 2008, SquareTrade, which sells warranties for
electronics, reported that there was a 16.4 percent failure rate for
Xbox 360 systems. Michael Pachter, an analyst at Wedbush Morgan,
estimated that the number is 3 million, or about 15 percent of the
total. It isn’t clear if the numbers should count consoles that fail
and aren’t repaired for free (only those with the three red lights are
replaced for free); also retailers such as GameStop have begun
repairing many of the units themselves. Bach said that Microsoft’s own
research shows that fans continued to love the console and buy games,
in spite of the problems with the defective units. Pachter believes
that is true.
“In the ordinary course of something like this, you would expect it to
show up in the customer reaction data,” Bach said. “We just haven’t
seen that. It speaks to the fact that they love their games and Xbox
Live. Does it frustrate them? Yes. On the other hand, they know we’re
taking care of them. People have a certain amount of respect for that.
If it had happened on a product that had less baseline customer
satisfaction, it would have had a bigger impact. We really haven’t
seen that.”
At this point, the Wii has begun to far outsell its rivals on a
worldwide basis. It’s not inconceivable that it will sell as many
units as its two rivals combined. It isn’t clear how much the RROD
problem hurt Microsoft, which remains ahead of Sony. The scorecard so
far: the Wii, 29 million; Xbox 360, 20 million; PS 3 (pictured left),
14.4 million.
Was it worth it to go first, despite the concerns of the engineers and
the inevitable consumer return problems?
“If you take the question of whether it was the right thing to try to
be first, the answer to that is definitely yes,” Bach said in July
2008. “It has given us a leg up in a number of places that are super
important. It has given us a leg up with game developers. It has given
us a leg up from an economics perspective. It helped us expand Xbox
Live quickly. At a strategy level, if you asked if we wanted to be
first again, I would say yes. It’s easy for me to go back and say, if
I knew what would transpire over the next two years, would I go back
and do something different, I think that’s an obvious answer. But the
fact is, based on the data we had at the time and all the hard work we
put into it, there was no way to see what actually happened.”
The problem for Microsoft was that it sacrificed something to rush
into the market and establish the Xbox 360 as the premiere platform of
the living room. What it sacrificed was the good will of consumers,
who are actually critical in terms of establishing a lasting platform.
As for profits, Microsoft has finally turned a fiscal-year profit in
the Entertainment & Devices group. It did so because it wrote off the
costs of the defective units and warranty program in the previous
fiscal year. (The $1.15 billion write-off covered several years of
repair costs, some of which would have otherwise been incurred during
the profitable fiscal year.) But Bach said that the business model
for the Xbox 360 isn’t permanently ruined.
“It doesn’t make it good,” he said. “Would I like the billion dollars
back? Of course, the answer is yes. But there is still plenty of
economics in the business for us to be successful.”
Microsoft hasn’t been as aggressive at cutting prices this generation,
but Sony has a more expensive box and so it hasn’t been putting price
pressure on Microsoft. Had Sony been more aggressive, the consequences
could have been much worse in terms of customer defections due to the
RROD. Even with the $50 to $79 price cuts announced by Microsoft on
Wednesday, it is taking a lot longer for prices to fall in this
generation.
Jerry Knoben, a former Intel manufacturing chief, joined Microsoft in
January this year as head of Xbox 360 manufacturing. He recently sent
out a memo to the staff saying that the yield on Xbox 360 production
had improved from the mid-80-percent-range to 92 percent. But he said
that there were still some surprises in manufacturing that Microsoft
had to fix.
The improved consoles use a new motherboard, dubbed Jasper. This one
has both a 65nm IBM microprocessor as well as a 65nm graphics chip
from AMD. The improved graphics chip may finally run at acceptable
thermal limits. But it’s very late in coming, compared to the rest of
the chip industry, in part because Taiwan Semiconductor Manufacturing
Co., the chip contract manufacturer making the graphics chips, was
slow to shift to 65nm factories. Late or not, Jasper is what enabled
Microsoft to cut its prices this week.
By next year at the earliest, Microsoft is scheduled to launch a new
iteration of the Xbox 360 with a board dubbed Valhalla. That board
will have a single chip that combines both the graphics chip and
microprocessor on a single device. This is the kind of technical tour
de force that Sony achieved with the chips for the PlayStation 2, a
move that allowed the company to significantly reduce manufacturing
costs. Microsoft still has an advantage over the PS 3 in manufacturing
costs, since it doesn’t have an expensive Blu-ray player.
Shane Kim, (pictured left) until recently head of Microsoft Game
Studios and now head of the game division’s business development
efforts, recently said that it was too early to call the console war
in favor of Nintendo. We won’t know the winner, he says, until
somebody sells 100 million units. Microsoft’s plan is to sell 75
million Xbox 360s. The only way they could ever do that is to be more
aggressive on their pricing, which means driving down the costs of the
console as low as possible. And by doing the right thing for their
customers from now on.
Sony and Nintendo aren’t safe just yet. Bach has always talked about
how Microsoft has a 10-year to 15-year plan for success. Maybe Xbox
3.0, just like Windows 3.1, will do the trick.
To conclude, the video game industry has never seen a consumer problem
as bad as the “red rings of death” and the size of the $1.15 billion
charge stands as one of the biggest liability glitches in consumer
electronics history. How Microsoft handled the flaw may provide a
lesson for all modern electronics companies; that is, if you are going
to promote the hell out of something, it better work the way you say
it does and you better have a strong customer support and engineering
debugging team to back it up.
Amazon and eBay have their periodic outages. Sony had its problems
with batteries, Intel had flawed Pentium chips that were bad at math,
Nvidia took a $200 million charge to deal with graphics failures on
laptops, and Apple has had its share of glitches on the iPhone. As
electronics become and more complicated, it becomes a practical
impossibility to test for every single flaw. But with the force of the
Internet behind them, angry customers can exact a revenge that can be
extremely damaging to a company’s brand image.
One thing is clear. Microsoft has to move beyond its mentality of
being a software company that can launch fast and fix later. With
global markets and global launches, the consequences of such a
cavalier approach to hardware quality can start to pile up. The
company clearly has the cash and the size to compete among the best
hardware companies out there. It cannot afford to tarnish its brand
when competitors on various fronts — such as Google on the web side
and Apple, Sony, and Nintendo on the hardware side — are there to
exploit its stumbles.
At this writing, it seems clear Microsoft’s hegemony in operating
systems and productivity software won’t last forever. The company has
limited time to get the rest of its house in order. It has to
diversify into fields such as video games and make them pay off with
profits. Microsoft has demonstrated an incredible tenacity in a lot of
markets. It took almost a decade for the company to march to victory
with its Office productivity software. I have no doubt the company can
execute. It just has to decide that quality is something that matters.
If Bach is correct and Xbox 360 gamers are loyal, then Microsoft has
been lucky. In the future, I suspect consumers won’t be as forgiving
as fans such as Chris Szarek, who doesn’t own any other console and
has bought three dozen Xbox 360 games.
Szarek said, “They got me. There was a time when I was furious with
them.”
So why did he stay with the Xbox 360?
“What can I say?,” he said. “I like the games.”