OPTERON 14x vs. 24x - is there a real difference?

[nobody]

Is there any real difference between 14x and 24x, aside from the
markings and the price? AFAIK, Athlon XP and MP are not different,
except for some bridges that are cut on XP and can be reconnected to
make XP work in dual config. The reason for the question is simple -
2 x 146 cost about as much as one 246. Does anybody have any
experience running 14x in dual config? Is any tweaking involved?
Any info will be greatly appreciated.

 

[Rob Stow]

Is there any real difference between 14x and 24x, aside from the
markings and the price? AFAIK, Athlon XP and MP are not different,
except for some bridges that are cut on XP and can be reconnected to
make XP work in dual config. The reason for the question is simple -
2 x 146 cost about as much as one 246. Does anybody have any
experience running 14x in dual config?

It can't be done. You genuinely need to use either
the Opty 2xx or 8xx if you want to build a dualie.

One of the differences between a 1xx and 2xx is
simply one more HT link - and without that additional
HT link, multiprocessing is not possible.

Is any tweaking involved?

It would have to be done by the motherboard manufacturers -
and none of them have the slightest interest in doing so.

With an Athlon or Xeon system, the glue logic in a multiprocessor
system is provided by the chipset on the motherboard. With
Opteron it is built into the processor and it is dependent on
having that additional HT link for interprocessor communications.
Nobody has any interest in developing a chipset for SMP
with Opty 1xx when the SMP capabilities built into the Opty
2xx and 8xx do the job so well already.

Any info will be greatly appreciated.

Lots more for you at AMD's site, as well as places like 2cpu.com.

 

[Black Jack]

Is there any real difference between 14x and 24x, aside from the
markings and the price? AFAIK, Athlon XP and MP are not different,
except for some bridges that are cut on XP and can be reconnected to
make XP work in dual config. The reason for the question is simple -
2 x 146 cost about as much as one 246. Does anybody have any
experience running 14x in dual config? Is any tweaking involved?
Any info will be greatly appreciated.

All Opterons have 3 Hypertransport links, but depending on which model
of Opteron some of the links are Coherent Hypertransport links while
others aren't. You need the cHT links to do "glueless"
multiprocessing. Opteron 1xx has no coherent links therefore you can't
do multiprocessing with it. (Opteron 2xx has one cHT link, while
Opteron 8xx has all three as cHT).

However, there is an exception to the case. If you're in the market
for a Cray Red Storm system, then you can use multiple Opteron 1xx
processors. That's because Cray implements its own coherency protocol.
Of course, this can't be considered "glueless" multiprocessing, since
you need outside circuitry to pull it off. But the option is there if
you need it.

Yousuf Khan

 

[The little lost angel]

All Opterons have 3 Hypertransport links, but depending on which model
of Opteron some of the links are Coherent Hypertransport links while
others aren't. You need the cHT links to do "glueless"
multiprocessing. Opteron 1xx has no coherent links therefore you can't
do multiprocessing with it. (Opteron 2xx has one cHT link, while
Opteron 8xx has all three as cHT).

Just curious, I've been looking at this and can't figure out why is it
limited to 8way? Is it due to some timing constraints that would be
exceeded if they went 9-way or 10-way or something? Could they get
around this simply by adding more cHT links in future Opterons?


--
L.Angel: I'm looking for web design work.
If you need basic to med complexity webpages at affordable rates, email me
Standard HTML, SHTML, MySQL + PHP or ASP, Javascript.
If you really want, FrontPage & DreamWeaver too.
But keep in mind you pay extra bandwidth for their bloated code

 

[Tony Hill]

Just curious, I've been looking at this and can't figure out why is it
limited to 8way? Is it due to some timing constraints that would be
exceeded if they went 9-way or 10-way or something? Could they get
around this simply by adding more cHT links in future Opterons?

8-way is the maximum number that you can connect the chips together,
using the 3 HT links, while keeping your worst-case number of "hops"
between chips to 3. For 9 processors your worst-case would require
you to go through 4 HT links to access remote memory. I would guess
that at this point your HT network would really start getting bogged
down with all the interprocessor memory requests. Some people have
actually suggested that even 8-way systems may got really bogged down
with remote memory requests.

There's also the simple matter of physical layout. Hypertransport is
designed for direct chip-to-chip communication on a single board.
Fitting 8 processors on a single board is REAL tough to do and fitting
more than 8 processors on a board might well be impossible. It's
somewhat unclear, from my reading of things, if it's possible for
hypertransport to work through daughterboard connectors, ie you have a
fairly bare-bones mainboard onto which you have 4 sockets to plug in
daughterboards, each with two processors and their accompanied memory.
If this is not possible, it's probably not going to be possible to
make a (glueless) 8-way Opteron system as it is, let alone anything
bigger.

Finally there's simply the marketing issue. As it stands now no one
has seen fit to make anything larger than a 4P Opteron server. Really
the 1-4P server market is the ONLY market segment that is growing, the
market for >4P servers is kind of disappearing. The number of tasks
that need such large servers is pretty darn small these days, and
while people talk a lot about virtualization (ie making one big server
look like lots of little servers), it seems to be all talk and no
action for the most part. Given that there isn't much of a market for
8P servers, there almost certainly won't be a market for >8P servers.

 

[Dorothy Bradbury]

8P does not seem to be a sweet spot on the SMP curve.

When you move into 8P, it's a system integration team:
o Buyers will probably want 2 servers
---- if important enough for 8P - uptime must be critical
o Buyers will want a lot of other powerful bits
---- very large number of dimm sockets = big VRM = backplane
---- very large number of drives = custom backplane
---- very large power demands = custom redundant PSUs for 8P
o Buyers are few
---- what business case gets past the economic buyer for 8P
---- a lot of R&D to spread over many parts & few sales
o Competitors are established
---- can an 8P box work re price / feature-set against SUN?

Cost benefit of glueless-8P is lost on many other area R&D costs.
Cost benefit of glueless-4P is here - cheap OTS 4P boards.

SMP wise, 4P Opteron is very attractive for dbase apps:
o Cheap OTS board/chips/PSU + lots of Ram + good CPU
o Cheap integration into low-U form-factor

Blades offer hundreds of CPUs cheaply, altho utilisation is moot.

I'm not sure a dbase app will be CPU-bound by 4P Opteron, so
going up to 8P is the wrong solution - probably focus on disk I/O.

Pity the dual-opteron board/CPU isn't a bit cheaper tho.

 

[nobody]

Yousuf,
Thank you for the reply. However, while it answers the main question
(and, unfortunately, kills my hope to get SMP system on the cheap), it
rises even more questions. I wonder how AMD makes 1xx different from
2xx different from 8xx. To me, the possibilities seem as follows:
1. It's etched into the die. While not totally impossible, very
improbable. AMD, unlike INTC, doesn't have so many fabs to dedicate
the whole line to produce only 1xx, the other for 2xx, and yet another
one for 8xx, not counting all the Athlons that are still the bulk of
AMD production.
2. Extra HT links burned out with a laser or otherwise disabled, just
like INTC used to castrate some 486DX to sell them as SX. Requires
extra equipment and operations that AMD could ill afford.
3. Some pins deliberately connected (or even more likely not
connected) during final assembly. The most probable way, but leaves
some hope to work around. Anybody knows the hack?
4. The difference is only in markings and price. From my previous
expreience with some other hardware, and even more from reading about
other folks' experience, I know that the fact that the maker did not
validate the hardware for certain mode of operation does not
necessarily preclude it from such operation.
So the question is how the difference between 1xx, 2xx, and 8xx is
implemented.
Please note that what I wrote is not an opinion of a professional, but
rather some amateur speculations. Whoever wants to correct me is
welcome to do so.
Thanks

 

[KR Williams]

Yousuf,
Thank you for the reply. However, while it answers the main question
(and, unfortunately, kills my hope to get SMP system on the cheap), it
rises even more questions. I wonder how AMD makes 1xx different from
2xx different from 8xx. To me, the possibilities seem as follows:

1. It's etched into the die. While not totally impossible, very
improbable. AMD, unlike INTC, doesn't have so many fabs to dedicate
the whole line to produce only 1xx, the other for 2xx, and yet another
one for 8xx, not counting all the Athlons that are still the bulk of
AMD production.

A "whole line" doesn't have to be dedicated to a single product,
particularly to products that are so similar. That said, I don't
buy this explanation either.

2. Extra HT links burned out with a laser or otherwise disabled, just
like INTC used to castrate some 486DX to sell them as SX. Requires
extra equipment and operations that AMD could ill afford.

Hardly. Exactly what equipment is needed here that AMD "couldn't
afford"? Compared to the cost of the fab, a little fuse-blow is
trivial.

3. Some pins deliberately connected (or even more likely not
connected) during final assembly. The most probable way, but leaves
some hope to work around. Anybody knows the hack?

Perhaps, though (anti)fuses are the tool to do these sorts of
things today.

4. The difference is only in markings and price. From my previous
expreience with some other hardware, and even more from reading about
other folks' experience, I know that the fact that the maker did not
validate the hardware for certain mode of operation does not
necessarily preclude it from such operation.

A possibility, but there is little reason to leave untested
functions enabled either. Perhaps...

So the question is how the difference between 1xx, 2xx, and 8xx is
implemented.

5. My bet is either #4 or (anti)fuses. Thse pups are becoming
common-place for other reasons (e.g. array redundancy). It's
trivial to castrate chips using fuses these days.

Please note that what I wrote is not an opinion of a professional, but
rather some amateur speculations. Whoever wants to correct me is
welcome to do so.

I don't think you're ever going to get an official answer here.
Perhaps some educated guesses though...

 

[Black Jack]

Just curious, I've been looking at this and can't figure out why is it
limited to 8way? Is it due to some timing constraints that would be
exceeded if they went 9-way or 10-way or something? Could they get
around this simply by adding more cHT links in future Opterons?

I think that's all it is, some timing constraints that they would
rather not deal with. Although a lot of people think of Opterons as
point-to-point multiprocessing chips, they are more like token-passing
ring architecture, rather than a star topology. The more nodes there
are on the token ring, the more points in between they have to pass
through.

Yousuf Khan

 

[Black Jack]

1. It's etched into the die. While not totally impossible, very
improbable. AMD, unlike INTC, doesn't have so many fabs to dedicate
the whole line to produce only 1xx, the other for 2xx, and yet another
one for 8xx, not counting all the Athlons that are still the bulk of
AMD production.

AMD as it now stands is producing a combination of K7 Athlons and K8
Athlon 64's and K8 Opterons out of the same plant in Dresden. And of
the K8 Athlons, there are variations which have 512K and 1MB cache,
and variations which have 64-bit mode turned off. You don't need
multiple plants to produce different lines of processors. In fact, the
contract chip makers like TSMC and UMC are producing completely
different chips for totally different fabless customers worldwide out
of individual fabs.

2. Extra HT links burned out with a laser or otherwise disabled, just
like INTC used to castrate some 486DX to sell them as SX. Requires
extra equipment and operations that AMD could ill afford.
3. Some pins deliberately connected (or even more likely not
connected) during final assembly. The most probable way, but leaves
some hope to work around. Anybody knows the hack?
4. The difference is only in markings and price. From my previous
expreience with some other hardware, and even more from reading about
other folks' experience, I know that the fact that the maker did not
validate the hardware for certain mode of operation does not
necessarily preclude it from such operation.

Although these are all possible, I think another possibility is far
simpler, AMD simply produces a different EPROM for each type of chip.
All K8's are produced identically (well except for cache size maybe),
but an EPROM tell its how many Hypertransport and Coherent
Hypertransport links it has available to it. Firmware in other words.

Yousuf Khan

 

[Tony Hill]

Yousuf,
Thank you for the reply. However, while it answers the main question
(and, unfortunately, kills my hope to get SMP system on the cheap), it
rises even more questions. I wonder how AMD makes 1xx different from
2xx different from 8xx. To me, the possibilities seem as follows:
1. It's etched into the die. While not totally impossible, very
improbable. AMD, unlike INTC, doesn't have so many fabs to dedicate
the whole line to produce only 1xx, the other for 2xx, and yet another
one for 8xx, not counting all the Athlons that are still the bulk of
AMD production.

This is extremely unlikely. All of the Opterons and Athlon64 chips,
and even a few of those new K8-core AthlonXP-M chips, use the same
die.

2. Extra HT links burned out with a laser or otherwise disabled, just
like INTC used to castrate some 486DX to sell them as SX. Requires
extra equipment and operations that AMD could ill afford.

"ill afford"? AMD's had this kind of equipment for ages and they use
it all the time to enable/disable all kinds of features on their
chips!

Of course, they don't burn out the entire HT link itself, just a fuse
(or not burn out an anti-fuse) to turn a particular feature on/off.

3. Some pins deliberately connected (or even more likely not
connected) during final assembly. The most probable way, but leaves
some hope to work around. Anybody knows the hack?

No, too tricky. Just burn a fuse on the chip. Easy, quick and cheap.

4. The difference is only in markings and price. From my previous
expreience with some other hardware, and even more from reading about
other folks' experience, I know that the fact that the maker did not
validate the hardware for certain mode of operation does not
necessarily preclude it from such operation.

This is a possibility as well. I'm not sure that anyone has really
been able to conclusively say if "cache coherent Hypertransport" is
electrically any different from regular old hypertransport in the
first place. It could just be that HT is being used to send different
data, and that is all that is needed to make it cache-coherent.
Still, I would expect that we would have heard by now if it were
possible to run 1xx Opterons in a dual-processor configuration.

There is a fifth option that you're missing, though it connects in to
the 4th. That is simply that the BIOS detects the type of processor
and multiprocessor support is enabled/disabled at this stage. This
would be the easiest to get around, just a hacked BIOS (though many
people are understandably a bit squeamish about using hacked BIOS
code).

 

[Dan Davis]

Sorry, but this is just not correct, or more accurately is too broad.

The market for > 4P systems in the x86 market may well be shrinking.
However, it is not shrinking in the overall market when you include
unix servers and mainframes. There are still a plethora of
applications, mostly production based with hundreds or even thousands
of users, that are not appropriate (read : are too big and resource
hungry) for any of the current x86 systems. I am not talking file
servers, mail servers, or SQL backends here, which is what most of the
windows world thinks when someone says 'server'. I'm talking about
centralized mission critical applications where the application and
the database all run on one server - the bread and butter of virtually
every major corporation in the world.

I'm talking about systems like these :
http://www.hp.com/products1/servers/scalableservers/superdome/specifications.html
http://www-132.ibm.com/content/home...eServer/pSeries/high_end/pSeries_highend.html

I really feel the x86 market has been semi-static for the last few
years. More power, but nothing to do with it of real benefit. Big
action has been happening in the high-end market though, and im
fortunate enough to be able to see some of it =)

 

[Rob Stow]

The way Opteron performance scales up so nicely as you go from
1P to 2P to 4P and - presumably - to 8P makes me wonder if an 8P
Opty system might have some tiny chance of turning around the
shrinking x86 8P market. Poor scaling of performance, especially
relative to price, as you move from 4P to 8P Xeon systems is
probably the biggest deterent to the purchase of 8P Xeons.

I don't think it is a matter of potential buyers not having lots
of tasks to run on an 8P x86 machine - it is that the number of
*economically feasible* tasks for an 8P Xeon system is small.
And unfortunately, everyone seems to be using 8P Xeon economics
to prejudge 8P Opty systems - in complete disregard for the fact
that the Opty appears to be much better suited for 8P systems.

8P x86 options currently are limited to Xeons, and for all but a
few things the extra cost for an 8P Xeon doesn't come close to
justifying the minimal increase in usable computing capability.
You pay 4 times as much as you would for a 4P Xeon to get a mere
20% performance gain.

I doubt an 8P Opty system would cost significantly less than an
8P Xeon, but if anticipated 4P to 8P Opty performance scaling
holds true, it should give the bang/buck ratio a huge boost.
Hence there should be (many ? a few ?) more tasks for which an
8P Opty is economically feasible compared to an 8P Xeon.

 

[Tony Hill]

Sorry, but this is just not correct, or more accurately is too broad.

The market for > 4P systems in the x86 market may well be shrinking.
However, it is not shrinking in the overall market when you include
unix servers and mainframes.

Yes it is, at least relative to the total size of the market. The big
Unix servers have seen a continuously shrinking portion of the total
server market, both in terms of units and dollars, for some time now.

There are still a plethora of
applications, mostly production based with hundreds or even thousands
of users, that are not appropriate (read : are too big and resource
hungry) for any of the current x86 systems. I am not talking file
servers, mail servers, or SQL backends here, which is what most of the
windows world thinks when someone says 'server'. I'm talking about
centralized mission critical applications where the application and
the database all run on one server - the bread and butter of virtually
every major corporation in the world.

I'm talking about systems like these :
http://www.hp.com/products1/servers/scalableservers/superdome/specifications.html
http://www-132.ibm.com/content/home...eServer/pSeries/high_end/pSeries_highend.html

I'm well aware of those high-end systems that IBM, HP, Sun, Fujitsu
et. al make. However I'm also aware that they are seeing a shrinking
portion of the worlds server purchases.

The simple fact of the matter is that, for a lot of major
corporations, x86 is able to run the VAST majority of their
applications. Sure, they may still have the odd IBM Regatta server or
Sun Starfire for one particular super-high-end, mission critical
application, but they've only probably only got one. Previously
companies had lots of these things for all kinds of uses, now it's
only the very top-end of the market that needs servers like that.
Everything else runs on x86 for a small fraction of the price.

I really feel the x86 market has been semi-static for the last few
years. More power, but nothing to do with it of real benefit. Big
action has been happening in the high-end market though, and im
fortunate enough to be able to see some of it =)

I think you're just totally missing all the stuff that actually is
happening with x86 servers. They are now getting MUCH better
operating system support, first and foremost. Even Windows has
improved dramatically from where it was in the WinNT days, not to
mention all the work that's gone on in the Linux world. There's also
much better I/O on high-end x86 systems now than there used to be.
Combine that with a lot of improved hardware reliability features,
like hotswappable parts, hardware diagnostics and lots of error
detection and correction stuff. Sure, the high-end IBM, HP and Sun
systems still lead the way here, but x86 is now "good enough" for a
LOT of tasks that it just couldn't handle even just 5 years ago.
There's also a lot of work being done on various types of clustering
that has really helped x86. A lot of pretty high-end systems can now
run rather effectively on a cluster of x86 systems where previously
they needed one single system. Clustering for redundancy has also
helped improve reliability of some of these setups.

Like it or not, small 1-4 processor x86 servers are rapidly
encroaching in on a lot of applications that used to be the exclusive
realm of the high-end Unix systems.