Bob said:
For a 24/7 server it's best to use registered ECC memory,
which is more stable and more error free than the more
commonly used unbuffered memory. Phenoms don't
support registered memory and Opterons do. Also,
I would recommend memory built with -x4 chips, and
a mainboard that supports Chipkill advanced ECC.
Use of -x4 memory in combination with Chipkill virtually
guarantees that a computer will never have an
uncorrectable memory error.
-- Bob Day
http://bobday.vze.com
Todays desktop processors normally have no provision for placing
multiple sockets on the motherboard. (Yes, we used to have
dual socket motherboards on the desktop, like dual AthlonMP
or dual Pentium machines of various sorts. That trend kinda
stopped when multi-core processors started showing up.)
The server processor (in this case, the Opteron), has
HyperTransport interfaces. If there are multiple of
those interfaces, and the interfaces support cache
coherency, you can build a multi-processor machine.
The three coherent interfaces on a high end Opteron, support
building up to an eight socket computer.
http://en.wikipedia.org/wiki/Opteron
To see an example of the wiring pattern of an Opteron
server motherboard, try this Tyan datasheet. Page 2
shows the coherent links which run between the processor
sockets. The links running to 2050 and 2200 chipset
chips (for I/O) don't have to be coherent as I understand
it. Coherency is needed, so that NUMA (non uniform
memory access) works seamlessly. Any processor, can access
the memory connected to any other processor, but with
some delay due to transport time. On this motherboard,
you could actually plug all the memory in the system,
to just one of the processors, and all the other processors
could run by using that memory. But the other processors
would have a couple HT hops, for each memory access.
I'm guessing, that what would travel over the HT links,
would be entire cache lines, rather than a byte here, a
byte there. So using a "distant" memory isn't all that bad.
There have even been motherboard designs, where some
processors don't even have their memory slots added
to the motherboard - asymmetry by design.
ftp://ftp.tyan.com/datasheets/d_s4985_100.pdf
Now, when you look at the pricing, you'll see that you
pay dearly for that privilege. On the older AMD Opteron
processors, you might see the price of an 800 series Opteron,
would be higher than a 200 series. And a 200 series higher than
a 100 series. And that is because each of the processors
supports a "different sized" computer build. If you
wanted to populate an eight socket machine, you'd
be filling it with 800 series processors (because they
have three coherent HT links). Using an 800 in a dual
socket computer would work, but would be a waste of
money, as a 200 would be cheaper (the 200 has one fewer
coherent links). But if you were planning on using the
800 in some other computer later, then it might still make
sense.
Of course it is a scam, a scam enforced by trivial hardware
differences. It allows that exponential pricing curve we've
all grown to love, to be extended as more processors are
used. (Normally, the exponential pricing curve is used
as a function of CPU clock speed. But with the scheme of
making processors with various numbers of coherent links,
you can charge more for the processors that make it possible
to build 8 socket computers.) It doesn't cost them any more
to make the various processors, but they charge a lot more
for the ones that enable the owner to build an 8P machine.
*******
The Phenom is a desktop processor, and as far as I know,
doesn't have coherent links on it pins. You use it to
build single socket computers.
So what the Opteron (server) processor buys you,
is various expansion stories.
You need to understand what kind of computing load
your server will have, to know whether you should
plan for more sockets or not. Using a board with a
Phenom, implies you know what the load will be, and
it will always be totally met by one socket. But
if the load will grow and grow with time, you
can buy a motherboard with say 8 sockets, populate
a minimum number of processors (which is dictated
by using enough processors, so the HT wiring works).
You'll notice that the I/O chipset devices, need to be
connected to the cluster of processors, and if you
pull out (don't populate) a particular socket, the
I/O functions running off that socket won't work.
When you're using Opteron, you need to understand
a good deal about the architecture, to be able to
plan its purchase and installation. I'm just going
on a few things I've read over the years, so I'm
not an authority on building servers. After looking
at prices, I know you could buy a very nice car,
instead of some of these Opteron based systems.
Another bit of trivia for you. One of the 8P solutions
that Tyan sold, used two motherboards. The motherboards
are stacked, one above the other. Two pieces of PCB
material ("FR4"), with contacts on either end, carry more
HT busses, from one motherboard to the other. So they
build an 8P machine, using two 4P motherboards, and
two extender cards between motherboards. I've never
seen one assembled, but the heat coming out of that
must be brutal. It is the kind of system that
might use a 1200W power supply.
*******
On Intel now, they've developed an idea that looks
like what AMD did. They've put memory interfaces
right on their processor. QPI interfaces on the
processor, allow building multi-socket computers.
I presume the QPI features coherency and the NUMA
memory model. On page 2 here, it looks like
Intel is designing for two socket systems. They'd
presumably need more QPI links, to scale to a larger
system. I don't know enough about Intel's plans and
products, to be able to tell you if there is a
larger scaling option or not.
http://www.tyan.com/datasheets/d_S7002.pdf
I used to enjoy looking at SPEC benchmarks, as a means
of telling people which platform might be better. The
last time I looked though, I couldn't make any sense out
of what I was seeing in the tables of results. So now
I can't even tell you how many "AMDs" it takes to
equal an "Intel".
While sites like Tomshardware have benchmark results
for desktops, I don't think you'll find a site doing
something similar for servers. That would be too easy
There are sites with more discussions along these lines.
It is too bad we can't get a better story from
AMD and Intel themselves. (There is a mention of
"4P Xeon based parts" here, so maybe there is a
plan for a larger Intel system.)
http://www.hardforum.com/showthread.php?t=1409990
Paul
