H
hackbox.info
no, "we are talking 5000 Opteron processors and 3.5 petabytes of disk
storage" overall - now it makes sense
no, "we are talking 5000 Opteron processors and 3.5 petabytes of disk
storage" overall - now it makes sense
Y
Yousuf Khan
hackbox.info said:
Wouldn't that many processors and disks be able to fit inside the space
of a truck trailer? Especially when the trailer is refrigerated?
Yousuf Khan
H
hackbox.info
can you imagine that? because I cant
Length of semi-trailer used in a commercial tractor-semi-trailer
combination 53'-0"
Height (of all vehicles) 13'-6" (minus the wheels = 10')
Width (except municipal bus or traction engine) 8'-6"
I'm a metric guy, so 16 x 3 x 2.5 = 120m3
U8 Box with 16TB and quad Opterons 19"(W) x 27"(D) x 14"(H)
0.48 x 0.68 x 0.35 =1.1m3
Lets assume we can fit 100 of those boxes, that gives us 1.6 petabytes and
400 Opterons
Whole "Google probably has between two and five petabytes altogether"
(April 06, 04)
T
The little lost angel
Hmm, the old folks in csiphc will know my maths isn't the best in the
world. But from my juvenile estimates, I don't see why not?
A standard 20 foot container is 6.1m x 2.4m x 2.9m = 42.456m^3
Let's say we use modules of quad processors on a board, with drives
slotted into some kind of mounting rack under the board so the whole
module can make a nice little rectangular box for easy
installation/removal.
Using a Tyan Quad Opt board as a guideline, holding 4 Opterons takes
0.14m^2 area (0.33m x 0.41m round up), the same area can fit 6 (8 max
mathematically) drives under the board with ample space to spare for
air flow. Using a 1U heatsink fan, on it, I expect the board + sink
height + drive to be max of 0.07m (43mm + 25mm rounded up). So for a
modular unit of 4 processor sharing a 4 Drive Raid 5 or perhaps 2x
3-drive Raid 5, it will take a volume of 0.0098m^3
Say we need some mounting and airspace for whatever cooling methods
they come up with, adding 30% to the volume, making it 0.01274. But
the things need to be removed so there has gotta be space for sliding
them in and out. I suppose if they are arranged in walls, two walls
can share one common empty space for this. So add 50% more space for
this. This makes the required volume per module to be 0.01911m^3.
Let's round it up to 0.02m^3 per module to allow for fractional
heights and stuff. This would allow (42.456/0.02) 2122 such 4CPU + 6
Drives module, allowing 8488 processors, 6366TB or 6.3Petabyte of
drive space using 500GB drives.
Say we take away 30% of the same for power and cooling thingy and it's
4.4PB of drive space and 5.9K of processors.
Which is pretty close to the numbers in the article.
H
hackbox.info
Let's round it up to 0.02m^3 per module to allow for fractional
heights and stuff. This would allow (42.456/0.02) 2122 such 4CPU + 6
Drives module, allowing 8488 processors, 6366TB or 6.3Petabyte of
drive space using 500GB drives.
Say we take away 30% of the same for power and cooling thingy and it's
4.4PB of drive space and 5.9K of processors.
Which is pretty close to the numbers in the article.
Ok, custom modules, but those should be submerged in some cooling fluid
In my calculations U8 Boxes were equipped with 1300W power supply units.
This makes it a one hungry trailer container (130KW).
Where do you take the power from in your calculations?
Dont even thinkabout one centralized low voltage solution, I cant imagine a single PSU
being able to cope with fluctuating voltages of something two orders of
magnitude smaller.
Y
Yousuf Khan
hackbox.info said:
How about with blades instead of 1U servers? And the disks packed
together in a storage array rather than spread out with disks
intermingling with CPU?
Basically a 5000-P Opteron cluster would be half the size of the Cray
Red Storm supercomputer at Sandia.
Yousuf Khan
T
The little lost angel
Ok, custom modules, but those should be submerged in some cooling fluid
In my calculations U8 Boxes were equipped with 1300W power supply units.
I'm not sure what a U8 box is. But generally from what I know, 1U rack
come with at best a 300W PSU.
But let's say all these 5000 Opteron are using the 848HE versions,
with a TDP of 50W. That's 200W for the processors alone, throw in 50W
for the motherboard and stuff, and then another 60W for the drives,
that's about the limit of what they can do with the 1U. So
theoretically, the power each module, only requires the space of 2
drives. Which is like 0.14m x 0.1m x 0.03m = 0.00042m^3 or 2.2% more
space per module, which I think is more than covered by the 30%
additional space I estimated for power and cooling.
Hmm by mine, it's about 460KW, not counting efficiency and cooling
yet. I suspect with so much waste heat, it should be possible to reuse
them somehow. Even getting 20% of the energy back would be useful!
Where do you take the power from in your calculations?
about one centralized low voltage solution, I cant imagine a single PSU
being able to cope with fluctuating voltages of something two orders of
magnitude smaller.
I suppose if we're talking about PC PSU, there wouldn't be any
possibility. But I somehow don't think Google would be using off the
shelf power solutions for this kind of thing.
H
hackbox.info
Cray XT3? ~50 cabinets for the Opterons. Those are 57cm wide, meaning you
can run 26 of them along the wall (16m is the longest legal trailer),
there should be aproriate spacing between the next row = no room for the
opterons alone, and those things are seriously packed. To put it another
way - 35 tons of Opterons alone, and where are the disks? 730KW for
Opterons alone. And how much is Cray XT3? :]
H
hackbox.info
8U
.. But generally from what I know, 1U rack
8U with 40 discs = 1300W
how exactly?
I am talking any kind of one central switching power supply. 5000 x 50W
TDP (~10W at idle?) equals 5000 x 7 to 35 Amps current draw jumps, and
those are cpus alone.
But I somehow don't think Google would be using off the
shelf power solutions for this kind of thing.
I somehow think the whole thing is a PR stunt and utter BS.
T
The little lost angel
I dunno, if I did, I would not had said "I suspect". I'm not exactly
the scientist sort, was and probably still am the village idiot in the
CSIPHC newsgroup.
Hmm, of course I expect AC power to be provided externally on the drop
site. This goes into some kind of central distribution panel and
deliver power to a small 1U kind of adapter for each module, wouldn't
that work?
Well, a lot of innovations start out as BS, I'm sure talk about
controlled flight for human was disregarded as BS at some point or
another
So I wouldn't quite rule it out, even though there areadmittedly a lot of utter BS around
PpP
T
Tony Hill
You can pack chips MUCH more densely than that. Given that they are
going for high density, the first place I would look would be
something like HP's BL35p Blade servers. You can fit 16 blades, each
with 2 dual-core Opterons, into a 6U enclosure. With power supplies
for all the blades you can easily get 6 enclosures per 19" and fit
well within the height confines of your tractor.
Now figure that your 19" racks are a bit over 50cm wide and 75cm deep,
you can pack in 30 of them along each side of the truck, leaving 25cm
in the back for cabling and a 50cm aisle down the middle for someone
to walk between them. No figure 26 racks for blade servers:
26 x 6 x 16 = 2,496 servers
Each server can hold 2 dual-core Opteron chips for a total of 4,992
processors (close enough to 5000 for my counting). Now lets figure on
the other 4 racks for networking equipment, redundancy, control, etc.
etc. That still leaves us with a full 30 racks for storage. Now,
getting 3.5 Petabytes of storage into 30 racks is still going to be
quite a challenge, but it's probably do-able. I did a quick search
through HP's website after checking out the blades above, and it seems
like their storage products could pretty easily get you up to at least
1.5 petabytes in that space.
Ohh, and all these calculations are assuming only 48U racks, which are
about 225cm tall, leaving you about 75cm at the top for your cooling
and venting.
Anyway, long story short, I would say that it is indeed very possible
that Google IS packing all that computing power into a single
transport trailer. It's not an easy task and it DEFINITELY is not
cheap, but I would say that it's possible with what's out there today.
For some references on the blade servers config, see here:
http://h18004.www1.hp.com/products/servers/proliant-bl/p-class/35p/index.html
And in particular, page 15 here:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00301687/c00301687.pdf
This shows a 42U rack with 5 blade enclosures, though 6 should be
possible with the same setup in a 48U enclosure (3 enclosures per
dual-power pack). As you can probably guess, the power requirements
of such a setup would be absolutely prodigious! (I count ~ 750KW for
the blade servers alone, and probably nearly as much for the hard
drives). However assuming you have your own power substation, it
could probably be done.