Virginia-Tech Apple G5 Cluster (1100 strong)

How did they get it back up to #3? I remember when they were building it they said it would be #3, but then it came out that they were basing that on 100% efficiency. Some tests were ran where it actually had something like 65% efficiency placing it at #20 in the world (or something like that). At the time when that was figured out, someone said they were tuning and hoping for 85% efficiency which would get them into the top 10. Now they are coming back and saying it is #3 again. Anyone know what's up?

Brian Taylor
 
That is the sexiest thing i've seen in a while .. :eek:

Now if they will just install folding@home Team #33 we would be all set
 
Humm, Hope that's not tax payers money hard at work..:rolleyes:

Doubtful Apple donated all those;)

BillR
 
Originally posted by darkmyth
waste of money if you ask me ;)
Click to expand...

not at all,

That cluster cost 5 Million to build, and placed #3 in the TOP 500 supercomputers running at like 10 TFlops . Number 2 cost 300M running at 13 TFlops, and number 1 cost 500M and runs at 35 TFlops.

I think they got a very good deal for the money.
 
I've known about this for a while, and it is indeed quite sexy:)

So what if it is taxpayer money(i'm not sure if it is or not)? $5 million for that(whether you like macs or not) is a much better value than the other super computers that the government has or will build for a while. To achive the highest speeds they will still have to rely on expensive custom made systems, but this definitely shows the potential of clusters of more generic components.
 
The only way this could be considered a waste of money is if you're talking about the fact that they chose to use Apples instead of another type of PC because it really is inexpensive for a supercomputer. If you were going to do the ultimate bang for the buck supercomputer, I can't imagine anything beating one based on AthlonXP's. I would think that an Opteron-based supercomputer cluster would be faster and on price parity with the G5 cluster. Then again, I'm also sure that it depends on exactly what you're going to be doing with it - some tasks just like some processors better. ;)
 
Originally posted by Mattman
The only way this could be considered a waste of money is if you're talking about the fact that they chose to use Apples instead of another type of PC because it really is inexpensive for a supercomputer. If you were going to do the ultimate bang for the buck supercomputer, I can't imagine anything beating one based on AthlonXP's. I would think that an Opteron-based supercomputer cluster would be faster and on price parity with the G5 cluster. Then again, I'm also sure that it depends on exactly what you're going to be doing with it - some tasks just like some processors better. ;)
Click to expand...

well considering its probably going to be for science, aka physics and some other things... im sure that the G5 with its 64bit and 128bit portions would seriously whip the crap out of an AXP

also most AXP boards only have 32bit 33mhz pci, g5's have PCI-X i believe
 
Originally posted by Mattman
The only way this could be considered a waste of money is if you're talking about the fact that they chose to use Apples instead of another type of PC because it really is inexpensive for a supercomputer. If you were going to do the ultimate bang for the buck supercomputer, I can't imagine anything beating one based on AthlonXP's. I would think that an Opteron-based supercomputer cluster would be faster and on price parity with the G5 cluster. Then again, I'm also sure that it depends on exactly what you're going to be doing with it - some tasks just like some processors better. ;)
Click to expand...

haha ya I hate mac's if you can't tell. Linux for me :D and my Winxp home box i just use for gameing
 
Originally posted by FLECOM
well considering its probably going to be for science, aka physics and some other things... im sure that the G5 with its 64bit and 128bit portions would seriously whip the crap out of an AXP

also most AXP boards only have 32bit 33mhz pci, g5's have PCI-X i believe
Click to expand...

Really? I still do wonder why they used Apples instead. I think they coulda built it with Opterons and it would have been cheaper and faster. Opteron has 64 bit as well... hmm.. any other reasons?

FOLD ON!!!
 
Do you know of a source to get 1100 decently priced custom Opteron based computers? I believe they had to do little to the machines themselves as far as hardware goes so buying from Apple probably saved them a bunch on labor costs
 
One of the reasons that they probably chose macs is that apple has supported clusters and other DC projects in the past. I dont have any documentation of this, I just remember reading stuff about it.

Also, the G5s probably had the best collection of technologies put together in one machine, they were probably more futureproof. This thing was built quite a while before the opterons were in as much use as they are now, and the motherboards manufacturers probably did not the capabilities and experience with these more recent technologies (eg. SATA, PCI-X, GigE, 800-1000mhz fsb, IEEE1394b, 64bit proc with 32bit ALL STANDARD) as apple has implemented them all together, and implemented them far before many other computer manufacturers. You have to remember that this was the computer available around 6 months ago, when this supercomputer was designed.

It's also worth noting that apple did need the exposure, and so they probably offered VTech a better deal as they were getting something out of it too. They needed this to support them as they try to convince the world that G5s are the fastest baddest sweetest bestest processors on the whole planet. It just naturally goes along with their "desktop supercomputer" and "worlds fastest personal computer" campaigns.

Don't think I'm just a mac fanatic. I buy whatever fills my needs, and for now, its not a mac. I'm really an AMD fan because they've delivered processors that perform well for low prices, <$100, something Intel and apple don't do as well.
 
Another factor to keep in mind here is that we're dealing with the world of academia, where Macs are still held in high regard, along with some in the graphics and desktop publishing world who are caught under Steve Jobs' spell. Real people made the decision to go with Macs and if I was an Apple fan (like I was in '84, hehe) and Apple set me up with a good deal on some new G5's, I'd probably give them a shot over something else just because I wanted to see my vision of an Apple supercomputer come to life. From a cost/performance perspective it may not be the best, but it's not far enough off to fault them too much.

Originally posted by darkmyth
haha ya I hate mac's if you can't tell. Linux for me :D and my Winxp home box i just use for gameing
Click to expand...
I was guessing that's where you were coming from in your first post. :) I don't hate Macs, but it is pretty frustrating when Apple gets away with bogus advertising like the whole "supercomputer on your desktop" crap. If they just stopped the blatantly stupid claims and went on their way building alternative PC's, I wouldn't have any problems with them at all.
 
i bet you they are kicking themselves now that g5 xserves are out!
who knew 1U could be so damn sexy!!

could you imagine how many more they could have fit into that room if they were all 1u rackmounts!??
 
They might not want to fit more into one room since there's only so much heat a single room can take. I didn't read this latest article, but I read a month or so ago about how much power and cooling it was taking to run this thing...something many of us know all too well on a smaller scale! ;)
 
Probably no less than 160 Watt per dual system. Add overhead for the networking equipment and surge protectors.

200 Watt x 1100 = 220,000 KWatt

$12/mo/node = $13200/mo = $158,400/yr in Electricity

I wonder how that fares against a comparable supercomputer.
 
I'm sure that like my (smaller) university VT generates most of their electricity locally at a much cheaper price than is avaliable to the public. In fact, I think that except when football games are happening here(stadium power) power is put back into the grid and money is made.
 
I agree on the power thing. The college I go to does that with water, I guess as a result of them being one of the top 3 ranked colleges for hydrology majors.....No, im not one of them.
 
Originally posted by Methodical
One thing i'd like to add about the opteron argument is that there is only one motherboard in existence for the opteron that uses PCI-X, its from tyan, and came out after the G5s.

And with a 1100 computer grid, you need as fast an interconnect as possible.

opterons werent a real candidate because of this.
Click to expand...

Originally posted by Methodical

G5 = 2 double precision FPUs each capable of 1 fused MADD (multiply-add - commonly used) per clock. x2 processors = 8 GFLOPS per machine.

so for 1100 machines youd have a theoretical peak of 17.2 its at 10 something.

for opteron = 4gflops.

you'd have to be past 100% efficiency to be at the terascales current performance.
Click to expand...

And in response to those who are whining about tax money: A university is a business, and a business has to spend money to make money. This project will obviously attract more students and give prestige by having a research tool that ranks #3 in the world. Tuition pays for this and allows for more research to be done. In fact, this may actually make the university money. My university has to wait months on a waiting list and pay ungodly ammounts of money to get a few hours on a machine that is a fraction as fast and is hosted at some other university's cluster. Important research is done on these machines form biology to weather to testing math theories. The list goes on. Head on over to http://www.top500.org/list/2003/11/ and see that #4 and 5 are xeon and itanium based systems that cost way more than vt's.
 
Originally posted by Mattman
Another factor to keep in mind here is that we're dealing with the world of academia, where Macs are still held in high regard, along with some in the graphics and desktop publishing world who are caught under Steve Jobs' spell. Real people made the decision to go with Macs and if I was an Apple fan (like I was in '84, hehe) and Apple set me up with a good deal on some new G5's, I'd probably give them a shot over something else just because I wanted to see my vision of an Apple supercomputer come to life. From a cost/performance perspective it may not be the best, but it's not far enough off to fault them too much.

I was guessing that's where you were coming from in your first post. :) I don't hate Macs, but it is pretty frustrating when Apple gets away with bogus advertising like the whole "supercomputer on your desktop" crap. If they just stopped the blatantly stupid claims and went on their way building alternative PC's, I wouldn't have any problems with them at all.
Click to expand...

I just really really really dislike macs. Ever since i had to use one in a buisness class in highschool last year I just can't stand um. To damn annoying! ;)
 
IBM Sells Supercomputer: on Wednesday that it sold a supercomputer based on microchips made by Advanced Micro Devices Inc. to Bristol-Myers Squibb, giving a boost to AMD's fledgling Opteron technology. The supercomputer, the price of which was not disclosed, is made up of 64 computer servers that each run on two Opteron microprocessors. Bristol-Myers will use it to help accelerate early discovery of new compounds used to make drugs
Click to expand...

Right off the [H]ardocp front page. There are going to be some fast cheap systems hitting the streets in the next few months based on AMD chips.

I'm beting that at the time they were planning the G5 system, that the best compromise between price, performance, reliability, availability, and OEM support probably was with Apple. I remember reading in the news stories when this first broke that Apple bent over backwards to make these people happy.

L8R

Don
 
Cray is getting back into the game as well with an Opteron supercomputer. Exciting times for high end computing.
 
So does this mean that they could build a cluster similar to the power of the Earth Simulator for around 20 million or are the physical limitations to how many of these G5's you can hook together such as bandwidth and latency.
 
not sure how scalable it is, but throw 25mil at it and im sure you could...

now remember the earth simulator is how old now?
 
There's tricks for getting high bandwidth and low latency out of "normal" switches. I can't find the link right now but there was a formula for how many switches, how many ports 64/96/128 and how many NICs each PC would need. For 1024 nodes it was something like 4 gigabit network cards in each PC. The switches are cross linked to minimize "hops" between each node. Quite the spider web once connected.
 
Originally posted by turmelle
Probably no less than 160 Watt per dual system. Add overhead for the networking equipment and surge protectors.

200 Watt x 1100 = 220,000 KWatt

$12/mo/node = $13200/mo = $158,400/yr in Electricity

I wonder how that fares against a comparable supercomputer.
Click to expand...

I was wondering that as well, on the top 500 page you see how much it cost to build the system but you don't get a TCO (total cost of Ownership) to go along with it.

I would assume that since so much money was spent on building these machines (the top 1 and 2, as well as others below VT) that they are custom RISC based processors, and thus may require less electricity overhead to run than VT's toss em on a shelf and plug em in method. This would <IN THEORY> actually be a good thing then that they went to all this trouble to custom design (and spent so much loot in the process) in the first place, because if they aren't planning on upgrading for say, twenty years or so, it may be worth it to them to spend more intial money to spend less in the long run or fit in with their current energy budget/requirements.

And yes Cray is definitely getting back into the supercomputer fray with their X1 (they were wilting under the ownership of Sun anyway, it's much better for them to be on their own). Some really interesting shit is going on with them right now, as they have already filled the order quota for 2k4, and I assume they are going to attempt to get back into the vaunted position they once held.

I don't see however where it says that Cray is using Opterons, I thought they were using their own Cray X1 SSP (single streaming processor).

Some good info (though slightly propaganda) on the Cray X1 here:

http://www.ahpcrc.org/publications/X1CaseStudies/

edit: I cant make the link to the pdf work right so fuck it second one down presentation. *sigh*

That X1 is one liquid cooled beast though, I wish I had the money to afford to turn it on, let alone to own one :D

Lata
 
I'm not a huge MAC fan myself, but if all the previous tests are right I'd bet this thing is Hell fast on photoshop:rolleyes:

BillR
 
Originally posted by DonDon
Right off the [H]ardocp front page. There are going to be some fast cheap systems hitting the streets in the next few months based on AMD chips.

I'm beting that at the time they were planning the G5 system, that the best compromise between price, performance, reliability, availability, and OEM support probably was with Apple. I remember reading in the news stories when this first broke that Apple bent over backwards to make these people happy.

L8R

Don
Click to expand...

As I recall, a few months ago, when the Opterons were first introduced, that the Chinese bought 10,000 of them for a big supercomputer project. 10,000 processors is pretty schweet. :)
 
The reason they went with the G5 is that it can perform 2 math calculations @ 64 bits in one clock cycle. X 2 for duals, so you get 4 complete calcs in one cycle running 64 bits. Neither the Opteron nor the Xeon can do that. The fused X + is the key, as the article states below.

For the type of work they expect the cluster to be doing, thats what was important to them. They are going to be swapping the desktop G5's for X serve G5's shortly, and probably adding units too.

Heres a excellent summary of why they went with G5's over opterons:

Also please note they dont even use the SIMD instructions in the chip. Can you imagine how fast it would be if they had work that used AltiVec?

March/April 2003---project begins

April/May--finances raised

June 23, 2003--Apple announces G5

June 26, 2003--VA Tech contacts Apple. Deal sealed with Apple in a few days. Apple is stunned :). Thought that Dr. Vanadarajan must be a Mac fanatic. Turns out he never touched a Mac before, but is certainly proficient in using Mach and Linux/Unix. VA Tech actually ordered the machines via the Apple Store mechanism!

September 5-11, 2003---G5's arrive

September 23, 2003--facility begins preliminary operations

October 1 through mid November will be performance optimizations

Experimental runs by users can begin today.

Full production use by the start of 2004.

Cost:

2 million for upgraded the facilities in which the system is housed.
5.2 million for hardware (besides computers, includes cards, cables, storage, etc)

Current facility scheduled to be followed by a 2nd system in a new building in 2006.

General design criteria:

--Major factor is Performance/Price
--64 bit design (32 bit systems need not apply)
--Benchmarks will depend on double-precision floating point--Altivec **not**being used in this case.
--Connectivity to Internet 1, Internet 2 (Abilene) and soon into NLR (National Lambda ???).
--High bandwidth with ultralow latency communication
--Infiniband switched network, 20 GB/sec/port full duplex, latency of less than 10 microseconds on top of MPI.

Platforms considered:

Some vendors proposed a variety of "turnkey" systems. This drove up the cost of some bids into the range of 9-12 million dollars which was well beyond the budget.

Dell with Itanium 2----lost on processor and system cost and to overall performance
IBM with Opteron---lost on performance and overall system cost
IBM with PowerPC 970--won on performance, but lost in delivery time (January 2004) and overall system cost
Sun with SPARC--lost on performance and cost
Apple with PowerPC 970--won on performance and overall system cost.

Opteron apparently does not support the "fused multiply-add" (I may have the spelling wrong) function which gives G5 an edge in floating point performance. As such, G5 can outpace an Opteron by a factor of 2 in floating point.

Itanium 2 apparently gives a GEMM efficiency (see Results section below) as much as 15 percent better than G5 right now. However it is very expensive, and also loses whatever GEMM efficiency advantage it has due to other things like its slower clock speed. That is definitely an ironics twist. :)

Software:

--Each G5 machine has a stock install of OSX 10.2.7
--Mellanox Infiniband drivers
--MPI implemented using MVAPICH from D.K. Panda's group at Ohio State University. Code ported from Linux with additions of message caching and dynamic memory management.

--Cache optimized memory manager for scientific apps written for OSX as a KEXT (written in-house)

--Scaleable job starting system for MVAPICH (written in-house).

--Deja Vu as a system for fault tolerance ported to the G5. Intended to be separate for ordinary application logic. (written in-house)

Compilers used:

For C and C++, IBM xlc and GCC 3.3
For Fortran, IBM xlf and NAGWare

Results:

--Mellanox driver version 1 started in July and finished in mid-august. Subsequent tweaks have improved things by around 10 percent.

--Benchmarked using LinPack
--G5 solved a system of equations at N = 500K
--dense matrix operations
--main phase is LU decomposition. Gaussian elimination with partial row pivoting 0 (n^3)
--back solution follows at a lower order 0(n^2)

--Used BLAS libraries

--Core routines--matrix multiply (GEMM) optimized by Kazushiga Goto in Japan. 84.1 percent efficiency at this time. Apple's veclib framework also used.

AND THE CURRENT RESULTS AS OF 10/28/2003 ARE .................

9.6 teraflops

So on the current list, this puts them at number 3.

Immediate future plans:

Upgrade G5's to Panther in the next couple of weeks. All codes compile fine under Panther.
Along with some other optimization tricks, anticipating for at least another 10 percent improvement in performance.

Expecting to make their MPI enhancements and in-house software open source. For the Infiniband drivers, Dr. Varadarajan could not speak for them, but is hopeful that those drivers will be made available as open source as well. But that is Mellanox's call.

Related Articles:

http://www.computing.vt.edu/ (Virginia Tech Project: Terascale Cluster)

http://don.cc.vt.edu/ (Pictures: Terascale Cluster)

http://www.computerweekly.com/ (Apple chosen for supercomputing cluster)

http://macslash.org/ (TenCon Keynote - Dr. Srinidhi Varadarajan)

http://www.wired.com/ (Mac Supercomputer Just Got Faster)

Keep on hating Macs all you want, but they were the best choice based on the results they got. IBM pulled Apple off life support and they are now a excellent choice for everything ( except playing BF1942 ) _:D
 
Originally posted by Bill Clo
As I recall, a few months ago, when the Opterons were first introduced, that the Chinese bought 10,000 of them for a big supercomputer project. 10,000 processors is pretty schweet. :)
Click to expand...

You use the a 10,000 processor server farm everytime you plug a search into Google and hit SEARCH.
 
You must log in or register to reply here.
Back
Top