Comments

By Gary Stiehr, June 28, 2009 2:37 pm

DOE's Roadrunner Supercomputer

DARPA has issued an RFI (pdf) to help enable what they are calling Ubiquitous High Performance Computing (UHPC).  According to the June 2009 TOP500 supercomputer list, the fastest supercomputer available, Roadrunner, runs at just over 1 PFLOP.  It uses around 2.5 million watts of electricity and requires around 278 racks of equipment [1].

DARPA would like to fit the same computational power into one air-cooled rack and use no more than 57,000 watts (including cooling).  That’s 100% of Roadrunner’s computational power in 0.4% of the space using 2% of the electrical power. Also, while the most energy efficient system now achieves 536 Mflops/watt [2], DARPA is looking for 50 Gflops/watt.

What’s more, is that they would like to minimize the overhead associated with thousand-way to billion-way parallelism.  Why billion-way parallelism?  I suppose this implies an anticipation of systems containing billions of execution units.  This  may not be unreasonable.  For example, take a look at the proposed Sequoia supercomputer, which is proposed to include one million cores.

Beyond these astounding requirements, there are also requests for a “Self Aware OS” that is introspective, goal-oriented, adaptive, self-healing and approximate.  I’d recommend reading page eight of the RFI above for more details.  The hope is that the system will be able to continue operations in the face of failures and “attack” (see page 4 of RFI).

Well, while the OS and application capabilities will be huge challenges, the restrictions put on the physical aspects of the hardware are also challenging.  With GPUs and Cell processors leading to increased computations per watt, perhaps we may be able to significantly improve overall system power efficiencies.  In addition, DARPA is looking for this to take place potentially in 9 years (proposals are due by July 27, 2009) if it is feasible.  With top supercomputers sometimes becoming more powerful than the 500 most powerful supercomputers combined from four years prior, we can definitely see overall computational ability increase quickly but this doesn’t necessarily translate into the density and energy efficiencies.

Aside from the RFI above, you can read more here or here.  Also, thanks to @HPC_Guru from whom I first heard about this RFI.

Comments

By Gary Stiehr, February 5, 2009 11:50 pm

IBM has won a contract to build a supercomputer, called Sequoia, for the DOE’s NNSA.  It is estimated to be installed and brought online in 2011 and 2012.  It will have 1.6 million cores (from potentially 16-core chips) within 96 racks (in about 3,400 sq. ft.).  It will have around 1.6 Petabytes of memory and achieve about 20 Petaflops.  It will require about 6 million watts of power to operate, which is around 3.3 billion operations per second per watt–very impressive.  I wonder if that includes the power needed for the cooling system.  And is that when the processors are at 100% or when the system is idle?

At 1.6 PB of memory for 1.6 million cores, that is a relatively low amount of memory per core.  If the memory is doubled, for example, the system may require a few more megawatts of power.  This is based off of very rough estimates of power needed per GB of memory based off of some recent commodity clusters.  Do you have any hard numbers on power per GB of memory today?  Any information on the type of memory that might be used in Sequoia?

For more information, see IBM to send blazing fast supercomputer to Energy Dept. and/or U.S. taps IBM for 20 petaflops computer.

Comments

By Gary Stiehr,

From Blue Data Center Will Be Powered by the Tides (found via @tkunau/@ecogeek):

At first, tidal power will only cover one-fifth of the data center’s needs, but Atlantis hopes that if the first phase is successful, they can expand the tidal array to make up the remaining wattage.

Comments

By Gary Stiehr, January 27, 2009 1:09 am

Sun’s Colorado Consolidation Saves Millions describes how Sun used Liebert’s XD rack cooling, clear vinyl cold isle curtains and flywheels to increase the density of its data center while also reducing energy consumption.  They reduced 165,000 square feet of data center space into 700 square feet while reducing their monthly power usage by one million kilowatt-hours.

When we considered the XD cooling units, there were two options: chilled water or refrigerant.  In the case of chilled water, there was the question of potential water leaks in these rack-attached units.  With the refrigerant option, there was the question of an increase in the number of condensers and where they would be placed and how much  more maintenance would be needed.  With either option, there is also an increase in the need for maintenance inside the server room amongst the servers, storage, switches, etc.  The obvious benefit of the XD units is the fact that they can provide enough cooling for up to 30 kW in a single rack.  Although, if I recall correctly, there is a limit to the total number of racks with the refrigerant-based version due to limits on the maximum pressure or capacity of the refrigerant in a single system.

As for the vinyl curtains, there is usually more of an objection to their aesthetics.  Personally, I would like to see them installed to help keep the cold air completely contained in the cold aisle, where it is intended.  Especially in raised floor environments with high velocity air flow where the cold air might be pushed outside the confines of the cold aisle without such containment.

One question about Sun’s use of the flywheel: How large are your flywheels?  Flywheels generally supply on the order of ten seconds or so of power, which is usually enough time for generators to kick on but cuts it very close.  What type of services run out of Sun’s Colorado facility?

Comments

By Gary Stiehr, January 13, 2009 3:11 am

Four of these drives deliver 4TB in the same space as a standard 3.5-inch HDD.

via HPCwire: pureSilicon Debuts 1TB 2.5-Inch SSD.

Is that right?  Four 2.5-inch drives in the place of one 3.5-inch drive?

If so, in systems that can hold 48 3.5-inch drives, then could we fit 192 of these 2.5-inch, 1 TB drives?  If those 48-drive systems fit in 4U of rack space and we put 10 of them in one rack, we could get 1,920 TB in one rack.  That’s incredible density.

According to the stats at the at the article above, this rack would require about 9.2 kW of power when active and only 192 Watts (yes, Watts) when idle.  Of course this considers only the drives’ power consumption.

At 240 MB/s read and 215 MB/s write per drive, we’d have incredible I/O rates per 192-drive system.  Imagine the performance of such systems for large OLTP databases, for example.

So what are the challenges with such a system (besides price, I’d imagine)?  With one drive potentially nearly saturating the theoretical SATAII bus capability, how could we take advantage of so many drives?

Instead of a 192-drive in a system in 4U then, what about 48 drives in a 1U system?  Are the same technical challenges there as far as getting more of the I/O potential out of these drives?