With their recently announced XK6 system, Cray states that they can scale their system to 500,000 scalar cores (16-core 64-bit AMD Opteron 6200 Series processors) and to 50 petaflops of peak hybrid performance when coupled with NVIDIA Tesla X2090 GPUs. Each cabinet will require 45-54 kW of power depending on the configuration (and usage), with cooling provided by either 3,000 cfm of air or Cray’s ECOphlex liquid cooling. The configuration seems to allow for up to one GPU per CPU with up to 96 CPUs and 96 GPUs per cabinet. Each cabinet will perform at 70+ teraflops according to the XK6 technical specifications. There will be one Gemini routing and communications ASIC per two compute nodes with 48 switch ports per Gemini chip (160 GB/s internal switching capacity per chip) to enable a 3-D torus topology.
So what would it take to build a system operating at 50 petaflops based on the Cray XK6? At 70 teraflops per cabinet, we’d need about 714 cabinets and about 38.6 megawatts of power to achieve 50 petaflops. But Cray’s XK6 specifications page states 70+ teraflops per cabinet…so perhaps fewer racks and energy would be needed. For comparison, scaling the numbers by 2.5 for the 1.6 million-core Sequoia supercomputer, a 50 petaflop machine made from Sequoia’s hardware would need 240 racks and consume 15 megawatts of power.
Oak Ridge National Lab (ORNL) will be building a 20 petaflop machine, called Titan, based off of the Cray XK6. HPCWire talks about Titan and how its design compares to the 20 petaflop Sequoia design.