A Close-Up Look at the World’s Largest HPC System for Commercial Research 

BP's Atlantis hub, Gulf of Mexico (source: BP)

For decades, high performance computing has driven more accurate, detailed and faster seismic exploration. As oil and gas have becomes harder to find, HPC has been essential to keeping energy supplies flowing – to the point where new discovery and extraction techniques have driven up supply, driving energy prices down and making the United States a net energy exporter.

Of the companies in the sector, BP is notable for its aggressive embrace of HPC. The team of seismic researchers and computer scientists at the company’s Center for High-Performance Computing (CHPC), Houston, has quadrupled its computing power and doubled its storage capacity since it opened in 2013. The CHPC recently announced a new HPE system that more than doubles the center’s compute capabilities, making it the most powerful supercomputer in the world used for commercial research, according to industry watcher Hyperion Research.

BP’s compute capabilities led to the discovery last year of potential oil reserves estimated at 200 million barrels located at the company’s “Atlantis” drill field in the Gulf of Mexico. Whether or when BP will extract those reserves hasn’t been decided, but the finding demonstrates the increasing power and sophistication of HPC hardware and software for finding increasingly elusive energy reserves. The new Atlantis reservoir is a case in point: it’s under 7,000 feet of water and 25,000 feet below the ocean floor – about seven miles.

BP’s HPE Apollo 6000-based supercomputers reveal a balance between compute power and memory capacity. The CHPC has two HPE systems, one with 2,700 Apollo 6000 nodes equipped with Intel “Haswell” processors; its new system has 2,700 Apollo 6000 nodes with updated Intel Knights Landing chips, along with a 100 GB low latency network, boosting BP’s processing speed from 4 PB to 9 PFLOPS.  Equipped with High Bandwidth Memory (HBM), the system has a total active memory of 1,140 TB, along with 30 PB of storage.   The Knights Landing-based system is 18 times more powerful than the world’s fastest supercomputer in 2007.

BP's Keith Gray

“This is a new architecture,” Keith Gray, BP’s technical director for supercomputing, told EnterpriseTech of the Knights Landing-based system.  “The computer scientists on our team worked closely with Intel and HPE to optimize our codes. Each Knights Landing processor has 16GB of memory glued on top of the processor, and that memory is five times faster than the standard. We worked with Intel to understand the optimal configuration for this, we spent close to a year in preparation for the delivery of the Knights Landing system.”

Helping BP find the potential new reserve at the Atlantis field is a proprietary algorithm using Full Waveform Inversion (FWI) technology. Gray said distributing that capability across the enormous HPE cluster is a major technical challenge.

“The problems that we’re working on continually get so large that we now have software that breaks the problem into their chunks that can be solved on a single node,” Gray said. “Our computational scientists and software engineers have written tools that distribute the work across the whole of the cluster and then sum it together to create the final output volume. There will be times we'll be using more than 80,000 cores on a single MPI process.”

Among the geological challenges encountered at the Atlantis field are massive salt “canopies” overlaying the oil reservoir. Full Waveform Inversion is used to overcome the obstruction and distortion of seismic images caused by the canopies. FWI uses advanced algorithms to iteratively refine models of the subsurface by generating seismic wave simulations and adjusting the values of subsurface properties based on the quality of the match between the simulated and recorded data. According to BP, FWI has identified an estimated 1 billion barrels of oil across the company’s four hubs in the Gulf.

Wave Form Inversion showing a salt canopy

“We’ll come in with potentially terabytes of data for a seismic acquisition, the largest will exceed a petabyte,” Gray said. “We’ll process it, look at it in many dimensions and it is common for these research steps to create 3 to 5 PB of intermediate results that we will then analyze to put together a volume of the earth.”

The ongoing identification of new energy reserves defies predictions dating back to the 1970s that the world would run out of oil within a few decades.

“The key message is this: as we create new technical capabilities, as we open up new areas, we’re able to produce from more challenging fields,” said Gray, “We’ve been able to continue to deliver energy needed for the world, even though there were skeptics that would have claimed we were reaching a peak of energy production and we wouldn’t be able to continue to meet the requirements.”