BAE Systems Arms Clusters With GPU And Xeon Phi Accelerators 

With over 88,000 employees and £18.2 billion in revenues, BAE Systems is one of the largest defense, aerospace, and security contractors in the world and a major supplier of technology to the governments of the United Kingdom, United States, Australia, and Saudi Arabia. Like other high-tech manufacturers, BAE Systems relies on simulation and modelling software to create its products, but its needs are sometimes so unique that it often has to create its own such software.

That job falls to the Advanced Technology Centre, which is the central research and development arm of BAE Systems that is located in the United Kingdom. The ATC employs around 350 scientists and engineers who are experts in biology, materials, acoustics, electromagnetics, and other physical sciences that are used in the creation of munitions and armor to protect against them as well as all manner of craft for the air, land, sea, and space. The ATC, which is spread across three sites in the United Kingdom, is vital for keeping BAE Systems in the running for defense contracts against its larger rivals, Lockheed Martin and Boeing, and ahead of its peers Raytheon, General Dynamics, Northrop Grumman, EADS, Finmeccanica, and United Technologies.

As you might expect, software plays a large role in what BAE Systems does, and the ATC has different teams developing software for specific fields. There are about 30 people working on computational fluid dynamics, electromagnetics, electrodynamics, acoustics, real-time image processing, and data analysis codes, many of which are developed in-house, Pierre Moinier, group leader for integrated modelling technologies at the ATC, tells EnterpriseTech.

The computational engineering department at the ATC that Moinier runs has a number of software development clusters, which range in size from 500 to 1,000 cores, and it is on these machines that software engineers create their own codes or tune up off-the-shelf codes so they can be used by engineers working for the various BAE Systems business lines. Those business lines have their own – and generally substantially larger – clusters to run the simulation and modelling applications.

BAE Systems using ANSYS Fluent and CFX for some computational fluid dynamics work, and Livermore Software Technology's LS-DYNA for structural dynamics. Moinier bragged a bit about BAE Systems' own CFD, electromagnetics, and acoustic simulation software, saying that it was highly scalable and did some things that the third party tools cannot do and that are specifically aimed at the kind of weapons it builds for the governments of the world.

ATC looks at any new technology that can help accelerate these modeling applications and that could also, in turn, possibly be deployed inside of the defense systems that it creates on behalf of its military customers.

"We have been heavily involved with the use of GPUs from the very beginning back in 2008," says Moinier.

The ATC was one of the first organizations in the world to port computational fluid dynamics modeling software to GPU coprocessors when Nvidia was just getting started with the Tesla line of parallel compute offload engines. "Our own CFD code is fully GPU-enabled, and we are cutting edge in this space." Moinier says that the software engineers at ATC were able to get a factor of 10X speedup using GPUs to accelerate the homegrown CFD software. Another program that does multi-camera tracking for real-time operations also saw a 10X speedup after being ported to Tesla GPUs, and a ray tracing program used for signature prediction (meaning the shape of something as it moves through a fluid or gas) saw a factor of 100X speedup on GPU accelerators. A weapons clearance dynamics application saw a 20X speedup and also allowed for the use of a higher-fidelity model.

"Basically, we got all of these very good performance achievements at a fraction of the cost of what we would have traditionally spent on conventional hardware," says Moinier.

Now, the ATC is taking a hard look at what the current generation of Xeon Phi parallel X86 coprocessors might do for its code and thinking about what the future "Knights Landing" Xeon Phi chips, which will be available as a standalone processor as well as a coprocessor that plugs into a PCI-Express 3.0 slot, might do to accelerate applications.

Moinier says it is too early to tell whether Tesla or Xeon Phi accelerators will prevail inside of the modelling and simulation systems at  BAE Systems.

"We just ported our CFD code to the Xeon Phi, and it wasn't a very difficult activity," he explains. "However, we have not achieved the top performance yet simply because we have not had the time to tune. In order to achieve the performance you need to spend a lot of time in order to make the best use of vectorization. The code has been ported but it needs to be optimized. The Tesla GPU and Xeon Phi both have clear benefits, but they both come at a price. Our code cannot run directly on GPUs, and with Xeon Phi it might be a little bit easier, but you still have to spend a lot of time in terms of optimization. So at the moment, we haven't made up our mind and we are not sure who is going to dominate the HPC market. But we should be pretty much ready for whatever will be the most appropriate at the time. The important thing is that both have performance gains and we want to have the choice. So we have invested so we have the right tools and the right expertise."

What is interesting about BAE Systems is that it wrote its CFD code, and a lot of its other applications for that matter, in C++ with just a smattering of Fortran here and there in some applications. Without giving away too many secrets, Moinier says that the C++ code was abstracted from the underlying iron in such a way that it doesn't matter too much if it is being deployed to Tesla GPUs or Xeon Phis. The ATC says that it chose C++ because code maintenance is easier and because it can draw from a larger pool of C++ experts when it is looking for staff to work on code.

Rather than using the kinds of integrated development environments typically used at enterprise shops, the programmers at the ATC working on its engineering codes tend to use the open source GNU compilers or Intel's Parallel Studio XE compiler suites. No fuss, no muss. There is no hard and fast rule about what compiler must be used where, but as you might expect, the Intel compilers are probably best suited to hybrid Xeon-Xeon Phi machines. The ATC engineering software team is a long-time user of the Allinea Software's DDT debugger and has just adopted Allinea MAP, a performance profilers to help with the optimization of parallel code running on X86 processors or hybrid CPU-accelerator setups. Developers work from Linux and Windows workstations, and developers tend to favor Linux machines when they write applications and they tend to prefer the GNU C++ compilers.

The key thing that ATC focuses on is trying to position BAE Systems so its simulation and modeling software will be able to take advantage of any technology that comes along – even quantum computing, as it turns out. BAE Systems has not bought a D-Wave Two System quantum computer, but it is going to be doing some collaboration with the company, just to give you a sense of how much on the leading edge BAE Systems likes to be.