Mellanox Raises SHIELD for Enhanced Network Resiliency
High-performance networking company Mellanox is announcing a new protocol it’s calling SHIELD, an acronym for “Self Healing Interconnect Enhancement for inteLligent Datacenters.” Enabled within Mellanox’s 100G EDR and 200G HDR InfiniBand solutions, SHIELD improves the ability for switches to exchange real-time information in order to overcome link failures and optimize data flows.
The functionality that SHIELD enables has been previously implemented under the covers in Mellanox switches (it’s documented as FLFN, short for fault leak failure notification), but Mellanox is bringing it to the forefront now as the massive scale of today’s biggest clusters means that even small numbers of failures on a network fabric can have a significant impact on the stability of a job.
Where much of Mellanox’s IP is focused on enabling faster computing, SHIELD is about improving the resiliency of the fabric. Link failures are not things that happen routinely in the fabric, but the chances of a link failing on a cluster go up as the cluster gets larger and larger. With a very busy exascale cluster, it can happen frequently enough to impact jobs routinely. Mellanox claims the interconnect technology improves network fault recovery by 5,000 times.
Jerry Lotto, director of HPC and technical computing at Mellanox, explained that when a communications channel is cut off, the subnet manager typically needs to recalculate a new topology for the fabric. “For very large fabrics this can take a significant amount of time, as long as 10-20 seconds,” he said. “Clearly, an inter-process communication that must complete in micoseconds or nanoseconds will time out and won’t survive that recalculation.”
Large enterprise datacenters and traditional HPC and supercomputing environments stand to benefit the most, since on a small fabric, the task of recalculating the routes that the subnet manager needs to perform is a fairly simple task that completes very quickly.
“The new generations of interconnects have to handle data analysis as well as increase the overall robustness of the system,” said Dan Olds, principal analyst at OrionX.net in the press announcement. He called SHIELD “a breakthrough technology [that] will enable users to increase their data throughput and even improve total cost of ownership.”
According to Lotto, there’s no downside to using SHIELD, no performance overhead. “The link failure notification protocol doesn’t even come into play until a switch notices that a link has failed, then at that point it starts to communicate with other switches informing them of the problem,” he told us, getting into the details of how SHIELD works.
“There really are two scenarios that can occur. One is when the switch that is connected to the failed cable has more than one possible route to a desired destination, then it can independently make the decision: if I have traffic that should have gone via the failed link, I’ll simply send it the other way. A much more insidious situation is where the failed link occurs and it’s the only available path to a destination route, then the switch has the ability to communicate upstream to its neighbors who will have the ability to select a viable path about the failed link and give those packets a chance to reach their destination,” he said.
SHIELD works hand-in-hand with Mellanox’s adaptive routing technology. Introduced to enable better bisectional bandwidth out of different topology fabrics, adaptive routing supports out of order packet reception on ConnectX-5 and later adapters. If a link fault causes a packet to take a different route, out-of-order packet reception ensures that packets re-ordered in transit don’t need to be discarded and re-transmitted.
To be clear this capability is not a product or an add-on. It just needs to be enabled and configured in the switch. Every customer with a Switch-IB or later generation switch will be able to take advantage of it, according to Mellanox. This includes the Summit and Sierra CORAL supercomputers that are coming online soon. Naturally the coming generation of supercomputers are seen as a primary use case. “SHIELD will have an important role to play in enabling capable exascale computing,” said Lotto, referring to a production supercomputing environment capable of running continuous exascale jobs, requiring a high confidence that jobs will complete.
