Intel to Pull Networking Into Xeon and Xeon Phi Chips 

Among other things, Moore's Law allows for the successive integration of system components onto processors. Various levels of cache memory as well as main memory controllers and peripheral controllers have been pulled onto most current CPUs, and the next obvious step is to bring network controllers onto the die.

At its annual industry analyst meeting late last week, Intel starting to talk broadly about its plans to put network adapters and fabrics onto its future Xeon processors and Xeon Phi coprocessors. The technical details are a bit sketchy, but this is the first time that Intel has put some timelines on its plans. The possibilities are intriguing, and could include electronics that support multiple network protocols.

The first server-class processor from Intel to get an integrated Ethernet network interface was the eight-core "Avoton" Atom C2000, announced back in early September. This chip has a single Ethernet controller hanging off one of its internal buses, in this case based on the "Powerville" i350 Ethernet controller chip that Intel puts on its discrete network cards that plug into PCI-Express slots. In this case, the on-chip Ethernet controller has four lanes that can be configured to run at 1 Gb/sec or 2.5 Gb/sec speeds. This is sufficient bandwidth for the kind of lightweight workloads that the Avoton chip is aimed at: cold storage, dedicated hosting, and analytics workloads where having lots of cores is more important than fast cores and low-latency networking.

"We will continue to integrate Ethernet into our product line and we will launch it into Xeon next year," said Diane Bryant, general manager of the company's Datacenter and Connected Systems group, in her presentation at the industry analyst meeting.

The company is going to continue to build discrete Ethernet cards, she added, because some customers need more Ethernet ports that Intel will be able to cram onto a CPU.

Bryant did not say what Ethernet interface would be added to the future Xeon processors in 2014 and beyond, but it stands to reason that it will be quite a bit peppier than the one put on the Avoton Atom chip. A good guess is that Xeon chips need ports running at 10 Gb/sec speeds. In that case, Intel could be etching its 82599 Ethernet controller chip onto the Xeon chips. The 82599 chip has variants with one or two ports running at 10 Gb/sec.

This Ethernet controller could be integrated with the standard "Broadwell" Xeon E3 processor due next year for single-socket machines as well as the system-on-chip version of the Broadwell expected next year, too. Both Broadwell chips are implemented in 14 nanometer processes, a shrink from the current 22 nanometer processes used for the current "Haswell" Xeon E3 chips. This shrink is what will allow for the Ethernet ports to be brought on die. It could turn out that the Broadwell SoC is the one Intel will sell with the integrated Ethernet ports activated and the plain vanilla Broadwell will not have them turned on. In past years, no one would have guessed that Intel would have two distinct Broadwell designs. But Intel is doing more customizations across its server chips these days. Remember that there were three distinct "Ivy Bridge-EP" Xeon E5-2600 v2 processors launched in September, each with very different mixes of cores and caches. So, the future Broadwell Xeons, if these are indeed the first to get integrated Ethernet, could both have on-die networking.

It seems unlikely that the future "Haswell" Xeon E5 and E7 processors, aimed at machines with two to eight sockets using Intel chipsets, will have an integrated Ethernet controller. Those chips are not expected until late 2014 for two-socket machines and even later in 2015 for four-socket and larger machines. The timing works for that vague roadmap above. But the Haswell design did not have Ethernet ports on the die and it is hard to believe that Intel would go back and tweak a design that already exists to add network interfaces. Again, anything is possible, particularly if the company is feeling a competitive threat from ARM chip makers from below. It seems far more likely that Intel will start with the low-end Broadwell Xeon E3s with Ethernet integration.

For customers who need higher performance networking, Intel is working on a next-generation fabric interconnect that will be integrated onto both Xeon processors and Xeon Phi coprocessors. This next-gen interconnect, as you can see from the roadmap above, will come out in 2015 and beyond.

The roadmap above is intentionally vague about what kind of interconnect is coming to the Xeon and Xeon Phi chips, and when they will be available. But Bryant did provide some clues, saying that it will be derived from the True Scale InfiniBand technology that Intel got when it acquired the InfiniBand adapter and switch from QLogic two years ago.

"We have taken that technology and we have advanced it and are integrating it both into Xeon as well as Xeon Phi," Bryant said. "These products will start coming out in 2015, and our customers agree that integration of the fabric for high performance computing is highly, highly valued because of lower power, higher density, and higher performance. Some will actually say it is actually a requirement if high performance computing to grow at the pace it has been growing. We do believe that this will be a clear competitive advantage for us going forward."

It is important to not take Intel too literally in these vague roadmaps. First of all, embedding the fabric on the chip almost certainly does not mean putting a distributed switch onto the Xeon and Xeon Phi dies, although that is of course a very interesting possibility. This is the approach that Calxeda is taking with its ARM-based server chips, and it has the goal of creating a fabric across that distributed L2 switch that can span 100,000 nodes without a single top-of-rack switch. Intel is using the word fabric above, and the chart seems to imply that this next-generation fabric will be a follow-on to InfiniBand, perhaps jumping from the current 40 Gb/sec to 100 Gb/sec.

While it is possible that Intel will plunk InfiniBand ports onto the Xeon and Xeon Phi chips, it is equally likely that Intel will come up with something a little more sophisticated than this. The company has its own Ethernet adapter business. It acquired Fulcrum Microsystems several years ago to get Ethernet switch ASICs. It bought QLogic for InfiniBand adapters and switch chips. And then it finished off by acquiring Cray's "Gemini" XE and "Aries" XC interconnects. Intel has one of the best teams in the world with which to create a new interconnect, perhaps one that can speak multiple protocols.

Networking rival Mellanox Technologies, for instance, already has ConnectX adapter chips and SwitchX switch chips that can speak either InfiniBand or Ethernet. So the idea is not totally without precedent.

Intel was mum on the subject of a polyglot network interface when approached by EnterpriseTech for more details, but a betting person would gamble that Intel would add a more generic controller to future Xeon and Xeon Phi chips so they could plug into Ethernet or InfiniBand networks. The adapters could even borrow ideas from the Aries interconnect to give it some extra goodies not found in standard Ethernet or InfiniBand controllers.

By the way, that next-generation Xeon Phi, code-named "Knights Landing," will be a multicore processor in its own right, plugging into its own socket and having its own main memory for processing. Intel will also continue to sell Xeon Phi cards based on Knights Landing as discrete coprocessors for Xeon-based systems.