Cisco Bakes SDN Into Switch Chips 

Cisco Systems is still the dominant supplier of switches and routers in the datacenter, but upstarts like Arista Networks and Juniper Networks have done well selling faster switches in flatter networks. Now it is Cisco's turn to push back, and it is doing so with a new line of high-end Nexus 9000 switches with software-defined networking functions etched onto chips.

Cisco's SDN approach is in stark contrast to the alternatives from Arista and Juniper, who are pushing SDN functions out into software. VMware's NSX network virtualization tools work on a variety of switches and is a software-only play as well.

If you boil it all down, the new Application Policy Infrastructure Controller that is an add-on for the Nexus 9000s is the secret sauce to the new product, but it is not required. The Nexus 9000s can also be run as a plain old switch, albeit one that has a substantially pared-down NX-OS operating system that was designed to work hand-in-glove with APIC.

APIC was developed by Cisco spinout Insieme Networks, and the networking giant says that the combination of the new switches and software can cut the cost of virtualizing the network by a factor of three or more. This is done by implementing software-defined networking in a collection of specialized chips and homegrown software rather than a software-only stack that companies like VMware are charging a hefty premium for these days. This strategy of embedding SDN software into a leaf-spine network is not necessarily going to be appealing to customers who want to separate hardware from software, but there are plenty of customers who tend to follow Cisco's lead.

"Today's network is basically a network of boxes," Ish Limkakeng, vice president at Insieme, tells EnterpriseTech. "It is stable and scalable. But if you look at applications, they have evolved separately. Applications are the important thing, but they have to be fit to the network."

The strategy behind the Nexus 9000s is called Application Centric Infrastructure, and with it, Cisco wants to have the network better fit the applications. You define everything about an application in the policy engine, which in turn pushes out that policy to all of the bits of infrastructure – switches, security software and appliances, servers, hypervisors, storage, and so on – where that application resides. When the application moves, say during a live migration of virtual machines, the policies travel along with it.

Cisco is perfectly happy for companies to use the APIs that it is publishing as to weave together hypervisors, cloud controllers, SDN controllers if they want to go with an open source stack. But what the company really wants its customers to do is to hook the APIC software embedded in the switches into OpenStack or other controllers as well as hypervisors and have it manage the network and security configuration for any application anywhere on the network. APIC does not provision systems or applications, but it has hooks so other tools, such as Chef or Puppet, can manage these functions under the guidelines set by APIC. The combination of APIC chips and software is designed to manage up to 1 million endpoints.

As in times past, Cisco funded a small development team outside of itself to come up with a big ideas and implement them. At the launch event in New York, Cisco said it was spinning Insieme Networks back into the company now that it has been successful, and that it would spend up to $863 million to acquire it, with the amount depending on how much revenue the APIC add-ons for the Nexus 9000 line and future switches generate. Cisco funded Insieme with $100 million in early 2012 and then kicked in another $35 million this time last year. So the total investment for Cisco for ACI hardware and software could be well north of $1 billion when this is all said and done.

This spinout/acquisition approach was used in the development of Cisco's Unified Computing System converged server-networking platforms, which launched in March 2009 and which have made Cisco a player in enterprise datacenters. The same core team of Luca Cafiero, Prem Jain, and Mario Mazzola ran both the UCS effort and Insieme.

The new Nexus 9000 switches are based on a mix of off-the-shelf and custom network ASICs, unlike a lot of Cisco's products, which are based on its own chips instead of merchant silicon. Limkakeng says that the Nexus 9000s are based on a combination of Trident-II ASICs from Broadcom, which are used for the basic switching, and custom chips that implement the APIC protocols for managing virtual machines and application templates, code-named "Northstar" and "Alpine." The Northstar chip is used in the leaf switches and the Alpine is used in the spine switches. The switches support VMware's VXLAN protocol, which creates an overlay for multiple and distributed Layer 2 networks over Layer 3 in the network stack, making it all look like one big, flat Layer 2 network as far as server virtualization hypervisors and virtual machines are concerned.

There are three new Nexus 9000 switches. Two top of rack switches are in the 9300 series and one modular switch with eight line cards is in the 9500 series. Cisco says it will launch Nexus 9500 series switches with four and sixteen line cards as well as a wider set of top-of-rack switches in the Nexus 9300 series in 2014.

The Nexus 9396PX has 48 SFP+ ports running at 10 Gb/sec and 12 QSFP+ ports running at 40 Gb/sec in its 2U rack enclosure. These 40 Gb/sec ports are in an uplink module that is independent of the 10 Gb/sec ports. It has a switching bandwidth of 960 Gb/sec. The Nexus 93128TX doubles up the 10 Gb/sec ports in a 3U rack enclosure and keeps the same dozen-port 40 Gb/sec uplink module. The switching bandwidth is boosted to a total of 1.28 Tb/sec. You can only use eight of the twelve 40 Gb/sec ports on the 93128TX, presumably because of bandwidth limitations. Port-to-port latencies are on the order of 1 to 2 microseconds.

The 93128TX was designed explicitly so it could provide all the switching for two racks of servers with its 96 downlinks and eight usable uplinks. A pair of these (for redundancy) takes up only 6U of space, while a pairs of 9396PX switches offering the same number of downlinks can only span 48 servers in their 4U of space. You save 2U of rack space across two racks if you go with the fatter Cisco Nexus 9300.

The Nexus 9508 is the big, bad modular switch that is designed to be the spine in a very large leaf-spine network; it can also be used as the core switch at the end of a row of racks if you want to build a traditional three-tier network with access, aggregation, and core tiers. The Nexus 9508 has eight line cards and takes up 13U of space in a rack. The modular switch has six fabric modules for linking line cards together, each rated at 5.12 Tb/sec, for a total of 30.7 Tb/sec of aggregate bandwidth. The switch can be configured with up to 1,152 10 Gb/sec ports or 288 40 Gb/sec ports.

The size of the leaf-spine network that can be built from these new Nexus 9000 switches is enormous, as you can see here, in this example that pairs up Nexus 9508s as spines to create a network with 55.296 10 Gb/sec ports to servers:

That full scalabillity to 55,296 servers is based on the future 9516 switch with sixteen line cards. Cisco is not providing pricing on specific switches, but says that a leaf-spine network with 288 ports – two spine switches plus three leaves – will cost on the order of $75,000. This is the cost without the APIC software and hardware enabled. Limkakeng says that adding the APIC controller and other software functions to the switch will add about 15 to 25 percent to the cost above and beyond the hardware.

But what Cisco is touting as the real benefit for the APIC setup with its new switches is all of the software that companies do not have to buy to implement SDN on top of all of the Nexus 9000 switches in the network. Depending on the server and network virtualization stack chosen, Limkakeng says that the APIC approach can deliver as much as a 75 percent reduction in the cost of a VM. The chart above shows the cost of a 2,000 port network with 1,000 server nodes, each with 20 virtual machines (10 VMs per port). The upshot, says Limkakeng, is that Cisco can deliver an application-aware SDN setup with 10 Gb/sec ports at the same price as it costs to build a 1 Gb/sec network with a stack of external SDN software from VMware.

The Nexus 9000 switch running the streamlined NX-OS network operating system is available now. The APIC software that accesses those embedded chips for the SDN functions is expected to be available in the second quarter of next year, with the target being sometime around April. Cisco has built a simulator for beta tester customers to use so they can see how APIC works ahead of its delivery.