Regional Datacenter Bracing for 40GbE Demand 

Datachambers, a datacenter operator that offers co-location and hosting services across North Carolina, has tapped Extreme Networks to be its Ethernet switch provider as part of a build out of its operations. The move displaces gear by Cisco Systems. Thanks to the partnership between Datachambers and Extreme, the datacenter operator is also getting its hands on two of the new Extreme Summit X770 40 Gb/sec Ethernet switches, announced today, to give them a test run.

Not that Datachambers is expecting to move to 40 Gb/sec switching as part of its current datacenter build out. EJ Schwartz, director of solution engineering, tells EnterpriseTech that the company can offer 1 Gb/sec or 10 Gb/sec to customers who co-locate their servers in its facilities and that demand is not yet there for 40 Gb/sec pipes. But Datachambers is getting ready. "Six months from now, someone could write a killer app that requires that," says Schwartz.

Datachambers is a twelve-year old datacenter operator that was founded by North State Telecommunications as it expanded from its telecommunications business out into datacenter services. North State is a hundred years older than that and provides television, voice, and data services in the central of North Carolina.

Datachambers started out with a 15,000 square foot datacenter in Winston-Salem, and doubled that again three years ago when it was operating at 85 percent capacity, says Schwartz. At the same time as the Winston-Salem facility was running out of space, customers from that area wanted to have remote disaster recovery sites. So Datachambers built a 30,000 square foot datacenter in Raleigh (shown above) after retrofitting a 50,000 square foot building.

Demand for services has compelled Datachambers to build its first datacenter from the ground up rather than retrofit an existing building to house servers, storage, and switches, and it will be located in Charlotte, located near the Charlotte Douglas Airport and the North Carolina. The facility will weigh in at 50,000 square foot facility in Charlotte and is expected to open in the third quarter of next year. The design is being created now, but Schwartz says that it will be modular so it can be expanded easily. This has not been the case with the retrofitted buildings Datachambers has used in Winston-Salem and Raleigh.

Once Datachambers committed to building a new datacenter, the company figured it was time to look at multiple vendors for the gear inside of the datacenter. The company's value-added reseller, Blue Door Networks, strongly suggested that Extreme be brought in to bid against Cisco, and Schwartz tells EnterpriseTech that this had not occurred to the company because "we thought that they were out of our price range." Extreme competed hard to win the deal, and not only is Extreme gear going into the new Charlotte datacenter, but the core and edge switches in the Winston-Salem and Raleigh datacenters have also been replaced with Extreme gear.

Specifically, each facility has two BlackDiamond 8806 modular core switches, which have 10 Gb/sec and 40 Gb/sec line cards. Each switch has 1.95 Tb/sec of aggregate switching bandwidth each and can handle 1.42 million packets per second of forwarding at Layer 2 and Layer 3 of the network stack. For the edge, Datachambers has an inner edge that its customers see and an outer edge that links to the data services from carriers Verizon, Level 3, DukeNet, Time Warner Cable, and AT&T. The outside edge is comprised of Summit X480 switches from Extreme, which have 48 Gigabit Ethernet ports and six 10 Gb/sec ports. These switches can be stacked and managed as a single unit with up to 384 ports. The inside edge that reaches out to the servers in each facility is made up of Summit X460 servers, which cram 52 1 Gb/sec Ethernet ports into a 1U enclosure and up to 416 ports in a stacked switch. The BlackDiamond 8806s sit between the inside edge and the outside edge, linking the two.

This week, concurrent with the launch of the Summit X770 switches, Datachambers is getting two of the devices from Extreme for testing purposes that the network admins can play with. The Summit X770 is designed explicitly to allow companies to support 10 Gb/sec connectivity today but to switch to 40 Gb/sec in the future while leaving the switch in place, and that is why it is interesting for Datachambers and similar customers who want to bridge between the two bandwidths.

"The 10GE port is really starting to take off on servers," says Todd Acree, director of product management at Extreme. Most of the tier-one server makers have a 10 Gb/sec LAN-on-motherboard option, either a direct port or one that snaps in via a mezzanine card, and 10 Gb/sec ports are already mainstream in converged systems as well as in the infrastructure underpinning public clouds. Perhaps more significantly, 40 Gb/sec is starting to get some ramp, too, with Mellanox Technologies shipping ConnectX-3 adapters. So the time is right to offer a 40 Gb/sec switch, according to Acree, but it has to be one that can also support 10 Gb/sec ports as a stopgap.

Rather than have two different switches using the same ASIC and offering different port types and counts, as many of its competitors do, Extreme has one switch with 40 Gb/sec ports and is encouraging customers who want 10 Gb/sec ports to use cable splitters, which impose no performance penalty in terms of port-to-port latency. This is, in fact, the way Extreme expects most customers to use the new X770, which can support up to 104 10 Gb/sec ports with splitters. That is the limit of the Broadcom "Trident-II" switch ASIC used in the chip, which grabs 24 ports for its own internal use. Pull the splitter cables out, and you have a 32 port 40Gb/sec switch, and that is a large number of ports for a 1U enclosure.

The Trident-II offers around 600 nanoseconds of latency on port hops, which is 30 percent lower than the X670 switch the X770 replaces. The ASIC has support for the VMware VXLAN and Microsoft NVGRE overlays for Layer 3 networks to convert them into giant virtual Layer 2 nets.

The base X770 switch is expected to sell for between $40,000 and $45,000. An additional set of software capabilities such as OpenFlow, TRILL, MPLS, and IEEE 1588 time stamping adds around 20 percent more to the cost of the switch. The X770 will start shipping in January.

Across says that about 70 percent of Extreme's business comes from selling into enterprise datacenters, with the remaining 30 percent coming from service providers like Datachambers as well as supercomputing centers in government and academia.