AutoTrader.com Shifts Up Automation Gears
Like many enterprises, AutoTrader.com is on a journey from physical machines running distinct parts of its application stack toward a virtualized and automated cloud that allows for more efficiency and flexibility. It is a long journey, and one that no company completes in a year.
AutoTrader.com is one of the first wave of Internet companies that used the Web to provide services to buyers and sellers, in this case those who were looking to buy or sell cars. Founded in 1997, the company has become a key player in the auto market, and it is now part of the Cox Automotive group, a collection of 20 companies that the Cox Enterprises media conglomerate based in Atlanta, Georgia owns. Cox Enterprises is privately held, and was founded 116 years ago by James Cox, the owner of the Dayton Daily News in the Ohio city of that name; the company has about 50,000 employees across its various groups and around $16 billion in annual revenues. Cox operates the third largest cable company in the United States, with about 6 million consumer and business users, and it owns the Atlanta Journal Constitution, the Austin American-Statesman, and a number of other regional papers, and a slew of radio and TV stations as well. The company's media arm reaches about 52 million Americans through these media outlets.
The AutoTrader.com unit is a key component of the Cox Automotive group, which has relationships with more than 40,000 dealers and around 250,000 private owners and which was just formed in August to bring various acquisitions into a single group. Between them, the AutoTrader.com site, which providers car listing services, and Kelley Blue Book site, which provides estimated values for second-hand vehicles, have about 32 million monthly viewers. The company says that about 67 percent of all car buyers make use of these sites when they go shopping. The Cox's Manheim unit, which is now part of the Cox Automotive group, processes car auction transactions with an aggregate value of $46 billion each year.
At the moment, the various divisions of the Cox Automotive group have relative autonomy when it comes to their IT platforms, and part of that is out of necessity given the fact that Cox acquired them over time. The AutoTrader.com unit has been gradually virtualizing its infrastructure and is also adding more automation to control its virtualized environment. The company is also beginning to experiment with Hadoop as a means of aggregating and correlating information across its automotive various sites and is thinking ahead to how it will make the full move to cloud.
Moore's Law has allowed Autotrader.com to compress its server footprint significantly over the years. Six years ago, the company had approximately 1,800 servers supporting its various units. Today, according to Chris Nakagaki, senior systems engineer for cloud infrastructure at the Autotrader.com group, the company has about 200 servers split across its two Georgia datacenters. The datacenters operate in an active-active fashion, with workloads distributed across the two, running on more than 3,000 virtual machines. So the server footprint has compressed in a physical sense, but the number of virtual servers has nearly doubled over that time.
A few years back, Autotrader.com deployed Hewlett-Packard's ProLiant DL360 G7 rack-mounted systems as its basic building block, and now it is in the middle of upgrading its systems to PowerEdge M1000e blade enclosures and PowerEdge M620 server nodes based on Intel's "Ivy Bridge" Xeon E5-2600 v2 processors. Each node has two 12-core processors. While AutoTrader has been virtualizing some of its workloads for many years, the virtualization effort really got underway in earnest a little more than three years ago atop VMware's ESXi hypervisor. The company is now using ESXi in the vSphere Enterprise Plus packing at the 5.0 release level and is getting ready to move to the 5.5 release. (VMware has not yet announced ESXi 6.0, which is still in beta testing.)
Today about 80 percent of the workloads behind the Autotrader.com site have been propped up onto virtual machines, and most of the applications are written in Java and run atop the JBoss application server that itself runs on Red Hat Enterprise Linux. (Red Hat controls JBoss, of course.) This is level of virtualization for workloads consistent with the level of virtualization out there across the enterprise at large.
At this point, says Nakagaki, the two big workloads that are not virtualized are its back-end Oracle databases and the Microsoft FAST search engine service that underlies its various web sites. Other divisions of Cox Enterprises have virtualized their databases, and Autotrader.com has not done so more because of licensing issues than technological ones, according to Nakagaki. Autotrader.com is replacing FAST with the open source Apache Solr search engine, and this will probably be run on bare metal because Solr was designed to run in a larger memory footprint than the PowerEdge M620 nodes can deliver. "That is one of the reasons that we went with blades," explains Nakagaki. "We can get some of the benefits of virtualization without having to virtualize. But it is definitely something I want to re-adjust down the road. It is a matter of sizing it and then architecting it so it is more of a cloud-based system and it fits into our virtualization."
The production machines in the Autotrader.com clusters tend to have about 15 virtual machines per host, but the six development machines in the company's clusters have a total of 855 virtual machines, for an average of 143 VMs per host, or nearly ten times the VM density of the production machines. Autotrader.com could cram more VMs onto its production environment, but it has a rule that CPU utilization on the production machines in the cluster cannot exceed 40 percent, thereby leaving room for the random spikes in traffic that can happen out there on the Internet.
Until recently, Autotrader.com relied on the vCenter console and system admin experience to manage its cluster. "We licked our thumbs and held them up in the air to see which way the wind was blowing," Nakagaki says with a laugh, but in all seriousness, it meant drilling down into vCenter and babysitting machines manually. The other divisions of Cox Enterprises were leveraging VMware virtualization and management tools, and so the parent company was able to negotiate a bigger enterprise license agreement for Autotrader.com. And with that came licenses to vRealize Operations Insight, which is the combination of vRealize Operations, which does performance and capacity management, and vRealize Log Insight, a log management and analytics tool. The log analysis is used to correlate the performance of VMs with events that are happening in the software stack, thereby giving admins a better sense of why something is happening, not just an alert when something is happening.
The entire cloud for the Autotrader.com group is managed by four admins, who provision the cloud itself and the virtual machines for new and existing workloads. An image processing cluster that takes in and renders pictures of the vehicles is separate from the core two clusters that host the AutoTrader.com and KBB sites.
At a certain level of spending, Nakagaki foresees that Autotrader.com could move up to the full vRealize suite of tools essentially convert its virtualized clusters into full-on clouds, complete with automation and disaster recovery. But other divisions of Cox Enterprises are using Red Hat's CloudForms, an orchestration overlay that is compatible with vCenter and ESXi, and it is equally likely that Autotrader.com could go in this direction. Cox Enterprises has two Ruby developers that actually contribute to the open source CloudForms effort at Red Hat. Before choosing CloudForms, the Cox companies were also looking at DynamicOps, a tool that was acquired by VMware a few years back and is now one of the key components in the vRealize suite. (vRealize Automation 6.2, to be precise.) Ultimately, the choice will be driven by how well either tool supports bursting to various public clouds, including VMware's ESXi-compatible vCloud Air as well as Microsoft Azure and Amazon Web Services. Nakagaki is also looking carefully at the OpenStack implementation that VMware is creating.
"I am kind of the opinion that if we are going to be developing any kinds of workflows, that we should be developing against OpenStack," he says. "That should give us breadth, and we can then develop once and we don't have to redevelop to deploy to any other platform on the backend. If we develop against OpenStack, it becomes a little bit easier." For now, Nakagaki says that developing applications on vCloud Air makes life easier because if they need to be brought back in house, this can be done relatively easily and using the existing VMware management tools.
Just Getting Started With Hadoop
The Cox Automotive group, which is comprised of Autotrader.com and Manheim and all of their subsidiaries, has another initiative that is just getting underway, says Nakagaki. And that is a baby Hadoop cluster that is being set up by the organization's Linux group. The Hadoop cluster has a dozen nodes and is running Cloudera's distribution of Hadoop.
The idea is do a lot more sophisticated analysis using data across the various sites that Cox Automotive operates. Manheim, ExportTrader.com, DealShield, Go Auto Exchange, and Ready Auto Transport provide auto auction and wholesale services. AutoTrader.com and Kelley Blue Book allow dealers and private sellers to list their cars for shoppers and to find out the book value of the cars. HomeNet Automotive, Haystak Digital Marketing, vAuto, VinSolutions provide various services and software to dealers. Nakagaki says that through all of these companies, Cox Automotive covers a lot of the car supply chain. As a simple example, using data across its companies, Cox Automotive could not only tell customers what price to sell a car at, but could use analytics to figure out when and where to sell that car to maximize the price. Moreover, you could also offer a service to car buyers where they could watch the vehicle they buy be manufactured, shipped to the dealer, delivered to them, maintained over the years, and then sold.
"There are a lot of data points that we are trying to pull together to provide that kind of value," Nakagaki says. With the various units of the Cox Automotive group having their own datacenters and IT operations, then Hadoop will become a kind of data glue that pulls them together.
