New Ranking of Public Cloud Providers Reveals Big Price/Performance Disparities
(Source: Shutterstock/wavebreakmedia)
As companies increasingly move portions of their advanced scale computing workloads to the cloud, at least one industry watcher warns that the lure of an instant, scalable virtual infrastructure with utility billing comes with plenty of caveat emptor price/performance variables.
In an update to its public cloud benchmark released today, Cloud Spectator, a Boston-based IaaS watcher, reports the current price/performance cloud services provider (CSP) leader is Germany-based 1&1 Internet, followed by ProfitBricks, Ubiquity, CloudSigma and Google Compute Engine. At the bottom of its ranking of 17 CSPs: Amazon Web Services and IBM Softlayer. Microsoft Azure comes in at 12th.
The report, updated frequently by Cloud Spectator, underscores not only price/performance differences in the public cloud market but also its volatility. While the report from a year ago also placed 1&1 in the top spot, it was followed by AWS, Aruba Cloud, CloudSigma and Azure.
Cloud Spectator's Kenny Li
The report ranks CSPs using a Cloud Spectator methodology measuring the price/performance of vCPU, memory, and block storage. CEO Kenny Li said his company’s study puts the lie to the assumption that one CSP is much like another. While CSPs often use the same terms to label technology resources, Li said, differences in the underlying hardware, architecture, and performance tuning lead to different results.
"When organizations select a cloud provider, they often overlook the impact performance can have on the total cost of ownership, due to the assumption that all clouds are similar,” he said. “Believe me when I say it isn't true; in our 2016 study, we witnessed a 10x difference in performance on block storage, and a 3.5x difference in VM among the top 10 public cloud providers benchmarked. If you don't apply this type of research in your cloud selection strategy, then your costs aren't optimized and you could miss huge potential savings."
Bottom line: While it may be true that “you get what you pay for” when it comes to support, security, location and managed services from a CSP, Cloud Spectator concludes there is little correlation between price and performance. Indeed, the best-value CSPs offer virtualized resources at the lowest prices. “Similarly sized VMs within the Top 10 IaaS providers displayed a spectrum of prices with a 5x difference between the least and most expensive CSPs.”
Cloud Spectator said the report debunks another widely held assumption: that resource contention, or the “Noisy Neighbor Effect,” is not a major concern. Because CSPs offer multi-tenant physical hosts, a business may share the same physical resources with different users on the same hardware. But “with a lack of understanding of other users’ activities, resource-hogging applications can affect the performance of other VMs on the host machine.” This impacts the performance of block storage offerings, which exhibited a 25X difference in IOPS during the 24-hour test period of the study, according to Cloud Spectator.
It’s “a lack of transparency in the public cloud IaaS marketplace for performance” has led to misinformation or false assumptions about public cloud price/performance.
“Users and potential users may be led to view cloud computing as a commodity, differentiated mostly by services. The reality of performance in cloud computing, though, impacts the user differently from CSP to CSP, involving everything from the physical hardware (e.g., Intel or AMD, SSD or spinning disk), to the cost of the virtualized resources.” The key to maximizing cloud value is to identify environments based on performance rather than resource count.
While Cloud Spectator intended to test 25 of the largest and well-known CSPs, the company could not complete testing on eight of them for reasons ranging from bugs and provisioning issues to resource limitations, “stressing the importance of testing before selecting a CSP.”
The study examined CSP price performance in three areas: vCPU, VM and block storage over a continuous 24-hour period. CSPs were ranked for VM and block storage price/performance along with an overall “CloudSpecs Score.” In ranking 1&1 on top in the overall category, Cloud Spectator noted this was “largely due to high VM performance and the most inexpensive packaged pricing found in the study.”
For performance combined with value in the vCPU and memory category, the results overlapped with the overall category: 1&1 on top followed by Ubiquity, CenturyLink, ProfitBricks and Google.
A similar mix of vendors was top-ranked in the block storage value category: ProfitBricks, CloudSigma, Rackspace, 1&1 and Google.
One of the most interesting of the study’s findings is the absence among Cloud Spectator’s top 10 CSPs of most of the major providers: Amazon, Microsoft and IBM. Cloud Spectator said this is because some of the smaller CSPs specialize in high performance and aggressive pricing, achieving higher scores as a result. Having said that, Cloud Spectator stated that, from a pure performance standpoint, SoftLayer and AWS achieved above-average or higher VM results, but their rankings were held down by higher costs combined, in the case of Amazon, with lower relative disk performance.
To be considered among the IaaS providers tested anonymously by Cloud Spectator, a CSP had to have: at least one data center in North American; self sign-up, without involving a sales representative; self-service, enabling the user to login to a portal to provision, manage and terminate virtual machines and other services; billing intervals by the hour or less; and persistent block storage.
