Cloudera, Databricks, IBM, Intel, and MapR Collaborate
Open source contributors, Cloudera, Databricks, IBM, Intel, and MapR announced that they are joining efforts to broaden support for Apache Spark (Spark), while simultaneously standardizing it as the framework of choice by bringing popular tools from the MapReduce world to this new engine.
Spark has quickly become a standard in many Hadoop distributions, with rapid customer adoption and use in a variety of use cases, ranging from machine learning to stream processing workloads. To further support this growth, these five vendors have come together to collectively broaden the range of tools and technologies in the Hadoop ecosystem that leverage Spark as an underlying processing engine.
Today, besides being used independently as a programming language, Spark is used as the basis for several projects including:
- Spark Streaming for continuous data processing
- MLLib for a machine learning toolkit
- GraphX for graph analytics capabilities
In recent months, other projects have also added support for Spark, as evinced by recent efforts to port Crunch, Mahout, and Concurrent’s Cascading framework to Spark.
This collaborative new effort expands upon the Spark momentum to include several key Hadoop projects - starting with the Apache Hive SQL engine (Hive). Using Spark as the underlying execution engine, this effort will improve the performance of batch SQL jobs in Hive, while seamlessly maintaining compatibility with the core Hive code base.
Simultaneously, the group is investigating ways to adapt Apache Pig to leverage Spark, as well as other popular tools, such as Sqoop and Search. By making Spark the execution layer of choice, this group is driving consolidation and standardization around Spark as the evolution of MapReduce for modern hardware.
This effort highlights the power of open source communities, with marketplace competitors coming together to help shape a common execution layer, thus creating a community standard. End users benefit by having a widely supported execution layer, preventing lock-in, while continuing to use their tools of choice. Further, the simplicity of having to manage and learn a single engine reduces operational costs.
Spark is an open source data analytics framework originally developed in the AMPLab at UC Berkeley. Quickly embraced for its inherent advantages, such as improved data processing and in-memory capabilities on Hadoop, Spark offers application performance gains up to 100 times faster than Hadoop MapReduce for certain applications. Spark has attracted the attention of the open source community and vendors alike.
Hive is a data warehouse infrastructure initially developed by Facebook Inc. and built on top of Hadoop. Hive was created to query and manage large datasets stored across a cluster of servers. Hive continues to remain a popular choice for SQL batch processing and it offers many advantages to customers. There is an active community including enterprise vendors Cloudera, IBM, Intel and MapR, committed to furthering Hive based on cutting edge industry standards.
Cloudera, Databricks, IBM, Intel, and MapR agree that by bringing Spark more widely to Hadoop communities, the outcome will be a rich and unified ecosystem that will deliver the next level of performance in Hadoop deployments. A proposal for Apache Hive on Apache Spark has been submitted and companies anticipate work immediately upon its approval. Hive on Spark will work within the context of the existing Hive community and establish Spark as the back end standard to improve Hive performance.
