Databricks Unveils Its Natural Language Interface, LakehouseIQ
SAN FRANCISCO, June 28, 2023 -- At the sold-out Data + AI Summit, Databricks today announced LakehouseIQ, a knowledge engine that learns what makes an organization's data, culture and operations unique. LakehouseIQ uses generative AI to understand jargon, data usage patterns, organizational structure, and more to answer questions within the context of a business.
Anyone in an organization can interact with LakehouseIQ using natural language to search, understand, and query data. LakehouseIQ is fully integrated with Databricks Unity Catalog to help ensure that democratizing access to data adheres to internal security and governance rules.
The AI-Powered Knowledge Engine that Uniquely Understands your Business
Today, every business must be data-driven. Yet, when employees need access to internal data to help complete their tasks, it can be difficult to quickly find and analyze the data they need. Whether the CEO is trying to build quarterly sales forecasts or a marketer is attempting to analyze campaign performance, knowledge workers rely on a small team of over-worked data scientists and programmers to find and query the relevant data sets. This bottleneck prevents businesses from truly embracing data and AI. Large language models (LLMs) promised to fix this problem, but so far, the results have been disappointing. General purpose models don't understand the unique language of every business: they cannot process jargon or internal acronyms; they are not trained on the company's unique data sets; and they do not understand organizational charts or know which teams should have access to what information.
LakehouseIQ learns from these signals within an organization using schemas, documents, queries, popularity, lineage, notebooks, and BI dashboards to gain intelligence as it answers more queries. LakehouseIQ helps employees get immediate answers to questions without requiring that they possess the technical skill required by traditional data analysis tools. The engine understands their unique business jargon and context to more accurately interpret the intent of the question, and can even generate additional insights that could spur new questions or lines of thinking. With LakehouseIQ, every employee — not just data scientists — can unlock the full potential of internal corporate data to make better, more informed decisions and drive new innovations.
"LakehouseIQ will help democratize data access for every company to improve better decision-making and accelerate innovation. With LakehouseIQ, an employee can simply write a question and find the data they need for a project, or get answers to questions relevant to their company's operations. It removes the roadblocks inherent in traditional data tools and doesn't require programming skills," said Ali Ghodsi, Co-Founder and CEO at Databricks. "Every employee knows the questions to ask to improve their day-to-day work and ultimately their business. With LakehouseIQ, they have the power to quickly and accurately discover the answers."
Securely Democratizing Data Access
LakehouseIQ is governed by Unity Catalog, Databricks' flagship solution for unified search and governance across data, analytics and AI. Organizations can be confident that their employees will only have access to the data they are authorized to use, so increasing data accessibility doesn't increase risk.
"LakehouseIQ solves two of the biggest challenges that businesses face in using AI: getting employees the right data while staying compliant and keeping data private when it should be," said Ghodsi. "It alleviates time-strapped engineers, eases the burden of data management, and empowers employees to take advantage of the AI revolution without jeopardizing the company's proprietary information."
Databricks continues to expand the Lakehouse Platform, recently announcing Lakehouse Apps and the general availability of Databricks Marketplace, Delta Lake 3.0, new governance capabilities, and a suite of data-centric AI tools for building and governing LLMs on the lakehouse.
Availability
The Databricks Assistant, powered by LakehouseIQ, is in preview.
Click here to learn more about LakehouseIQ, watch the Data + AI Summit live.
About Databricks
Databricks is the Data and AI company. More than 10,000 organizations worldwide — including Comcast, Condé Nast, and over 50% of the Fortune 500 — rely on the Databricks Lakehouse Platform to unify their data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe. Founded by the original creators of Delta Lake, Apache Spark, and MLflow, Databricks is on a mission to help data teams solve the world's toughest problems.
Source: Databricks
