AI-focused lakehouse supplier Databricks is partnering with Amazon so customers can develop customized GenAI apps using Databricks-held data, Amazon Bedrock models, and AWS Trainium chips.
Through this Strategic Collaboration agreement, joint customers will be able to use Databricks’ Mosaic AI to pre-train, fine-tune, augment, and serve large language models (LLMs) on their private data, backed by the scale, performance, and security of AWS.
Ali Ghodsi, co-founder and CEO at Databricks, stated: “Generative AI is disrupting every industry and a company’s data will ultimately be its competitive advantage. Strengthening our collaboration with AWS allows us to provide customers with unmatched scale and price-performance so they can bring their own generative AI applications to market more rapidly.”
Mosaic AI capabilities include Mosaic AI Model Serving, which is designed to support a variety of model providers, including the broad selection of models available via Amazon Bedrock. Customers can develop customized AI apps without sacrificing control over their data or intellectual property.
AWS says Trainium benefits from the Nitro System collection of hardware and software components designed to provide high performance, availability, privacy, and security for cloud-based workloads.
Matt Garman, AWS CEO, said: “By using AWS Trainium to power Mosaic AI, Databricks will make it cost-effective for customers to build and deploy generative AI applications on top of their analytics workflows, regardless of their industry or use case.”
AWS and Databricks will work with systems integrator (SI) partners to migrate data estates running in on-premises datacenters to AWS. Databricks will build technical solutions and implementation resources for SI partners to help customers identify GenAI workloads and on-premises migrations that can be optimized with Databricks on AWS.
The pair will continue to develop joint industry solutions for industries like media and entertainment and financial services.
Databricks has been rapidly building up its GenAI capabilities in the last 18 months. The Dolly AI chatbot for analyzing customer data was announced in April last year. In June 2023, it bought GenAI startup MosaicML and its LLM development functions for $1.3 billion. It then bought Tabular for its data management software layer based on Iceberg tables for a billion dollars or more in June this year.
The biz wants to provide everything a customer needs to store massive amounts of data suitable for GenAI model mining and building, and provide facilities for customers to develop their own models to extract insights from their data.
Databricks’ AWS business has passed the $1 billion run rate, helped by the AWS Marketplace, where total contract value has doubled on an annualized basis over the past two years. The expanded partnership will also include new integrations for Databricks on the AWS Marketplace, including simplified onboarding and configuration and serverless compute powered by AWS.
The Databricks Data Intelligence Platform is available on AWS and has achieved FedRAMP High Agency Authority to Operate for its cloud services on AWS GovCloud.
We anticipate similar partnerships between Databricks and other major tech players – such as Microsoft with Azure and OpenAI, as well as Google with Anthropic.
