Datalake supplier Databricks has signed a deal with Palantir, developed a better way to fine-tune large language models (LLMs), and is including Anthropic’s Claude LLMs with its datalake in a five-year alliance.
Databricks bought generative AI model maker MosaicML for $1.3 billion in June 2023 and developed its technology into Mosaic AI. This is a set of tools and services to help customers build, deploy, and manage GenAI models with retrieval-augmented generation (RAG) applied to their proprietary data. It is integrated into Databricks’ datalake, the Data Intelligence Platform. Now it is also integrating the Palantir Artificial Intelligence Platform (AIP) with the Data Intelligence Platform and offering Anthropic’s reasoning-level AI models through it. The TAO (Test-time Adaptive Optimization) initiative uses test-time compute to augment and simplify model tuning without needing labeled data.
Ali Ghodsi, Databricks co-founder and CEO, stated: “We are bringing the power of Anthropic models directly to the Data Intelligence Platform – securely, efficiently, and at scale – enabling businesses to build domain-specific AI agents tailored to their unique needs. This is the future of enterprise AI.”
Anthropic co-founder and CEO Dario Amodei said: “This year, we’ll see remarkable advances in AI agents capable of working independently on complex tasks, and with Claude now available on Databricks, customers can build even more powerful data-driven agents to stay ahead in this new era of AI.”
Palantir’s AIP is a workflow builder that uses GenAI models to analyze natural language inputs, generate actionable responses, and, in military scenarios where it could analyze battlefield data, suggest strategies and tactical responses. Earlier Palantir AI systems, Gotham and Foundry, are data integration and analytics systems designed to help users analyze complex, disparate datasets and produce action-oriented responses, particularly in Gotham’s case for military and national security datasets. AIP combines Palantir’s ontology (Foundry) framework, a digital representation of an organization’s operations, to bring more context to its request responses.
Databricks is bringing Palantir’s military-grade security to its Unity Catalog and Delta Sharing capabilities. The Unity Catalog is a unified governance software layer for data and AI within the Databricks platform. Delta Sharing is Databricks’ way of sharing third-party data. Databricks says that joint Databricks and Palantir customers will be able to use GenAI, machine learning, and data warehousing within a secure, unifiedm and scalable environment.
Rory Patterson, chairman of the board of Databricks Federal, said the combination of Unity Catalog and Delta Sharing with the Palantir system “will deliver the best of both worlds to our joint customers.”
The Anthropic deal enables Databricks to offer Anthropic models and services, including its newest frontier model, Claude 3.7 Sonnet, natively through the Databricks Data Intelligence Platform, available via SQL query and model endpoint. This means that “customers can build and deploy AI agents that reason over their own data.” Claude can “handle large, diverse data sets with a large context window to drive better customization.”
Databricks says its Mosaic AI provides the tools to build domain-specific AI agents on customers’ own data “that deliver accurate results with end-to-end governance across the entire data and AI lifecycle, while Anthropic’s Claude models optimize for real-world tasks that customers find most useful.”
The Unity Catalog works with Anthropic Claude, providing the ability for users to enforce access controls, set rate limits to manage costs, track lineage, implement safety guardrails, monitor for potential misuse, and ensure their AI systems operate within defined ethical boundaries. Customers can customize Claude models with RAG by automatically generating vector indexes or fine-tuning models with enterprise data.
The fine-tuning angle brings us to TAO and its way of side-stepping labeled data needs. An LLM trained on general data can be fine-tuned through additional training by using input items paired with output items, a label, clearly indicating a desired response. This “teaches” the model to generate better responses by adjusting its internal parameters when comparing its internal predictions against the labels. For example, the input could be “Is rain wet?” for a weather-related session with an answer of “Yes.”
However, this can involve a huge amount of human labeling effort, with tens or even hundreds of thousands of input-output labels. A Databricks blog explains that TAO uses “test-time compute … and reinforcement learning (RL) to teach a model to do a task better based on past input examples alone, meaning that it scales with an adjustable tuning compute budget, not human labeling effort.”
The model “then executes the task directly with low inference costs (i.e. not requiring additional compute at inference time).” Unexpectedly, TAO can achieve better model response quality than traditional fine-tuning. According to Databricks, TAO can bring inexpensive, source-available models like Llama close to the performance of proprietary models models like GPT-4o and o3-mini and also Claude 3.7 Sonnet.
The blog says: “On specialized enterprise tasks such as document question answering and SQL generation, TAO outperforms traditional fine-tuning on thousands of labeled examples. It brings efficient open source models like Llama 8B and 70B to a similar quality as expensive models like GPT-4o and o3-mini without the need for labels.” TAO will enable Databricks users to use less expensive and capable GenAI models and improve their abilities.
To learn more about the Databricks and Anthropic partnership, sign up for an upcoming webinar with Ghodsi and Amodei. Anthropic’s Claude 3.7 Sonnet is now available via Databricks on AWS, Azure, and Google Cloud.
