Databricks data lake analysts can use Prophecy software to build their own data prep pipelines for downstream analytics and AI processing.
Prophecy, which describes itself as a data transformation copilot company, is a Databricks-focused AI and analytics data pipeline development tool that collects unstructured data from multiple corporate data sources – structured or unstructured, on-premises or in the cloud. The software then transforms it and delivers it to Databricks SQL queries. Its AI-powered visual designer generates standardized, open code that extracts, transforms, and loads the required data. It automatically builds the necessary data pipelines and tests, generates documentation, and suggests fixes for errors. The v4.0 ETL product delivers self-service, production-ready data preparation that operates within guardrails defined by central IT.
Roger Murff, VP of Technology Partners at Databricks, said: “Organizations have put their most valuable data assets into Databricks, and Prophecy 4.0 makes it easier than ever to make that data available to analysts. And because Prophecy is natively integrated with the Databricks Data Intelligence Platform, platform teams get centralized visibility and control over user access, compute costs, and more.”
Prophecy v4.0 features include:
- Secure data loading from commonly used sources such as files via SFTP (Secure File Transfer Protocol), SharePoint, Salesforce, and Excel or CSV files from analysts’ desktops
- Last-mile data operations, allowing analysts to send results to Tableau and notify stakeholders via email
- Built-in automation with a drag-and-drop interface that lets analysts run and validate pipelines without the need for separate tools
- Data profiles showing distribution, completeness, and other attributes
- Packages of reusable, governed components
- Simplified version control
- Real-time pipeline observability to track performance and detect failures, thereby reducing downtime
A blog by Prophecy’s Mitesh Shah, VP for marketing and analyst relations, declares: “Self-service data preparation has been a game-changer for accelerating AI and analytics.” Analysts can get data prepared themselves and don’t have to hand the task off to data engineers, theoretically saving time and duplicated effort.
Shah says that as “Prophecy is deeply integrated with Databricks, we allow organizations to enforce cluster limits and cost guardrails automatically.” That prevents data prep costs from getting out of hand. Prophecy works with Databricks’ Unity Catalog and “analysts inherit existing permissions from Databricks.”
Prophecy was founded by CEO Raj Bains and Vikas Marwaha in 2017. It raised $47 million in a B-round in January, justified by 3.5x revenue growth in 2024 with 160 percent net revenue retention from existing customers. It has raised a total of $114 million, $35 million in a 2023 B-round, $25 million in a 2022 A-round, and $7 million before that.
Bains stated: “Analysts can design and publish pipelines whenever they want, with security, performance, and data access standards predefined by IT. We’ve visited companies where analysts would outline data workflows in their data prep tools and then engineers downstream would recode the entire pipeline from scratch with their ETL software. It was a huge waste of time and energy. With Prophecy 4.0, everything is done once.”
There will be a Prophecy v4.0 in action webinar on April 24; those looking to attend can register here.
