We launched Lakeflow Designer to general availability in June, and since then, over 3,000 customers have used it to address one of the biggest bottlenecks in working with data: the technical barrier to entry. Teams migrating from legacy self-service data preparation tools have streamlined governance, simplified their data stacks, accelerated their path to production, and cut costs dramatically. With consumption-based pricing and no per-user licensing fees, we’ve seen many customers reduce their spend on no-code tooling by 90% after adopting Lakeflow Designer.
Working closely with these customers has helped us identify ways to make Designer even more useful. Our latest updates bring new no-code operators, enhancements to existing operators, more output options, and an improved canvas for building and managing larger workflows.
Built directly in Databricks, Lakeflow Designer is a visual, no-code, AI-native experience for data preparation and analytics. It lets analysts, domain experts, and other less technical users prepare and explore data through a drag-and-drop canvas and natural language.
What’s New?
Here are some of our favorite enhancements we've made to Lakeflow Designer over the last 4 months.
New no-code operators for data cleaning and quality checks
Genie Code, Python, and SQL operators have always made Designer flexible. However, we heard from early customers that they wanted more no-code ways to handle common tasks like deduplication, data cleaning, and data quality checks.
Prepare operator: Clean and standardize data
Clean and standardize your data with the Prepare operator. Change text case, convert data types, replace values, apply formulas, and more, all in a single operator. Learn more about Prepare.
Unique operator: Remove duplicate records
Quickly remove duplicates across the entire dataset or a selected set of columns. Set an order to control which record you keep, such as the most recently updated record for each customer. Learn more about Unique.
Guardrails operator: Add data quality checks
Add data quality checks directly to your workflow with Guardrails. Check for missing values, duplicates, unexpected values, and more, then choose how failures affect the workflow: continue with passing rows, block the passed output, or stop execution. Learn more about Guardrails.
Enhancements to the existing Join, Filter, and Select operators
We’ve expanded existing operators to better handle advanced use cases and branching.
Split and multi-join
The Join operator now provides separate outputs for matched, left unmatched, and right unmatched rows by default, allowing you to branch workflows and trigger alerts on unexpected values. It also supports joining more than two tables in a single operator. Learn more about Join.
Dynamic column selection
Select columns by data type, name pattern, or formula, so your workflow adapts as the input schema changes. For example, keep every column whose name starts with sales_, including new matching columns added later. Learn more about Select.
Parameterize inputs and outputs
Source and Output operators now support parameters in their configuration, letting you change these values without reconfiguring the operator. Parameters can be overridden based on individual schedules, such as runs in test and production environments. Learn more about parameters.
Branching in the Filter operator
The Filter operator now has separate outputs for rows that match your conditions and rows that don't. Use both outputs to build separate processing paths from a single filter. Learn more about Filter.
Expanded outputs for materialized views, tables, and files
The Output operator now gives you more ways to save results, update existing data, and create files for downstream use.
Create materialized views
Write your results as a materialized view in Unity Catalog for fast computation in downstream processes like AI/BI Dashboards or even other visual data prep files. The view refreshes automatically when your visual data prep runs. Learn more about materialized view outputs.
Merge into existing tables
Keep an existing table up to date by merging results using the keys you specify. Matching records are updated, and new records are inserted. You can also append results to the end of an existing table. Learn more about table outputs.
Write directly to files
Write CSV, Excel, or JSON files directly to Unity Catalog volumes. For Excel outputs, you can also write dynamically to different worksheets. Learn more about file outputs.
Canvas improvements for large workflow navigation and collaboration
We've updated the canvas to make larger workflows easier to navigate, understand, and maintain.
Navigate larger workflows with a visually redesigned canvas
The canvas has a refreshed design with more visual contrast, larger connection targets, and clearer cues for unconfigured operators and missing inputs. We've also improved scrolling and panning on canvases with 1000s of operators, making large workflows more responsive to navigate.
Find operators with a searchable table of contents
Find any operator using the new searchable table of contents, then select it to jump to its location on the canvas. You can also expand its Generated code section to inspect the underlying code.
Review workflow changes with a visual version history
See what changed between versions directly on the canvas. Version history now highlights operators that were added, removed, or modified, making workflow changes easier to review. Learn more about version history.
Follow collaborators with live cursor presence
Live cursor presence lets you collaborate and follow what coworkers are doing when you're working in the same visual data prep file.
What’s next for Lakeflow Designer?
We are continuing to invest in making Lakeflow Designer the best self-service data preparation and automation tool to work with data. The following are a few upcoming improvements we’re excited about.
Databricks Apps integration
We're integrating Designer with Databricks Apps so authors can share their work with non-technical users through a simple app interface. Users will be able to configure inputs, run the workflow, and view results without viewing the underlying workflow.
Data profiling on the full output
We're expanding Designer's existing data profiling capabilities so you can run a profile across all data returned by an operator and view the results directly in the output table header.
Create custom reports from multiple outputs with the Report operator
A new Report operator will let you combine results from multiple outputs with custom graphics like company logos. You'll be able to save reports directly to the workspace or deliver them via email through a native notification operator.
Native migration tooling with Lakebridge
Native migration tooling powered by Databricks Lakebridge will let you bulk migrate workflows from legacy self-service data preparation tools directly into Lakeflow Designer.
Frequently asked questions about Lakeflow Designer
Is Lakeflow Designer available in Databricks Free Edition?
Yes. Select Visual Data Prep in the left navigation to get started in Free Edition.
How is Lakeflow Designer priced?
There are no per-user licenses or separate licensing fees for Lakeflow Designer. You only pay for the compute you use.
Where can I see the generated code?
You can view the underlying code for any operator by opening the table of contents and expanding Generated code. You can also export the entire file through File > Export to inspect the underlying code.
Does Lakeflow Designer work with Git?
Yes. Place your Lakeflow Designer files in a Databricks Git folder to track changes and sync with your Git provider. Each file contains both the visual layout and the underlying Python code, so both are versioned together.
Can I use Lakeflow Designer for HIPAA workloads?
Yes. Lakeflow Designer supports HIPAA workloads in workspaces configured to meet Databricks HIPAA requirements, including the required compliance controls and an active Business Associate Agreement (BAA). See the HIPAA documentation for requirements and regional availability.
Try it, and tell us what you think
Lakeflow Designer is generally available in all Databricks workspaces. To try it, click + New in the top left of the workspace and select Visual data prep.
To learn more, check out these other resources:
Instagram Engineering
Shopify Engineering