Delta Live Tables vs Lakeflow Pipelines what changed and how to build today
Delta Live Tables is now Lakeflow pipelines. Existing DLT code still works, while current APIs make dataset types, CDC patterns and pipeline intent more explicit.
The declarative model remains
Dependencies, managed refresh, expectations and event-log operations continue. Modernize the API surface when it adds clarity or future compatibility.
Short answer
Lakeflow pipelines is the current name and the next API surface for DLT
The product was renamed, but the declarative pipeline model did not disappear. Existing DLT code can keep running. New code should use current Lakeflow terminology and the pyspark.pipelines API.
No mandatory code migration
The rename itself does not require you to rebuild working DLT pipelines.
Modern APIs are clearer
Streaming tables and materialized views now have distinct Python decorators.
Lakeflow adds beyond open Spark SDP
Databricks adds AUTO CDC, expectations, event logs, continuous mode and other production features.
What stays
What changes
pyspark.pipelinesTreat the rename, API modernization and legacy publishing mode as separate topics. They do not all require the same migration.
API map
Map the old DLT surface to the current Lakeflow API
The important migration detail is dataset intent. Older @dlt.table code represented both batch and streaming outputs. Current Python makes that distinction explicit.
import dlt
from pyspark import pipelines as dp@dlt.table + streaming DataFrame
@dp.table@dlt.table + batch DataFrame
@dp.materialized_view@dlt.view
@dp.temporary_viewdlt.apply_changes(...)
dp.create_auto_cdc_flow(...)APPLY CHANGES INTO
AUTO CDC INTOStart with how the output must stay correct.
What does the target need?
Process each new row once
Append-heavy ingestion or streaming transformations.
Streaming table
Keep a query result current
Joins, aggregates and derived analytical state.
Materialized view
Reuse logic inside the pipeline
Intermediate logic that does not need a published dataset.
Temporary view
CDC is a flow decision: use AUTO CDC to apply ordered inserts, updates and deletes into a streaming table.
Dataset choice
Do not choose between streaming tables and materialized views by layer name
A Bronze table is often streaming, while Silver and Gold can contain either type. The correct choice depends on source behaviour, transformation semantics and how historical changes must be reflected.
Streaming tables are inherently incremental
Normal refreshes process records that arrived since the previous update. Older rows are not automatically reprocessed.
Materialized views preserve query correctness
Databricks can refresh incrementally when possible and fall back to a full recompute when required.
Single SQL object can be standalone
Use a full Lakeflow pipeline when you need multi-stage dependencies, Python, sinks or pipeline-only CDC patterns.
Current syntax
Use current dataset names in SQL and Python
SQL uses explicit streaming-table and materialized-view DDL. Python now uses separate decorators so the dataset type is visible in code.
CREATE OR REFRESH STREAMING TABLE bronze_mes_events
AS SELECT *
FROM STREAM read_files(
'/Volumes/factory/raw/mes_events',
format => 'json'
);
CREATE OR REFRESH MATERIALIZED VIEW gold_daily_output
AS
SELECT
production_date,
line_id,
SUM(good_quantity) AS good_quantity
FROM silver.production_runs
GROUP BY production_date, line_id;from pyspark import pipelines as dp
@dp.table(name="bronze_mes_events")
def bronze_mes_events():
return (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/Volumes/factory/raw/mes_events")
)
@dp.materialized_view(name="gold_daily_output")
def gold_daily_output():
return (
spark.read.table("silver.production_runs")
.groupBy("production_date", "line_id")
.sum("good_quantity")
)The old CREATE OR REFRESH LIVE TABLE form is deprecated. For new SQL, use CREATE OR REFRESH MATERIALIZED VIEW or CREATE OR REFRESH STREAMING TABLE.
Source behaviour
Choose the ingestion pattern from how the source changes
Lakeflow pipelines can handle files, event streams, CDC feeds and snapshots, but those sources should not be forced through the same pattern.
Supported database or SaaS source
managed ingestion
Pattern
Lakeflow Connect
Target
managed streaming tables
ADLS files or append-only events
new records arrive
Pattern
Auto Loader / streaming read
Target
streaming table
Ordered inserts, updates and deletes
CDC feed
Pattern
AUTO CDC
Target
streaming table · SCD1 / SCD2
Periodic full snapshots only
no native CDC
Pattern
AUTO CDC FROM SNAPSHOT
Target
streaming table · Python only
Lakeflow Connect and Lakeflow pipelines solve different parts of the stack. Managed connectors add source-specific authentication, CDC handling, retries and schema evolution. Lakeflow pipelines provide the declarative transformation and processing framework underneath and downstream.
Execution mode
Streaming table does not mean continuous compute
Dataset type and execution mode are independent. Start with triggered execution for most manufacturing workloads and increase freshness only when a faster decision actually changes the outcome.
Triggered is the default
Refresh available data, then stop compute. This fits most hourly, daily and on-demand pipelines.
Continuous is for proven latency needs
For new continuous workloads, Databricks recommends the continuous Lakeflow Job pattern over the built-in continuous pipeline setting.
Real-time is specialized
Millisecond-range real-time mode is currently a Public Preview feature and is aimed at operational streaming use cases.
Freshness ladder
Pick the lowest latency that the business decision needs.
Planning, inventory, finance, daily production performance
Near-real-time operational monitoring where latency changes the response
Specialized event-driven operational workloads
Compute: Databricks recommends serverless for almost all new Lakeflow pipelines. Use classic where network, instance or environment requirements demand it.
Bronze events
streaming
Silver valid
2 expectations
Gold output
materialized
Passed
99.7%
Dropped
186
Backlog
0
Event log
update status · quality metrics · lineage · errors · resource events
Illustrative pipeline metrics. Use the actual event log and pipeline UI as the operational source.
Quality & observability
Preserve behaviour, not just syntax
A DLT modernization is only equivalent when data-quality actions, refresh behaviour and monitoring still produce the same operational result.
Expectations remain a Lakeflow feature
Retain invalid rows for metrics, drop them, fail the update or implement a quarantine path when investigation matters.
The event log is the programmatic record
Use it for update outcomes, expectation trends, errors, lineage and historical operational analysis.
Streaming metrics expose backlog
For streaming flows, monitor backlog records, bytes, files and seconds where available.
Migration
Separate API modernization from publishing-mode migration
The DLT rename does not force a migration. Legacy publishing mode is different. Pipelines created before February 5, 2025 can still use the LIVE virtual schema and should be reviewed for migration to the current default publishing mode.
DLT API modernization
Update names and APIs as you touch the codebase. Validate behaviour instead of performing a blind rewrite.
Inventory DLT imports, decorators, APPLY CHANGES and old terminology.
Classify each dataset as streaming table, materialized view or temporary view.
Refactor only where current APIs improve clarity or future compatibility.
Reconcile outputs, expectations, latency and restart behaviour before promotion.
Default publishing mode
This changes name resolution and pipeline metadata. It is separate from replacing import dlt.
Legacy
LIVE virtual schema
legacy publishing mode
Current
Catalog + schema
default publishing mode
Review unqualified reads that previously resolved through workspace defaults.
Use fully qualified identifiers for datasets outside the configured pipeline catalog and schema.
The publishing-mode migration updates pipeline metadata. It does not move or rewrite the underlying datasets.
Common mistakes
Avoid migrations that change semantics by accident
The risky changes are rarely the product name. They are incorrect assumptions about dataset type, source behaviour, execution mode or name resolution.
Avoid
Rewriting a working DLT pipeline only because the name changed
Better default
Modernize APIs when the code is already being changed
Avoid
Replacing every @dlt.table with @dp.table
Better default
Classify batch outputs as materialized views first
Avoid
Assuming a streaming table needs continuous mode
Better default
Choose dataset type and execution mode independently
Avoid
Streaming from a source that updates old rows without a plan
Better default
Use AUTO CDC or another pattern that matches the source semantics
Avoid
Using APPLY CHANGES in new examples
Better default
Use AUTO CDC for current Lakeflow code
Avoid
Ignoring LIVE and unqualified names in legacy publishing mode
Better default
Review name resolution when moving to default publishing mode
How Food For Analytics implements it
Titan uses Lakeflow patterns according to source behaviour and decision freshness
We do not default every manufacturing dataset to the same pipeline object. Titan separates ingestion behaviour, transformation semantics and execution mode so each data product can use the simplest reliable pattern.
Source behaviour
Classify the change pattern
Files, events, CDC feeds and snapshots require different ingestion semantics.
Titan on Azure Databricks
Choose the Lakeflow pattern
Use the dataset and flow type that preserves correctness with the lowest operational complexity.
Operational outcome
Run at the required freshness
Triggered by default, continuously where latency matters, with expectations and event-log monitoring around the data product.
FAQ
Frequently asked questions
Practical answers about moving from Delta Live Tables terminology to current Lakeflow pipelines.
Is Delta Live Tables discontinued?
Delta Live Tables is the former product name. Databricks now calls the product Lakeflow pipelines. Existing DLT code continues to work, so the rename does not require a rewrite.
Do I need to replace import dlt immediately?
No. The dlt Python APIs remain supported, but Databricks recommends using from pyspark import pipelines as dp for new code and when existing code is being modernized.
What replaces @dlt.table?
The replacement depends on the dataset. Use @dp.table for a streaming table and @dp.materialized_view for a materialized view. This makes the intended dataset type explicit in current Python code.
What replaces APPLY CHANGES?
Databricks recommends AUTO CDC. AUTO CDC handles ordered change feeds and supports SCD Type 1 and Type 2. AUTO CDC FROM SNAPSHOT is available in Python when only periodic source snapshots are available.
Does a streaming table require a continuous pipeline?
No. Dataset type and pipeline execution mode are separate decisions. Streaming tables and materialized views can both run in triggered or continuous Lakeflow pipelines.
What happened to the LIVE schema?
The LIVE virtual schema belongs to legacy publishing mode. New pipelines use the default publishing mode, where LIVE is ignored and the configured catalog and schema determine unqualified references. Legacy publishing mode should be reviewed separately from the DLT naming change.
Practical next step
Modernize Lakeflow pipelines without rewriting what already works
We can review an existing DLT codebase, source semantics, publishing mode and operational behaviour, then define the smallest safe modernization path.