Skip to main content
Pillar guide · Azure Databricks + food manufacturing

Databricks for Food Manufacturing from fragmented factory data to trusted production decisions

Connect ERP, MES / OT and WMS on Azure Databricks, preserve food-specific batch, recipe, quality and shelf-life context, and publish governed data products for production, yield, inventory and AI.

See the reference architecture
ERP + MES / OT + WMS Batch · recipe · quality · shelf life
Food manufacturing data to decisions

ERP

MES / OT

WMS

Titan

Food manufacturing context on Azure Databricks

Production

Yield

Inventory

Reference architecture

Turn ERP, MES and WMS data into food-manufacturing decisions on Databricks

Food manufacturing is primarily a context problem. Orders, line events, stock records and quality results can all be correct while describing different parts of the same physical production process.

A useful Databricks architecture therefore preserves source truth, then connects order, line, batch, SKU, recipe, shift and time before exposing business-facing data products. Copying every source table into cloud storage creates a larger data estate; contextualization creates a shared operating model.

A typical end-to-end food manufacturing data path

Operational systems → contextualize in Titan → governed production decision

Operational systems

ERP

Orders · recipes · product

MES / OT

Events · stops · output

WMS

Stock · batches · expiry

Contextualize in Titan

One governed food manufacturing language

Built on Azure Databricks

Match · order, line, batch, SKU, recipe
Align · time, units, quality and process context
Standardize · output, losses, yield and shelf life
Delta Lake · Unity Catalog · Lakeflow

Governed manufacturing data product

Production order performance

Plan attainment

88.2%

Largest loss

45 min wait

Microsoft TeamsAsk Titan
Why did production order PO-260827-184 miss plan?

Main driver: packaging material availability

The order achieved 88.2% of plan. The largest loss was 45 min WAIT_MATERIAL, matching the TRAY-500 stock-out until 07:40.

Uses governed Titan definitions and access rules.

Why food manufacturing is different

Food decisions depend on context generic manufacturing models often miss

Two pallets with the same SKU and quantity are not necessarily interchangeable. Batch identity, expiry, quality release, customer shelf-life requirements, recipe version and production sequence can all change whether stock is usable or an order is feasible.

Physical product context

Batch / lot genealogy

Supplier lot → material issue → intermediate batch → finished batch → shipment.

Recipe version

Effective-dated material and process specifications need to match the actual production run.

Handling unit & quality state

Pallet, tote or case identity matters when physical stock exists but remains blocked or unreleased.

Decision context

Shelf life & FEFO

Expiry date, quality release and customer minimum remaining life determine commercial availability.

Allergen & sequence constraints

The next feasible SKU can depend on the previous run, cleaning requirement, packaging and allergen transition.

Yield, rework & giveaway

Nominal recipe, actual consumption, good output, rework and waste connect operations to margin.

A useful governed relationship

Product Recipe version Production order Batch / lot Quality state Customer / shelf-life eligibility

Where it creates value

Three high-value food manufacturing data products on Databricks

Start where several systems and food-specific rules must be combined for one recurring decision. Build the context once, then reuse the governed product across analytics and AI.

Production performance

Combine ERP plan, MES states, target rate, packaging/material availability and released output to explain plan attainment, OEE and the losses behind a missed production target.

ERP · MES / OT · WMS · Quality

Yield & giveaway

Relate recipe targets, actual material consumption, good output, rework, waste and quality results to product, batch and line context so yield variance becomes explainable.

ERP · MES · WMS · Quality

Inventory & shelf life

Combine batch stock, expiry, quality status, FEFO, demand and customer shelf-life rules to expose ageing, expiry risk and material readiness before they become operational losses.

ERP · WMS · Quality · Planning

The same governed Databricks foundation can support production planning, batch traceability, maintenance analytics, margin visibility and customer-service decisions without rebuilding the source context in every report.

Implementation guidance

How we would ingest ERP, MES / OT / IoT and WMS data into Databricks

Do not force every food-manufacturing source through the same connector. Choose the ingestion pattern from how the source changes, the latency the decision needs, the load the source can tolerate and whether you need recoverable history in Delta.

ERP

Preserve transactional history without stressing the ERP

Orders, recipes/BOMs, products, purchasing, suppliers and customer rules usually need durable incremental history rather than repeated full-table extracts.

Recommended

Use Lakeflow Connect managed CDC when the database and network pattern are supported.

When CDC is not available

Use query-based incremental ingestion when a reliable cursor exists. For legacy platforms, controlled JDBC/ODBC extraction to ADLS can remain the pragmatic bridge.

Food context to retain

Product, recipe version, order, customer, supplier and original source identifiers.

MES / OT / IoT

Separate execution records from high-frequency telemetry

MES events and OT signals are different workloads. Production events are business records; telemetry is a stream that should only be retained at the granularity the decision requires.

MES database

Use CDC or query-based ingestion for downtime, reason codes, output, consumption and production-order events stored relationally.

Historian / files

Land repeatable exports in ADLS and use Auto Loader with Lakeflow pipelines for incremental file discovery and schema handling.

Live events

Use Kafka or Azure Event Hubs through its Kafka-compatible endpoint when event-level freshness is genuinely required. Aggregate or filter high-frequency signals upstream when possible.

WMS

Preserve stock movement, lot, expiry and quality-state history

Current on-hand stock is not enough for FEFO, expiry risk or material readiness. You need the events that explain when a batch became available, blocked, released, moved or consumed.

Recommended

Use managed CDC for a supported relational WMS, or query-based incremental ingestion when the source exposes a reliable cursor or modification timestamp.

Files / SaaS

Use ADLS + Auto Loader for exports. For SaaS WMS platforms, use a supported managed connector or land API/export data before contextualization.

Food context to retain

Warehouse, location, handling unit, lot/batch, expiry, quality status and movement timestamps.

Technical decision guide

Compare Databricks ingestion patterns

Open the guide when you need to choose between CDC, query-based ingestion, files, streaming or federation.
Source behaviourPreferred Databricks patternUse it when
Transactional DB with supported CDCLakeflow Connect managed CDCYou need reliable incremental history and want change capture managed for a supported source.
Database with a reliable cursorLakeflow query-based ingestionScheduled incremental extraction is sufficient and the source can tolerate direct queries.
Files in ADLSAuto Loader + Lakeflow pipelinesLegacy systems, historians or integrations deliver repeatable file drops.
Continuous event streamKafka / Event Hubs + streamingThe operational decision genuinely needs event-level or near-real-time freshness.
Reference / exploratory accessLakehouse FederationIn-place access is useful and dependency on the remote system is acceptable.

Connector coverage, feature maturity and network requirements vary by source and workspace. Validate the source-specific Databricks support level before locking a production design.

Practical advice

Four decisions prevent most food-manufacturing ingestion problems

Choose freshness from the decision

Daily yield does not need millisecond streaming. A live line-stop use case might. Do not pay for latency nobody acts on.

Benchmark the source before scaling Databricks

Source concurrency, fetch size, predicates, network throughput and transaction logging can bottleneck before Databricks compute does.

Do not stream the entire factory

Retain high-frequency OT / IoT only where it supports a defined analytic or operational use case; aggregate or filter at the edge or historian when possible.

Govern connections and food definitions together

Unity Catalog governs access and lineage; the business must still own definitions for good output, quality release, batch availability, FEFO, shelf life, yield and giveaway.

FAQ

Frequently asked questions

Practical answers about Databricks for food manufacturing.

Why is food manufacturing different on Databricks?

Food decisions depend on context that generic manufacturing models often miss: batch and lot genealogy, recipe versions, quality release, expiry, customer shelf-life rules, FEFO and allergen or sequence constraints. These relationships should be modeled explicitly rather than recreated separately in reports.

What is Databricks used for in food manufacturing?

Food manufacturers use Databricks to ingest, contextualize and govern data from ERP, MES / OT, WMS, quality and selected IoT sources, then publish reusable data products for production performance, yield, inventory, traceability, planning, analytics and AI.

Does Databricks replace an ERP, MES or WMS?

No. ERP, MES and WMS remain systems of record and execution. Databricks provides the governed data and AI layer that connects their data across orders, lines, batches, recipes, shifts and time.

How should ERP, MES / OT / IoT and WMS data be ingested differently?

Use managed CDC where supported for transactional databases, query-based incremental ingestion when a reliable cursor exists, Auto Loader for repeatable files, and Kafka or Azure Event Hubs for continuous events. Choose the pattern from freshness, recoverability and source load rather than forcing every system through one connector.

Does every machine or IoT signal need to be ingested?

No. Retain high-frequency telemetry only when it changes an operational or analytical decision. Many use cases need line events, stop reasons or aggregated process values rather than every raw millisecond signal.

How should batch and lot genealogy be modeled?

Preserve original source identifiers and explicit parent-child relationships from supplier lot through material issue, production order, intermediate and finished batch, handling unit and customer shipment. Keep event time and quality state so the genealogy remains auditable.

How do shelf life and FEFO fit into the Databricks model?

Model expiry, quality status, stock movements, customer minimum remaining life and demand together. A batch can exist physically while being unavailable for a specific customer or production decision, so availability must be derived from governed rules rather than quantity alone.

Where does Unity Catalog fit in a food manufacturing architecture?

Unity Catalog governs catalogs, schemas, tables, views, permissions and lineage. It provides the technical governance boundary, while the business still owns definitions such as good output, quality release, FEFO, yield, giveaway and shelf-life availability.

Can Power BI and Ask Titan use the same Databricks data products?

Yes. A governed data product can expose the same business definitions to Power BI, SQL and question-driven AI experiences such as Ask Titan, reducing the need to rebuild KPI and access logic in each consumption layer.

Practical next step

Map your first food-manufacturing data product on Databricks

Bring one recurring production, yield, stock or planning decision and your ERP, MES and WMS landscape. We map the minimum source scope, food-specific context and Azure Databricks architecture needed to support it.

Bring one decision
For example: why did this order miss plan, or which batches are at expiry risk?
Map the minimum source scope
ERP, MES / OT, WMS and quality only where the decision needs them.
Leave with an architecture direction
Ingestion, food context, Unity Catalog and the first reusable data product.