ERP, MES and WMS integration with Databricks for food manufacturing
Choose CDC, incremental ingestion, files or event streaming from how the source changes and how fresh the decision needs to be. Then contextualize ERP, MES and WMS data into one governed manufacturing model.
ERP database
orders · master · transactions
MES database
runs · output · status
WMS
stock · lots · movements
Historian files
CSV · JSON · Parquet
OT events
state · alarms · counters
Snapshots
legacy full exports
Choose the pattern by source behaviour
The system name does not determine the ingestion architecture.
Use the simplest replayable pattern that meets the business freshness requirement and protects the operational source.
The short answer
Do not choose an ERP, MES or WMS ingestion pattern from the application name. Choose it from how the data changes: CDC for mutable database state, incremental queries where CDC is unavailable, Auto Loader for arriving files and event streaming only where low latency changes the decision.
Land each source independently first. Then align identity, time, grain, units and status in Databricks so reporting, analytics and AI reuse the same manufacturing context.
Integration problem
The connector is rarely the hardest part
Most integration risk comes from choosing the wrong ingestion behaviour, putting too much load on the source or losing the history needed to reconcile systems later.
One system, several behaviours
A MES can expose mutable tables, historian files and a live event stream at the same time.
Freshness is often over-specified
Historical yield does not need the same architecture as a live machine-event decision.
The source sets the ceiling
Concurrency, fetch size, query shape and network throughput can matter before Databricks compute does.
Why food is different
Food manufacturing context has to survive the integration
A technically correct pipeline is not enough if batch, shelf-life, quality or temporal relationships disappear on the way into the analytical model.
Batch & lot genealogy
Orders, production runs and warehouse movements create many-to-many batch relationships.
Shelf life
Expiry and remaining shelf life have to stay linked to stock, demand and production timing.
Quality state
Blocked, released, rejected and rework states change the meaning of available stock and output.
Temporal reality
Event time, posting time and late arrival can place the same record in different operational windows.
Databricks ingestion patterns
Match the pattern to how the source changes
Start with the change mechanism and the required freshness. The same ERP, MES or WMS landscape can use several patterns side by side.
- Change Tracking / CDC for mutable database rows where updates and deletes matter.
- Query-based incremental ingestion where CDC is unavailable but a reliable cursor exists.
- Auto Loader for historian, middleware and equipment files arriving in cloud storage.
- Event Hubs / Kafka for genuine event streams where seconds-to-minutes freshness changes the decision.
Choose the simplest pattern that meets the required freshness.
Latency and reliability
Streaming is only one part of a production-grade design
Choose the lowest latency that changes the decision, then make event time, replay and source protection explicit.
Hourly / daily
Planning, yield, margin and historical traceability.
5–30 minutes
Production progress, material readiness and stock position.
Seconds–minutes
Active run state, alarms and latency-sensitive event decisions.
Machine control
Deterministic safety and control stay at PLC, SCADA or edge level.
Event time
Track when it happened, not only when the record arrived.
Replay
Keep checkpoints and source-aligned history so pipelines can recover safely.
Data quality
Apply expectations and quarantine bad records where appropriate.
Source load
Benchmark fetch size, query shape, partitioning and concurrency.
Practical workflow
Implement the integration in five practical steps
Start with one cross-system decision. Classify the required sources by behaviour, ingest them independently and only then build the shared manufacturing context.
Choose
The decision and freshness.
Ingest
Each source safely.
Contextualize
Identity, time and grain.
Select the lowest latency that actually changes the business decision.
Preserve source keys, event timestamps and source history before cross-system joins.
Publish reusable production, yield, inventory and traceability models instead of rebuilding joins in every report.
From a business decision to governed manufacturing data.
Select one decision
Define the required facts and freshness before choosing technology.
Classify source behaviour
CDC, incremental table, arriving files, event stream or snapshot.
Land sources independently
Protect operational systems and preserve replayable source history.
Contextualize the process
Relate ERP order, MES run and WMS lot by identity, time and grain.
Publish and activate
Serve Power BI, SQL and Ask Titan from the same governed data products.
Production guardrails
Make the pipeline safe before you make it fast
A production integration needs more than throughput. Source protection, replay, late-data handling, observability and ownership are part of the architecture.
Protect sources
Dedicated identity, least privilege and measured concurrency.
Replay safely
Keep checkpoints, source history and idempotent processing.
Handle bad data
Use expectations and quarantine where records need review.
Handle late data
Define event-time, watermark and late-arrival policies explicitly.
Operate it
Monitor lag, update duration, failures and deployments like a product.
Common mistakes
Five integration anti-patterns to avoid
Most integration problems come from overengineering the ingestion path or losing the source truth needed later.
Avoid
Streaming the whole factory
Better default
Stream only where latency changes the decision
Avoid
One connector per application
Better default
Choose the pattern per dataset behaviour
Avoid
Joining sources during ingestion
Better default
Land first, contextualize afterwards
Avoid
Scaling Databricks first
Better default
Benchmark the operational source first
Avoid
Ignoring replay and corrections
Better default
Preserve enough history to rebuild state
Good integration is usually boring: preserve the source, choose the simplest reliable ingestion pattern, and add complexity only when the decision requires it.
How Food For Analytics implements it
Titan turns manufacturing data into plant optimizations
Built on Azure Databricks, Titan connects operational systems, creates shared manufacturing context and publishes reusable data products.
Operational systems
ERP
orders · recipes · cost
MES / OT
runs · output · events
WMS
stock · lots · movements
Other sources
quality · planning · files
Titan
Governed manufacturing context
Built on Azure Databricks
Connect
CDC · files · events
Govern
identity · time · definitions
Decide
trusted data for daily decisions
Lakeflow · Delta Lake · Unity Catalog
Data products
Production performance
Yield & giveaway
Inventory & shelf life
Traceability
Reusable across Power BI, SQL, analytics and Ask Titan.
Ask Titan example
The integration becomes useful when people can use the context
Once ERP, MES and WMS facts are contextualized in Titan, Ask Titan can answer business questions in Microsoft Teams using the same governed definitions, filters and access rights.
Ask a cross-system question
Ask about production, stock, yield or another governed business concept.
Use governed evidence
Answers use the Titan model instead of rebuilding ERP, MES and WMS joins in the chat layer.
Keep definitions and access consistent
Filters, definitions and access rights remain part of the governed model.
Order 4711 released 10,580 kg against a 12,000 kg plan.
The largest explainable loss was a 45-minute material wait on Line 4 during MES run R9834. WMS movement history shows the required lot was issued after the planned staging window.
Evidence: ERP order · MES run/events · WMS lot movement · governed Titan definitions.
Example only. Ask Titan answers questions from governed Titan data; it does not replace the underlying data model.
FAQ
ERP, MES and WMS integration questions
Practical answers about ingestion patterns, latency and governed manufacturing context on Azure Databricks.
Should MES data always be streamed into Databricks?
No. A MES can expose database tables, files and events. Use CDC or incremental ingestion for database state, Auto Loader for arriving files and continuous streaming only where low latency changes the decision.
When should we use CDC or Change Tracking?
Use a managed change-feed pattern where mutable database rows need updates and deletes preserved. For supported SQL Server tables with a primary key, Change Tracking can reduce source overhead compared with full CDC.
What if the source has no CDC capability?
Use query-based incremental ingestion when a reliable monotonic cursor such as a timestamp or sequence exists. If only periodic full snapshots are available, snapshot-based change processing can be used instead.
Should ERP, MES and WMS be joined during ingestion?
Usually not. Land each source independently first so source keys, timestamps and history remain reproducible. Build identity, temporal and grain relationships in the governed contextual layer afterwards.
How fresh should manufacturing data be?
Start from the business decision. Historical yield or margin can be hourly or daily, operational progress may need 5 to 30 minute freshness, and genuine event decisions may require seconds-to-minutes streaming. Machine control and safety normally remain at PLC, SCADA or edge level.
How do late machine events affect the model?
Keep event time separate from source-recorded and ingestion timestamps. Stateful streaming designs should define lateness and watermark rules so delayed events do not silently distort run, shift or production-order context.
How does Titan use Azure Databricks for these integrations?
Titan uses Azure Databricks as its data and AI foundation to ingest source data, contextualize ERP, MES and WMS relationships, govern definitions and publish reusable data products for Power BI, SQL, analytics and Ask Titan.
How does Ask Titan use the integrated data?
Ask Titan answers user questions from the governed Titan data model. It uses the same definitions, filters and access rights rather than rebuilding ERP, MES and WMS joins in the conversational layer.
Next step
Start with one integration decision
Pick one decision that needs ERP, MES or WMS data, define the required freshness and then choose the smallest reliable Databricks ingestion pattern around it.
1. Pick the decision
What needs to become faster or more reliable?
2. Classify the sources
CDC, cursor, files, events or snapshots.
3. Contextualize
Identity, time, grain and business definitions.
4. Activate it
Power BI, SQL, analytics and Ask Titan.