FFA logo wit
FFA logo wit
FFA logo wit

Food For Thought | Stream the dream

HomeFood For Thought | Stream the dream
31 oktober 2023

Consider a large food manufacturer that produces beef products on a global scale. A typical process of a food manufacturer is depicted below:

Production pipeline

This is a simplified production process, which can easily be swapped for any other type of production process. Most companies allocate specific systems and functionalities for each step in this process to store information effectively at that moment. In this example, a MES system is directly connected to the machines, which are part of the production process, and logs each production step together with its processing- and machine-data. An ERP system is used for administrative tasks and contains processing planning-, sales-, and inventory data. Combining data from these systems (and potential other in- or external data sources) into a generic BI system leads to a batch-driven solution, providing business with valuable actionable insights on a daily basis. A schematic overview of this typical dataflow is represented in the figure below.

Stream platform 1

However, having numerous information systems, each having a specific purpose and dealing with their respective granularity, timeliness, and data formats, leads to a variety of integration patterns (i.e. file-based, ETL, batch- and event-based-patterns). This results in unwanted complexity and dependencies in an organization’s dataflow. Above all, does it answer all the users’ questions at all organizational levels? Instant action on production or health-safety issues in food production is of utmost importance for effective control and response. For example, the team lead should be able to act directly whilst management should be informed to adapt strategy in the shortest amount of time possible. Hence, an Enterprise Streaming Platform (ESP) is the solution!

An ESP enables a food manufacturer to have (near) real-time insights and availability of all data within the organization. ESP considers everything to be an “event”. This can be a goods receiving, inventory movement, production registration, invoicing, or financial posting.

An ESP differs from an Eventbus-platform by persisting events and making them available in a "topic" for every information system. Looking at an Eventbus-platform, it puts events on a queue and waits until an information system consumes it. Subsequently it deletes the event from the queue.

An ESP persisting events allows other systems to consume events in the exact order as the preceding information system. Thereby, providing each information system with an identical set of data and dataflow. This gives the ESP a significant advantage compared to an Eventbus-platform.

Plotting an ESP on the food manufacturer layout we can observe the following schematic overview:

Stream platform 2

Comparing layouts, switching to a single ESP reduced the data platform's complexity, streamlining integration to one pattern in food manufacturing. As a result, we moved from a batch-driven environment to an event-based-environment across all levels of the organization. Secondly, we have made all data readily and rapidly available to any information system or team that wants to consume or produce data. This will enable a food manufacturer to become truly data-driven, resulting in a competitive advantage. In addition, an ESP provides teams with the opportunity to create innovative solutions and the ability to apply data science or machine learning techniques on real-time data, leading to new actionable insights, process optimizations, and potentially increased margins.  

When using an ESP it becomes extremely easy to integrate various existing and new information systems through a single integration pattern and platform. Leveraging the ESP-capabilities simplifies things drastically as there are no more custom point-2-point-application-integration patterns required (e.g., SAP with Exact Online; CSB-System with external MES, etc.).

Enabling everyone via a single (near)-real-time ESP transforms how we work and think about data, unlocking its untapped potential. At Food For Analytics, we believe in the value of an ESP—a Scalable, Flexible, and Robust real-time data platform. We see it as a must to become data-driven and move towards industry 4.0.

Interested? Reach out to us directly!

Sjors Otten

“Insights without action is worthless”

Sjors Otten is a pragmatic and passionate data & analytics architect. He excels in leveraging the untapped potential of your organization’s data. Sjors has a solid background in Business Informatics and Software Development.

With his years of experience on all levels of IT, Sjors is the go-to-person for breaking down business- and IT-strategies in workable and understandable data & analytics solutions for all levels within your organization whilst maintaining alignment with the defined corporate strategies.

Related blogs

Food For Thought | Cloud vs. On-premise

Navigating tech choices and the cloud vs. on-premise decision in our data-driven world.

Read more
Food For Thought | Architectural guidelines

Discover why it's essential for clarity, cost-effectiveness, and a unified data vision strategy.

Read more
Food For Thought | Stream the dream

Discover how Food For Analytics is revolutionizing food manufacturing with data insights.

Read more

Ready to become a titan in the food industry?

Get your own Titan
Food For Analytics © 2023 All rights reserved
Chamber of Commerce 73586218
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram