Food For Thought | Stream the Dream

Author: Sjors Otten

Consider a large food manufacturer that produces beef products on a global scale. A typical process of a food manufacturer is depicted below:

This is a simplified production process, which can easily be swapped for any other type of  production process. In most companies all steps in this process require their own system and functionality to store the information of that moment. In this example, a MES system is directly connected to the machines, which are part of the production process, and logs each production step together with its processing- and machine-data. An ERP system is used for administrative tasks and contains processing planning-, sales-, and inventory data. Combining data from these systems (and potential other in- or external data sources) into a generic BI system leads to a batch-driven solution, providing business with valuable actionable insights on a daily basis. A schematic overview of this typical dataflow is represented in the figure below.

However, having numerous information systems, each having a specific purpose and dealing with their respective granularity, timeliness, and data formats, leads to a variety of integration patterns (i.e. file-based, ETL, batch- and event-based-patterns). This results in unwanted complexity and dependencies in an organization’s dataflow. Above all, does it answer all the users’ questions at all organizational levels? It is of the upmost importance to have the ability to act instantly on any production- or health-safety-issue in a food production process. For example, the team lead should be able to act directly whilst management should be informed to adapt strategy in the shortest amount of time possible. Hence, an Enterprise Streaming Platform (ESP) is the solution!

An ESP enables a food manufacturer to have (near) real-time insights and availability of all data within the organization. ESP considers everything to be an “event”. This can be a goods receiving, inventory movement, production registration, invoicing, or financial posting.

An ESP differs from an Eventbus-platform as it persists events and makes it available in a “topic” for every information system in the organization. Looking at an Eventbus-platform, it puts events on a queue and waits until an information system consumes it. Subsequently it deletes the event from the queue. As an ESP persists events, other information systems can also consume events in the exact same order as the preceding information system. Thereby, providing each information system with an identical set of data and dataflow. This gives the ESP a significant advantage compared to an Eventbus-platform.

Plotting an ESP on the food manufacturer layout we can observe the following schematic overview:

By comparing the typical and proposed food manufacturing layout, we have drastically reduced the complexity of the data platform as we switched to a single ESP and thereby one integration pattern. As a result, we moved from a batch-driven environment to an event-based-environment across all levels of the organization. Secondly, we have made all data readily and rapidly available to any information system or team that wants to consume or produce data. This will enable a food manufacturer to become truly data-driven, resulting in a competitive advantage. In addition, an ESP provides teams with the opportunity to create innovative solutions and the ability to apply data science or machine learning techniques on real-time data, leading to new actionable insights, process optimizations, and potentially increased margins.  

When using an ESP it becomes extremely easy to integrate various existing and new information systems through a single integration pattern and platform. Leveraging the ESP-capabilities simplifies things drastically as there are no more custom point-2-point-application-integration patterns required (e.g., SAP with Exact Online; CSB-System with external MES, etc.).

To conclude, enabling everything and anyone at the food manufacturer via a single (near)-real-time ESP will drastically change the way we work with data, but also how we think about data and its untapped potential. At Food For Analytics we are strong believers of everything data and the added value of an ESP, which enables a Scalable, Flexible and Robust real-time data platform. We see it as a must to become data-driven and move towards industry 4.0.

In our next Food For Tech blog-series we will demonstrate the true power of an ESP as we will apply our vision on an actual use case! Please do keep an eye on this site for our next blog.

Interested? Reach out to us directly!