Food For Thought | Stream the dream

Consider a large food manufacturer that produces beef products on a global scale. A typical process of a food manufacturer is depicted below:

This is a simplified production process, which can easily be swapped for any other type of production process. Most companies allocate specific systems and functionalities for each step in this process to store information effectively at that moment. In this example, a MES system is directly connected to the machines, which are part of the production process, and logs each production step together with its processing- and machine-data. An ERP system is used for administrative tasks and contains processing planning-, sales-, and inventory data. Combining data from these systems (and potential other in- or external data sources) into a generic BI system leads to a batch-driven solution, providing business with valuable actionable insights on a daily basis. A schematic overview of this typical dataflow is represented in the figure below.

However, having numerous information systems, each having a specific purpose and dealing with their respective granularity, timeliness, and data formats, leads to a variety of integration patterns (i.e. file-based, ETL, batch- and event-based-patterns). This results in unwanted complexity and dependencies in an organization’s dataflow. Above all, does it answer all the users’ questions at all organizational levels? Instant action on production or health-safety issues in food production is of utmost importance for effective control and response. For example, the team lead should be able to act directly whilst management should be informed to adapt strategy in the shortest amount of time possible. Hence, an Enterprise Streaming Platform (ESP) is the solution!

An ESP enables a food manufacturer to have (near) real-time insights and availability of all data within the organization. ESP considers everything to be an “event”. This can be a goods receiving, inventory movement, production registration, invoicing, or financial posting.

An ESP differs from an Eventbus-platform by persisting events and making them available in a "topic" for every information system. Looking at an Eventbus-platform, it puts events on a queue and waits until an information system consumes it. Subsequently it deletes the event from the queue.

An ESP persisting events allows other systems to consume events in the exact order as the preceding information system. Thereby, providing each information system with an identical set of data and dataflow. This gives the ESP a significant advantage compared to an Eventbus-platform.

Plotting an ESP on the food manufacturer layout we can observe the following schematic overview:

Comparing layouts, switching to a single ESP reduced the data platform's complexity, streamlining integration to one pattern in food manufacturing. As a result, we moved from a batch-driven environment to an event-based-environment across all levels of the organization. Secondly, we have made all data readily and rapidly available to any information system or team that wants to consume or produce data. This will enable a food manufacturer to become truly data-driven, resulting in a competitive advantage. In addition, an ESP provides teams with the opportunity to create innovative solutions and the ability to apply data science or machine learning techniques on real-time data, leading to new actionable insights, process optimizations, and potentially increased margins.  

When using an ESP it becomes extremely easy to integrate various existing and new information systems through a single integration pattern and platform. Leveraging the ESP-capabilities simplifies things drastically as there are no more custom point-2-point-application-integration patterns required (e.g., SAP with Exact Online; CSB-System with external MES, etc.).

Enabling everyone via a single (near)-real-time ESP transforms how we work and think about data, unlocking its untapped potential. At Food For Analytics, we believe in the value of an ESP—a Scalable, Flexible, and Robust real-time data platform. We see it as a must to become data-driven and move towards industry 4.0.

Interested? Reach out to us directly!