Food For Tech | Robust and scalable data solution | part 2

Author: Benito van Breugel

Continuing our Data & Analytics journey we will make a stop at the platform side. We will describe the Robust and Scalable data solution which Food For Analytics (FFA) provides, from a platform-perspective. Infrastructure- and platform-components, required to run a Data & Analytics solution, are a costly endeavor for an organization, especially if storage-, compute-, and data-capability-requirements are rapidly growing.

To be able to run your Scalable, Flexible and Robust Data & Analytics solution, as described in the previous blog, you’ll need a platform that is build according to the same principles. For this, Food For Analytics provides a full-fledged data-platform that can be deployed in either Amazon Web Services and Microsoft Azure, or any other cloud for that matter. In other words, it's cloud agnostic. Our platform consists of a fully workable stack comprising: Spark, Kafka, Airflow, different RDBMS (MySql, SQL Server, PostgresSQL), Druid, Zeppelin, and above all, integration with Amazon Web Services: S3 and Microsoft Azure blob storage. Allowing you to scale both storage- and compute-capacity independently from each other. The platform-components run in docker-containers and are orchestrated by Kubernetes, including required security and reliability. Sounds cool right? But why would you need such a platform? And what are the advantages? Let’s dive in right away!

Running the FFA-platform stack in docker-containers on Kubernetes significantly aids in increasing the scalability and flexibility of your Data & Analytics solution. Since the FFA platform is cloud-agnostic it can be deployed to various cloud vendors. As it is built in docker-containers, it can run anywhere! With this, we achieve full independence, and guarantee the platform’s functionality regardless of the underlying infrastructure. Let’s say, if you decide to change your cloud provider for your data platform from Amazon Web Services to Microsoft Azure, we can easily pick up the FFA-platform (with each solution running on it) and migrate it to Azure, whilst spinning down the old AWS instance. With proper preparation this can be done in days rather than weeks in comparison to migrating your on-premise data center.

Another advantage of using docker-containers is that they use less resources. Once effectively used and setup, this will drastically reduce platform costs, increase flexibility, and scalability. Hence, you will only pay for using resources of the cloud provider, during the actual data-processing. Besides that, you can schedule up and down times and increase or decrease the  platform’s capacity at any time, catering to your organization’s current and future needs!

In comparison to buying and setting up on-premise hardware, our platform is flexible, and above all focusses on speed and efficiency. Combined with the orchestration-capabilities of Kubernetes you can start, create, replicate, or destroy containers in seconds. This will make new software releases or version updates quick and easy.

In addition to the platform-benefits, docker-containers also enable developers, such as Data Engineers, to improve their productivity and use continuous integration and continuous delivery (CI/CD). Having your code for data-pipelines being pushed to a central repository, enables us to trigger a CI/CD pipeline for automated validation, testing and deployment to any desired environment. This ensures that all required components, like spark and airflow, contain the data-pipeline-code so that it is ready to be executed and perform “the actual data-processing-work". Letting us focus on delivering business value!

A high-level overview of our platform’s data-capabilities is depicted in the figure above. The FFA-platform can be used together with the FFA Automation Framework, as they integrate with each other out-of-the-box.

Using our platform and automation framework together, enables your organization to kickstart a data-centricity-program that accelerates digital transformation. Thereby, enabling you to quickly derive valuable insights from your organization’s data and serve your clients even better.

Feel free to reach out in case you have any questions! Stay Tuned for more soon!