To be able to run your Scalable, Flexible and Robust Data & Analytics solution, as described in the previous blog, you’ll need a platform that is build according to the same principles. For this, Food For Analytics provides a full-fledged data-platform that can be deployed in either Amazon Web Services and Microsoft Azure, or any other cloud for that matter. In other words, it's cloud agnostic. Our platform consists of a fully workable stack comprising: Spark, Kafka, Airflow, different RDBMS (MySql, SQL Server, PostgresSQL), Druid, Zeppelin, and above all, integration with Amazon Web Services: S3 and Microsoft Azure blob storage. Allowing you to scale both storage- and compute-capacity independently from each other. The platform-components run in docker-containers and are orchestrated by Kubernetes, including required security and reliability. Sounds cool right? But why would you need such a platform? And what are the advantages? Let’s dive in right away!

Running the FFA-platform stack in docker-containers on Kubernetes significantly aids in increasing the scalability and flexibility of your Data & Analytics solution. Since the FFA platform is cloud-agnostic it can be deployed to various cloud vendors. As it is built in docker-containers, it can run anywhere! With this, we achieve full independence, and guarantee the platform’s functionality regardless of the underlying infrastructure. Let’s say, if you decide to change your cloud provider for your data platform from Amazon Web Services to Microsoft Azure, we can easily pick up the FFA-platform (with each solution running on it) and migrate it to Azure, whilst spinning down the old AWS instance. With proper preparation this can be done in days rather than weeks in comparison to migrating your on-premise data center.

Another advantage of using docker-containers is that they use less resources. Once effectively used and setup, this will drastically reduce platform costs, increase flexibility, and scalability. Hence, you will only pay for using resources of the cloud provider, during the actual data-processing. Besides that, you can schedule up and down times and increase or decrease the  platform’s capacity at any time, catering to your organization’s current and future needs!

In comparison to buying and setting up on-premise hardware, our platform is flexible, and above all focusses on speed and efficiency. Combined with the orchestration-capabilities of Kubernetes you can start, create, replicate, or destroy containers in seconds. This will make new software releases or version updates quick and easy.

In addition to the platform-benefits, docker-containers also enable developers, such as Data Engineers, to improve their productivity and use continuous integration and continuous delivery (CI/CD). Having your code for data-pipelines being pushed to a central repository, enables us to trigger a CI/CD pipeline for automated validation, testing and deployment to any desired environment. This ensures that all required components, like spark and airflow, contain the data-pipeline-code so that it is ready to be executed and perform “the actual data-processing-work". Letting us focus on delivering business value!