Migrating on premise hadoop infrastructure to the cloud in a highly secure/regulated environment

Business context

KBC decided to migrate their on premise big data hadoop platform to a modern data platform running in the cloud. Having several hundreds of data pipelines currently running on premise, migrating their current workloads is not a trivial task. As a big Belgian bank, KBC has to take into account a lot of security, compliance and governance requirements. These need to be translated into a data platform that is compliant, secure and user friendly for their hundreds of data engineers and data scientists. They need to work efficiently on many different projects for many different BU’s and legal entities, with minimal migration effort.

Scope & objectives

  • Design and build the new data platform based on cloud technologies that is fit for purpose for the coming 5 years

  • Offer an efficient end-to-end development flow for data engineers and data scientists to reduce the time to market for many new use cases in parallel

  • Abstract away infrastructure and cloud complexities so data engineers can focus on their tasks at hand

  • Use best practices to guarantee data security, least privileged principles, data leakage prevention and regulatory requirements like GDPR, SHREMS II, ...

  • Use simple but powerful practical concepts to easily integrate the data platform with KBC internal processes

Key results

Data Minded performed PoC’s to help assist KBC making the right technological choices for their data platform that should be valid for at least the next 5 years. Together we defined the architecture and set the groundwork for their future capabilities of their data engineering and data science workflows. We introduced new concepts that covered both their requirements and that are practically implementable with cloud concepts. During that time we also executed on that plan and were able to scale from a small core team to a larger team that is capable of maintaining, running and extending the data platform for many different legal entities.

Impact

  • We embedded AWS knowledge and expertise within their teams to get started very quickly

  • Introducing best development practices like Infrastructure as code that are now being used everywhere and allow us to scale the platform more easily

  • Dramatically improved developer experience for both Data engineers and data scientists without compromising on security and compliance rules

  • Help design and implement a data platform that got approved by all internal compliance bodies and external security audits