NEO data platform

Business context

Luminus, a prominent player in the Belgian energy market, embarked on a transformative journey, venturing into the realm of machine learning. As they embraced this new frontier, Luminus faced the challenge of transitioning from a technology stack tailored for traditional business intelligence on on-premise infrastructure to a more advanced machine learning environment.

Scope & objectives

The primary goal was to establish an effective operating model and a cloud-native data and analytics platform that empowers the development and execution of machine learning applications. This approach greatly emphasized self-service data science, enabling data scientists to harness the full potential of their work.

Key results

Self-service experimentation by data scientists with Sagemaker notebooks, seamless data integration through Amazon Appflow and DMS (Database Migration Service), and real-time analysis, particularly in response to sensor and metering data, through Amazon MSK (Managed Streaming for Apache Kafka) were essential components of this transition. Furthermore, ad-hoc querying of S3 data, registered in the AWS Glue catalog, through AWS Athena provided additional capabilities to the data engineering landscape.

As a result of the transformative efforts, an entirely new IT-oriented Landing Zone and Data & Analytics Platform was implemented, leveraging the Conveyor. This comprehensive platform covered the complete machine learning workflow, from initial experimentation to real-time monitoring, while employing essential design patterns for both batch processing and streaming data.

Significant structural changes were introduced, leading to the creation of long-lived business-oriented use case teams, consisting of new roles across various business lines and IT. This innovative approach not only improved ownership and innovation but also fostered a culture of continuous learning.

Impact

Within a single year, more than 25 users, including scientists and engineers, were successfully onboarded, leading to the deployment of seven live use cases. Additionally, 18 projects were initiated, and connections were established with 21 data sources, enhancing data accessibility and utilization.

The implementation of these advanced capabilities resulted in a positive return on investment after the delivery of the first use cases. The data platform witnessed widespread adoption across different business lines, ultimately reinforcing the value-driven industrialization of new use cases.

Continuous growth and enhancement were achieved through the Dataminded Academy, facilitating the cultivation of expertise and knowledge among Luminus’s workforce.