Understanding the real challenges faced by data professionals is essential for strategic planning. Our team at Conveyor Data recently completed an extensive qualitative research project, conducting 32 hour-long conversations with data engineers, architects, and leaders across various industries and organization sizes. I was one of the people who conducted the interviews, together with Pascal Knapen, Niels Claeys, Lieven Verswyvel, Kristof Martens and colleagues from Dataminded.

What emerged from our candid talks was a snapshot of the data landscape in 2025 — not theoretical predictions, but ground-level realities from those doing the work every day. This article highlights key findings from our research, offering insights into how successful teams are navigating today’s complex data challenges.

Role: Data leader | Organisation size: Enterprise

Balancing Innovation and Governance

Enterprise data leaders in 2025 find themselves at a critical intersection of competing demands. The AI disruption has dramatically changed their landscape, creating both opportunities and significant challenges.

Managing the growing flood of AI and data demands has become increasingly difficult. This disconnect creates a real prioritization problem, with leaders struggling to separate genuine innovation opportunities from technology-for-technology’s-sake requests. “I find it quite upside-down that I have to do use case workshops..,” one data leader told us. “If you really believe in data and AI, look for where AI can add value in the core of your domain.. you are the subject matter expert.”

Simultaneously, these leaders are working to enable data autonomy while maintaining quality. The push for self-service analytics continues to accelerate, but implementing it effectively requires a delicate balance. One healthcare data executive described it as “walking a tightrope between empowerment and protection,” noting that current implementations often lack integration and process maturity.

Regulatory navigation adds another layer of complexity. Meeting compliance requirements while maintaining innovation speed creates constant tension, especially with new regulations like the EU AI Act. “I don’t see readiness in the market,” shared one experienced data leader, highlighting the challenge of satisfying both legal requirements and business innovation needs without letting either side completely dominate the conversation.

Role: Data leader | Organisation size: SME

Pragmatic Solutions with Growth Potential

For data leaders and architects in small and medium enterprises (where these roles often overlap), the challenges take on a different character. Unlike their enterprise counterparts with specialized teams and substantial budgets, SME leaders often wear multiple hats and must be extraordinarily pragmatic with limited resources.

Technical complexity presents a significant hurdle for these organizations. Modern data solutions that might work well in enterprise environments often prove too complex for SMEs to implement effectively. The SME challenge is designing architectures that balance immediate business needs with long-term scalability. “We need architectures that work for the company we are today and the company we want to be in three years, without requiring a complete rebuild in between.” one technology company founder explained.

“Domain experts struggle with technical aspects while bringing solutions to production. We need solutions that can deliver 80% of the value at 20% of the enterprise cost and complexity.” one architect explained.

“We must create solutions that business users can actually operate day-to-day.” another added.

Download the full report >>

This complexity challenge is directly tied to cost management concerns. “Enterprise-grade solutions like Databricks and Snowflake frequently exceed SME budgets and needs. Organizations struggle to find right-sized, cost-effective alternatives,” as one automotive industry data manager put it. Several leaders described creating hybrid approaches — using simplified versions of enterprise platforms for core functions while adopting specialized tools only where they deliver clear competitive advantage.

Data quality control becomes particularly challenging in resource-constrained environments. Without dedicated quality teams, SMEs must find creative ways to maintain standards. “Upstream data quality issues and bypassed SDLC practices create downstream problems. Limited resources make maintaining quality standards particularly challenging,” noted one SME leader.
What makes these challenges particularly difficult is how they often blend together in SMEs. The technical complexity, cost constraints, and quality concerns create a complex puzzle that requires innovative approaches to solve with limited resources.

Role: Data architect | Organisation size: Enterprise

Bridging Technical and Business Worlds

Data architects in enterprise environments described a role that has evolved significantly in recent years. Far from being purely technical positions, these roles now require a unique blend of technical expertise and business communication skills.

Low data literacy among stakeholders emerged as a persistent challenge. “Data literacy gaps can impact the pace of decision-making and change adoption. As a result, architects frequently prioritize concept education over strategic initiative development,” explained one telecommunications architect.“We lost a few good months before we realised we were speaking different languages, what Leadership said and what I thought they said were very different things,” shared an automotive industry architect, highlighting the communication challenges around using data concepts.Data quality issues create cascading problems throughout organizations. Poor upstream quality from source systems and inconsistent metadata management impact everything downstream.“Poor upstream data quality cascades throughout organizations. Unreliable source systems and inconsistent metadata management impact service quality & operational effectiveness,” noted one enterprise architect.The absence of clear processes and ownership creates significant organizational friction. “Absence of formal processes and unclear ownership creates organizational paralysis. Cloud cost ownership, architect roles, system accountability remain undefined, causing project delays,” explained one architect, describing the organizational challenges they face.

Again, it’s the interconnected nature of these challenges that creates an extra layer of difficulty. Low data literacy makes it harder to establish clear processes, while unclear processes make it difficult to address quality issues effectively. Successful enterprise data architects need to learn to address these challenges holistically rather than in isolation.

Role: Data Engineer | Organisation size: Enterprise

Implementation Reality

Data engineers in large organizations paint a picture of increasing complexity and collaboration challenges. Their role has evolved from building pipelines to orchestrating complex systems across multiple teams and departments.

Cross-team collaboration emerged as a significant pain point. Engineers struggle to adopt standardized best practices and collaboration frameworks across different teams (and on different continents, as is often the case in corporates), leading to inefficiencies and duplicated work.“We have five different teams building essentially the same data pipelines with five different approaches. The lack of standardization makes maintenance a nightmare,” explained one engineer at a major retailer.Navigating the complex data landscape presents ongoing challenges. As one telecommunications engineer told us, “They spend a lot of time to keep versions of their internal libraries, Spark clusters up to date as a platform team. By now they have 100+ projects that use them, making changes to all of them takes time.”Data governance and quality monitoring create additional complexity. “If we get a question it takes easily a couple of hours to find out,” shared one media engineer, describing the time-consuming process of tracing data quality issues.Platform complexity creates tension between functionality and usability. Modern data platforms offer powerful capabilities but often become too complex for end-users to navigate effectively.“That’s just painful. Your feature is ready in week one. And then you have to wait until week four or five before it gets into the cycle,” noted one financial services engineer, describing the challenges of balancing stability with rapid feature delivery. Working with third-party vendors adds another layer of difficulty. “Using tooling/interacting with 3rd party vendors is always a chore. Support is flaky, timelines for fixes are not always clear,” explained one telecommunications engineer.

The most successful engineering teams need to find ways to balance these competing demands. Focusing on governance outcomes rather than tools, and creating abstraction layers that shield users from underlying complexity while still delivering needed functionality are two tactics with direct positive impact mentioned by the enterprise data engineers we talked to.

Role: Data Engineer | Organisation size: SME

Building Foundations with Limited Resources

For data engineers in smaller organizations, the challenges reflect their need to establish fundamental capabilities with constrained resources. Technical foundation building dominates their work. Engineers in SMEs often struggle to establish robust data infrastructure, with limited resources making it difficult to implement best practices from the start.

Unlike enterprise teams with dedicated specialists for each technology, SME engineers often handle everything from database administration to API development. One healthcare startup engineer described their role as “wearing all the hats in the data stack.”

“Good engineers need to care for what they build and be perfectionist. Good engineers need to be able to be independent and able to ramp up productivity fast without onboarding, I don’t have time for that,” explained one lead engineer at a small MLOps company, highlighting both the talent challenges and resource constraints they face.

Finding and retaining skilled talent presents a significant challenge for smaller organizations. “We’ve had 6 hires in 4 years, 4 fired again,” shared the same lead engineer, emphasizing how critical each hire becomes when resources are limited.|
Impact achievement takes on heightened importance in resource-constrained environments. Engineers in SMEs can’t afford to work on projects without clear business outcomes. Their focus constantly shifts between technical excellence and business value creation, requiring a pragmatic approach to every decision.“In a small company, there’s no room for technical projects without clear business outcomes. Everything we do must directly impact the bottom line,” noted one engineer.

Cost management becomes particularly critical. “Initial step to the cloud was lift and shift, secondly we now need to stabilize the platform and make it run well,” explained one engineer, highlighting the evolution from simply moving to the cloud to optimizing for cost efficiency.What distinguishes successful SME engineering teams is their ability to be selective about which enterprise best practices to adopt. Rather than attempting to implement comprehensive frameworks, they focus on the core principles that deliver the most value with the least overhead, creating lightweight approaches that can scale as the organization grows.
The Absolute Universal Challenge: Data Quality

Across all roles and organization sizes, one challenge emerged consistently: data quality. However, the nature of this challenge varies significantly based on organizational context.

For enterprises, data quality issues typically stem from complexity and scale. With data flowing through numerous systems and touchpoints, maintaining consistent quality becomes exponentially more difficult. Several enterprise leaders described implementing federated governance models where quality responsibility is distributed to data domain owners rather than centralized teams.

For SMEs, data quality challenges often reflect resource constraints and technical debt. Without dedicated data quality teams, these organizations must embed quality practices into everyday workflows. One retail analytics leader described their approach: “We’ve made data quality everyone’s job by making it visible to everyone — dashboards showing quality metrics are as prominent as business KPIs.”

What makes data quality particularly challenging is its cascading nature. Poor quality in upstream systems inevitably impacts downstream analytics and decision-making.

“If we get a question, it often takes at least a couple of hours to find out the answer,” shared one engineer, describing the time-consuming process of tracing data quality issues.

Regardless of organization size, successful teams are moving away from treating data quality as a technical problem and instead approaching it as an organizational capability.This shift recognizes that sustainable quality requires alignment across people, processes, and technology.

Key Takeaways for Data Professionals in 2025

Our research revealed several patterns that transcend specific roles and organization sizes.

1. Organizations face a surge in AI and data initiatives but struggle to identify valuable use cases, prioritize them effectively, and demonstrate clear ROI. The most successful teams have implemented structured processes for evaluating and selecting use cases based on business impact rather than technical interest.

2. Teams are working to make quality data more accessible across their organizations, with self-service capabilities recognized as a top priority but challenging to implement well. Success requires balancing accessibility with appropriate governance and quality controls.

3. Collaboration across different roles and departments remains challenging, with teams struggling to establish common language and shared objectives. Organizations making the most progress have invested in building data literacy across all functions, not just within data teams.

Modern data solutions bring significant added complexity that must be managed differently based on organization size and available resources. The most effective approaches are those tailored to an organization’s specific context rather than following generic best practices.

The Complete Picture

Our research summary report provides a clean overview of these key findings, with specific recommendations for each organizational role and size. Download the full report here to see where your organization fits in this landscape and what targeted actions you can take to address your specific challenges in 2025.

Whether you’re leading a data team, designing architectures, or implementing solutions, understanding these patterns can help you focus your efforts where they’ll deliver the most value.