Migrating a Data Platform: 20% technology, 80% everything else

At some point, when the bills become too high, or the platform is not scalable anymore, almost every organisation with a data platform reaches the same crossroads: “We need to migrate”. Sometimes it is out of the hands of the engineers if a vendor sunsets a tool you built everything on. Or a license renewal comes with a heavy price tag that nobody wants to sign off on. Whatever happens after is rarely about the technology itself. It is about whether the organisation and its people are ready to go through a migration.

I have gone through this experience more than once myself, moving pipelines, tables, and being in the middle of different delivery teams that are expecting their dashboards to just keep working, regardless of the backend changing. If there is one thing I keep relearning, it is this: Migrations do not fail because the SQL was hard to translate. Migrations fail because nobody agreed upfront about when is it ”done”, who owns what, or what happens when the numbers do not match.

Why is this more relevant now?

Today’s platforms are far more complex than 10 years ago . They are made up of orchestrators, lakehouses, dozens/hundreds of dashboards, APIs, and legacy pipelines that nobody remembers building but are crucial to the everyday work of different teams. Moreover, platforms are being retired at a much faster pace. Every vendor now seems to a have a new “unified” offering they want you to move into. Whether or not that pressure is justified is a separate discussion.

Who this is actually for?

This is written for the people who actually sit around the table during a migration, and what is at stake for each of them specifically, not just the engineers writing the code

If you are a migration architect or a data platform owner, this is about not reinventing the wheel every time, and making sure that the strategy and success criteria are defined from the beginning.

If you are a platform architect, this is about knowing early whether the target platform can truly support what you already have, before the time gap starts to grow

If you are a data engineer doing the actual migration, this is not about being handed ambiguous work like migrate everything and being expected to figure out priority and validation thresholds on your own.

Finally, if you are a data owner or a consumer team, and your dependency sits downstream, this concerns you more than you might think. You are the one that will be asked whether those numbers look right to you or not. It is much better to be part of that conversation early on than to be surprised by it before decommissioning of the old platform.

What tends to go wrong?

The technical part of a migration, in my experience, is rarely the hardest part, and it is becoming easier. A few years ago, translating SQL dialects or rewriting a pipeline for a different scheduler was a lot of manual work: reading legacy code line by line, guessing the intent where comments were missing and heavily testing to make sure nothing broke. With AI-assisted coding on the rise, giving agents the task of migrating from platform A to B or between different SQL dialects has become quite trivial. What is much harder to solve is the stuff that should have been agreed on before a single pipeline was touched:

• Nobody defined what “success” really means for a given pipeline. Does it need to match the legacy output, or is a 95% similarity fine? If this gets decided later in the project you end up negotiating thresholds with a stakeholder while the clock could potentially be running on a deadline.

• The discovery phase usually gets rushed or entirely skipped because it does not feel like actual progress being made. A dependency diagram or a source-to-target mapping sheet is not super impressive. Teams are usually tempted to jump right into execution. Migration is usually close to 75% analysis and 25% execution with AI-assisted coding. Spending more time here can save up a lot of time down the line.

Building the priority list as part of the Discovery phase.

• Access to the legacy system and outdated documentation is essential to the engineering team. This alone can stall migration for quite some time.

• Consumer teams and data owners end up being part of the loop quite late. By then, the trust relationship is already broken, and you end up trying to repair it instead of repairing their broken pipelines.

• In some cases, other teams are still developing on the legacy systems while the migration team is still trying to map out what exists. Every new table or new job being introduced is a new dependency that might be missed.

None of these are technology problems. They are alignment problems and they are exactly the kind of thing that is cheap to fix in advance, but expensive to fix mid-project.

Why I ended up writing a guidebook about it

After going through the cycle myself a couple of times across different projects, the pattern became too repetitive to ignore. So I put together a small migration guidebook that breaks the whole process into five phases: Vision, Governance, Discovery, Execution, and Cutover. At a high level, it walks through who should be involved at each stage, and what decisions should come out before moving forward to the next phase. It also includes where AI agents can meaningfully take over at the execution level with some examples.

In Conclusion

A failed migration might look differently depending on the situation. Nobody sends out an email saying “the migration didn’t work.” Instead, a dashboard quietly starts showing numbers people stop trusting. A consumer team builds a workaround instead of raising the discrepancy, because raising it feels like more friction than it’s worth. It shows up months later, as a slow decay of confidence in the platform itself, and that is a hard thing to win back.

The technical side of a migration is the part getting easier every year, and I expect that trend to continue. The part that decides whether people trust the platform afterwards is the part that has nothing to do with code. This includes agreeing early on what success looks like, giving discovery the time it actually needs, and bringing in the teams downstream early on. Get that right and the execution tends to take care of itself. Get it wrong, and no amount of clever tooling will fix it after the fact. If you are about to start one, treat scope, ownership, and validation as the actual project, and treat the migration itself as the execution of that plan. The guidebook can be found here. If you are about to start a migration, or you are trying to convince your organisation that a migration is needed, this is where I would point you next.