Conveyor vs Do-it-yourself
One of the first things that comes to mind to enable a team of data engineers and data scientists is to build your own data platform. Let us have a look at the pros and cons of that approach and how Conveyor can bring a fresh point of view to this subject.
It’s the build-versus-buy story once again
So you are about to start a new data project. They come in different shapes, but often, they resemble one of the following scenarios: Data pipelines, Machine learning, Data warehouse modernization.
Each scenario requires you to build the data project itself and all the underlying infrastructure. From source control, to packaging and publishing your code, all the way to deployment, scheduling, and operations (logs, metrics, documentation, ...).
DIY
Conveyor
Data Pipelines
Create a batch pipeline often used for analytics to periodically collect, transform and move data to a data warehouse according to business needs
Cross cutting concerns
Once your use case is live, you need to look at cross-cutting concerns like cost management, update management, troubleshooting, security, access-control, ...
Risks
We all know, the devil is in the details. Building a self-service infrastructure, making sure your developers don't spend all their time juggling between 10 different heterogeneous systems to work on their data product is not as easy as it sounds. It will take at least a few months and dozen of iterations to get it right.
Costs
Operating, maintaining and extending your data platform comes at a significant cost
Time
Creating a full-fledge data platform takes a huge amount of time
Firefighting
Once the infrastructure has landed, it's even harder to keep your projects live
Conveyor wants to make your journey easier
Conveyor is meant to be a centralized home for all your data projects, while preserving the freedom of each engineer or team to use their favorite tools or frameworks. You can get a head start on your data use-cases right using templates favoring software engineering best-practices.
Speed up data projects
Use scaffolding and templates for projects as well as abstract away infrastructure
Decrease time-to-market
Decrease time to market by streamlining application lifecycle
Reduce costs
Use monitoring and evergreen strategies to keep costs under control