Conveyor vs Do-it-yourself

One of the first things that comes to mind to enable a team of data engineers and data scientists is to build your own data platform. Let us have a look at the pros and cons of that approach and how Conveyor can bring a fresh point of view to this subject.

It’s the build-versus-buy story once again

So you are about to start a new data project. They come in different shapes, but often, they resemble one of the following scenarios: ​Data pipelines, Machine learning, Data warehouse modernization.

Each scenario requires you to build the data project itself and all the underlying infrastructure. From source control, to packaging and publishing your code, all the way to deployment, scheduling, and operations (logs, metrics, documentation, ...).

DIY

pf_solution_data_pipelines_600dpi.png

Conveyor

pf_solution_data_pipelines_600dpi.png

Data Pipelines

Create a batch pipeline often used for analytics to periodically collect, transform and move data to a data warehouse according to business needs

Cross cutting concerns

Once your use case is live, you need to look at cross-cutting concerns like cost management, update management, troubleshooting, security, access-control, ...

Risks

We all know, the devil is in the details. Building a self-service infrastructure, making sure your developers don't spend all their time juggling between 10 different heterogeneous systems to work on their data product is not as easy as it sounds. It will take at least a few months and dozen of iterations to get it right.

cost_600dpi.png

Costs

Operating, maintaining and extending your data platform comes at a significant cost

time_snail_600dpi.png

Time

Creating a full-fledge data platform takes a huge amount of time

DIY_overall_600dpi.png

Firefighting

Once the infrastructure has landed, it's even harder to keep your projects live

Conveyor wants to make your journey easier

Conveyor is meant to be a centralized home for all your data projects, while preserving the freedom of each engineer or team to use their favorite tools or frameworks. You can get a head start on your data use-cases right using templates favoring software engineering best-practices.

DMC_gain_velocity_600dpi.png

Speed up data projects

Use scaffolding and templates for projects as well as abstract away infrastructure

pf_metric_icon_lead_time_600dpi.png

Decrease time-to-market

Decrease time to market by streamlining application lifecycle

DMC_icon_cloud_cost_reduction_600dpi.png

Reduce costs

Use monitoring and evergreen strategies to keep costs under control