Determining whether to create or purchase new software is a decision that data engineers often face. In the realm of data engineering, building data pipelines was a common choice, as it only required a few data ingestion scripts. However, with the rise of big data, the landscape is changing rapidly.
We, as data engineers, now must handle high volumes of data from constantly changing sources, and the latency of real-time data use cases is more critical than ever. There are various approaches to developing data pipeline architectures in this new world. If we choose to build our own data pipelines, we end up with data integration systems that are pieced together by multiple engineers over time, leading to systems that resemble existing frameworks such as Airflow.
At times, we tend to approach problems as a choice between building and buying, without weighing the opportunity costs. Depending on the company’s overall goals and data maturity journey, building may not always be the best option.
This article aims to discuss the decision of building versus buying for both real-time data streams and batch processing pipelines (ETL/ELT). This will aid teams in making the right choice for their next data infrastructure component.
Challenges of building and maintaining data pipelines:
Building and maintaining data infrastructure is a lengthy process and time-consuming. Even small requests can pose challenges, particularly when dealing with various data types and dozens of sources with different schema.
Addressing ad hoc requests for new data sets while maintaining the current code base creates redundancy and slows down business impact and scalability.
Benefits of buying:
Quick turn around: Bought solutions meet the majority of company’s use cases more quickly and can be implemented immediately.
Less maintenance: Maintenance costs are handled by the solution provider, freeing your team to focus on adding value rather than maintenance.
No need to keep up with APIs: Tools provide connectors out of the box, removing the need to rebuild and maintain them.
New features don’t need to be built: Purchased solutions are continuously improved by the provider, saving time and resources.
Challenges that come with buying:
Less flexibility: Bought tools limit the extent of editing or modification in terms of functionality.
Less control: Future use cases or small edits may not be adaptable, and your team is not responsible for building new features.
Vendor lock-in: Monthly invoice payments and multi-year agreements may tie the company to the vendor.
Lots of different tools leads to multiple learning curves and slow initial development.
Buy or build considerations:
When to buy:
Team’s main focus is not building software and they have budget limitations.
Tight timeline and the need to turn around value quickly.
Limited resources or technical knowledge for the specific solution they would need to build.
When to build:
Executive team needs a unique function or ability that no solutions offer.
Bigger scope and vision for the solution, with plans to sell it externally.
No tight timeline and a proficient team delivering large-scale projects.
The final decision:
In conclusion, it is essential to balance build versus buy in the modern data stack era. While there are pros and cons to both options, there is a growing array of pre-built tools in the cloud with sensible pricing models that can be tested with free trials. Most companies are too occupied with other operational needs to fully commit to internally building data tools.