It was 2013 and my job was to sell a fleet management solution to the largest motorbike manufacturer in India but the deal would only go through if they first invested in a new database system that would make our solution viable. Together with a partner vendor, I found myself pitching SAP Hana as a modern database solution that is ideal for both transactional and analytic workloads.
At the time, collecting, storing, and analyzing data at scale required significant investment and was viable only for enterprises that had budgeted a few hundred thousand dollars and a few months in human capital. Fast forward to the present and it is table stakes for an organization serious about building a data culture to invest in a data warehouse. It’s no surprise that this shift is being driven by the innovation taking place in the market that has led to better, more affordable data warehousing solutions.
Before exploring the market for cloud data warehousing tools, it’s worth shedding some light on why companies need a data warehouse as well as the factors leading to its rapid adoption.
Due to an explosion of purpose-built SaaS tools, there is more data and more data sources than ever before. At the same time, organizations big and small have come to terms with the fact that providing personalized experiences powered by data is the only way to attract and retain customers.
This is made possible only when teams are able to analyze and activate data collected across every customer touchpoint, and while there are suites of SaaS products that can help with that, storing a copy of one’s data in a warehouse can unlock so many more possibilities. Moreover, not storing a copy of the data leads to vendor lock-ins, inefficiencies, and frustration for teams that need ad-hoc analyses or a specific dataset to be made available in an external tool they rely on.
Let me now highlight the obvious benefits of adopting a data warehouse.
The usage-based pricing and separation of storage and compute (more on that below) have made it extremely affordable for businesses small and big to adopt a warehouse even with limited data volume and few data sources. In fact, most data warehousing vendors offer trials and free monthly quotas for storage and analysis.
In terms of implementing a data warehouse, you can spin one up in a matter of hours (or minutes if you’ve done it before) without the need to write any code whatsoever. And using an ELT tool such as Fivetran or Airbyte, you can ingest data into the warehouse from pretty much every data source imaginable. Needless to say that one has to write SQL to run queries on the data or to build data models that can be consumed in downstream applications. But speaking as a non-engineer, SQL isn’t hard and here are some great resources to learn SQL.
One of the biggest benefits of cloud data warehouses is that, unlike their predecessors, modern warehousing tools are built on an architecture that separates compute from storage.
It implies that what you pay to store data is separate from the cost to run queries on the data. This not only brings cost benefits but also makes cloud data warehouses more performant with the ability to concurrently run hundreds of queries. I recommend this quick read to learn more about concurrency in cloud data warehouses.
Moreover, with storage becoming cheaper every day, companies are able to replicate their production databases in their data warehouses, making the warehouse the source of truth for all data.
Let’s look at how the market for data warehousing solutions has evolved in the last decade and where it is headed.
At the beginning of this post, I mentioned that a deal I was trying to close relied on the buyer investing in SAP Hana which at the time was available as an on-premise solution with an upfront cost of a couple of hundred thousand dollars. You can imagine, therefore, that it took more than a few sales meetings to get companies to part with that kind of money.
(In case you’re curious, that deal of mine was stuck in a loop for months until I stopped following up.)Other leading warehousing solutions were offered by Oracle, IBM, and Vertica. These were deployed on-premise and combined transactional (OLTP) and analytical (OLAP) processing. The key differences between OLTP and OLAP are explained in this short article from IBM.