4 ms·
The market for new proprietary databases is both large and lucrative -- it is printing money right now. What has changed is the structure of the market and the
by jandrewrogers 6y ago
The market for new proprietary databases is both large and lucrative -- it is printing money right now. What has changed is the structure of the market and the product economics. The classic marketing-driven retail database product model, the kind that puts billboards on the 101, stopped working as a financial model a decade ago -- development costs are too high for the marginal return. That has been replaced by models that produce much higher margins with much lower development cost. Unlike database companies of yore such as Oracle, they use little overt marketing because they don't sell through the public. It is a completely different value proposition for the customer too. And this market is growing.
Proprietary databases have found their niche with qualitatively higher performance and throughput -- often 10-100x -- than the best open source has to offer on the same hardware footprint. As data continues to grow exponentially and cloud costs increase, operational efficiency has a much bigger impact on the bottom line than licensing cost. This gap has only been increasing over time. For most open source databases this is a fundamental architectural issue, and you can't fix architecture, so they are defenseless against highly optimized competitors.
I was on a project a year or two ago where a particular ElasticSearch workload was replaced by a proprietary solution that demonstrably offered 100x higher throughput. It was apparently worth tens of millions of dollars to the customer for that improvement.
The reason big retail databases went away is that 90% of the development cost is in the "big retail" part and not the "database" part. At the same time, development costs for a state-of-the-art database engine have become extremely high due to wage costs (worse than FAANG), so the industry needed to find a business model that would allow those costs to be contained. It is still around but much more invisible. It would be nearly impossible to properly finance a database company as a startup, so a different model is used.
- totoglazer 6y agoThis is fascinating! Could you share some names of these databases? I’d love to learn more.
- LukeEF 6y agoThere is a Stonebraker quote somewhere (can't find now) that basically says 'give me the workload and I will build you an optimised database'. The point with building a database enterprise is to build something that is sufficiently general purpose - not just optimised for one workload even if that workload is valuable for one company. In that world, there is only open source. Even snowflake, which is cited as a good example of a new proprietary, is built on FoundationDB and is like a hosted version with lots of bells and whistles. We are building a DB (TerminusDB) and started as prop and we couldn’t see a way to make it work without going open source.
- jandrewrogers 6y agoThis is essentially correct. Trying to be all things to all people massively increased development costs for diminishing returns, and not infrequently forces significant architectural tradeoffs too. The new databases tend to be narrower, hyper-optimized data model and workload specialists. It requires a small fraction of the code and you know exactly who your target market is. I much prefer this model from an engineering perspective. It is also what drives the interest in database meta-programming scaffolds that can codegen narrowly optimized database engines from something that approximates a specification. It greatly reduces cost/time to market in this new environment. I'm not sure how you would open source a codegen implementation. I don't use any open source inside high-performance implementations myself for good reason, though I occasionally do PostgreSQL mods for companies for fun. Customers care less about open source on the very high-end because vanishingly few people could credibly maintain it in any case.
- hodgesrm 6y ago> Proprietary databases have found their niche with qualitatively higher performance and throughput -- often 10-100x -- than the best open source has to offer on the same hardware footprint. This is not true in general. It depends on the workload. Here's an example: MySQL 8.0 is benchmarked at over 1M QPS on I/O bound reads using Optane SSDs. [1] You would need to show that proprietary software hits 10-100M QPS for this workload and hardware configuration. Oracle and SQL Server are often faster because they have better query planning but it's not orders of magnitude. [1] http://dimitrik.free.fr/blog/posts/mysql-performance-1m-iobound-qps-with-80-ga-on-intel-optane-ssd.html http://dimitrik.free.fr/blog/posts/mysql-performance-1m-iobo...
- jandrewrogers 6y agoThis really isn't comparing against Oracle or SQL Server, which are both legacy architectures. And you don't need Optane SSDs; people do better than that with cheap NVMe JBODs like you get on EC2. Also, something is only "I/O bound" if it is literally running at the theoretical limits of the storage hardware. And even that doesn't imply another database couldn't drive much more workload with the same theoretical bandwidth; many popular databases employ architectures that unnecessarily burn storage bandwidth. It is pretty rare for these days for a modern database engine to be I/O bound in a storage sense, even with cheap solid-state storage. Certainly not for fast-twitch query workloads. For most data models and workloads, the bottleneck is either memory or network bandwidth if you are doing everything else right. MySQL isn't typically held up as a paragon of query performance. It is solidly middle of the road. It is part of a generation of architectures that can't get anywhere close to current high-performance architectures. Every database architecture suffers this fate as they get older.
- hodgesrm 6y agoRight, I realized I put in the Oracle comparison which was beside the point but decided not to edit it out.