7 ms·
Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (a
by jamesblonde 1y ago
Data warehousing is quickly becoming a commodity through open-source.
I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking.
It's not just Trino. StarRocks and Clickhouse have enterprise grade k8s helm charts/operators. That 60bn valuation is an albtross on Databrick's neck - their pricing will have to justify it, and their core business is commoditizing.
Neon filled their product gap of not having an operational (row-oriented) DB.
- hlpn 1y agoTotally agree. Happy open source StarRocks user here using the k8s operator for customer-facing analytics on terabytes of data. There's very little need for Databricks in my world.
- anilshanbhag 1y agoLooking at StarRocks site (https://www.starrocks.io/ https://www.starrocks.io/), they compare against Clickhouse, Druid and Trino. Don't even compare against Spark/Databricks! Guess Spark is just not competitive.
- jflkdsjlcsuq 1y agoDon't fall for benchmarketing. Do your own benchmarks for your use case.
- lars_francke 1y agoAnyone looking for an open-source Cloudera alternative based on Kubernetes operators. We're building one (~5 years old now): https://stackable.tech/ https://stackable.tech/ & https://github.com/stackabletech/ https://github.com/stackabletech/ On-premise open-source S3 is a problem though. MinIO is not something we're touching and other than that it looks a bit empty with enterprise ready solutions.
- matt-p 1y agoWhat's wrong with minio out of curiosity? Ceph an option?
- lars_francke 1y agoThis is at least partially subjective. https://news.ycombinator.com/item?id=32148007 https://news.ycombinator.com/item?id=32148007 https://news.ycombinator.com/item?id=35299665 https://news.ycombinator.com/item?id=35299665 Ceph would be a theoretical option, but a) we don't have a lot of experience with it and b) it's relatively complex to operate. We'd really love to add a lighter option to our stack that's under the stewardship of a foundation.
- positisop 1y agoTry expanding a cluster, or changing erasure coding configuration, or using anything that needs random access within a file (parquet), or any day 2 operation.
- kwillets 1y agoIt's been a commodity for decades now. Metrics like price-performance have a long history, but the SnowBricks products fail at them quite dramatically. The difference is hard-sell vs. soft or no-sell.
- datadrivenangel 1y agoNot having to buy an appliance and pay for it up front is quite a valuable option. Also the split between processing and storage allows for better archival and scaling strategies.
- datadrivenangel 1y agoBut why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.
- antruok 1y agoIn addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so. One example from Snowflake is hybrid tables which adds rowstore next to columnar. OLAP + OLTP = HTAP
- bbohan 1y agoSingleStore been doing that for years . Unistore been strugglin
- antruok 1y agoI wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.
- orangechairs 1y agoSingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.
- moonikakiss 1y agoit was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead https://www.mooncake.dev/blog/htap-is-dead
- swyx 1y agoif Databricks just wanted a row DB they couldve done postgres themselves. paying this much for Neon i think is a sign that Neon has something special they want (which, knowing their marketing line, is "independently scalable storage and compute for postgres")
- dustingetz 1y ago"Time is the denominator"
- t0mas88 1y agoThat sounds like AWS Aurora?
- swyx 1y agomashed w a lil bit of snowflake arch
- yencabulator 1y agoEasy quick cheap forks of database state for AI agents to muck with.
- richardw 1y agoNot commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it. It’s SO expensive for an enterprise to rejig for a new platform that you can’t rely on (insert open source project here). I managed to add one startup and so far it’s done very well, but it was an exceptional case and the global CEO wanted the functionality. But it used MongoDB and ops team didn’t have any skills, so rather than learn one tiny thing for an irrelevant data store they added cash to use Atlas with all the support and RBAC etc etc. They couldn’t use the default Azure firewall because they only know one firewall, so added one of those too. Also loaded with contracts. Kept hiring load down, one number to call, job done. Startups cost is $5-10k per year. Support BS about $40k. (I forget the exact numbers but it dwarfed the startup costs.) Startups are from Venus, enterprise are from Jupiter.
- antruok 1y agoEnterpise also often wants a full data platform (like Databricks), not a plain data warehouse.
- CharlesW 1y agoAre there even any "plain data warehouse" vendors left? Even the oldest of the old-school providers seem to have crawled up the modern data stack since the data warehouse glory days, refocused on lakehouse + Open Table Formats either in their core platform or complementary products.
- tqi 1y agoI'm curious - as someone who hasn't been paying super close attention to the space, what does it mean to be a full data platform? Is it just having all the different flavors of DB you might need all from one vendor, or is there also tighter integration than if you cobbled it together from multiple vendors?
- jflkdsjlcsuq 1y agoI have been happily using ClickHouse for the past couple of years without any issue. Rock solid database with wide variety of features and fulfills all my needs. My favourite is the "external dictionary" feature which easily allows me to integrate it with other datastores like Postgres and Redis.
- data_marsupial 1y agowhat do they use for ETL?