2 ms·
This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t h
by georgewfraser 8y ago
This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t hear about using Snowflake and BigQuery to analyze giant datasets. They don’t blog about it because they’re just using boring commercial tech. I think the real reason companies like Uber build their own stack is just good old not-invented-here.
- carlineng 8y agoTough to tell from the diagrams exactly, but it looks like the majority of their data is stored in their own data centers, which might mean some reluctance to migrate and ship data to the cloud (“cloud storage” only makes an appearance in the Gen 1 chart). There also exists at least one public example where they’ve bought a commercial database to build on (https://www.memsql.com/blog/real-time-analytics-at-uber-scale/ https://www.memsql.com/blog/real-time-analytics-at-uber-scal...). I’d be willing to concede that there might be legit reasons for not wanting to use BQ or Snowflake. That being said, we use Snowflake heavily at Strava and are very happy with it.
- jfim 8y agoThey are, but their costs also scale somewhat linearly (ie. for CPU core-based licensing, 10x the cores means ~10x the cost, give or take a little based on the vendor discount). At some point, it makes having an engineering team working on a custom solution the cheaper option: why pay vendor X $n$ millions of dollars per year when you could have $n \times 4$ engineers working on a system that's perfectly adapted to your needs? That being said, there is definitely a large amount of NIH promotion-ware being developed at large tech companies, as you mention.