4 ms·

Reading this, I can’t help but think Uber would be better off adopting one of the commercial data warehouses that separates compute from storage: Snowflake or B

Reading this, I can’t help but think Uber would be better off adopting one of the commercial data warehouses that separates compute from storage: Snowflake or BigQuery. They have full support for updates, they support huge scale, and because they’re more efficient the cost is comparable to Presto in spite of the margin. You can ingest huge quantities of updates if you batch them up correctly, and there are commercial tools that will do the entire ingest for you (cough Fivetran). Disclosure: am CEO of Fivetran.

sixdimensional 8y ago
The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well. Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings. At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive. If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.). I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).
- jfim 8y ago
  The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.
  - georgewfraser 8y ago
    This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t hear about using Snowflake and BigQuery to analyze giant datasets. They don’t blog about it because they’re just using boring commercial tech. I think the real reason companies like Uber build their own stack is just good old not-invented-here.
    - carlineng 8y ago
      Tough to tell from the diagrams exactly, but it looks like the majority of their data is stored in their own data centers, which might mean some reluctance to migrate and ship data to the cloud (“cloud storage” only makes an appearance in the Gen 1 chart). There also exists at least one public example where they’ve bought a commercial database to build on (https://www.memsql.com/blog/real-time-analytics-at-uber-scale/ https://www.memsql.com/blog/real-time-analytics-at-uber-scal...). I’d be willing to concede that there might be legit reasons for not wanting to use BQ or Snowflake. That being said, we use Snowflake heavily at Strava and are very happy with it.
    - jfim 8y ago
      They are, but their costs also scale somewhat linearly (ie. for CPU core-based licensing, 10x the cores means ~10x the cost, give or take a little based on the vendor discount). At some point, it makes having an engineering team working on a custom solution the cheaper option: why pay vendor X $n$ millions of dollars per year when you could have $n \times 4$ engineers working on a system that's perfectly adapted to your needs? That being said, there is definitely a large amount of NIH promotion-ware being developed at large tech companies, as you mention.
kerng 8y ago
My take is that companies like Uber, Google, Facebook see and encounter issues ahead of the curve - so they build their own stuff and luckily open source it, which is good. Maybe others will run into similar issues and can leverage those solutions in case they work for them also.
indogooner 8y ago
How does BigQuery support updates? IIRC in BigQuery you have to overwrite whole partitions. At least in Hive you can have secondary partitions to mitigate this.
- georgewfraser 8y ago
  BigQuery has had DML for over a year. There are quotas but the quotas keep rising and they can manually increase your quota on request.
  - indogooner 8y ago
    Thanks. I checked the quotas. They seem to have increased to 200 per table. Last time I checked (last year) they were pretty limiting. I like the MERGE statement though. Hope it comes out of Beta soon. The pricing is for full scan so it is not just for in-place update but charged with bytes scanned for selected columns for scanned partitions.
curiousDog 8y ago
Isn't a DW just part of the overall equation? These days I assume you'd want to do a lot more like train ML models as part of your pipeline, things that Spark/Scala allow you to do that'd be harder than just SQL. I think most customers use both.
- indogooner 8y ago
  Yes but BigQuery provides read/write libraries. So you can use Spark or Apache Beam to read the data into your cluster and then process it using your framework of choice - XGBoost/SparkML etc. The predictions can then be written back to BigQuery using those same libraries.
  - curiousDog 8y ago
    Sure but that would mean copying the data in/out. Instead you could get data-compute co-location with a proper HDFS cluster. But I think BigQuery launched SQL ML APIs recently to train simple distributed ML models on the fly.
    - azurezyq 8y ago
      Actually running and scaling a hdfs cluster is no easy task and quite expensive than cloud based object storage options. Just list a few: namenode HA, unbalanced r/w (a lot of old data but less used, wasting cpus for just making the machines up), cross dc replication. You may be able to get some of them from cloudera, but would that be cheap?
    - smueller1234 8y ago
      A "proper" HDFS cluster also wastes incredible amounts of disk with it's naive redundancy. It's a bear to maintain (see sibling comment) and the disaster recovery story (at a DC level) is non-existent. If you invest a chunk of that money saved by not having to do 3 or 4 way data replication into good network, a lot of that move-compute-to-data complexity becomes unnecessary. The second big efficiency promise of the basic map reduce is doing lots of sequential I/O on those pesky disks which can do 100x sequential throughput compared to their pithy random read performance. Alas, in today's mixed workloads, you're certainly not getting those >100MB/s from a disk that might make a benchmark scream. So the network savings (you're still going to do that shuffle before your reduce anyway, aren't you?) from compute-next-to-storage becomes less useful. Best I can tell, the compute-next-to-storage trick does still matter as you get to large data sets (PB in a job), but then as you keep growing your infrastructure (and I'd wager Uber would be just about at this point), the benefits of disaggregation start weighing increasingly heavily. In particular, fleet management becomes considerably easier with less resource stranding.

sixdimensional 8y ago

The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well. Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings. At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive. If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.). I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).

jfim 8y ago
The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.
- georgewfraser 8y ago
  This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t hear about using Snowflake and BigQuery to analyze giant datasets. They don’t blog about it because they’re just using boring commercial tech. I think the real reason companies like Uber build their own stack is just good old not-invented-here.
  - carlineng 8y ago
    Tough to tell from the diagrams exactly, but it looks like the majority of their data is stored in their own data centers, which might mean some reluctance to migrate and ship data to the cloud (“cloud storage” only makes an appearance in the Gen 1 chart). There also exists at least one public example where they’ve bought a commercial database to build on (https://www.memsql.com/blog/real-time-analytics-at-uber-scale/ https://www.memsql.com/blog/real-time-analytics-at-uber-scal...). I’d be willing to concede that there might be legit reasons for not wanting to use BQ or Snowflake. That being said, we use Snowflake heavily at Strava and are very happy with it.
  - jfim 8y ago
    They are, but their costs also scale somewhat linearly (ie. for CPU core-based licensing, 10x the cores means ~10x the cost, give or take a little based on the vendor discount). At some point, it makes having an engineering team working on a custom solution the cheaper option: why pay vendor X $n$ millions of dollars per year when you could have $n \times 4$ engineers working on a system that's perfectly adapted to your needs? That being said, there is definitely a large amount of NIH promotion-ware being developed at large tech companies, as you mention.

kerng 8y ago

My take is that companies like Uber, Google, Facebook see and encounter issues ahead of the curve - so they build their own stuff and luckily open source it, which is good. Maybe others will run into similar issues and can leverage those solutions in case they work for them also.

indogooner 8y ago

How does BigQuery support updates? IIRC in BigQuery you have to overwrite whole partitions. At least in Hive you can have secondary partitions to mitigate this.

georgewfraser 8y ago
BigQuery has had DML for over a year. There are quotas but the quotas keep rising and they can manually increase your quota on request.
- indogooner 8y ago
  Thanks. I checked the quotas. They seem to have increased to 200 per table. Last time I checked (last year) they were pretty limiting. I like the MERGE statement though. Hope it comes out of Beta soon. The pricing is for full scan so it is not just for in-place update but charged with bytes scanned for selected columns for scanned partitions.

curiousDog 8y ago

Isn't a DW just part of the overall equation? These days I assume you'd want to do a lot more like train ML models as part of your pipeline, things that Spark/Scala allow you to do that'd be harder than just SQL. I think most customers use both.

indogooner 8y ago
Yes but BigQuery provides read/write libraries. So you can use Spark or Apache Beam to read the data into your cluster and then process it using your framework of choice - XGBoost/SparkML etc. The predictions can then be written back to BigQuery using those same libraries.
- curiousDog 8y ago
  Sure but that would mean copying the data in/out. Instead you could get data-compute co-location with a proper HDFS cluster. But I think BigQuery launched SQL ML APIs recently to train simple distributed ML models on the fly.
  - azurezyq 8y ago
    Actually running and scaling a hdfs cluster is no easy task and quite expensive than cloud based object storage options. Just list a few: namenode HA, unbalanced r/w (a lot of old data but less used, wasting cpus for just making the machines up), cross dc replication. You may be able to get some of them from cloudera, but would that be cheap?
  - smueller1234 8y ago
    A "proper" HDFS cluster also wastes incredible amounts of disk with it's naive redundancy. It's a bear to maintain (see sibling comment) and the disaster recovery story (at a DC level) is non-existent. If you invest a chunk of that money saved by not having to do 3 or 4 way data replication into good network, a lot of that move-compute-to-data complexity becomes unnecessary. The second big efficiency promise of the basic map reduce is doing lots of sequential I/O on those pesky disks which can do 100x sequential throughput compared to their pithy random read performance. Alas, in today's mixed workloads, you're certainly not getting those >100MB/s from a disk that might make a benchmark scream. So the network savings (you're still going to do that shuffle before your reduce anyway, aren't you?) from compute-next-to-storage becomes less useful. Best I can tell, the compute-next-to-storage trick does still matter as you get to large data sets (PB in a job), but then as you keep growing your infrastructure (and I'd wager Uber would be just about at this point), the benefits of disaggregation start weighing increasingly heavily. In particular, fleet management becomes considerably easier with less resource stranding.