4 ms·

The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go op

The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well. Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings. At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive. If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.). I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).

jfim 8y ago
The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.
- georgewfraser 8y ago
  This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t hear about using Snowflake and BigQuery to analyze giant datasets. They don’t blog about it because they’re just using boring commercial tech. I think the real reason companies like Uber build their own stack is just good old not-invented-here.
  - carlineng 8y ago
    Tough to tell from the diagrams exactly, but it looks like the majority of their data is stored in their own data centers, which might mean some reluctance to migrate and ship data to the cloud (“cloud storage” only makes an appearance in the Gen 1 chart). There also exists at least one public example where they’ve bought a commercial database to build on (https://www.memsql.com/blog/real-time-analytics-at-uber-scale/ https://www.memsql.com/blog/real-time-analytics-at-uber-scal...). I’d be willing to concede that there might be legit reasons for not wanting to use BQ or Snowflake. That being said, we use Snowflake heavily at Strava and are very happy with it.
  - jfim 8y ago
    They are, but their costs also scale somewhat linearly (ie. for CPU core-based licensing, 10x the cores means ~10x the cost, give or take a little based on the vendor discount). At some point, it makes having an engineering team working on a custom solution the cheaper option: why pay vendor X $n$ millions of dollars per year when you could have $n \times 4$ engineers working on a system that's perfectly adapted to your needs? That being said, there is definitely a large amount of NIH promotion-ware being developed at large tech companies, as you mention.

jfim 8y ago

The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.

georgewfraser 8y ago
This makes sense in theory but in practice I have observed that commercial data warehouses are incredibly well optimized. There are lots of companies we don’t hear about using Snowflake and BigQuery to analyze giant datasets. They don’t blog about it because they’re just using boring commercial tech. I think the real reason companies like Uber build their own stack is just good old not-invented-here.
- carlineng 8y ago
  Tough to tell from the diagrams exactly, but it looks like the majority of their data is stored in their own data centers, which might mean some reluctance to migrate and ship data to the cloud (“cloud storage” only makes an appearance in the Gen 1 chart). There also exists at least one public example where they’ve bought a commercial database to build on (https://www.memsql.com/blog/real-time-analytics-at-uber-scale/ https://www.memsql.com/blog/real-time-analytics-at-uber-scal...). I’d be willing to concede that there might be legit reasons for not wanting to use BQ or Snowflake. That being said, we use Snowflake heavily at Strava and are very happy with it.
- jfim 8y ago
  They are, but their costs also scale somewhat linearly (ie. for CPU core-based licensing, 10x the cores means ~10x the cost, give or take a little based on the vendor discount). At some point, it makes having an engineering team working on a custom solution the cheaper option: why pay vendor X $n$ millions of dollars per year when you could have $n \times 4$ engineers working on a system that's perfectly adapted to your needs? That being said, there is definitely a large amount of NIH promotion-ware being developed at large tech companies, as you mention.