5 ms·

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Goo

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB and DynamoDB. We ended up doing a lot of sharded MySQL in our own data centers instead. This on-prem decision led to a lot challenges internally where we would adopt OSS and then have difficulty scaling it to our needs. For some tech like Kafka it worked out, and we hired Kafka contributors who helped us scale it. For other tech like Cassandra it was a pretty epic failure. I am sure more of these war stories exist that I wasn’t privy to myself. Coupled with the fact that we were early adopters into Golang which had its own OSS ecosystem, we found that writing a lot of our own infrastructure solutions was the only viable option at our scale. What you are seeing now is a lot of that home grown infrastructure being open sourced in big way as people who have left Uber continue to see value in investing in the tech that they worked so hard to build. There is probably a nontrivial amount of work to scale the Uber OSS down for smaller use cases but some startups are emerging to make that happen. Source: I worked at Uber from 2015-2019 on product and platform teams and had several close colleagues in infra.

pas 7y ago
Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39 https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-and-version-4-wishlist-vinay-chella https://www.datastax.com/resources/video/cassandra-netflix-a...
- roskilli 7y ago
  Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes). https://github.com/Netflix/atlas https://github.com/Netflix/atlas
  - ckdarby 7y ago
    > which was too expensive for Uber to run (since the dataset is in petabytes). Ok, but I am fairly confident Netflix also is at that kind of scale. Netflix has a section on Atlas's documentation about how they get around this: https://github.com/Netflix/atlas/wiki/Overview#cost https://github.com/Netflix/atlas/wiki/Overview#cost They also did this nice video that outlines their entire operation including how they do rollups: https://www.youtube.com/watch?v=4RG2DUK03_0 https://www.youtube.com/watch?v=4RG2DUK03_0 This is how they do the rollup but keep their tails accurate to parts per million and the middle to be parts per hundred: https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest
    - roskilli 7y ago
      I want to first say, I have a great amount of respect for Netflix's engineering and for Atlas itself, it's great that it exists and is more accessible than other scalable in-memory TSDBs open sourced by large companies. A few of my thoughts on this, and this has come up before. Firstly Netflix self-identifies it is expensive to run an in-memory TSDB for metrics - for instance Roy's talk on Atlas mentions this as such[0] at the 37min mark of his Operations Engineering talk "It scales kind of efficiently. I'd love to say efficiently instead of efficiently-ish however that's hard to claim when my platform until this last quarter cost Netflix more than any other element of the cloud ecosystem ... Atlas and the associated telemetry costs Netflix 100s of thousands of dollars a week". At Uber M3 cost a significant amount to run as well at first and that is why M3DB was born to drive down that cost as much as it could and still provide a ton of instrumentation to engineers. Either way, giving engineers tons of room to instrument their code will result in a high cost no matter what since it will be viewed as a free lunch, that is why squeezing the economics on this matters since you want to provide as much instrumentation as possible at the lowest cost. Regarding your points about their documentation on cost: 1) Yes reducing cardinality by dropping node dimension on metrics, etc is possible to save cost - but also keeping things on disk is an alternate and great way to save cost too and keep the data at high fidelity. The challenge is making on disk lookup fast too, which with M3DB is what we were focused on doing. 2) Dropping replication of the data to a single replica is another way to save cost, however also comes with operational complexity as now you need to do backup/restore if you lose data and lose the ability to query that data in the meantime. This is why M3DB always is recommended (as per documentation) to run at RF=3 with quorum reads and writes so losing a single machine does not impact the availability of your operational monitoring and alerting platform. 3) Regarding rollups and tail solutions accurate, we always push for people to use histograms as that can be aggregated over any arbitrary time window and across time series. T-Digests are much more expensive to store raw and aggregate later. Bjorn talked about histograms, their use in Prometheus at FOSDEM[1] and why they're more desirable than t-digests or other similar aggregations. [0]: https://www.infoq.com/presentations/netflix-monitoring-system/ https://www.infoq.com/presentations/netflix-monitoring-syste... (video, quote is at 37minutes in) [1]: https://fosdem.org/2020/schedule/event/histograms/ https://fosdem.org/2020/schedule/event/histograms/ (slides and videos)
- remote_phone 7y ago
  It was an epic failure because you need a team to support and guide Cassandra use properly but no one wanted to do the grunt work. The VP of infrastructure MM openly called it “toil vs talent”, meaning those that did the grunt work would be held in high esteem and get yearly bonuses, but the promotions would go to those with “talent”, ie creating new things. When people are openly and stupidly incentivized like this, expect those people to behave in a predictable way. People started building new services to get promotions instead of “toiling” at supporting their fellow engineers. It affected most of engineering but especially in teams like Cassandra, where you needed guidance and support to properly use it effectively, it was a disaster. There should have been open office hours to help people with questions and to ensure that teams were using it properly but there wasn’t. Instead people were left to do what they wanted with no structure or guidance and Cassandra was completely misused. Productions problems ensued, people left the team because they didn’t want to be oncall fixing fires all the time, and eventually it came to the point where they decided to stop supporting it altogether. It was a complete disaster caused by very poor engineering management. We all knew that Netflix and Facebook use it without issues, but because of stupid management, it failed at Uber.
  - pas 7y ago
    Oh, wow, that sounds worse than bad. Thanks for elaborating on the "root cause" too!
- deleted 7y ago
  [deleted]

pas 7y ago

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39 https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-and-version-4-wishlist-vinay-chella https://www.datastax.com/resources/video/cassandra-netflix-a...

roskilli 7y ago
Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes). https://github.com/Netflix/atlas https://github.com/Netflix/atlas
- ckdarby 7y ago
  > which was too expensive for Uber to run (since the dataset is in petabytes). Ok, but I am fairly confident Netflix also is at that kind of scale. Netflix has a section on Atlas's documentation about how they get around this: https://github.com/Netflix/atlas/wiki/Overview#cost https://github.com/Netflix/atlas/wiki/Overview#cost They also did this nice video that outlines their entire operation including how they do rollups: https://www.youtube.com/watch?v=4RG2DUK03_0 https://www.youtube.com/watch?v=4RG2DUK03_0 This is how they do the rollup but keep their tails accurate to parts per million and the middle to be parts per hundred: https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest
  - roskilli 7y ago
    I want to first say, I have a great amount of respect for Netflix's engineering and for Atlas itself, it's great that it exists and is more accessible than other scalable in-memory TSDBs open sourced by large companies. A few of my thoughts on this, and this has come up before. Firstly Netflix self-identifies it is expensive to run an in-memory TSDB for metrics - for instance Roy's talk on Atlas mentions this as such[0] at the 37min mark of his Operations Engineering talk "It scales kind of efficiently. I'd love to say efficiently instead of efficiently-ish however that's hard to claim when my platform until this last quarter cost Netflix more than any other element of the cloud ecosystem ... Atlas and the associated telemetry costs Netflix 100s of thousands of dollars a week". At Uber M3 cost a significant amount to run as well at first and that is why M3DB was born to drive down that cost as much as it could and still provide a ton of instrumentation to engineers. Either way, giving engineers tons of room to instrument their code will result in a high cost no matter what since it will be viewed as a free lunch, that is why squeezing the economics on this matters since you want to provide as much instrumentation as possible at the lowest cost. Regarding your points about their documentation on cost: 1) Yes reducing cardinality by dropping node dimension on metrics, etc is possible to save cost - but also keeping things on disk is an alternate and great way to save cost too and keep the data at high fidelity. The challenge is making on disk lookup fast too, which with M3DB is what we were focused on doing. 2) Dropping replication of the data to a single replica is another way to save cost, however also comes with operational complexity as now you need to do backup/restore if you lose data and lose the ability to query that data in the meantime. This is why M3DB always is recommended (as per documentation) to run at RF=3 with quorum reads and writes so losing a single machine does not impact the availability of your operational monitoring and alerting platform. 3) Regarding rollups and tail solutions accurate, we always push for people to use histograms as that can be aggregated over any arbitrary time window and across time series. T-Digests are much more expensive to store raw and aggregate later. Bjorn talked about histograms, their use in Prometheus at FOSDEM[1] and why they're more desirable than t-digests or other similar aggregations. [0]: https://www.infoq.com/presentations/netflix-monitoring-system/ https://www.infoq.com/presentations/netflix-monitoring-syste... (video, quote is at 37minutes in) [1]: https://fosdem.org/2020/schedule/event/histograms/ https://fosdem.org/2020/schedule/event/histograms/ (slides and videos)
remote_phone 7y ago
It was an epic failure because you need a team to support and guide Cassandra use properly but no one wanted to do the grunt work. The VP of infrastructure MM openly called it “toil vs talent”, meaning those that did the grunt work would be held in high esteem and get yearly bonuses, but the promotions would go to those with “talent”, ie creating new things. When people are openly and stupidly incentivized like this, expect those people to behave in a predictable way. People started building new services to get promotions instead of “toiling” at supporting their fellow engineers. It affected most of engineering but especially in teams like Cassandra, where you needed guidance and support to properly use it effectively, it was a disaster. There should have been open office hours to help people with questions and to ensure that teams were using it properly but there wasn’t. Instead people were left to do what they wanted with no structure or guidance and Cassandra was completely misused. Productions problems ensued, people left the team because they didn’t want to be oncall fixing fires all the time, and eventually it came to the point where they decided to stop supporting it altogether. It was a complete disaster caused by very poor engineering management. We all knew that Netflix and Facebook use it without issues, but because of stupid management, it failed at Uber.
- pas 7y ago
  Oh, wow, that sounds worse than bad. Thanks for elaborating on the "root cause" too!
deleted 7y ago
[deleted]