6 ms·

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely w

by tnolet 7y ago

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

buro9 7y ago
It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things that seem to truly work at every scale, Kafka and ClickHouse come to mind for wholly different use cases than a time series database.
- 1996 7y ago
  > ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty
  - valyala 7y ago
    There is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282 https://medium.com/@valyala/how-victoriametrics-makes-instan...
    - 1996 7y ago
      "-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month." Not sure I want that in a TSDB! "Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur." Ouch. But ok, disk space is cheap. The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.
- manigandham 7y ago
  Clickhouse (and other columnstore RDBMS) are all perfectly fine for time-series and usually better than the standard options because they have SQL querying.
- linuxhansl 7y ago
  We run OpenTSDB (which stores its data in Apache HBase) at scale. 150m-200m events/minute and about 20-30 trillion (10^12) events stored. Doubling about every 12-18 months or so. While it's true that things start to creak at scale, this has worked remarkably well for us so far. I doubt M3DB is somehow magical in this regard.
  - roskilli 7y ago
    M3DB ingested 30 million datapoints per second (so 1.8 billion per minute) with each node writing hundreds of thousands of writes per second. The dataset was in the petabytes. For us the cost savings vs OpenTSDB (millions of dollars of hardware), the faster query time and the reduction in oncall overhead was worthwhile.
    - shaklee3 7y ago
      Hundreds of thousands per second isn't very high when you compare that to clickhouse or kdb+.
hkarthik 7y ago
I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB and DynamoDB. We ended up doing a lot of sharded MySQL in our own data centers instead. This on-prem decision led to a lot challenges internally where we would adopt OSS and then have difficulty scaling it to our needs. For some tech like Kafka it worked out, and we hired Kafka contributors who helped us scale it. For other tech like Cassandra it was a pretty epic failure. I am sure more of these war stories exist that I wasn’t privy to myself. Coupled with the fact that we were early adopters into Golang which had its own OSS ecosystem, we found that writing a lot of our own infrastructure solutions was the only viable option at our scale. What you are seeing now is a lot of that home grown infrastructure being open sourced in big way as people who have left Uber continue to see value in investing in the tech that they worked so hard to build. There is probably a nontrivial amount of work to scale the Uber OSS down for smaller use cases but some startups are emerging to make that happen. Source: I worked at Uber from 2015-2019 on product and platform teams and had several close colleagues in infra.
- pas 7y ago
  Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39 https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-and-version-4-wishlist-vinay-chella https://www.datastax.com/resources/video/cassandra-netflix-a...
  - roskilli 7y ago
    Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes). https://github.com/Netflix/atlas https://github.com/Netflix/atlas
roskilli 7y ago
As per sibling comment, they do most definitely work until they don’t. M3 actually started with ElasticSearch and Cassandra for index and storage respectively but then were replaced with M3DB. I mentioned the FOSDEM talk elsewhere in the thread but you might be interested in the evolution segment where it’s mentioned ”With M3DB 7x less servers from Cassandra, while increasing RF=2 to RF=3” and something that’s not on the slides but is in the talk is a reference to an order of magnitude reduction in operational overhead (incidents/oncall debugging). Both slides and video is linked from the FOSDEM talk’s page https://fosdem.org/2020/schedule/event/m3db/ https://fosdem.org/2020/schedule/event/m3db/.
whyreplicate 7y ago
Cassandra and ElasticSearch would probably have been fine except that Uber dramatically under-provisioned the hardware used for them. The database redundancy was so low that any minor hardware issue could quickly turn into a major outage for all of Uber's monitoring services.
- roskilli 7y ago
  Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware. It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.
  - apta 7y ago
    What do you think of ScyllaDB then? Would it have been able to do the job
    - roskilli 7y ago
      I think ScyllaDB would have definitely done better than Cassandra (which we were using alongside ElasticSearch), although another thing I mention in this thread is that a lot of existing distributed databases do not have a multi-dimensional inverted index available that can index keys in there primary storage engine. This makes it tough to use solely either ScyllaDB, ClickHouse or Cassandra for that matter for metrics workloads at scale since they need to find a needle in a haystack - a few thousand time series amongst a set of millions to billions, where users only specify a subset of the dimensions on the metrics in any order they want to. This is hard to do without an inverted index.
cube2222 7y ago
Having deployed m3 recently, I’ve not found an alternative which is cost effective and fast at the same time. Granted, it uses a lot of memory, but other than that I’ve been incredibly happy with it.
- hagen1778 7y ago
  Since you've done your research, would you mind to post a short list of alternatives and reasons why have they been rejected? Thanks!
  - cube2222 7y ago
    This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it. Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element. Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it. M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that. For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.
jandrewrogers 7y ago
It is relatively common for companies to not use open source for this type of application above a certain scale, even if they use open source for most other things. I've seen it happen at multiple companies big and small. There are two major reasons for this. The first reason is that open source platforms struggle beyond a certain scale due to architectural weaknesses, which becomes an ongoing operational headache. Most companies just deal with it but it gets worse as the workload grows. The second reason is that it is expensive to run the open source platforms due to their very low efficiency. I've seen companies reduce their hardware footprint by a factor of 10 by rolling their own metrics/time-series implementations due solely to superior software design. When you are running a petabyte of metrics per day through these systems, that adds up to a lot of money. tl;dr: it is technically straightforward for a company to design their own metrics infrastructure that massively outperforms the open source tooling, and the limitations of the open source implementations are often painful enough as the data models scale up that many companies do.
manigandham 7y ago
Any columnstore RDBMS would work great. Clickhouse, MemSQL, KDB, Greenplum, Vertica, etc. Fast, efficient, and with the full flexibility of SQL queries.

by tnolet 7y ago

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

buro9 7y ago

It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things that seem to truly work at every scale, Kafka and ClickHouse come to mind for wholly different use cases than a time series database.

1996 7y ago
> ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty
- valyala 7y ago
  There is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282 https://medium.com/@valyala/how-victoriametrics-makes-instan...
  - 1996 7y ago
    "-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month." Not sure I want that in a TSDB! "Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur." Ouch. But ok, disk space is cheap. The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.
manigandham 7y ago
Clickhouse (and other columnstore RDBMS) are all perfectly fine for time-series and usually better than the standard options because they have SQL querying.
linuxhansl 7y ago
We run OpenTSDB (which stores its data in Apache HBase) at scale. 150m-200m events/minute and about 20-30 trillion (10^12) events stored. Doubling about every 12-18 months or so. While it's true that things start to creak at scale, this has worked remarkably well for us so far. I doubt M3DB is somehow magical in this regard.
- roskilli 7y ago
  M3DB ingested 30 million datapoints per second (so 1.8 billion per minute) with each node writing hundreds of thousands of writes per second. The dataset was in the petabytes. For us the cost savings vs OpenTSDB (millions of dollars of hardware), the faster query time and the reduction in oncall overhead was worthwhile.
  - shaklee3 7y ago
    Hundreds of thousands per second isn't very high when you compare that to clickhouse or kdb+.

hkarthik 7y ago

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB and DynamoDB. We ended up doing a lot of sharded MySQL in our own data centers instead. This on-prem decision led to a lot challenges internally where we would adopt OSS and then have difficulty scaling it to our needs. For some tech like Kafka it worked out, and we hired Kafka contributors who helped us scale it. For other tech like Cassandra it was a pretty epic failure. I am sure more of these war stories exist that I wasn’t privy to myself. Coupled with the fact that we were early adopters into Golang which had its own OSS ecosystem, we found that writing a lot of our own infrastructure solutions was the only viable option at our scale. What you are seeing now is a lot of that home grown infrastructure being open sourced in big way as people who have left Uber continue to see value in investing in the tech that they worked so hard to build. There is probably a nontrivial amount of work to scale the Uber OSS down for smaller use cases but some startups are emerging to make that happen. Source: I worked at Uber from 2015-2019 on product and platform teams and had several close colleagues in infra.

pas 7y ago
Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39 https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-and-version-4-wishlist-vinay-chella https://www.datastax.com/resources/video/cassandra-netflix-a...
- roskilli 7y ago
  Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes). https://github.com/Netflix/atlas https://github.com/Netflix/atlas

roskilli 7y ago

As per sibling comment, they do most definitely work until they don’t. M3 actually started with ElasticSearch and Cassandra for index and storage respectively but then were replaced with M3DB. I mentioned the FOSDEM talk elsewhere in the thread but you might be interested in the evolution segment where it’s mentioned ”With M3DB 7x less servers from Cassandra, while increasing RF=2 to RF=3” and something that’s not on the slides but is in the talk is a reference to an order of magnitude reduction in operational overhead (incidents/oncall debugging). Both slides and video is linked from the FOSDEM talk’s page https://fosdem.org/2020/schedule/event/m3db/ https://fosdem.org/2020/schedule/event/m3db/.

whyreplicate 7y ago

Cassandra and ElasticSearch would probably have been fine except that Uber dramatically under-provisioned the hardware used for them. The database redundancy was so low that any minor hardware issue could quickly turn into a major outage for all of Uber's monitoring services.

roskilli 7y ago
Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware. It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.
- apta 7y ago
  What do you think of ScyllaDB then? Would it have been able to do the job
  - roskilli 7y ago
    I think ScyllaDB would have definitely done better than Cassandra (which we were using alongside ElasticSearch), although another thing I mention in this thread is that a lot of existing distributed databases do not have a multi-dimensional inverted index available that can index keys in there primary storage engine. This makes it tough to use solely either ScyllaDB, ClickHouse or Cassandra for that matter for metrics workloads at scale since they need to find a needle in a haystack - a few thousand time series amongst a set of millions to billions, where users only specify a subset of the dimensions on the metrics in any order they want to. This is hard to do without an inverted index.

cube2222 7y ago

Having deployed m3 recently, I’ve not found an alternative which is cost effective and fast at the same time. Granted, it uses a lot of memory, but other than that I’ve been incredibly happy with it.

hagen1778 7y ago
Since you've done your research, would you mind to post a short list of alternatives and reasons why have they been rejected? Thanks!
- cube2222 7y ago
  This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it. Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element. Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it. M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that. For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.

jandrewrogers 7y ago

It is relatively common for companies to not use open source for this type of application above a certain scale, even if they use open source for most other things. I've seen it happen at multiple companies big and small. There are two major reasons for this. The first reason is that open source platforms struggle beyond a certain scale due to architectural weaknesses, which becomes an ongoing operational headache. Most companies just deal with it but it gets worse as the workload grows. The second reason is that it is expensive to run the open source platforms due to their very low efficiency. I've seen companies reduce their hardware footprint by a factor of 10 by rolling their own metrics/time-series implementations due solely to superior software design. When you are running a petabyte of metrics per day through these systems, that adds up to a lot of money. tl;dr: it is technically straightforward for a company to design their own metrics infrastructure that massively outperforms the open source tooling, and the limitations of the open source implementations are often painful enough as the data models scale up that many companies do.

manigandham 7y ago

Any columnstore RDBMS would work great. Clickhouse, MemSQL, KDB, Greenplum, Vertica, etc. Fast, efficient, and with the full flexibility of SQL queries.