25 ms·

M3DB, a distributed timeseries database

320 points by Anon84 7y ago

missosoup 7y ago
Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.
- gtirloni 7y ago
  It's opensource. Why should Uber give any guarantees? They are not in the business of selling software. Unless Uber is actively blocking contributions, it's not Uber's fault if no community formed around something they opensourced. As for this being a PR piece, they could have achieved the same with just a detailed blog post and no code. It looks like a expensive PR piece if they have to opensource work that took probably hundreds of development hours.
  - missosoup 7y ago
    I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that resembles their own operating model. Maybe this project will set a new trend, but so far nothing they put out gained any traction or became suitable for general purpose production use. In that regard, H3 and their other projects have remained at the level of decent 'show HN' pieces rather than something you'd ever use professionally. In other words, marketing. Based on previous news coming out e.g. https://news.ycombinator.com/item?id=20931644 https://news.ycombinator.com/item?id=20931644 It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.
    - mamon 7y ago
      Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)
      - cfors 7y ago
        Maybe I’m a cynic of all big corp companies but if you’ve ever worked with a big corp open source department that’s almost the entire point is to build PR for the engineering department. Same goes for tech blogs. These things will be PR pieces first, and valid production tools/frameworks second (mostly).
        closeparen 7y ago
        In my experience software gets written in the first place for the usual internal reasons. Corporate or individual prestige may be the driving factor in open sourcing, though, rather than a genuine interest in having it used externally.
      - lopsidedBrain 7y ago
        Pretty much every successful open source project that people pay attention to is one that has had long-term support behind it. Linux, Mozilla, gcc, clang, git. Almost always, that support begins with the original author. Protects that don't do that are therefore unlikely to remain interesting for long.
    - carlisle_ 7y ago
      >It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case. Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed. Whether or not the work we were undertaking was meritious and valuable is an entire branch of philosophy I'm pretty sure. Part of the struggle at big companies is that a lot of existing solutions just don't work. Let me use an example with chat. A few years ago Slack was evaluated as a replacement for HipChat, since Atlassian's outages had finally started affecting us during our own outages. Everybody wanted to go to Slack, but the cost of Slack was tremendously prohibitive and the state of the service then (as I was told) was such that it could not support a company of Uber's size. Tremendous effort would have been undertaken by Slack to support Uber and they didn't want to expend that effort for a single customer. This was late 2015 early 2016. There were tons of options, but ultimately an in-house chat software was created. At the time it seemed required to make our own highly reliable chat, considering how distributed engineering teams were. I think if you talk to anybody without the background of how chat evolved at Uber they would think the in-house chat project would have been a boondoggle. Not all over-scoped engineering projects are actually so noble. There was certainly a ton of "reinventing wheels" going on. There was significantly more "these problems are really hard and I only have bad solutions." Though if the result is ultimately, "nothing more than interesting repos amongst a sea of interesting repos" sign me up.
  - _jal 7y ago
    > Why should Uber give any guarantees? First, "guarantee" is the wrong word to take too literally here. Depending on how you want to look at it, there are no guarantees, even with guarantees. But looked at more loosely, answering that is really Uber's responsibility. Why did they release it? If it is just a PR release, fire-and-forget works fine for that. If they want to see wider adoption outside of their firm, there are some fairly obvious things they should do to foster that. Sometimes you release just the right thing at just the right time and everyone else does your evangelism and support work for you, but it is much more normal for your next great thing to take a while to build a user base.
    - tylerl 7y ago
      There are lots of reasons to open source internal software, and only a minority of them involve establishing a serious community and driving significant adoption. But the PR claim you're making isn't particularly credible. The ROI is abysmal if that's all you're after, and there are easier ways to get it. Based on your specific complaints, it sounds like your opinion doesn't matter in this case; you're not the audience. You want support and a predictable future: you're looking for a product, not for technology. This isn't a product, and it's not a platform. If instead you represented another company looking into solving this same problem yourself, and are looking at starting points, then you're the perfect audience. In that case, you'd have time and motivation to contact the developers directly rather than gripe on HN. You'd be less interested in whether there was an organized community, and more interested in how to directly influence the roadmap. You'd care about what the code looks like, how they solved Problem X and Problem Y, that kind of thing.
  - deleted 7y ago
    [deleted]
  - ForHackernews 7y ago
    Everything you say is true, but tossing useless code releases over the wall isn't really participating in the open source community, either. It looks to me like maybe their engineers internally are fans of the idea of "open source", and the PR department is happy to try and get some good press out of it, but the company culture isn't really set up to develop in public or maintain these things they've nominally "released". Sadly, this isn't unusual among tech companies, but it'd be more obvious what's happening if they just put up a bare-bones FTP with a README: "Here's some code under <LICENSE>. Use it at your own risk."
  - papito 7y ago
    Even I have half a mind about releasing something that I cannot commit to for a little bit - the initial bugfix stage, at least. I would say a giant like Uber hurts themselves more in terms of PR when not being conservative enough about putting source out there. People inevitably gravitate towards big player "open sauce" as it implies some commitment. A company should first put their own system through hell and decide that "yeah, this is good, we are sticking with this", before luring people to use it.
  - grogenaut 7y ago
    Open source for the steward is limping the project along is the worst type of open source because the steward usually isn't going to make any real decisions around the project the people are reticent to fork it and drive it because there is a Steward doing some activity. Selenium was in the state for years.
  - Udik 7y ago
    > They are not in the business of selling software. But they are probably in the business of selling themselves as a software/ tech company. People value tech companies, so you'd better be one, even if you do taxi services, produce and distribute tv series, or rent office space.
- reichardt 7y ago
  Why do you consider H3 to be on life support? It's basically a finished spec with actively developed implementations. https://github.com/uber/h3 https://github.com/uber/h3
  - deleted 7y ago
    [deleted]
  - ForHackernews 7y ago
    So is Gopher, but almost nobody uses it.
- richieartoul 7y ago
  Chronosphere, a startup founded by two of the early M3 engineers, just raised 11 million dollars to build a monitoring platform based around M3DB: https://techcrunch.com/2019/11/05/chronosphere-launches-with-11m-series-a-to-build-scalable-cloud-native-monitoring-tool https://techcrunch.com/2019/11/05/chronosphere-launches-with... Uber also uses M3DB extensively internally and the project is nowhere near being abandoned or on life support: https://github.com/m3db/m3/commits/master https://github.com/m3db/m3/commits/master
  - bfung 7y ago
    Yep, another way to pre-market/pre-signal to investors. A pretty common story these days: 1. I built an X to solve Unicorn U’s problem. 2. Open source it, give talks on it 3. Leave Unicorn U and start a company based on X 4. ... 5. Profit(???) Also, debatable whether or not Unicorn U actually needed a freshly built X instead of using existing tools/tech.
    - sieabahlpark 7y ago
      Seems genius imo
  - StreamBright 7y ago
    >> While the founders, CEO Martin Mao and CTO Rob Skillington, were working at Uber, they recognized a gap in the monitoring industry, particularly around cloud-native technologies like containers and microservices What is the actual gap that is not addressed with one or all of the following? - Prometheus / Grafana [1] - Datadog [2] - Cloudwatch [3] 1. https://docs.docker.com/config/thirdparty/prometheus/ https://docs.docker.com/config/thirdparty/prometheus/ 2. https://www.datadoghq.com/blog/introducing-live-container-monitoring/ https://www.datadoghq.com/blog/introducing-live-container-mo... 3. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
    - opsunit 7y ago
      Wavefront: https://www.wavefront.com/ https://www.wavefront.com/
- api 7y ago
  Many large companies push this stuff out for publicity and recruitment. Sometimes employees are encouraged to spend a little bit of their time on it or to brand extracurricular activities with the company name for publicity. The test for open source is if it keeps getting maintained and supported for years. That only happens when the project is a core business effort, has some direct means of support (e.g. dual licensing or SaaS), or happens to be one of the few genuinely volunteer driven large scale open source projects.
- scrappyjoe 7y ago
  Huh? The last commit to the H3 github repo was 3 days ago. In what sense is it abandoned? Genuinely interested as we are considering using it as a core library.
- ajfriend 7y ago
  We're definitely still working on H3. We just got a nice, new domain: h3geo.org We're also basically done with a new Python wrapper written in Cython. https://github.com/uber/h3-py/tree/cython https://github.com/uber/h3-py/tree/cython We could probably use some help with the last step of packaging, if anyone is interested!
- excerionsforte 7y ago
  https://github.com/facebookarchive https://github.com/facebookarchive - 11 Pages of unsupported open source software. Beringei, a TSDB, (https://github.com/facebookarchive/beringei https://github.com/facebookarchive/beringei) in particular with what you are saying was a PR piece (https://engineering.fb.com/core-data/beringei-a-high-performance-time-series-storage-engine/ https://engineering.fb.com/core-data/beringei-a-high-perform...) since it was never really used by anyone outside of FB. I really don't see the negative part of free code that you can learn from and/or incorporate at all.
  - Fellshard 7y ago
    Free code is good, yes. When you rely on the systems themselves, and do so with expectation of support from the originating company, your expectations will almost certainly be broken. I think that's the simplest takeaway - not to run away from any open-sourced project, but to take into proper consideration if/how they plan on supporting the tool, and how much you would be capable of adapting and owning yourself if the worst happened.
    - excerionsforte 7y ago
      Yeah exactly. If one wants support they can pay for it i.e. SaaS if available. If an open source project has not created a contract with any user then there is no guarantee of support. I don't believe any company creates a contract with users automatically because they made source code available. That is unsustainable. Chronosphere is the SaaS part for M3DB in this case. The negativity around someone open sourcing code for PR is nuts especially when all of the code is available. I love reading the code and getting ideas about how things work.
- iblaine 7y ago
  Open sourcing projects is the new merit badge for engineers. But at least there’s more good than bad from it. Hudi is at least one Uber project I can point to off the top of my head that is a great idea.
- parentheses 7y ago
  The issue here is not the company but the fact that the owners of the original library did not figure out how to “disown” the library. Open sourcing something is naturally more expensive than not. It’s seldom the case that impact to the community triggers contributions that outweigh that cost. The fallacy we hold is that companies will prop up software that is open source for everyone to use despite the lacking community contribution. We as engineers should push ourselves to contribute when we find issues - rather than simply create tickets that represent work were want to have done for free. This is how open source software dies. There is a minority that does this.
- throwaway5752 7y ago
  Lots of good open source projects fail. But not every company is willing to open source code like this, though, and I'm very happy that Uber did so in this case. I get your frustration, but everyone should remember there are never any promises of support with open source software, regardless of how well supported it is at a particular time.
roskilli 7y ago
Thanks for the interest, I just did a talk at FOSDEM a few weeks ago on the subject of querying over large datasets that M3DB can warehouse and query in real-time here: Slides https://fosdem.org/2020/schedule/event/m3db/attachments/audio/4032/export/events/attachments/m3db/audio/4032/FOSDEM_2020_Querying_millions_to_billions_of_metrics_with_M3DBs_inverted_index.pdf https://fosdem.org/2020/schedule/event/m3db/attachments/audi... Video https://video.fosdem.org/2020/UD2.120/m3db.mp4 https://video.fosdem.org/2020/UD2.120/m3db.mp4
clircle 7y ago
Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.
- idunno246 7y ago
  A db that backs lots of time based graphs. They generally hit some pathological cases for general dbs. Frequent write of small data, most recent time is very hot so sharding is tricky, queries generally pull lots of little bits of data in from long time ranges, resolution of past data can often be lowered, etc. Opentsdb for instance was built on top of Hbase because implementing one naively in hbase hits tons of performance issues.
- steve_adams_86 7y ago
  There many flavours of time series databases, but my understanding is that at their core they're optimised for storing and querying based on time stamp/time series data. Typically very quickly in large volumes. That's maybe the primary criteria to define a database as a time series database. Someone could probably elaborate on this a massive amount. I'm sure there is some nuance and a lot of relevant details around how that optimization is done.
- namanaggarwal 7y ago
  It's not a marketing term. It's a database optimised for storing and querying time based metrics. Uber didn't invent the term, there are a lot of existing products in market. Question is why none of them worked for them. I have used OpenTSDB and it worked great at Mastercard scale. What issues did Uber had?
  - rossjudson 7y ago
    "Mastercard scale" doesn't mean anything in particular, unless you quantify it. How many metrics? Write rate? Query rate? Query complexity?
- abvdasker 7y ago
  A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_series_database https://en.m.wikipedia.org/wiki/Time_series_database
tnolet 7y ago
I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.
- buro9 7y ago
  It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things that seem to truly work at every scale, Kafka and ClickHouse come to mind for wholly different use cases than a time series database.
  - 1996 7y ago
    > ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty
    - valyala 7y ago
      There is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282 https://medium.com/@valyala/how-victoriametrics-makes-instan...
      - 1996 7y ago
        "-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month." Not sure I want that in a TSDB! "Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur." Ouch. But ok, disk space is cheap. The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.
MichaelRazum 7y ago
Ok everything open source was not good enough. Please make a simple benchmark. Without it, it is so hard to make decisions
- hagen1778 7y ago
  I'm aware of only one public benchmark including some competitors - https://promcon.io/2019-munich/talks/remote-write-storage-wars/ https://promcon.io/2019-munich/talks/remote-write-storage-wa... Would like to see more of this.
TheRealPomax 7y ago
admins/mods: this needs an apostrophe to turn it into "Uber's M3DB". For anyone who's never heard of M3DB, and lives in a place where Uber doesn't operatore or is even banned (and so isn't part of daily life or conversation) "Ubers" might just as easily be some db researcher affiliated with the university of who knows where showing off something they came up with last summer and got a grant for.
- tlb 7y ago
  Fixed, thanks
jmakov 7y ago
So how does this compare to e.g. Clickhouse?
- bdcravens 7y ago
  Clickhouse is an analytic column-based RDBMS. It's not a timeseries database. Each class of product is used to solve different problems.
  - aeyes 7y ago
    Clickhouse works exceptionally well as a TSDB.
    - jmakov 7y ago
      That is also my experience. Also in bencmark it is almost as fast as GPU analytical DBs or KDB+.
    - roskilli 7y ago
      While this is true, for a metrics workload it does not work great I have both seen and heard from others, mainly due to the fact it does not have an inverted index - so finding a small subset of metrics in a dataset of billions of metrics ends up taking significant time due to the scan required to find the timeseries matching the arbitrary number of dimensions specified to find the timeseries you're looking for. If you're building it with a specific application and a concrete schema you can create which will result in fast queries and don't have requirements for arbitrary dimensions being specified for lookup, then yes it's great as a TSDB. Prometheus, M3DB, etc all use an inverted index alongside the column store TSDB to help with metrics workloads.
      - mbell 7y ago
        Most practical applications using Clickhouse for metrics data store the metric index separately. What index you want really depends on the metric system, e.g. with graphite data you don't want an inverted index, you want a trie.
heliodor 7y ago
When the Android app is broken in so many easy-to-fix ways that blatantly interfere with usability, how does a company allow its developers to spend time on making custom internal tools or even spend time open-sourcing them? The company has so much money and yet seems so utterly mismanaged.
- rossjudson 7y ago
  Sounds like off-the-shelf tooling just didn't work. What's your solution for that?
  - heliodor 7y ago
    In the grander scheme that I'm discussing, the solution you're asking about is to gather no more metrics than the off-the-shelf tools allow and spend the engineering effort on fixing the simple bugs that prevent customers from getting cars. Why do they need a perfect ton of metrics while they can't fix simple things such as: - the car icon doesn't move as location updates come in (as evidenced by the route line getting shorter) - the on-screen keyboard does not allow me to type anything after the first message (no other app on my phone has this problem) - after I rate a driver, the app shows a map and none of the UI. I have to kill the app and restart it in order for it to be usable again. Picture this: I'm trying to get a ride, I open the app, I get nagged to rate the last driver. I agree just to be nice to the driver. (I should skip instead and get back to my task of getting a ride). After putting up with the nagware, the app fails 100% and I cannot complete the original task! Alternatively, if they demand the collection of so many metrics, why don't they collect the metrics that would show them just how pathetically broken their Android app is?
deleted 7y ago
[deleted]
monstrado 7y ago
On a related note, one of their engineers wrote a POC that uses FoundationDB instead of their custom storage engine. https://github.com/richardartoul/tsdb-layer https://github.com/richardartoul/tsdb-layer The README does a really good job explaining the internals and motivation.
ksec 7y ago
How does it compare to TimescaleDB [1] ? [1] https://www.timescale.com https://www.timescale.com
- akulkarni 7y ago
  TimescaleDB co-founder here. TimescaleDB is a more versatile time-series database. It supports a variety of datatypes (text, ints, floats, arrays, json), allows for out-of-order writes and backfilling of old data, supports full SQL, JOINs between tables (eg for metadata), flexible continuous aggregates, native compression, and is backed by the reliability of Postgres. [0] M3DB seems much more limited in scope [1]: "Current Limitations Due to the nature of the requirements for the project, which are primarily to reduce the cost of ingesting and storing billions of timeseries and providing fast scalable reads, there are a few limitations currently that make M3DB not suitable for use as a general purpose time series database. The project has aimed to avoid compactions when at all possible, currently the only compactions M3DB performs are in-memory for the mutable compressed time series window (default configured at 2 hours). As such out of order writes are limited to the size of a single compressed time series window. Consequently backfilling large amounts of data is not currently possible. The project has also optimized the storage and retrieval of float64 values, as such there is no way to use it as a general time series database of arbitrary data structures just yet." [0] https://www.timescale.com/ https://www.timescale.com/ [1] https://m3db.github.io/m3/m3db/#current-limitations https://m3db.github.io/m3/m3db/#current-limitations
synack 7y ago
I setup a lot of Uber's early metrics infrastructure, so I can speak to how they got to the place where building a custom solution was the right answer. In the beginning, we didn't really have metrics, we had logs. Lots of logs. We tried to use Splunk to get some insight from those. It kinda worked and their sales team initially quoted a high-but-reasonable price for licensing. When we were ready to move forward, the price of the license doubled because they had missed the deadline for their end of quarter sales quota. So we kicked Splunk to the curb. Having seen that the bulk of our log volume was noise and that we really only cared about a few small numbers, I looked for a metrics solution at this point, not a logs solution. I'd operated RRDtool based systems at previous companies, and that worked okay, but I didn't love the idea of doing that again. I had seen Etsy's blog about statsd and setup a statsd+carbon+graphite instance on a single server just to try out and get feedback from the rest of the engineering team. The team very quickly took to Graphite and started instrumenting various codebases and systems to feed metrics into statsd. statsd hit capacity problems first, as it was a single threaded nodejs process and used UDP for ingest, so once it approached 100% CPU utilization, events got dropped. We switched to statsite, which is pretty much a drop-in replacement written in C. The next issue was disk I/O. This was not a surprise. Carbon (Graphite's storage daemon) stores each metric in a separate file in the whisper format, which is similar to RRDtool's files, but implemented in pure Python and generally a bit easier to interact with. We'd expected that a large volume of random write ops on a spinning disk would eventually be a problem. We ordered some SSDs. This worked okay for a while. At this point, the dispatch system was instrumented to store metrics under keys with a lot of dimensions, so that we could generate per-city, per-process, per-handler charts for debugging and performance optimization. While very useful for drilling down to the cause of an issue, this led to an almost exponential growth in the number of unique metrics we were ingesting. I setup carbon-relay to shard the storage across a few servers- I think there were three, but it was a long time ago. We never really got carbon-relay working well. It didn't handle backend outages and network interruptions very well, and would sometimes start leaking memory and crash, seemingly without reason. It limped along for a while, but wasn't going to be a long-term solution. We started looking for alternatives to carbon, as we wanted to get away from whisper files... SSDs were still fairly expensive, and we believed that we should be able to store an append-only dataset on spinning disks and do batch sequential writes. The infrastructure team was still fairly small and we didn't have the resources to properly maintain a HBase cluster for OpenTSDB or a Cassandra cluster, which would've required adapting carbon- I understand that Cassandra is a supported backend these days, but it was just an idea on a mailing list at that point. InfluxDB looked like exactly what we wanted, but it was still in a very early state, as the company had just been formed weeks earlier. I submitted some bug reports but was eventually told by one of the maintainers that it wasn't ready yet and I should quit bugging them so they could get to MVP. Right around this time, we started having serious availability issues with metrics, both on the storage side- I estimated we were dropping about 60% of incoming statsd events, and on the query side- Graphite would take seconds-to-minutes to render some charts and occasionally would just time out. We had also built an ad-hoc system for generating Nagios checks that would poll Graphite every minute to trigger threshold-based alerts, which would make noise if Graphite was down and the monitored system was not. This led to on-call fatigue, which made everybody unhappy. We started running an instance of statsite on every server which would aggregate the individual events for that server into 10 second buckets with the server's hostname as a key prefix, then pushed those to carbon-relay. This solved the dropped packets issue, but carbon-relay was still unreliable. We were pretty entrenched in the statsd+graphite way of doing things at this point, so switching to OpenTSDB wasn't really an option and we'd exhausted all of the existing carbon alternatives, so we started thinking about modifying carbon to use another datastore. The scope of this project was large enough that it wasn't going to get built in a matter of days or weeks, so we needed a stopgap solution to buy time and keep the metrics flowing while we engineered a long term solution. I hacked together statsrelay, which is basically a re-implementation of carbon-relay in C, using libev. At this point, I was burned out and handed off the metrics infrastructure to a few teammates that ran with statsrelay and turned it into a production quality piece of code. Right around the same time, we'd begun hiring for an engineering team in NYC that would take over responsibility for metrics infrastructure. These are the people that eventually designed and built M3DB.
katzgrau 7y ago
Nice... patiently waits for AWS to create a managed version of it...
- staticassertion 7y ago
  https://aws.amazon.com/timestream/ https://aws.amazon.com/timestream/
  - katzgrau 7y ago
    Yep, I registered for the "preview" ages ago, so I figured it was vaporware at this point

320 points by Anon84 7y ago

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

gtirloni 7y ago
It's opensource. Why should Uber give any guarantees? They are not in the business of selling software. Unless Uber is actively blocking contributions, it's not Uber's fault if no community formed around something they opensourced. As for this being a PR piece, they could have achieved the same with just a detailed blog post and no code. It looks like a expensive PR piece if they have to opensource work that took probably hundreds of development hours.
- missosoup 7y ago
  I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that resembles their own operating model. Maybe this project will set a new trend, but so far nothing they put out gained any traction or became suitable for general purpose production use. In that regard, H3 and their other projects have remained at the level of decent 'show HN' pieces rather than something you'd ever use professionally. In other words, marketing. Based on previous news coming out e.g. https://news.ycombinator.com/item?id=20931644 https://news.ycombinator.com/item?id=20931644 It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.
  - mamon 7y ago
    Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)
    - cfors 7y ago
      Maybe I’m a cynic of all big corp companies but if you’ve ever worked with a big corp open source department that’s almost the entire point is to build PR for the engineering department. Same goes for tech blogs. These things will be PR pieces first, and valid production tools/frameworks second (mostly).
      - closeparen 7y ago
        In my experience software gets written in the first place for the usual internal reasons. Corporate or individual prestige may be the driving factor in open sourcing, though, rather than a genuine interest in having it used externally.
    - lopsidedBrain 7y ago
      Pretty much every successful open source project that people pay attention to is one that has had long-term support behind it. Linux, Mozilla, gcc, clang, git. Almost always, that support begins with the original author. Protects that don't do that are therefore unlikely to remain interesting for long.
  - carlisle_ 7y ago
    >It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case. Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed. Whether or not the work we were undertaking was meritious and valuable is an entire branch of philosophy I'm pretty sure. Part of the struggle at big companies is that a lot of existing solutions just don't work. Let me use an example with chat. A few years ago Slack was evaluated as a replacement for HipChat, since Atlassian's outages had finally started affecting us during our own outages. Everybody wanted to go to Slack, but the cost of Slack was tremendously prohibitive and the state of the service then (as I was told) was such that it could not support a company of Uber's size. Tremendous effort would have been undertaken by Slack to support Uber and they didn't want to expend that effort for a single customer. This was late 2015 early 2016. There were tons of options, but ultimately an in-house chat software was created. At the time it seemed required to make our own highly reliable chat, considering how distributed engineering teams were. I think if you talk to anybody without the background of how chat evolved at Uber they would think the in-house chat project would have been a boondoggle. Not all over-scoped engineering projects are actually so noble. There was certainly a ton of "reinventing wheels" going on. There was significantly more "these problems are really hard and I only have bad solutions." Though if the result is ultimately, "nothing more than interesting repos amongst a sea of interesting repos" sign me up.
- _jal 7y ago
  > Why should Uber give any guarantees? First, "guarantee" is the wrong word to take too literally here. Depending on how you want to look at it, there are no guarantees, even with guarantees. But looked at more loosely, answering that is really Uber's responsibility. Why did they release it? If it is just a PR release, fire-and-forget works fine for that. If they want to see wider adoption outside of their firm, there are some fairly obvious things they should do to foster that. Sometimes you release just the right thing at just the right time and everyone else does your evangelism and support work for you, but it is much more normal for your next great thing to take a while to build a user base.
  - tylerl 7y ago
    There are lots of reasons to open source internal software, and only a minority of them involve establishing a serious community and driving significant adoption. But the PR claim you're making isn't particularly credible. The ROI is abysmal if that's all you're after, and there are easier ways to get it. Based on your specific complaints, it sounds like your opinion doesn't matter in this case; you're not the audience. You want support and a predictable future: you're looking for a product, not for technology. This isn't a product, and it's not a platform. If instead you represented another company looking into solving this same problem yourself, and are looking at starting points, then you're the perfect audience. In that case, you'd have time and motivation to contact the developers directly rather than gripe on HN. You'd be less interested in whether there was an organized community, and more interested in how to directly influence the roadmap. You'd care about what the code looks like, how they solved Problem X and Problem Y, that kind of thing.
- deleted 7y ago
  [deleted]
- ForHackernews 7y ago
  Everything you say is true, but tossing useless code releases over the wall isn't really participating in the open source community, either. It looks to me like maybe their engineers internally are fans of the idea of "open source", and the PR department is happy to try and get some good press out of it, but the company culture isn't really set up to develop in public or maintain these things they've nominally "released". Sadly, this isn't unusual among tech companies, but it'd be more obvious what's happening if they just put up a bare-bones FTP with a README: "Here's some code under <LICENSE>. Use it at your own risk."
- papito 7y ago
  Even I have half a mind about releasing something that I cannot commit to for a little bit - the initial bugfix stage, at least. I would say a giant like Uber hurts themselves more in terms of PR when not being conservative enough about putting source out there. People inevitably gravitate towards big player "open sauce" as it implies some commitment. A company should first put their own system through hell and decide that "yeah, this is good, we are sticking with this", before luring people to use it.
- grogenaut 7y ago
  Open source for the steward is limping the project along is the worst type of open source because the steward usually isn't going to make any real decisions around the project the people are reticent to fork it and drive it because there is a Steward doing some activity. Selenium was in the state for years.
- Udik 7y ago
  > They are not in the business of selling software. But they are probably in the business of selling themselves as a software/ tech company. People value tech companies, so you'd better be one, even if you do taxi services, produce and distribute tv series, or rent office space.
reichardt 7y ago
Why do you consider H3 to be on life support? It's basically a finished spec with actively developed implementations. https://github.com/uber/h3 https://github.com/uber/h3
- deleted 7y ago
  [deleted]
- ForHackernews 7y ago
  So is Gopher, but almost nobody uses it.
richieartoul 7y ago
Chronosphere, a startup founded by two of the early M3 engineers, just raised 11 million dollars to build a monitoring platform based around M3DB: https://techcrunch.com/2019/11/05/chronosphere-launches-with-11m-series-a-to-build-scalable-cloud-native-monitoring-tool https://techcrunch.com/2019/11/05/chronosphere-launches-with... Uber also uses M3DB extensively internally and the project is nowhere near being abandoned or on life support: https://github.com/m3db/m3/commits/master https://github.com/m3db/m3/commits/master
- bfung 7y ago
  Yep, another way to pre-market/pre-signal to investors. A pretty common story these days: 1. I built an X to solve Unicorn U’s problem. 2. Open source it, give talks on it 3. Leave Unicorn U and start a company based on X 4. ... 5. Profit(???) Also, debatable whether or not Unicorn U actually needed a freshly built X instead of using existing tools/tech.
  - sieabahlpark 7y ago
    Seems genius imo
- StreamBright 7y ago
  >> While the founders, CEO Martin Mao and CTO Rob Skillington, were working at Uber, they recognized a gap in the monitoring industry, particularly around cloud-native technologies like containers and microservices What is the actual gap that is not addressed with one or all of the following? - Prometheus / Grafana [1] - Datadog [2] - Cloudwatch [3] 1. https://docs.docker.com/config/thirdparty/prometheus/ https://docs.docker.com/config/thirdparty/prometheus/ 2. https://www.datadoghq.com/blog/introducing-live-container-monitoring/ https://www.datadoghq.com/blog/introducing-live-container-mo... 3. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
  - opsunit 7y ago
    Wavefront: https://www.wavefront.com/ https://www.wavefront.com/
api 7y ago
Many large companies push this stuff out for publicity and recruitment. Sometimes employees are encouraged to spend a little bit of their time on it or to brand extracurricular activities with the company name for publicity. The test for open source is if it keeps getting maintained and supported for years. That only happens when the project is a core business effort, has some direct means of support (e.g. dual licensing or SaaS), or happens to be one of the few genuinely volunteer driven large scale open source projects.
scrappyjoe 7y ago
Huh? The last commit to the H3 github repo was 3 days ago. In what sense is it abandoned? Genuinely interested as we are considering using it as a core library.
ajfriend 7y ago
We're definitely still working on H3. We just got a nice, new domain: h3geo.org We're also basically done with a new Python wrapper written in Cython. https://github.com/uber/h3-py/tree/cython https://github.com/uber/h3-py/tree/cython We could probably use some help with the last step of packaging, if anyone is interested!
excerionsforte 7y ago
https://github.com/facebookarchive https://github.com/facebookarchive - 11 Pages of unsupported open source software. Beringei, a TSDB, (https://github.com/facebookarchive/beringei https://github.com/facebookarchive/beringei) in particular with what you are saying was a PR piece (https://engineering.fb.com/core-data/beringei-a-high-performance-time-series-storage-engine/ https://engineering.fb.com/core-data/beringei-a-high-perform...) since it was never really used by anyone outside of FB. I really don't see the negative part of free code that you can learn from and/or incorporate at all.
- Fellshard 7y ago
  Free code is good, yes. When you rely on the systems themselves, and do so with expectation of support from the originating company, your expectations will almost certainly be broken. I think that's the simplest takeaway - not to run away from any open-sourced project, but to take into proper consideration if/how they plan on supporting the tool, and how much you would be capable of adapting and owning yourself if the worst happened.
  - excerionsforte 7y ago
    Yeah exactly. If one wants support they can pay for it i.e. SaaS if available. If an open source project has not created a contract with any user then there is no guarantee of support. I don't believe any company creates a contract with users automatically because they made source code available. That is unsustainable. Chronosphere is the SaaS part for M3DB in this case. The negativity around someone open sourcing code for PR is nuts especially when all of the code is available. I love reading the code and getting ideas about how things work.
iblaine 7y ago
Open sourcing projects is the new merit badge for engineers. But at least there’s more good than bad from it. Hudi is at least one Uber project I can point to off the top of my head that is a great idea.
parentheses 7y ago
The issue here is not the company but the fact that the owners of the original library did not figure out how to “disown” the library. Open sourcing something is naturally more expensive than not. It’s seldom the case that impact to the community triggers contributions that outweigh that cost. The fallacy we hold is that companies will prop up software that is open source for everyone to use despite the lacking community contribution. We as engineers should push ourselves to contribute when we find issues - rather than simply create tickets that represent work were want to have done for free. This is how open source software dies. There is a minority that does this.
throwaway5752 7y ago
Lots of good open source projects fail. But not every company is willing to open source code like this, though, and I'm very happy that Uber did so in this case. I get your frustration, but everyone should remember there are never any promises of support with open source software, regardless of how well supported it is at a particular time.

roskilli 7y ago

Thanks for the interest, I just did a talk at FOSDEM a few weeks ago on the subject of querying over large datasets that M3DB can warehouse and query in real-time here: Slides https://fosdem.org/2020/schedule/event/m3db/attachments/audio/4032/export/events/attachments/m3db/audio/4032/FOSDEM_2020_Querying_millions_to_billions_of_metrics_with_M3DBs_inverted_index.pdf https://fosdem.org/2020/schedule/event/m3db/attachments/audi... Video https://video.fosdem.org/2020/UD2.120/m3db.mp4 https://video.fosdem.org/2020/UD2.120/m3db.mp4

clircle 7y ago

Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.

idunno246 7y ago
A db that backs lots of time based graphs. They generally hit some pathological cases for general dbs. Frequent write of small data, most recent time is very hot so sharding is tricky, queries generally pull lots of little bits of data in from long time ranges, resolution of past data can often be lowered, etc. Opentsdb for instance was built on top of Hbase because implementing one naively in hbase hits tons of performance issues.
steve_adams_86 7y ago
There many flavours of time series databases, but my understanding is that at their core they're optimised for storing and querying based on time stamp/time series data. Typically very quickly in large volumes. That's maybe the primary criteria to define a database as a time series database. Someone could probably elaborate on this a massive amount. I'm sure there is some nuance and a lot of relevant details around how that optimization is done.
namanaggarwal 7y ago
It's not a marketing term. It's a database optimised for storing and querying time based metrics. Uber didn't invent the term, there are a lot of existing products in market. Question is why none of them worked for them. I have used OpenTSDB and it worked great at Mastercard scale. What issues did Uber had?
- rossjudson 7y ago
  "Mastercard scale" doesn't mean anything in particular, unless you quantify it. How many metrics? Write rate? Query rate? Query complexity?
abvdasker 7y ago
A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_series_database https://en.m.wikipedia.org/wiki/Time_series_database

tnolet 7y ago

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

buro9 7y ago
It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things that seem to truly work at every scale, Kafka and ClickHouse come to mind for wholly different use cases than a time series database.
- 1996 7y ago
  > ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty
  - valyala 7y ago
    There is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282 https://medium.com/@valyala/how-victoriametrics-makes-instan...
    - 1996 7y ago
      "-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month." Not sure I want that in a TSDB! "Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur." Ouch. But ok, disk space is cheap. The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.

MichaelRazum 7y ago

Ok everything open source was not good enough. Please make a simple benchmark. Without it, it is so hard to make decisions

hagen1778 7y ago
I'm aware of only one public benchmark including some competitors - https://promcon.io/2019-munich/talks/remote-write-storage-wars/ https://promcon.io/2019-munich/talks/remote-write-storage-wa... Would like to see more of this.

TheRealPomax 7y ago

admins/mods: this needs an apostrophe to turn it into "Uber's M3DB". For anyone who's never heard of M3DB, and lives in a place where Uber doesn't operatore or is even banned (and so isn't part of daily life or conversation) "Ubers" might just as easily be some db researcher affiliated with the university of who knows where showing off something they came up with last summer and got a grant for.

tlb 7y ago
Fixed, thanks

jmakov 7y ago

So how does this compare to e.g. Clickhouse?

bdcravens 7y ago
Clickhouse is an analytic column-based RDBMS. It's not a timeseries database. Each class of product is used to solve different problems.
- aeyes 7y ago
  Clickhouse works exceptionally well as a TSDB.
  - jmakov 7y ago
    That is also my experience. Also in bencmark it is almost as fast as GPU analytical DBs or KDB+.
  - roskilli 7y ago
    While this is true, for a metrics workload it does not work great I have both seen and heard from others, mainly due to the fact it does not have an inverted index - so finding a small subset of metrics in a dataset of billions of metrics ends up taking significant time due to the scan required to find the timeseries matching the arbitrary number of dimensions specified to find the timeseries you're looking for. If you're building it with a specific application and a concrete schema you can create which will result in fast queries and don't have requirements for arbitrary dimensions being specified for lookup, then yes it's great as a TSDB. Prometheus, M3DB, etc all use an inverted index alongside the column store TSDB to help with metrics workloads.
    - mbell 7y ago
      Most practical applications using Clickhouse for metrics data store the metric index separately. What index you want really depends on the metric system, e.g. with graphite data you don't want an inverted index, you want a trie.

heliodor 7y ago

When the Android app is broken in so many easy-to-fix ways that blatantly interfere with usability, how does a company allow its developers to spend time on making custom internal tools or even spend time open-sourcing them? The company has so much money and yet seems so utterly mismanaged.

rossjudson 7y ago
Sounds like off-the-shelf tooling just didn't work. What's your solution for that?
- heliodor 7y ago
  In the grander scheme that I'm discussing, the solution you're asking about is to gather no more metrics than the off-the-shelf tools allow and spend the engineering effort on fixing the simple bugs that prevent customers from getting cars. Why do they need a perfect ton of metrics while they can't fix simple things such as: - the car icon doesn't move as location updates come in (as evidenced by the route line getting shorter) - the on-screen keyboard does not allow me to type anything after the first message (no other app on my phone has this problem) - after I rate a driver, the app shows a map and none of the UI. I have to kill the app and restart it in order for it to be usable again. Picture this: I'm trying to get a ride, I open the app, I get nagged to rate the last driver. I agree just to be nice to the driver. (I should skip instead and get back to my task of getting a ride). After putting up with the nagware, the app fails 100% and I cannot complete the original task! Alternatively, if they demand the collection of so many metrics, why don't they collect the metrics that would show them just how pathetically broken their Android app is?

deleted 7y ago

[deleted]

monstrado 7y ago

On a related note, one of their engineers wrote a POC that uses FoundationDB instead of their custom storage engine. https://github.com/richardartoul/tsdb-layer https://github.com/richardartoul/tsdb-layer The README does a really good job explaining the internals and motivation.

ksec 7y ago

How does it compare to TimescaleDB [1] ? [1] https://www.timescale.com https://www.timescale.com

akulkarni 7y ago
TimescaleDB co-founder here. TimescaleDB is a more versatile time-series database. It supports a variety of datatypes (text, ints, floats, arrays, json), allows for out-of-order writes and backfilling of old data, supports full SQL, JOINs between tables (eg for metadata), flexible continuous aggregates, native compression, and is backed by the reliability of Postgres. [0] M3DB seems much more limited in scope [1]: "Current Limitations Due to the nature of the requirements for the project, which are primarily to reduce the cost of ingesting and storing billions of timeseries and providing fast scalable reads, there are a few limitations currently that make M3DB not suitable for use as a general purpose time series database. The project has aimed to avoid compactions when at all possible, currently the only compactions M3DB performs are in-memory for the mutable compressed time series window (default configured at 2 hours). As such out of order writes are limited to the size of a single compressed time series window. Consequently backfilling large amounts of data is not currently possible. The project has also optimized the storage and retrieval of float64 values, as such there is no way to use it as a general time series database of arbitrary data structures just yet." [0] https://www.timescale.com/ https://www.timescale.com/ [1] https://m3db.github.io/m3/m3db/#current-limitations https://m3db.github.io/m3/m3db/#current-limitations

synack 7y ago

I setup a lot of Uber's early metrics infrastructure, so I can speak to how they got to the place where building a custom solution was the right answer. In the beginning, we didn't really have metrics, we had logs. Lots of logs. We tried to use Splunk to get some insight from those. It kinda worked and their sales team initially quoted a high-but-reasonable price for licensing. When we were ready to move forward, the price of the license doubled because they had missed the deadline for their end of quarter sales quota. So we kicked Splunk to the curb. Having seen that the bulk of our log volume was noise and that we really only cared about a few small numbers, I looked for a metrics solution at this point, not a logs solution. I'd operated RRDtool based systems at previous companies, and that worked okay, but I didn't love the idea of doing that again. I had seen Etsy's blog about statsd and setup a statsd+carbon+graphite instance on a single server just to try out and get feedback from the rest of the engineering team. The team very quickly took to Graphite and started instrumenting various codebases and systems to feed metrics into statsd. statsd hit capacity problems first, as it was a single threaded nodejs process and used UDP for ingest, so once it approached 100% CPU utilization, events got dropped. We switched to statsite, which is pretty much a drop-in replacement written in C. The next issue was disk I/O. This was not a surprise. Carbon (Graphite's storage daemon) stores each metric in a separate file in the whisper format, which is similar to RRDtool's files, but implemented in pure Python and generally a bit easier to interact with. We'd expected that a large volume of random write ops on a spinning disk would eventually be a problem. We ordered some SSDs. This worked okay for a while. At this point, the dispatch system was instrumented to store metrics under keys with a lot of dimensions, so that we could generate per-city, per-process, per-handler charts for debugging and performance optimization. While very useful for drilling down to the cause of an issue, this led to an almost exponential growth in the number of unique metrics we were ingesting. I setup carbon-relay to shard the storage across a few servers- I think there were three, but it was a long time ago. We never really got carbon-relay working well. It didn't handle backend outages and network interruptions very well, and would sometimes start leaking memory and crash, seemingly without reason. It limped along for a while, but wasn't going to be a long-term solution. We started looking for alternatives to carbon, as we wanted to get away from whisper files... SSDs were still fairly expensive, and we believed that we should be able to store an append-only dataset on spinning disks and do batch sequential writes. The infrastructure team was still fairly small and we didn't have the resources to properly maintain a HBase cluster for OpenTSDB or a Cassandra cluster, which would've required adapting carbon- I understand that Cassandra is a supported backend these days, but it was just an idea on a mailing list at that point. InfluxDB looked like exactly what we wanted, but it was still in a very early state, as the company had just been formed weeks earlier. I submitted some bug reports but was eventually told by one of the maintainers that it wasn't ready yet and I should quit bugging them so they could get to MVP. Right around this time, we started having serious availability issues with metrics, both on the storage side- I estimated we were dropping about 60% of incoming statsd events, and on the query side- Graphite would take seconds-to-minutes to render some charts and occasionally would just time out. We had also built an ad-hoc system for generating Nagios checks that would poll Graphite every minute to trigger threshold-based alerts, which would make noise if Graphite was down and the monitored system was not. This led to on-call fatigue, which made everybody unhappy. We started running an instance of statsite on every server which would aggregate the individual events for that server into 10 second buckets with the server's hostname as a key prefix, then pushed those to carbon-relay. This solved the dropped packets issue, but carbon-relay was still unreliable. We were pretty entrenched in the statsd+graphite way of doing things at this point, so switching to OpenTSDB wasn't really an option and we'd exhausted all of the existing carbon alternatives, so we started thinking about modifying carbon to use another datastore. The scope of this project was large enough that it wasn't going to get built in a matter of days or weeks, so we needed a stopgap solution to buy time and keep the metrics flowing while we engineered a long term solution. I hacked together statsrelay, which is basically a re-implementation of carbon-relay in C, using libev. At this point, I was burned out and handed off the metrics infrastructure to a few teammates that ran with statsrelay and turned it into a production quality piece of code. Right around the same time, we'd begun hiring for an engineering team in NYC that would take over responsibility for metrics infrastructure. These are the people that eventually designed and built M3DB.

katzgrau 7y ago

Nice... patiently waits for AWS to create a managed version of it...

staticassertion 7y ago
https://aws.amazon.com/timestream/ https://aws.amazon.com/timestream/
- katzgrau 7y ago
  Yep, I registered for the "preview" ages ago, so I figured it was vaporware at this point