roskilli

9 ms·

31.
by roskilli 7y ago
M3DB ingested 30 million datapoints per second (so 1.8 billion per minute) with each node writing hundreds of thousands of writes per second. The dataset was in the petabytes. For us the cost savings vs OpenTSDB (millions of dollars of hard
32.
by roskilli 7y ago
Whether it saves space or not, looking at metrics over period of months or years when the data is raw is far more slow/expensive than looking at downsampled data. If you still want to be able to quickly graph and view old data, downsam
33.
by roskilli 7y ago
I think ScyllaDB would have definitely done better than Cassandra (which we were using alongside ElasticSearch), although another thing I mention in this thread is that a lot of existing distributed databases do not have a multi-dimensional
34.
by roskilli 7y ago
Yes I've seen that also work, it's a lot of stitching together things yourself and we had to put a lot of caching in front of the inverted index we were using, however definitely plausible. ClickHouse doesn't do any streamin
35.
by roskilli 7y ago
I want to first say, I have a great amount of respect for Netflix's engineering and for Atlas itself, it's great that it exists and is more accessible than other scalable in-memory TSDBs open sourced by large companies. A few of m
36.
by roskilli 7y ago
While this is true, for a metrics workload it does not work great I have both seen and heard from others, mainly due to the fact it does not have an inverted index - so finding a small subset of metrics in a dataset of billions of metrics e
37.
by roskilli 7y ago
Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware. It’s hard to justify using tens of millions of doll
38.
by roskilli 7y ago
I touch on this a little in the podcast I did with Jeff[0], but it boils down to OpenTSDSB for us when we benchmarked could only do low tens of thousands of writes per second per node, whereas M3DB is hyper optimized and can do hundreds of
39.
by roskilli 7y ago
Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store componen
40.
by roskilli 7y ago
As per sibling comment, they do most definitely work until they don’t. M3 actually started with ElasticSearch and Cassandra for index and storage respectively but then were replaced with M3DB. I mentioned the FOSDEM talk elsewhere in the th
41.
by roskilli 7y ago
Thanks for the interest, I just did a talk at FOSDEM a few weeks ago on the subject of querying over large datasets that M3DB can warehouse and query in real-time here: Slides https://fosdem.org/2020/schedule/event
42.
by roskilli 7y ago
Oh heh, I declared another method then used that to set the passcode var (without the var being shadowed), return it to the hardcoded method name, and returned that. // readCodesFromKeypad - get codes from keypad input func
43.
by roskilli 7y ago
Some of us (read... one of us, and not me, yet) uses TLA+ when theorizing changes to parts of M3DB, which is a distributed time series database. You can see the specs here, the current TLA+ models the consistency model of data being persist
44.
by roskilli 7y ago
It's not raw speed or raw performance on a single node that M3DB is optimized for, it's for a reliable scale out story when you have a considerable number of instances required to collect the raw data you operate on (organizations
45.
by roskilli 7y ago
So we collected and aggregated more than 1 billion samples of metrics per second, which resulted in writing more than 30-40 million unique metric datapoints per second to storage. This resulted in more than 10 billion unique time series bei
46.
by roskilli 7y ago
What's interesting about some of the more modern monitoring systems like M3 and Prometheus is that they have a reverse index on top of the column store entries to very quickly find the relevant metrics for a multi-dimensional query. In
47.
by roskilli 7y ago
Haha TY for the kind words, I had to stop playing years ago now unfortunately with family commitments - also I mainly enjoyed playing on a team than actually developing real soccer skills (which I relied on others in the team to pull me upw
48.
by roskilli 7y ago
Hey Rob a co-founder and M3DB creator here, more than happy to answer any queries anyone might have. We're committed on continuing M3 being 100% apache 2 licensed, clustering and all other M3 features included. We're focused on pr
49.
by roskilli 7y ago
Rob, co-founder and M3DB creator here, Uber collected billions of metric samples and we had tens of billions of metrics in M3 at Uber. Netflix for reference has not published any numbers higher than single digit billions of time series. The
50.
Chronosphere launches with $11M Series A to build scalable monitoring tool(techcrunch.com)
120 points by roskilli 7y ago|35 comments
51.
by roskilli 7y ago
Can confirm that your confirmation of M3DB being cancelled is incorrect and is actively being worked on at Uber and elsewhere. I do agree there were definitely projects that were overly ambitious and optimistic that likely that should have
52.
by roskilli 7y ago
Seems like the flamegraph he took of his Rust program shows it using backtracking, maybe for the specific regexp he is using it's unable to use a DFA: https://i.imgur.com/9lx42Tu.png
53.
Time Series Databases Deep Dive [audio](softwareengineeringdaily.com)
23 points by roskilli 7y ago|1 comments
54.
by roskilli 7y ago
Might be using browser local storage APIs? Hah. I imagine you could also do some kind of web socket/long poll and try to fingerprint the connection a little so there’s high likelihood it’s the same session when it reconnects between pa
55.
by roskilli 7y ago
Very cool, great that the portal is a Typescript Nodejs open source project itself too. Hope that companies can adopt this, it's pretty tedious setting up open source review committees and making it simple/keeping the friction low
56.
by roskilli 7y ago
It is definitely a huge step forward for time series data collection at scale, great props to the authors. With M3TSZ vs vanilla TSZ the major difference is that instead of XORing the value component of each datapoint it is determined if th
57.
by roskilli 7y ago
Those interested in TSDBs might also be interested in M3DB.io which is Apache 2 licensed, supports a modified TSZ compression implementation with 11x compression ratio, replication of data in the cluster, streaming of data between nodes on
58.
by roskilli 7y ago
Basically it's been deprecated in favor of using projects that are more mature like Apache Helix (but conversely more operationally intense to get started, requires Zookeeper and a bunch of things on top if you want to use it in a non-
59.
by roskilli 7y ago
It's to avoid the complicated routing problem (i.e. building a router microservice that sits in front of it), and encapsulate that in the service itself (by proxying internally to another part of the service after inspecting the reques
60.
by roskilli 7y ago
I don't want to come across as negative, but just an observation and to play devil's advocate - wouldn't it be better to fix the flaky test or delete it entirely instead of build a feature to disable it during a test run in a

More ›