8 ms·

DalmatinerDB: A fast, distributed metric store

96 points by sciurus 11y ago

dataloopio 11y ago
Some more info here: https://dataloopio.wordpress.com/2016/03/21/playing-with-dalmatiner-db/ https://dataloopio.wordpress.com/2016/03/21/playing-with-dal...
im_down_w_otp 11y ago
Yes! Oh sweet wonderful gift from the Gods! I can't wait to dig around in the KV data model for this.
- im_down_w_otp 11y ago
  Ah, this is based on Riak Core, not Riak KV as is suggested by the Wordpress blog post announcing DalmatinerDB. That actually makes me even happier. :-)
  - Licenser 11y ago
    The storage engine is a separate library, it might be of interest for you: https://github.com/dalmatinerdb/mstore https://github.com/dalmatinerdb/mstore
    - im_down_w_otp 11y ago
      So much so that I'd already been poking around there. :-) I like to see new riak_core applications being made. For the most part it's a pretty clean framework to bootstrap something with. I have a current particular interest in riak_core over the top of a pile of disk_log because I want something Kafka shaped, but where topics are cheap, and sloppy-leader-esque behavior is the default without destroying data. Optimizing for availability over everything else and guaranteed delivery of all ACK'd messages put into the system if the end of all logs have been reached (via continuation).
  - dataloopio 11y ago
    Sorry, blog corrected.
rdtsc 11y ago
Very nice. I like how it delegates some of the stuff to the filesystem (compression, checksumming).
chimeracoder 11y ago
Is this an alternate spelling of "Dalmatian" I'm not aware of?
- bajsejohannes 11y ago
  It's German for Dalmatian.
marknadal 11y ago
Could somebody explain this? "No guarantee of storage. DalmatinerDB offers a 'best effort' on storing the metrics" https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of-storage https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of...
- querulous 11y ago
  it means the transport is UDP and there's no ack on write
- metasean 11y ago
  I think it's acceptable data loss because "It Probably Works" - https://www.youtube.com/watch?v=FSlPU5Nrvds https://www.youtube.com/watch?v=FSlPU5Nrvds
- Licenser 11y ago
  This explains a bit on the take of loosing data: https://vimeo.com/148514080 https://vimeo.com/148514080
bajsejohannes 11y ago
Nomenclature nitpick: This is more clearly called a time series database. A "metric" doesn't imply multiple samples over time (although researching for this comment, there doesn't seem to be a strict definition)
- toomim 11y ago
  Yeah, what is a "metric"? Does that mean a "number"?
  - devdas 11y ago
    http://blog.lusis.org/blog/2011/07/21/monitoring-sucks-watch-your-language/ http://blog.lusis.org/blog/2011/07/21/monitoring-sucks-watch...
- Licenser 11y ago
  I feel that time series implies different tradeoffs, this is very specifically build for metrics in the sense of application or hardware metrics which usually have a time time dimension but yes there are no general valid definitions
okdtsk 11y ago
Good dog. 101 Dalmatians.
losvedir 11y ago
When would you want to use something like this over just ordinary postgresql? Say you're trying to measure/analyze the use of a feature like clicking a button. You want to record every time a user clicks the button, their user ID, and some attributes of the user (maybe in bucket A or bucket B of a test). Later you'd want to be able to answer questions like "how many times was this button pressed?" or "how many unique users pressed a button?" or "how many times did users in bucket A press the button?". Is that an "appropriate" use for postgres? Or is that the kind of "metrics" that a DB like Dalmatiner would be more optimized for? Is there a nice overview of the various kinds of DBs out there and what use cases they're optimized for?
- Licenser 11y ago
  The situation you describe is more an event then it is an metric, and indeed postgres is much better suited for it, if asked that is what I'd recommend for this kind of issue. What DalmatinerDB was build of is handling high numbers of metrics like CPU usage, memory usage, etc, everything where for every point in time you have exactly 1 value. In that space it can handle millions of metrics (read millions of inserts) at the same time where a system like postgres would stat to have problems in my experience.
  - losvedir 11y ago
    I see, thanks for your response!
  - TheLogothete 11y ago
    I wouldn't use postgres for that at all! A NoSQL store is much better suited for that task.
    - AlterEgo20 11y ago
      There is not suck thing as "NoSQL store". NoSQL is just a buzzword that is used to describe dozens (hundreds?) of very different solutions.
ipalreadytaken 11y ago
Another exciting and related project is Tachyon, which allows for rapid collection and sending of metrics to Dalmatiner. https://docs.project-fifo.net/docs/tachyon https://docs.project-fifo.net/docs/tachyon
whatnotests 11y ago
I see there's a Python client -- which looks fairly straightforward. I'd like to offer a ruby client, but would prefer not to duplicate an effort already underway. What's the status of an official Ruby client?
- shiftoutbox 11y ago
  Go write a ruby client.
rdtsc 11y ago
Wow, saw the benchmarks: https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html "8,500,000 and 9,000,000 metrics per second" on a 5 node cluster. This is impressive. The 1 node is impressive as well (and it shows how it scales): 1,500,000 metrics per second
- fasteo 11y ago
  They are impressive, but let's put some context: No guarantee of storage DalmatinerDB offers a 'best effort' on storing the metrics, the ingress transport is UDP and there is no log for writes (there is the ZIL if enabled in ZFS) or forced sync after each write. This means that if your network fails packets can get lost, if your server crashes unwritten data can be lost. [1] https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of-storage https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of...
  - Licenser 11y ago
    Actually let me put that paragraph fully in context with the reality. UDP is no longer used (this is outdated sorry for that), the connection is TCP now as it turned out over all the performance was better. Dataloss can still occur since DalmatinerDB keeps a cache (which other metric stores might also do). A lot of that can be mitigated by using N=2 (or 3) to store data in multiple nodes that will reduce the chance of dataloss significantly. Keeping caches isn't uncommon however, to ensure full consistency it requires a kind of transaction from that goes from client to server to client, which is 'really' costly and I am convinced not worth it for metrics, a few seconds of lost metrics doesn't warrant the cost of that.
  - Licenser 11y ago
    The tests were run with TCP, they do not include non stored metrics.
joshfraser 11y ago
There's a lot of overhead from disk seeks that plague a lot of traditional databases. Especially if you have a physical arm moving around. Any database like this that can write data sequentially and forgo update and delete operations can really speed things up. Also at scale you start worrying about things like storage costs. Sequential reads/writes make it reasonable to use spinning disks for the cost savings as SSDs cost more and lose their order-of-magnitude advantage when comparing sequential writes.
johlo 11y ago
The docs strongly recommend running on ZFS. What is the current state of ZFS for linux? Is it common in production today?
- dataloopio 11y ago
  I know quite a few companies running ZFSOL in production. In terms of our own experiences I blogged about them here: https://dataloopio.wordpress.com/2016/03/07/zfs-on-linux/ https://dataloopio.wordpress.com/2016/03/07/zfs-on-linux/
phxql 11y ago
If you are interested in storing time series, check out http://chronix.io/ http://chronix.io/. A colleague of mine is working on his PhD about time series storage and created it.
crocdundee 11y ago
Here is a great video talking about the decisions made when designing DalmatinerDB - https://vimeo.com/148514080 https://vimeo.com/148514080
gtirloni 11y ago
Tying itself to a technology that faces some legal challenges (ZFS) doesn't look like a good thing [1]. I understand the benefits of avoiding duplicating features that are already present in the ZFS layer, but it's a roadblock today and in the foreseeable future. 1 - https://news.ycombinator.com/item?id=11176107 https://news.ycombinator.com/item?id=11176107
- philipcristiano 11y ago
  That's the case if you were going to deploy this on Linux. The packages provided are for SmartOS.
varg 11y ago
Looks great! ..I like the fact its built on top of proven technology like ZFS and Riak core.

96 points by sciurus 11y ago

Some more info here: https://dataloopio.wordpress.com/2016/03/21/playing-with-dalmatiner-db/ https://dataloopio.wordpress.com/2016/03/21/playing-with-dal...

im_down_w_otp 11y ago

Yes! Oh sweet wonderful gift from the Gods! I can't wait to dig around in the KV data model for this.

im_down_w_otp 11y ago
Ah, this is based on Riak Core, not Riak KV as is suggested by the Wordpress blog post announcing DalmatinerDB. That actually makes me even happier. :-)
- Licenser 11y ago
  The storage engine is a separate library, it might be of interest for you: https://github.com/dalmatinerdb/mstore https://github.com/dalmatinerdb/mstore
  - im_down_w_otp 11y ago
    So much so that I'd already been poking around there. :-) I like to see new riak_core applications being made. For the most part it's a pretty clean framework to bootstrap something with. I have a current particular interest in riak_core over the top of a pile of disk_log because I want something Kafka shaped, but where topics are cheap, and sloppy-leader-esque behavior is the default without destroying data. Optimizing for availability over everything else and guaranteed delivery of all ACK'd messages put into the system if the end of all logs have been reached (via continuation).
- dataloopio 11y ago
  Sorry, blog corrected.

rdtsc 11y ago

Very nice. I like how it delegates some of the stuff to the filesystem (compression, checksumming).

chimeracoder 11y ago

Is this an alternate spelling of "Dalmatian" I'm not aware of?

bajsejohannes 11y ago
It's German for Dalmatian.

marknadal 11y ago

Could somebody explain this? "No guarantee of storage. DalmatinerDB offers a 'best effort' on storing the metrics" https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of-storage https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of...

querulous 11y ago
it means the transport is UDP and there's no ack on write
metasean 11y ago
I think it's acceptable data loss because "It Probably Works" - https://www.youtube.com/watch?v=FSlPU5Nrvds https://www.youtube.com/watch?v=FSlPU5Nrvds
Licenser 11y ago
This explains a bit on the take of loosing data: https://vimeo.com/148514080 https://vimeo.com/148514080

bajsejohannes 11y ago

Nomenclature nitpick: This is more clearly called a time series database. A "metric" doesn't imply multiple samples over time (although researching for this comment, there doesn't seem to be a strict definition)

toomim 11y ago
Yeah, what is a "metric"? Does that mean a "number"?
- devdas 11y ago
  http://blog.lusis.org/blog/2011/07/21/monitoring-sucks-watch-your-language/ http://blog.lusis.org/blog/2011/07/21/monitoring-sucks-watch...
Licenser 11y ago
I feel that time series implies different tradeoffs, this is very specifically build for metrics in the sense of application or hardware metrics which usually have a time time dimension but yes there are no general valid definitions

okdtsk 11y ago

Good dog. 101 Dalmatians.

losvedir 11y ago

When would you want to use something like this over just ordinary postgresql? Say you're trying to measure/analyze the use of a feature like clicking a button. You want to record every time a user clicks the button, their user ID, and some attributes of the user (maybe in bucket A or bucket B of a test). Later you'd want to be able to answer questions like "how many times was this button pressed?" or "how many unique users pressed a button?" or "how many times did users in bucket A press the button?". Is that an "appropriate" use for postgres? Or is that the kind of "metrics" that a DB like Dalmatiner would be more optimized for? Is there a nice overview of the various kinds of DBs out there and what use cases they're optimized for?

Licenser 11y ago
The situation you describe is more an event then it is an metric, and indeed postgres is much better suited for it, if asked that is what I'd recommend for this kind of issue. What DalmatinerDB was build of is handling high numbers of metrics like CPU usage, memory usage, etc, everything where for every point in time you have exactly 1 value. In that space it can handle millions of metrics (read millions of inserts) at the same time where a system like postgres would stat to have problems in my experience.
- losvedir 11y ago
  I see, thanks for your response!
- TheLogothete 11y ago
  I wouldn't use postgres for that at all! A NoSQL store is much better suited for that task.
  - AlterEgo20 11y ago
    There is not suck thing as "NoSQL store". NoSQL is just a buzzword that is used to describe dozens (hundreds?) of very different solutions.

ipalreadytaken 11y ago

Another exciting and related project is Tachyon, which allows for rapid collection and sending of metrics to Dalmatiner. https://docs.project-fifo.net/docs/tachyon https://docs.project-fifo.net/docs/tachyon

whatnotests 11y ago

I see there's a Python client -- which looks fairly straightforward. I'd like to offer a ruby client, but would prefer not to duplicate an effort already underway. What's the status of an official Ruby client?

shiftoutbox 11y ago
Go write a ruby client.

rdtsc 11y ago

Wow, saw the benchmarks: https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html "8,500,000 and 9,000,000 metrics per second" on a 5 node cluster. This is impressive. The 1 node is impressive as well (and it shows how it scales): 1,500,000 metrics per second

fasteo 11y ago
They are impressive, but let's put some context: No guarantee of storage DalmatinerDB offers a 'best effort' on storing the metrics, the ingress transport is UDP and there is no log for writes (there is the ZIL if enabled in ZFS) or forced sync after each write. This means that if your network fails packets can get lost, if your server crashes unwritten data can be lost. [1] https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of-storage https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of...
- Licenser 11y ago
  Actually let me put that paragraph fully in context with the reality. UDP is no longer used (this is outdated sorry for that), the connection is TCP now as it turned out over all the performance was better. Dataloss can still occur since DalmatinerDB keeps a cache (which other metric stores might also do). A lot of that can be mitigated by using N=2 (or 3) to store data in multiple nodes that will reduce the chance of dataloss significantly. Keeping caches isn't uncommon however, to ensure full consistency it requires a kind of transaction from that goes from client to server to client, which is 'really' costly and I am convinced not worth it for metrics, a few seconds of lost metrics doesn't warrant the cost of that.
- Licenser 11y ago
  The tests were run with TCP, they do not include non stored metrics.

joshfraser 11y ago

There's a lot of overhead from disk seeks that plague a lot of traditional databases. Especially if you have a physical arm moving around. Any database like this that can write data sequentially and forgo update and delete operations can really speed things up. Also at scale you start worrying about things like storage costs. Sequential reads/writes make it reasonable to use spinning disks for the cost savings as SSDs cost more and lose their order-of-magnitude advantage when comparing sequential writes.

johlo 11y ago

The docs strongly recommend running on ZFS. What is the current state of ZFS for linux? Is it common in production today?

dataloopio 11y ago
I know quite a few companies running ZFSOL in production. In terms of our own experiences I blogged about them here: https://dataloopio.wordpress.com/2016/03/07/zfs-on-linux/ https://dataloopio.wordpress.com/2016/03/07/zfs-on-linux/

phxql 11y ago

If you are interested in storing time series, check out http://chronix.io/ http://chronix.io/. A colleague of mine is working on his PhD about time series storage and created it.

crocdundee 11y ago

Here is a great video talking about the decisions made when designing DalmatinerDB - https://vimeo.com/148514080 https://vimeo.com/148514080

gtirloni 11y ago

Tying itself to a technology that faces some legal challenges (ZFS) doesn't look like a good thing [1]. I understand the benefits of avoiding duplicating features that are already present in the ZFS layer, but it's a roadblock today and in the foreseeable future. 1 - https://news.ycombinator.com/item?id=11176107 https://news.ycombinator.com/item?id=11176107

philipcristiano 11y ago
That's the case if you were going to deploy this on Linux. The packages provided are for SmartOS.

varg 11y ago

Looks great! ..I like the fact its built on top of proven technology like ZFS and Riak core.