3 ms·

Author here and a few other M3ers around too, please do ask any questions. Ultimately we'd just like to make the project useful to others, so any and all feedb

Author here and a few other M3ers around too, please do ask any questions. Ultimately we'd just like to make the project useful to others, so any and all feedback is appreciated, thank you!

adrianratnapala 8y ago
> Avoid compactions where possible, including the downsampling path, to increase the utilization of host resources for more concurrent writes and provide steady write/read latency. What does that mean? What kind of compaction?
- roskilli 8y ago
  So if you look at a lot of LSM databases they basically optimize for write throughput by design but also ultimately spend a lot of time on compaction of sstable like file volumes. So we avoid a lot of compaction of volumes together by making volumes time window based rather than other limits like heap size, etc. This means that your resources ultimately do less work and should cost less. The flip side is that you do need to make sure you have enough heap for your mutable data (which is fine in the steady write case, but for back fills means you should be more careful and use libraries to coordinate the backfills)
pininja 8y ago
Is it possible to use M3 without a retention downsampling policy? For example, to retain timeseries metrics at the 1 second resolution indefinitely for one or two days worth of data. Edit: I was interested in M3 for this instead of just a time series DB because I’d also like to aggregate the metrics at lower resolutions and higher resolutions.
- roskilli 8y ago
  Hey, it is yes - by default all metrics are unaggregated, it’s only until you set aggregation and mapping rules that downsampling occurs. It’s mainly just disk space that is the primary bottleneck.
  - pininja 8y ago
    Oh ok, is it possible to also maintain an aggregated “view” at the same time as an event view?
    - roskilli 8y ago
      Yup indeed, we have to make this more friendly because you have to curl an HTTP/JSON endpoint right now to set the configuration (we’ll release an embedded UI for policy rules editing soon, called m3ctl). Basically though when you set retention mapping rules, metrics are downsampled to whatever resolution you choose (10s, 1min, 10min, etc) and then stored for whatever time the retention the namespace you have setup for that policy. And with respect to my comment about disk space being the bottleneck - unless you have a huge dataset you actually might not be bottlenecked by disk, it’s just we’ve encountered that at times with our really high cardinality data sets.
plasma 8y ago
Congrats Rob!
- roskilli 8y ago
  Wow you have some epic karma AA - haha, cheers!

Author here and a few other M3ers around too, please do ask any questions. Ultimately we'd just like to make the project useful to others, so any and all feedback is appreciated, thank you!

adrianratnapala 8y ago

> Avoid compactions where possible, including the downsampling path, to increase the utilization of host resources for more concurrent writes and provide steady write/read latency. What does that mean? What kind of compaction?

roskilli 8y ago
So if you look at a lot of LSM databases they basically optimize for write throughput by design but also ultimately spend a lot of time on compaction of sstable like file volumes. So we avoid a lot of compaction of volumes together by making volumes time window based rather than other limits like heap size, etc. This means that your resources ultimately do less work and should cost less. The flip side is that you do need to make sure you have enough heap for your mutable data (which is fine in the steady write case, but for back fills means you should be more careful and use libraries to coordinate the backfills)

pininja 8y ago

Is it possible to use M3 without a retention downsampling policy? For example, to retain timeseries metrics at the 1 second resolution indefinitely for one or two days worth of data. Edit: I was interested in M3 for this instead of just a time series DB because I’d also like to aggregate the metrics at lower resolutions and higher resolutions.

roskilli 8y ago
Hey, it is yes - by default all metrics are unaggregated, it’s only until you set aggregation and mapping rules that downsampling occurs. It’s mainly just disk space that is the primary bottleneck.
- pininja 8y ago
  Oh ok, is it possible to also maintain an aggregated “view” at the same time as an event view?
  - roskilli 8y ago
    Yup indeed, we have to make this more friendly because you have to curl an HTTP/JSON endpoint right now to set the configuration (we’ll release an embedded UI for policy rules editing soon, called m3ctl). Basically though when you set retention mapping rules, metrics are downsampled to whatever resolution you choose (10s, 1min, 10min, etc) and then stored for whatever time the retention the namespace you have setup for that policy. And with respect to my comment about disk space being the bottleneck - unless you have a huge dataset you actually might not be bottlenecked by disk, it’s just we’ve encountered that at times with our really high cardinality data sets.

plasma 8y ago

Congrats Rob!

roskilli 8y ago
Wow you have some epic karma AA - haha, cheers!