3 ms·
FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate mu
by roskilli 6y ago
FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single DB node by using a limit on the number of datapoints (inferred by blocks of datapoints per series). M3DB also uses an inverted index using FST segments that are mmap'd[2] similar to Apache Lucene and Elastic Search to make multi-dimensional searches on very large data sets fast (hundreds of trillions of datapoints, petabytes of data) which is a bit different to traditional columnar databases which focus on column stores and rarely is accompanied by a full text search inverted index.
[0]: https://docs.m3db.io/m3db/architecture/engine/ https://docs.m3db.io/m3db/architecture/engine/
[1]: https://docs.m3db.io/operational_guide/resource_limits/ https://docs.m3db.io/operational_guide/resource_limits/
[2]: https://fosdem.org/2020/schedule/event/m3db/ https://fosdem.org/2020/schedule/event/m3db/, https://fosdem.org/2020/schedule/event/m3db/attachments/audio/4032/export/events/attachments/m3db/audio/4032/FOSDEM_2020_Querying_millions_to_billions_of_metrics_with_M3DBs_inverted_index.pdf https://fosdem.org/2020/schedule/event/m3db/attachments/audi... (PDF)
- ignoramous 6y agoRecommended reading on FST for the curious: https://blog.burntsushi.net/transducers/ https://blog.burntsushi.net/transducers/
- roskilli 6y agoThank you for mentioning that, Andrew's post is really fantastic covering many things altogether: fundamentals, data structure, real world impact and examples.
- FridgeSeal 6y agoI love FST's and similar structures, they're just such a cool idea. Anyone know if there are any other similar interesting blog posts/articles about FST's?
- hintymad 6y agoThanks, @roskilli! Nice documentation.