4 ms·

The hot new technology for metrics is Prometheus and it's ilk which is pull-based.

• bostik 6y ago
  At this point Prometheus is pretty close to becoming the boring technology. The latest versions have finally brought in the plumbing and tuning knobs to protect against [most] overly expensive queries. So you can't easily take it down anymore. The single-binary approach is still a problem, though. In my mind any serious telemetry collection stack should separate the query engine and ingestion path from each other - Prometheus has both the query interface and the ingestion/writing subsystem in the same process.[ß] As for the parent poster: you certainly want to push telemetry out on every event, but the mechanism has to be VERY lightweight. With prometheus the solution is to have a telemetry collection/aggregation agent on the host, feed it with the event data and have prometheus scrape the agent. Statsd with the KV extension is a great protocol for shoveling the telemetry out from the process and into the agent. ß: you can get around this with Thanos + Trickster to take care of the read path only, but it's quite a bit more complex than plain Prometheus.
  • wikibob 6y ago
    See the Cortex project
  • roskilli 6y ago
    M3 separates query and ingestion if you're interested in clustered storage for metrics, slide in question here: https://www.slideshare.net/RobSkillington/fosdem-2019-m3-prometheus-and-graphite-with-metrics-and-monitoring-in-an-increasingly-complex-world/31 https://www.slideshare.net/RobSkillington/fosdem-2019-m3-pro...
    • bostik 6y ago
      Oh, nice. Thank you. I had bookmarked M3 earlier on, but never got around to really looking into it with proper thought.
  • NikolaeVarius 6y ago
    I heavily disagree with Prometheus being boring tech. The storage backends still have heavy churn.