6 ms·

Prometheus is not easy to run at scale on the storage side.

• pram 6y ago
  This is all relative but I don't personally think so. Not on EC2+EBS, anyway. Certainly not as difficult as running/scaling an ES or Kafka cluster.
  • Thaxll 6y ago
    It's a completely different problem because by default Prometheus does not shard anything so you're bound to a single instance, where ES and Kafka are cluster based.
  • smcleod 6y ago
    Out of interest what do you find hard about running ElasticSearch clusters? In my experience ES has been one of the easiest clustered / highly available and sharded systems I've ever run - especially for how incredibly performant and reliable it is. I've generally found that beyond right sizing your nodes, indexes and shard configuration - it pretty much just works without ever really having issues.
• shitloadofbooks 6y ago
  Victoria Metrics is an absolutely superb drop in replacement.
  • jrv 6y ago
    It's not a drop-in replacement (even though it tries to sell itself as such), it's incompatible in a significant number of ways and throws away part of your data.
    • SkyRocknRoll 6y ago
      We use Victoria Metrics in Prod for more than 6 months. It is very reliable and scalable. Victoria metrics handles more than 2B+ series in our setup without breaking a sweat.
    • hagen1778 6y ago
      VictoriaMetrics (VM) is incompatible with Prometheus only in things which seemed to be "wrong" or "not good enough" in original. There was a very very long discussion about rate/irate implementation in Prometheus long before VM existence, and VM just fixed that and a bunch of other things. And about "throws away part of your data" - let's be clear here. It is a precision loss while converting floating point numbers, not data dropping. Instead, user gets a superior compression and query performance which is a good trade off in my opinion. For those who are interested in details of comparison VM vs Prometheus pls read the following article https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87 https://valyala.medium.com/evaluating-performance-and-correc...
    • valyala 6y ago
      VictoriaMetrics provides its own query lagnuage - MetricsQL. It is backwards compatible with PromQL with some exceptions. These exceptions are aimed towards solving practical issues experienced by users when working with PromQL. All these "incompatibilities" are well documented [1]. [1] https://victoriametrics.github.io/MetricsQL.html https://victoriametrics.github.io/MetricsQL.html
    • heliodor 6y ago
      They made some pragmatic optimizations by dropping part of the data where it mostly doesn't matter. Who's actually affected by this? 0.01% of users?
    • pointestimate 6y ago
      No, it's better. You can focus on the thing you're measuring rather than the bloody platform.