7 ms·

Did you raise the challenges you had with Prometheus to the dev team and community? They are usually quite responsive and open for contributions. I'd be interes

by mk_ 7y ago

Did you raise the challenges you had with Prometheus to the dev team and community? They are usually quite responsive and open for contributions. I'd be interested in the discussion around the reasons you brought up for forking.

chaps 7y ago
Not OP, but one of the biggest issues that I've had with prometheus is that doesn't support the backfilling of timeseries data. Bringing it up to the prometheus devs led to oddly indignant responses that they had no intention of supporting backfilling, because that's not what prometheus was designed for. In large production systems, like prometheus is meant for, it was a very disappointing missing feature, and I could absolutely see why someone would want to fork if nothing else, just to get away from that level of indignancy.
- hnarn 7y ago
  I don't know if the Prometheus devs care about it being an enterprise level monitoring software, but not having any type of backfilling on the road map pretty much disqualifies it. Accurate reporting is not a nice-to-have for enterprise grade monitoring.
  - chaps 7y ago
    FWIW, from the lead dev's own description on his "up and running" book, pretty much implying that they do, in fact, care about its use in large production systems: "Get up to speed with Prometheus, the metrics-based monitoring system used by tens of thousands of organizations in production. This practical guide provides application developers, sysadmins, and DevOps practitioners with a hands-on introduction to the most important aspects of Prometheus, including dashboarding and alerting, direct code instrumentation, and metric collection from third-party systems with exporters."
  - sagichmal 7y ago
    Backfilling isn't necessary for "accurate reporting"?
    - hnarn 7y ago
      Is that a question? I think I pretty plainly stated that the answer to that question is yes.
      - sagichmal 7y ago
        I'm confused because the statement doesn't make sense. Accurate reporting is orthogonal to the ability to backfill.
        chaps 7y ago
        Take a breather. Pretty sure the confusion here is in the overloading and different contexts of "reporting". One kind of reporting is post-hoc, and the other is "real time" reporting for outages and such. There are probably other ways to define it, but let's try to figure out where the confusion is first. How are you defining "reporting" to where it's orthogonal to the ability to backfill?
  - SuperQue 7y ago
    Backfilling data is on the roadmap. It's already possible, the tools are being worked on. Also, accurate reporting and backfill are not related.
    - chaps 7y ago
      It's been on the road map for at least two and a half years, so that's not very convincing: https://web.archive.org/web/20170708101602/https://prometheus.io/docs/introduction/roadmap/ https://web.archive.org/web/20170708101602/https://prometheu... Curious to know why you don't think accurate reporting and backfilling aren't related. In my experience, they absolutely are -- mostly during disaster scenarios, where that information is more critical than any other time.
    - hnarn 7y ago
      > accurate reporting and backfill are not related. I strongly disagree. If you are unable to backfill data then report accuracy will be affected during multiple scenarios, including migrations from other products, unplanned downtime etc. Not having backfill is accepting that reports will suffer as soon as historical data is inaccurate and basically saying that you don't care.
  - deleted 7y ago
    [deleted]
- omeze 7y ago
  By backfilling I assume you mean retroactively applying labels to a time series? Can you link to the thread with more context if you have it around? As a prometheus user myself at a 100+ eng org company I've never had a use case for a flow that would require editing old timeseries apart from having to restore data after a crash
  - chaps 7y ago
    What I mean by that is, I have an exporter that (for example) might be reading a bunch of logs and exposing some inferred metrics information through the exporter. Two days later, I realize that there's some useful information from those logs or another log file, or hell, another server, that wasn't inferred. That way, in some monitoring analysis I'm running, both data points would be side by side. Prometheus doesn't allow for that, because opinions. Just a toy example, so please don't HN-penanticize it :) Not sure why restoring data after a crash isn't a bigger problem in your eyes, though. In the context of fully understanding system stability through prometheus, that's a pretty giant gap in your post portem analysis without any way to correct it. It forces you into either using two tools to be able to do that sort of analysis, or accepting that you're always going to risk losing valuable data.
    - omeze 7y ago
      Gotcha, it seems like you want to run backtests from within Prometheus for richer post-hoc analysis. That sounds like a normal use case, I often do fairly non-trivial post hoc analysis in Prometheus (e.g. Z scores and stddevs of metrics) since the query language is quite well suited to it, so I can see why you want to leverage it more. Re: restoration, we use facilities like EBS snapshots, so its separated from prometheus. It could theoretically be an issue for post mortem analysis but data loss is relatively rare so in practice the ability to run post mortems has never been affected. Generally we don't consider our metrics as "business-critical" as production data (e.g. we wouldn't consider the above acceptable for MySQL) so that's where the leniency comes in. I can see different orgs weighing this differently -- we're not really an "infrastructure" company so I think the tradeoff is right for us, but if I was at a PaaS or IaaS I would feel differently.
- icook 7y ago
  FWIW, backfilling is on their roadmap[0]. There's also a ticket open about it[1]. [0] https://prometheus.io/docs/introduction/roadmap/#backfill-time-series https://prometheus.io/docs/introduction/roadmap/#backfill-ti... [1] https://github.com/prometheus/prometheus/issues/535 https://github.com/prometheus/prometheus/issues/535
  - chaps 7y ago
    Hah, yeah I remember seeing that ticket. It's been open for over five years now. The dev's.... attitude... is clear as day.
- Rapzid 7y ago
  Yeah, if that were the only reason given I would 100% be like "Ah, yep, good call".
- mekster 7y ago
  VictoriaMetrics seems to support that. I feel this is quite technically superior to other remote storages. https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ (Not affiliated, just impressed with the author's blog posts explaining how it's better than competitors.)
greendave 7y ago
Disclosure: Zebrium employee. We've had a great experience with prometheus - both the community and the project. Building what we did would have been immeasurably harder without them. That said, this has been a rapid process for us. We are starting to engage and see what parts of our fork they would be interested for upstream. (In particular our storage needs are a bit different, so we'll see how that fits.)

by mk_ 7y ago

chaps 7y ago

Not OP, but one of the biggest issues that I've had with prometheus is that doesn't support the backfilling of timeseries data. Bringing it up to the prometheus devs led to oddly indignant responses that they had no intention of supporting backfilling, because that's not what prometheus was designed for. In large production systems, like prometheus is meant for, it was a very disappointing missing feature, and I could absolutely see why someone would want to fork if nothing else, just to get away from that level of indignancy.

hnarn 7y ago
I don't know if the Prometheus devs care about it being an enterprise level monitoring software, but not having any type of backfilling on the road map pretty much disqualifies it. Accurate reporting is not a nice-to-have for enterprise grade monitoring.
- chaps 7y ago
  FWIW, from the lead dev's own description on his "up and running" book, pretty much implying that they do, in fact, care about its use in large production systems: "Get up to speed with Prometheus, the metrics-based monitoring system used by tens of thousands of organizations in production. This practical guide provides application developers, sysadmins, and DevOps practitioners with a hands-on introduction to the most important aspects of Prometheus, including dashboarding and alerting, direct code instrumentation, and metric collection from third-party systems with exporters."
- sagichmal 7y ago
  Backfilling isn't necessary for "accurate reporting"?
  - hnarn 7y ago
    Is that a question? I think I pretty plainly stated that the answer to that question is yes.
    - sagichmal 7y ago
      I'm confused because the statement doesn't make sense. Accurate reporting is orthogonal to the ability to backfill.
      - chaps 7y ago
        Take a breather. Pretty sure the confusion here is in the overloading and different contexts of "reporting". One kind of reporting is post-hoc, and the other is "real time" reporting for outages and such. There are probably other ways to define it, but let's try to figure out where the confusion is first. How are you defining "reporting" to where it's orthogonal to the ability to backfill?
- SuperQue 7y ago
  Backfilling data is on the roadmap. It's already possible, the tools are being worked on. Also, accurate reporting and backfill are not related.
  - chaps 7y ago
    It's been on the road map for at least two and a half years, so that's not very convincing: https://web.archive.org/web/20170708101602/https://prometheus.io/docs/introduction/roadmap/ https://web.archive.org/web/20170708101602/https://prometheu... Curious to know why you don't think accurate reporting and backfilling aren't related. In my experience, they absolutely are -- mostly during disaster scenarios, where that information is more critical than any other time.
  - hnarn 7y ago
    > accurate reporting and backfill are not related. I strongly disagree. If you are unable to backfill data then report accuracy will be affected during multiple scenarios, including migrations from other products, unplanned downtime etc. Not having backfill is accepting that reports will suffer as soon as historical data is inaccurate and basically saying that you don't care.
- deleted 7y ago
  [deleted]
omeze 7y ago
By backfilling I assume you mean retroactively applying labels to a time series? Can you link to the thread with more context if you have it around? As a prometheus user myself at a 100+ eng org company I've never had a use case for a flow that would require editing old timeseries apart from having to restore data after a crash
- chaps 7y ago
  What I mean by that is, I have an exporter that (for example) might be reading a bunch of logs and exposing some inferred metrics information through the exporter. Two days later, I realize that there's some useful information from those logs or another log file, or hell, another server, that wasn't inferred. That way, in some monitoring analysis I'm running, both data points would be side by side. Prometheus doesn't allow for that, because opinions. Just a toy example, so please don't HN-penanticize it :) Not sure why restoring data after a crash isn't a bigger problem in your eyes, though. In the context of fully understanding system stability through prometheus, that's a pretty giant gap in your post portem analysis without any way to correct it. It forces you into either using two tools to be able to do that sort of analysis, or accepting that you're always going to risk losing valuable data.
  - omeze 7y ago
    Gotcha, it seems like you want to run backtests from within Prometheus for richer post-hoc analysis. That sounds like a normal use case, I often do fairly non-trivial post hoc analysis in Prometheus (e.g. Z scores and stddevs of metrics) since the query language is quite well suited to it, so I can see why you want to leverage it more. Re: restoration, we use facilities like EBS snapshots, so its separated from prometheus. It could theoretically be an issue for post mortem analysis but data loss is relatively rare so in practice the ability to run post mortems has never been affected. Generally we don't consider our metrics as "business-critical" as production data (e.g. we wouldn't consider the above acceptable for MySQL) so that's where the leniency comes in. I can see different orgs weighing this differently -- we're not really an "infrastructure" company so I think the tradeoff is right for us, but if I was at a PaaS or IaaS I would feel differently.
icook 7y ago
FWIW, backfilling is on their roadmap[0]. There's also a ticket open about it[1]. [0] https://prometheus.io/docs/introduction/roadmap/#backfill-time-series https://prometheus.io/docs/introduction/roadmap/#backfill-ti... [1] https://github.com/prometheus/prometheus/issues/535 https://github.com/prometheus/prometheus/issues/535
- chaps 7y ago
  Hah, yeah I remember seeing that ticket. It's been open for over five years now. The dev's.... attitude... is clear as day.
Rapzid 7y ago
Yeah, if that were the only reason given I would 100% be like "Ah, yep, good call".
mekster 7y ago
VictoriaMetrics seems to support that. I feel this is quite technically superior to other remote storages. https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ (Not affiliated, just impressed with the author's blog posts explaining how it's better than competitors.)

greendave 7y ago

Disclosure: Zebrium employee. We've had a great experience with prometheus - both the community and the project. Building what we did would have been immeasurably harder without them. That said, this has been a rapid process for us. We are starting to engage and see what parts of our fork they would be interested for upstream. (In particular our storage needs are a bit different, so we'll see how that fits.)