9 ms·

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheu

• jeffbee 5y ago
  Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
  • buro9 5y ago
    That's what Mimir solves
    • deepsun 5y ago
      How does it compare to VictoriaMetrics?
      • buro9 5y ago
        100% Prometheus compatible, proven to scale to 1B active series. It's not about comparisons, every tool has it's own place and feature set that may be right for you depending on what you're doing. But if you've reached the end of the road with Prometheus due to scale and you need massive scale and perfect compatibility... Then Mimir stands out.
  • halfmatthalfcat 5y ago
    Can you elaborate? I’ve ran Prometheus at some scale and it’s performed fine.
    • lokar 5y ago
      You pretty quickly exceed what one instance can handle for memory, cpu or both. At that point you don't have any real good options to scale while maintaining a flat namespace (you need to partition).
      • preseinger 5y ago
        Sure? Prometheus scales with a federation model, not a single flat namespace.
        • gttalbot 5y ago
          This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.
          • preseinger 5y ago
            Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.
            • gttalbot 5y ago
              Monarch can support both ad-hoc and periodic, standing queries of arbitrarily large size, and has the means to spread the computation out over many intermediate mixer and leaf nodes. It does query push-down so that the "expensive" parts of aggregations, joins, etc., can be done in massively parallel fashion at the leaf level. It scales so well that many aggregations are set up and computed for every service across the whole company (CPU, memory usage, error rates, etc.). For basic monitoring you can run a new service in production and go and look at a basic dashboard for it without doing anything else to set up monitoring.
            • gttalbot 5y ago
              Also, very large ad-hoc queries are supported, with really good user isolation, so that (in general) users don't harm each other.
            • xyzzy_plugh 5y ago
              That's a choice Prometheus has made, not an invariant. Many scalable systems support arbitrarily large queries as they themselves scale. Prometheus pretty much prevents creating arbitrarily large datasets to begin with, so the point is kind of moot. Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.
  • dilyevsky 5y ago
    You can set up dist eval similar to how it was done in borgmon but you gotta do it manually (or maybe write an operator to automate). One of Monarchs core ideas is to do that behind the scenes for you
    • jeffbee 5y ago
      Prometheus' own docs say that distributed evaluation is "deemed infeasible".
      • dilyevsky 5y ago
        Thats just like.. you know.. their opinion, man. https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/
        • bpicolo 5y ago
          Prometheus federation isn't distributed evaluation. It "federates" from other nodes onto a single node. > Federation allows a Prometheus server to scrape selected time series from another Prometheus server
          • dilyevsky 5y ago
            Collect directly from shard-level prometheus then aggregate using /federate at another level. That’s how thanos also works afaik
            • hiptobecubic 5y ago
              This still requires the query to fit on a single node at some point unless you're doing multi-level aggregation. Monarch does that. Being able to push expensive, highly-parallelizable aspects of the query down into the leaves is a massive benefit.
              • 5y ago
  • preseinger 5y ago
    Prometheus is highly scalable?? What are you talking about??
    • dijit 5y ago
      It is not. It basically does the opposite of what every scalable system does. To get HA you double you’re number of pollers. To get scale your queries you aggregate them into other prometheii. If this is scalability: everything is scalable.
      • preseinger 5y ago
        I don't understand how the properties you're describing imply that Prometheus isn't scalable. High Availability always requires duplication of effort. Scaling queries always requires sharding and aggregation at some level. I've deployed stock Prometheus at global scale, O(100k) targets, with great success. You have to understand and buy into Prometheus' architectural model, of course.
        • dijit 5y ago
          The ways in which you can scale Prometheus: you can scale anything. It does not; itself, have highly scalable properties built in. It does not do sharding, it does not do proxying, it does not do batching, it does not do anything that would allow it to run multiple servers and query over multiple servers. Look. I’m not saying that it doesn’t work; but when I read about borgmon and Prometheus: I understood the design goal was intentionally not to solve these hard problems, and instead use them as primitive time series systems that can be deployed with a small footprint basically everywhere (and individually queried). I submit to you, I could also have an influxdb in every server and get the same “scalability”. Difference being that I can actually run a huge influxdb cluster with a dataset that exceeds the capabilities of a single machine.
          • preseinger 5y ago
            It seems like you're asserting a very specific definition of scalability that excludes Prometheus' scalability model. Scalability is an abstract property of a system that can be achieved in many different ways. It doesn't require any specific model of sharding, batching, query replication, etc. Do you not agree?
  • gttalbot 5y ago
    This. Any new large query or aggregation in the Borgmon/Prometheus model requires re-solving federation, and continuing maintenance of runtime configuration. That might technically be scalable in that you could do it but you have to maintain it, and pay the labor cost. It's not practical over a certain size or system complexity. It's also friction. You can only do the queries you can afford to set up. That's why Google spent all that money to build Monarch. At the end of the day Monarch is vastly cheaper in person time and resources than manually-configured Borgmon/Prometheus. And there is much less friction in trying new queries, etc.
• lokar 5y ago
  It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.
  • Jedd 5y ago
    So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges. I haven't read the full paper, but do you know if the push model was revisited mostly for auto-registration / discovery, or performance bottlenecks at the server, or some other concern? Typically for us, once we've got the hard part - an entity registered - we're happy with pull only. A no-response from a prod end-point is an automatic critical. I guess at their scale there's more nuance around one or more agents being non-responsive. EDIT: Oh, there's not much in the paper on the subject, as it happens. And yes, it's vanilla discovery woes. "Push-based data collection improves system robustness while simplifying system architecture. Early versions of Monarch discovered monitored entities and “pulled” monitoring data by querying the monitored entity. "This required setting up discovery services and proxies, complicating system architecture and negatively impacting overall scalability. Push-based collection, where entities simply send their data to Monarch, eliminates these dependencies."
    • gttalbot 5y ago
      Also, I gave this talk a couple of years ago, though I'm not sure how deeply I talked about collection models. https://youtu.be/2mw12B7W7RI https://youtu.be/2mw12B7W7RI
    • lokar 5y ago
      Push can help a bit, but you still have to know which endpoints you expect to hear from (if you want to detect they are missing).
      • 5y ago
• dilyevsky 5y ago
  It was originally push but i think they went back to sort of scheduled pull mode after a few years. There was a very in depth review doc written about this internally which maybe will get published some day
  • atdt 5y ago
    What's the go/ link?
    • dilyevsky 5y ago
      Can’t remember - just search on moma /s
• gttalbot 5y ago
  Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the metric measurements according to the schedules that the leaves sent. Several problems. First, the leaf-side data structures and TCP connections become very expensive. If that leaf process is connecting to many many many thousands of monitored entities, TCP buffers aren't free, keep-alives aren't free, and a host of other data structures. Eventually this became an...interesting...fraction of the CPU and RAM on these leaf processes. Second, this implies a service discovery mechanism so that the leaves can find the entities to monitor. This was a combination of code in Monarch and an external discovery service. This was a constant source of headaches an outages, as the appearance and disappearance of entities is really spiky and unpredictable. Any burp in operation of the discovery service could cause a monitoring outage as well. Relatedly, the technical "powers that be" decided that the particular discovery service, of which Monarch was the largest user, wasn't really something that was suitable for the infrastructure at scale. This decision was made largely independently of Monarch, but required Monarch to move off. Third, Monarch does replication, up to three ways. In the pull-based system, it wasn't possible to guarantee that the measurement that each replica sees is the same measurement with the same microsecond timestamp. This was a huge data quality issue that made the distributed queries much harder to make correct and performant. Also, the clients had to pay both in persistent TCP connections on their side and in RAM, state machines, etc., for this replication as a connection would be made from each backend leaf processes holding a replica for a given client. Fourth, persistent TCP connections and load balancers don't really play well together. Fifth, not everyone wants to accept incoming connections in their binary. Sixth, if the leaf process doesn't need to know the collection policies for all the clients, those policies don't have to be distributed and updated to all of them. At scale this matters for both machine resources and reliability. This can be made a separate service, pushed to the "edge", etc. Switching from a persistent connection to the clients pushing measurements in distinct RPCs as they were recorded eventually solved all of these problems. It was a very intricate transition that took a long time. A lot of people worked very hard on this, and should be very proud of their work. I hope some of them jump in to the discussion! (At very least they'll add things I missed/didn't remember... ;^)