3 ms·

At that point you're just treating Prometheus as an agent, like telegraf or collectd, running in the customers infrastructure to collect metrics from it and pub

At that point you're just treating Prometheus as an agent, like telegraf or collectd, running in the customers infrastructure to collect metrics from it and publish them to Cortex. That gets to my first point that Prometheus does a lot more than that, so maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.

slgeorge 8y ago
maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. The way the user runs it means that they don't store any metrics locally. The Prometheus agent on the cluster does scraping and then sends the data immediately up to the central SaaS instance (Cortex). what if your potential customer runs on a PaaS like Heroku The range of exporters is very extensive (https://prometheus.io/docs/instrumenting/exporters/ https://prometheus.io/docs/instrumenting/exporters/). And, as K8s has become so important lots of other people are publishing Prometheus metrics. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you. Cortex provides a scalable backend storage for Prometheus metrics, it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.
- sciurus 8y ago
  > The way the user runs it means that they don't store any metrics locally Ah, that's great! I was mistakenly assuming that Prometheus would still have to write the data to disk; I didn't realize you could disable local storage. > The range of exporters is very extensive Sure, but that doesn't solve the problem that your heroku dyno, or lambda function, etc cannot expose metrics for Prometheus to pull, but it could easily push metrics somewhere. > it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. That's fair; I was thinking of Cortex as something to adopt off the shelf, instead of something to integrate with. However, as a SaaS provider you can always build more custom functionality alongside Cortex instead of making it a direct part of the open-source project.
  - bboreham 8y ago
    > I didn't realize you could disable local storage. You are correct: there is no option to disable so we set to discard after 2 hours. Prometheus doesn’t write continuously to disk so the io is not noticeable. To your original question, yes we could produce another agent from the Prometheus code which did discovery, scraping and sending but not disk storage. We choose not to as it’s another thing to maintain. Also we want an option to queue on disk when the network link to push is down. (I am a Cortex maintainer)
netingle 8y ago
> I'd expect you'd have to run an API that your customers can code against to push data to you. Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.

slgeorge 8y ago

maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. The way the user runs it means that they don't store any metrics locally. The Prometheus agent on the cluster does scraping and then sends the data immediately up to the central SaaS instance (Cortex). what if your potential customer runs on a PaaS like Heroku The range of exporters is very extensive (https://prometheus.io/docs/instrumenting/exporters/ https://prometheus.io/docs/instrumenting/exporters/). And, as K8s has become so important lots of other people are publishing Prometheus metrics. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you. Cortex provides a scalable backend storage for Prometheus metrics, it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.

sciurus 8y ago
> The way the user runs it means that they don't store any metrics locally Ah, that's great! I was mistakenly assuming that Prometheus would still have to write the data to disk; I didn't realize you could disable local storage. > The range of exporters is very extensive Sure, but that doesn't solve the problem that your heroku dyno, or lambda function, etc cannot expose metrics for Prometheus to pull, but it could easily push metrics somewhere. > it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. That's fair; I was thinking of Cortex as something to adopt off the shelf, instead of something to integrate with. However, as a SaaS provider you can always build more custom functionality alongside Cortex instead of making it a direct part of the open-source project.
- bboreham 8y ago
  > I didn't realize you could disable local storage. You are correct: there is no option to disable so we set to discard after 2 hours. Prometheus doesn’t write continuously to disk so the io is not noticeable. To your original question, yes we could produce another agent from the Prometheus code which did discovery, scraping and sending but not disk storage. We choose not to as it’s another thing to maintain. Also we want an option to queue on disk when the network link to push is down. (I am a Cortex maintainer)

netingle 8y ago

> I'd expect you'd have to run an API that your customers can code against to push data to you. Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.