6 ms·

CNCF's Cortex v1.0: scalable, fast Prometheus implementation

181 points by netingle 7y ago

netingle 7y ago
Hi! Tom, one of the Cortex authors here. Super proud of the team and this release - let me know if you have any questions!
- ctovena 7y ago
  Great job Cortex team, Do you think this means Cortex will move to incubation in the CNCF landscape ?
  - netingle 7y ago
    I hope so! Goutham is apply for incubation as we speak..
    - RichiH 7y ago
      This will also depend on SIG o11y, the creation of which is currently being voted on by CNCF TOC. TOC vote is looking good and projects which have been in sandbox for some time are obvious candidates for early review.
- number101010 7y ago
  Hey Tom! Can you outline how Cortex differs from some of the other available Prometheus backends?
  - netingle 7y ago
    Sure, check out this talk from PromCon I did with Bartek, the Thanos author: https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/ https://grafana.com/blog/2019/11/21/promcon-recap-two-househ...
    - MetalMatze 7y ago
      Love that talk. :)
  - valyala 7y ago
    VictoriaMetrics FAQ contains comparisons to Cortex [1] and Thanos [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-cortex https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#... [2] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-thanos https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...
rfratto 7y ago
Great job Cortex team!
mattmendick 7y ago
Really exciting! Well done
ones_and_zeros 7y ago
Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?
- gouthamve 7y ago
  Yes, you're running the Prometheus server. But what Cortex is a Prometheus API compatible service that horizontally scales and has multi-tenancy and other things built in.
- netingle 7y ago
  Yes, Prometheus is an implementation - the HN text has a limited number of words, so I thought "Prometheus implementation" conveyed the fact Cortex was trying to be a 100% API compatible implementation of Prometheus, but with scalability, replication etc
  - cat199 7y ago
    how about: CNCF's Cortex v1.0: scalable, fast Prometheus API implementation ready for prod (grafana.com) saves 1 char.
- ownagefool 7y ago
  It's kinda several things - The OSS product - The Storage Format (I guess) - The Interface for pulling metrics (https://github.com/OpenObservability/OpenMetrics https://github.com/OpenObservability/OpenMetrics) I haven't dug into cortex even a little, but the other comments are suggesting it's API compatible but essentially claiming they're production ready because they'll give you things the OSS project won't give you out of the box, i.e. long term storage and RBAC. Looks like a good thing.
  - netingle 7y ago
    > wrapping prometheus and giving you that production readyness that they're claiming the OSS project won't give you out of the box No! Prometheus is and has been production ready for many years. Cortex is a clustered/horizontally scalable implemention of the Prometheus APIs, and Cortex has just gone production ready. Sorry for the confusion.
stuff4ben 7y ago
This was a Weaveworks project right?
- gouthamve 7y ago
  Yes, it was created at Weaveworks, but it was later donated to CNCF and now the community is much bigger! Having said that Weaveworks is still a major contributor!
kapilvt 7y ago
also props to https://weave.works https://weave.works for creating cortex, open-sourcing it and moving it under cncf, something this blog post leaves out.
Rapzid 7y ago
Dat architecture tho: https://cortexmetrics.io/docs/architecture/ https://cortexmetrics.io/docs/architecture/ . Holy bi-gebus.
- netingle 7y ago
  Thats the "microservices" mode - you can run it as a single process and the architecture becomes super boring. Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.
  - valyala 7y ago
    According to Cortex docs [1], a single-process Cortex isn't production ready. It is intended for development and testing only. [1] https://cortexmetrics.io/docs/configuration/single-process-config/ https://cortexmetrics.io/docs/configuration/single-process-c...
nopzor 7y ago
awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fact, cortex vendors in prometheus. - you can think of cortex as a scale-out, multi-tenant, highly available "implementation" of prometheus itself. - the reason grafana labs put so much resources into cortex is because it powers our grafana cloud product (which offers a prometheus backend). like grafana itself, we are also actively working on an enterprise edition of cortex that is designed to meet the security and feature requirements of the largest companies in the world. - yes, cortex was born at weaveworks in 2016. tom wilkie (vp of product at grafana labs) co-created it while he worked there. after tom joined grafana labs in 2018, we decided to pour a lot more resources into the project, and managed to convince weave.works to move it to the cncf. this was a great move for the project and the community, and cortex has come a long long way in the last 2 years. once again, a big hat tip to everyone who made this release possible. a big day for the project, and for prometheus users in general! [edit: typos]
- Florin_Andrei 7y ago
  I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-started-chunks-storage/ https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?
  - ecnahc515 7y ago
    Thanos is probably one of the other popular choices. It's being heavily used in production by a number of companies, but I don't think they've branded it at "Prod ready" in a 1.0 release though.
    - sciurus 7y ago
throwaway50203 7y ago
Reminder: github star history is in no way a measure of quality.
zytek 7y ago
Congrats to Grafana Team! If you're looking at scaling your Prometheus setup - check out also Victoria Metrics. Operational simplicity and scalability/robustness are what drive me to it. I used to to send metrics from multiple Kubernetes clusters with Prometheus - each cluster having Prom with remote_write directive to send metrics to central VictoriaMetrics service. That way my "edge" prometheus installations are practically "stateless", easily set up using prometheus-operator. You don't even need to add persistent storage to them.
mmcclellan 7y ago
New to Cortex but when looking at a comparison of Prometheus and InfluxDB (like https://prometheus.io/docs/introduction/comparison/#prometheus-vs-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...) it appears that Cortex offers similar horizontal scalability features to the InfluxDB Enterprise offering. The linked comparison does note the difference between event logging and metrics recording but I am curious (choosy beggar that I am) whether others consider them separate tooling or whether it is possible to remain performant using one solution.
demilich 7y ago
Good job, excited!

181 points by netingle 7y ago

Hi! Tom, one of the Cortex authors here. Super proud of the team and this release - let me know if you have any questions!

ctovena 7y ago
Great job Cortex team, Do you think this means Cortex will move to incubation in the CNCF landscape ?
- netingle 7y ago
  I hope so! Goutham is apply for incubation as we speak..
  - RichiH 7y ago
    This will also depend on SIG o11y, the creation of which is currently being voted on by CNCF TOC. TOC vote is looking good and projects which have been in sandbox for some time are obvious candidates for early review.
number101010 7y ago
Hey Tom! Can you outline how Cortex differs from some of the other available Prometheus backends?
- netingle 7y ago
  Sure, check out this talk from PromCon I did with Bartek, the Thanos author: https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/ https://grafana.com/blog/2019/11/21/promcon-recap-two-househ...
  - MetalMatze 7y ago
    Love that talk. :)
- valyala 7y ago
  VictoriaMetrics FAQ contains comparisons to Cortex [1] and Thanos [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-cortex https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#... [2] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-thanos https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...

rfratto 7y ago

Great job Cortex team!

mattmendick 7y ago

Really exciting! Well done

ones_and_zeros 7y ago

Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?

gouthamve 7y ago
Yes, you're running the Prometheus server. But what Cortex is a Prometheus API compatible service that horizontally scales and has multi-tenancy and other things built in.
netingle 7y ago
Yes, Prometheus is an implementation - the HN text has a limited number of words, so I thought "Prometheus implementation" conveyed the fact Cortex was trying to be a 100% API compatible implementation of Prometheus, but with scalability, replication etc
- cat199 7y ago
  how about: CNCF's Cortex v1.0: scalable, fast Prometheus API implementation ready for prod (grafana.com) saves 1 char.
ownagefool 7y ago
It's kinda several things - The OSS product - The Storage Format (I guess) - The Interface for pulling metrics (https://github.com/OpenObservability/OpenMetrics https://github.com/OpenObservability/OpenMetrics) I haven't dug into cortex even a little, but the other comments are suggesting it's API compatible but essentially claiming they're production ready because they'll give you things the OSS project won't give you out of the box, i.e. long term storage and RBAC. Looks like a good thing.
- netingle 7y ago
  > wrapping prometheus and giving you that production readyness that they're claiming the OSS project won't give you out of the box No! Prometheus is and has been production ready for many years. Cortex is a clustered/horizontally scalable implemention of the Prometheus APIs, and Cortex has just gone production ready. Sorry for the confusion.

stuff4ben 7y ago

This was a Weaveworks project right?

gouthamve 7y ago
Yes, it was created at Weaveworks, but it was later donated to CNCF and now the community is much bigger! Having said that Weaveworks is still a major contributor!

kapilvt 7y ago

also props to https://weave.works https://weave.works for creating cortex, open-sourcing it and moving it under cncf, something this blog post leaves out.

Rapzid 7y ago

Dat architecture tho: https://cortexmetrics.io/docs/architecture/ https://cortexmetrics.io/docs/architecture/ . Holy bi-gebus.

netingle 7y ago
Thats the "microservices" mode - you can run it as a single process and the architecture becomes super boring. Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.
- valyala 7y ago
  According to Cortex docs [1], a single-process Cortex isn't production ready. It is intended for development and testing only. [1] https://cortexmetrics.io/docs/configuration/single-process-config/ https://cortexmetrics.io/docs/configuration/single-process-c...

nopzor 7y ago

awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fact, cortex vendors in prometheus. - you can think of cortex as a scale-out, multi-tenant, highly available "implementation" of prometheus itself. - the reason grafana labs put so much resources into cortex is because it powers our grafana cloud product (which offers a prometheus backend). like grafana itself, we are also actively working on an enterprise edition of cortex that is designed to meet the security and feature requirements of the largest companies in the world. - yes, cortex was born at weaveworks in 2016. tom wilkie (vp of product at grafana labs) co-created it while he worked there. after tom joined grafana labs in 2018, we decided to pour a lot more resources into the project, and managed to convince weave.works to move it to the cncf. this was a great move for the project and the community, and cortex has come a long long way in the last 2 years. once again, a big hat tip to everyone who made this release possible. a big day for the project, and for prometheus users in general! [edit: typos]

Florin_Andrei 7y ago
I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-started-chunks-storage/ https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?
- ecnahc515 7y ago
  Thanos is probably one of the other popular choices. It's being heavily used in production by a number of companies, but I don't think they've branded it at "Prod ready" in a 1.0 release though.
  - sciurus 7y ago

throwaway50203 7y ago

Reminder: github star history is in no way a measure of quality.

zytek 7y ago

Congrats to Grafana Team! If you're looking at scaling your Prometheus setup - check out also Victoria Metrics. Operational simplicity and scalability/robustness are what drive me to it. I used to to send metrics from multiple Kubernetes clusters with Prometheus - each cluster having Prom with remote_write directive to send metrics to central VictoriaMetrics service. That way my "edge" prometheus installations are practically "stateless", easily set up using prometheus-operator. You don't even need to add persistent storage to them.

mmcclellan 7y ago

New to Cortex but when looking at a comparison of Prometheus and InfluxDB (like https://prometheus.io/docs/introduction/comparison/#prometheus-vs-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...) it appears that Cortex offers similar horizontal scalability features to the InfluxDB Enterprise offering. The linked comparison does note the difference between event logging and metrics recording but I am curious (choosy beggar that I am) whether others consider them separate tooling or whether it is possible to remain performant using one solution.

demilich 7y ago

Good job, excited!