5 ms·

Prometheus, but Bigger

57 points by kiyanwang 5y ago

deleted 5y ago
[deleted]
benchess 5y ago
The key constraint here is that the author has no access to persistent disks and can only use object storage for persistence. Otherwise Thanos would be extreme overkill for this number of metrics. Single-node VictoriaMetrics can easily handle 1M metrics/sec
- zzyzxd 5y ago
  > Thanos would be extreme overkill for this number of metrics Data volume is just one thing. Thanos makes Prometheus stateless and easy to shard, all in a non-invasive approach that is solid, boring, and just works. The architecture works well in small scale systems. I even use it in a single node k8s cluster in my homelab, pays only about ~$1 a month for Backblaze B2 so I never worry about data retention or disk usage. > Single-node VictoriaMetrics can easily handle 1M metrics/sec Even if I have disk access, I would think twice before deploying a database and manage it myself when I don't have to. Besides the maintenance burden and potential scaling issues in the future, it may cost you more to use block storage like EBS than S3. Also, Prometheus memory usage overhead for remote write was wild[1], so, good luck with capacity planning and config tweaking. 1. https://prometheus.io/docs/practices/remote_write/ https://prometheus.io/docs/practices/remote_write/
- znpy 5y ago
  > Single-node VictoriaMetrics can easily handle 1M metrics/sec yeah sure but then what will you be posting on Linkedin with buzzwords? "I have installed a boring software on a single machine because it works, performs well enough and it's cheap" ? what are you going to say, "I take daily snapshots of the vm because 1-day rpo is fine for me"? How will you get them likes?? (this is an ironic comment, before anyone starts a flame)
Ne02ptzero 5y ago
> For the monthly expenses, with most of the components running on-premises, there was a 90.61% cost reduction, going from US$ 38,421.25 monthly to US$ 3,608.99, including the AWS services cost. I might be missing something, but $3k/month (let alone $38k/month) sounds absolutely insane to me for how little metrics they're collecting (4k5 metrics per second, 2.7TB of data per year). Is the money going for network bandwidth or something along those lines?
- ericbarrett 5y ago
  AWS, for example, charges for cross-AZ data transfers. Naive setups with multiple AZs (us-west-1a, 1b, etc.) and a centralized Prometheus setup will rack up quite the cost.
  - gregimba 5y ago
    The reducing cross-AZ data transfer savings on one service resulted in a low 6 figure per year savings. Its something we overlooked during initial setup and now its something I check for when dealing with AWS networking.
    - otterley 5y ago
      Out of curiosity, what's your plan for recovery during an AZ or regional outage?
      - manyxcxi 5y ago
        If RDS or other “hard to replicate very quickly during disaster” infra is being run I personally would still have cross A-Z replication at minimum, to reduce network costs I would configure the “other zone” as a backup replica only and not for performance clustering. With automation we can spin up full new compute stacks, including load balancers and DNS in about 5-10 minutes per “unique” environment configuration. While it guarantees we could never have a no downtime failover, we’re okay with it and have more than halved our network costs (which admittedly were about number 8 on our AWS bill by cost).
nwmcsween 5y ago
I dont understand the need to NiH things, why not clickhouse with weekly, monthly, etc aggregation. It comes with built in sharding, can store time series data somewhat efficiently and doesn't have some hack h/a setup (in general not thanos).
- BurritoAlPastor 5y ago
  Wouldn’t Clickhouse be the NIH solution? Thanos supports Prometheus natively, but Clickhouse doesn’t (or at least doesn’t appear on the integrations list in the Prometheus docs), so you’d have to write an adapter for it.
  - nwmcsween 5y ago
    I'll bite, no because Prometheus, Thanos, etc all have to reinvent sharding (Thanos), a query language (Promql), vectorizing (Prometgeus can't), transactions (Prometheus internals), etc when it all exists in clickhouse and uses somewhat standard SQL. There are trade-offs but imo it's not worth NiHing something
luizrojo 5y ago
The solution here is meant to be expanded easily and to keep things at a low cost. The “insane” prices were from NewRelic, using the Kubernetes integration and infrastructure monitoring. I am based in Brasil, so every price is “times 5” plus taxes. Right now we have a small number of clusters, but it will be growing on a fast pace in the future. For us, monitoring the new clusters will come easy as deploying a couple of new prometheus PODs inside the new cluster, adding a couple lines to our deploy pipeline. As for doing in house, since we have a complete private cloud solution, it makes sense for us, as mentioned by zzyzxd in another comment, Thanos is pretty straight forward and just works, it demands almost zero maintenance, we just keep an eye for storage consumption and new releases.

57 points by kiyanwang 5y ago

deleted 5y ago

[deleted]

The key constraint here is that the author has no access to persistent disks and can only use object storage for persistence. Otherwise Thanos would be extreme overkill for this number of metrics. Single-node VictoriaMetrics can easily handle 1M metrics/sec

zzyzxd 5y ago
> Thanos would be extreme overkill for this number of metrics Data volume is just one thing. Thanos makes Prometheus stateless and easy to shard, all in a non-invasive approach that is solid, boring, and just works. The architecture works well in small scale systems. I even use it in a single node k8s cluster in my homelab, pays only about ~$1 a month for Backblaze B2 so I never worry about data retention or disk usage. > Single-node VictoriaMetrics can easily handle 1M metrics/sec Even if I have disk access, I would think twice before deploying a database and manage it myself when I don't have to. Besides the maintenance burden and potential scaling issues in the future, it may cost you more to use block storage like EBS than S3. Also, Prometheus memory usage overhead for remote write was wild[1], so, good luck with capacity planning and config tweaking. 1. https://prometheus.io/docs/practices/remote_write/ https://prometheus.io/docs/practices/remote_write/
znpy 5y ago
> Single-node VictoriaMetrics can easily handle 1M metrics/sec yeah sure but then what will you be posting on Linkedin with buzzwords? "I have installed a boring software on a single machine because it works, performs well enough and it's cheap" ? what are you going to say, "I take daily snapshots of the vm because 1-day rpo is fine for me"? How will you get them likes?? (this is an ironic comment, before anyone starts a flame)

Ne02ptzero 5y ago

> For the monthly expenses, with most of the components running on-premises, there was a 90.61% cost reduction, going from US$ 38,421.25 monthly to US$ 3,608.99, including the AWS services cost. I might be missing something, but $3k/month (let alone $38k/month) sounds absolutely insane to me for how little metrics they're collecting (4k5 metrics per second, 2.7TB of data per year). Is the money going for network bandwidth or something along those lines?

ericbarrett 5y ago
AWS, for example, charges for cross-AZ data transfers. Naive setups with multiple AZs (us-west-1a, 1b, etc.) and a centralized Prometheus setup will rack up quite the cost.
- gregimba 5y ago
  The reducing cross-AZ data transfer savings on one service resulted in a low 6 figure per year savings. Its something we overlooked during initial setup and now its something I check for when dealing with AWS networking.
  - otterley 5y ago
    Out of curiosity, what's your plan for recovery during an AZ or regional outage?
    - manyxcxi 5y ago
      If RDS or other “hard to replicate very quickly during disaster” infra is being run I personally would still have cross A-Z replication at minimum, to reduce network costs I would configure the “other zone” as a backup replica only and not for performance clustering. With automation we can spin up full new compute stacks, including load balancers and DNS in about 5-10 minutes per “unique” environment configuration. While it guarantees we could never have a no downtime failover, we’re okay with it and have more than halved our network costs (which admittedly were about number 8 on our AWS bill by cost).

nwmcsween 5y ago

I dont understand the need to NiH things, why not clickhouse with weekly, monthly, etc aggregation. It comes with built in sharding, can store time series data somewhat efficiently and doesn't have some hack h/a setup (in general not thanos).

BurritoAlPastor 5y ago
Wouldn’t Clickhouse be the NIH solution? Thanos supports Prometheus natively, but Clickhouse doesn’t (or at least doesn’t appear on the integrations list in the Prometheus docs), so you’d have to write an adapter for it.
- nwmcsween 5y ago
  I'll bite, no because Prometheus, Thanos, etc all have to reinvent sharding (Thanos), a query language (Promql), vectorizing (Prometgeus can't), transactions (Prometheus internals), etc when it all exists in clickhouse and uses somewhat standard SQL. There are trade-offs but imo it's not worth NiHing something

luizrojo 5y ago

The solution here is meant to be expanded easily and to keep things at a low cost. The “insane” prices were from NewRelic, using the Kubernetes integration and infrastructure monitoring. I am based in Brasil, so every price is “times 5” plus taxes. Right now we have a small number of clusters, but it will be growing on a fast pace in the future. For us, monitoring the new clusters will come easy as deploying a couple of new prometheus PODs inside the new cluster, adding a couple lines to our deploy pipeline. As for doing in house, since we have a complete private cloud solution, it makes sense for us, as mentioned by zzyzxd in another comment, Thanos is pretty straight forward and just works, it demands almost zero maintenance, we just keep an eye for storage consumption and new releases.