7 ms·

Using AZs can eat up your budget – From Prometheus to VictoriaMetrics

71 points by shscs911 2y ago

tomalaci 2y ago
I've used VictoriaMetrics in past (~4 years ago) for collection of not just service monitoring data but also for network switch and cell tower module metrics. At the time I found it to be the most efficient Prometheus-like service in terms of query speed, data compression and, more importantly, being able to handle high cardinality (over 10s or 100s of millions of series). However, I later switched to Clickhouse because I needed extra flexibility of running occasional async updates or deletes. In VictoriaMetrics you usually need to wipe out the entire series and re-ingest it. That may not be possible or would be quite annoying if you are dealing with a long history and you just wanted to update/delete some bad data in a month. So, if you want a more efficient Prometheus drop-in replacement and don't think limited update/delete ability is an issue then I highly recommend VictoriaMetrics. Otherwise, Clickhouse (larger scale) or Timescale (smaller scale) has been my go to for anything time series.
- brunoqc 2y ago
  Btw, both clickhouse and timescale are open core. If you care about that.
  - thayne 2y ago
    So is VictoriaMetrics
    - brunoqc 2y ago
      You are right. I guess I just saw the Apache 2 license and assumed it was foss.
  - hipadev23 2y ago
    Is there a reason you drop this comment on every product mention that’s not 100% OSS
    - presspot 2y ago
      Because it's helpful and adds context? Why do you care?
      - hipadev23 2y ago
        Because it's frustrating. They do it to belittle the projects and shame the authors for trying to make a living. Not every project wants to end up as another bloated abandonware in Apache Software Foundation.
        simfree 2y ago
        FOSS washing software is similarly frustrating. When I see a license on a project I expect that project will provide the code under that license and function fully at runtime, not play games of "Speak to a sales rep to flip that bit or three to enable that codepath".
        anonzzzies 2y ago
        I find it frustrating it is not immediately clear it is open core (in which case we shall never touch it as per our lawyers). So hopefully people will keep commenting on that.
- aramattamara 2y ago
  VictoriaMetrics does support occasional updates/deletes (e.g. you may need it for GDPR compliance).
raffraffraff 2y ago
I'd love to see a comparison with Mimir. Some of the problems that this article describes with Prometheus are also solved by Mimir. I'm running it in single binary mode, and everything is stored in S3. I'm deploying Prometheus in agent mode so it just scrapes and remote writes to Mimir, but doesn't store anything. The helm chart is a bit hairy because I have to use a fork for single binary mode, but it has actually been extremely stable and cheap to run. The same AZ cost saving rules apply, but my traffic is low enough right now for it not to matter. But I suppose I could also run ingesters per AZ to eliminate cross-AZ traffic.
- thelittleone 2y ago
  Interesting. I'm fairly new to the field, but would this configuration help reduce the cost of logging security events from multiple zones/regions/providers to a colocated cluster?
  - raffraffraff 2y ago
    Not really. On AWS, you're always going to pay an egress cost to get those logs out of AWS to your colo. If you were to ship your security logs to S3 and host your security log indexing and search services on EC2 within the same AWS region as the S3 bucket, you wouldn't have to worry about egress.
- FridgeSeal 2y ago
  I was on a team once where we ran agent-mode Prometheus into a Mimir cluster and it was endless pain and suffering. Parts of it would time out and blow up, one of the dozen components (slight hyperbole) they have you run would go down and half the cluster would go with it. It often had to be nursed back to health by hand, it was expensive to run, queries ate not even that fast. Absolutely would not repeat the experience. We cheered the afternoon we landed the PR to dump it.
  - raffraffraff 2y ago
    I definitely think that running the microservices deployment of Mimir (and Loki) looks hairy. But the monolithic deployments can handle pretty large volumes.
dantillberg 2y ago
This excessive inter-AZ data transfer pricing is distorting engineering best practices. It _should_ be cheap to operate HA systems across 2-3 AZs, but because of this price distortion on inter-AZ traffic charges, we lean towards designs that either silo data within an AZ, or that leverage S3 or other hosted solutions as a sort of accounting workaround (i.e. there are no data transfer charges to read/write an S3 bucket from any AZ in the same region). While AWS egress pricing gets a lot of attention, I think that the high cost of inter-AZ traffic is much less defensible. This is transfer on short fat pipes completely owned by Amazon. And at $0.01/GB, that's 2~10X what smaller providers charge for _internet_ egress.
- hipadev23 2y ago
  I assume it’s to discourage people from architecting designs that abuse the network. A good example would be collecting every single metric, every second, across every instance for no real business reason.
  - thayne 2y ago
    Or maybe it is price discrimination. A way to extract more money from customers that need higher availability and probably have higher budgets.
    - hipadev23 2y ago
      Price discrimination is when you charge different amounts for the same thing to different customers. And usually the difference in those prices are not made apparent. Like when travel websites quote iOS users more than Android because they generally can afford to pay more. This is just regular ole pricing.
      - thayne 2y ago
        So what is the correct term for "charge an extremely high markup for a feature that some, but not all, of your customers need"?
thayne 2y ago
> while it’s tempting to use the infinitely-scalable object storage (like S3), the good old block storage is just cheaper and more performant How is it cheaper? Object storage is cheaper per GB. Does using s3 have another component that is more expensive, maybe a caching layer? Is the storage format significantly less efficient? Are you not using a vpc enpoint to avoid egress charges?
- jdreaver 2y ago
  You are correct that storage is cheaper in S3, but S3 charges per request to GET, LIST, POST, COPY, etc objects in your bucket. Block storage can be cheaper when you are frequently modifying or querying your data.
  - thayne 2y ago
    That's a lot of requests.
    - hansvm 2y ago
      It is, but it's not _that_ many. AWS pricing is complicated, but for fairly standard services and assuming bulk discounts at ~100TB level, your break-even points for requests/network vs storage happens at: 1. (modifications) 4200 requests per GB stored per month 2. (bandwidth) Updating each byte more than once every 70 days You'll hit the break-even sooner, typically, since you incur both bandwidth and request charges. That might sound like a lot, but updating some byte in each 250KB chunk of your data once a month isn't that hard to imagine. Say each user has 1KB of data, 1% are active each month, and you record login data. You'll have 2.5x the break-even request count and pay 2.5x more for requests than storage, and that's only considering the mutations, not the accesses. You can reduce request costs (not bandwidth though) if you can batch them, but that's not even slightly tenable till a certain scale because of latency, and even when it is you might find that user satisfaction and retention are more expensive than the extra requests you're trying to avoid. Batching is a tool to reduce costs for offline workloads.
      - 2y ago
nathan_jr 2y ago
Is it possible to query cloud watch to calculate the current cost attributed to inter AZ traffic ?
sgarland 2y ago
My only hope is that as more and more companies find stuff like this out, there will be a larger shift towards on-prem / colo. Time is a circle.
- brunkerhart 2y ago
  Running on-prem/colo across two sites linked with 1 digit latency link won’t be free of charge either
remram 2y ago
(bit of a tangent) I like the new generation of metric database, but are there systems that allow for true distributed deployments? E.g. where some machines might be offline for a few days, and need to sync (send some/receive some) metrics with other machines when back online?
jiggawatts 2y ago
Azure inter-AZ traffic has been free for a while. AWS is not the only public cloud.

71 points by shscs911 2y ago

I've used VictoriaMetrics in past (~4 years ago) for collection of not just service monitoring data but also for network switch and cell tower module metrics. At the time I found it to be the most efficient Prometheus-like service in terms of query speed, data compression and, more importantly, being able to handle high cardinality (over 10s or 100s of millions of series). However, I later switched to Clickhouse because I needed extra flexibility of running occasional async updates or deletes. In VictoriaMetrics you usually need to wipe out the entire series and re-ingest it. That may not be possible or would be quite annoying if you are dealing with a long history and you just wanted to update/delete some bad data in a month. So, if you want a more efficient Prometheus drop-in replacement and don't think limited update/delete ability is an issue then I highly recommend VictoriaMetrics. Otherwise, Clickhouse (larger scale) or Timescale (smaller scale) has been my go to for anything time series.

brunoqc 2y ago
Btw, both clickhouse and timescale are open core. If you care about that.
- thayne 2y ago
  So is VictoriaMetrics
  - brunoqc 2y ago
    You are right. I guess I just saw the Apache 2 license and assumed it was foss.
- hipadev23 2y ago
  Is there a reason you drop this comment on every product mention that’s not 100% OSS
  - presspot 2y ago
    Because it's helpful and adds context? Why do you care?
    - hipadev23 2y ago
      Because it's frustrating. They do it to belittle the projects and shame the authors for trying to make a living. Not every project wants to end up as another bloated abandonware in Apache Software Foundation.
      - simfree 2y ago
        FOSS washing software is similarly frustrating. When I see a license on a project I expect that project will provide the code under that license and function fully at runtime, not play games of "Speak to a sales rep to flip that bit or three to enable that codepath".
      - anonzzzies 2y ago
        I find it frustrating it is not immediately clear it is open core (in which case we shall never touch it as per our lawyers). So hopefully people will keep commenting on that.
aramattamara 2y ago
VictoriaMetrics does support occasional updates/deletes (e.g. you may need it for GDPR compliance).

raffraffraff 2y ago

I'd love to see a comparison with Mimir. Some of the problems that this article describes with Prometheus are also solved by Mimir. I'm running it in single binary mode, and everything is stored in S3. I'm deploying Prometheus in agent mode so it just scrapes and remote writes to Mimir, but doesn't store anything. The helm chart is a bit hairy because I have to use a fork for single binary mode, but it has actually been extremely stable and cheap to run. The same AZ cost saving rules apply, but my traffic is low enough right now for it not to matter. But I suppose I could also run ingesters per AZ to eliminate cross-AZ traffic.

thelittleone 2y ago
Interesting. I'm fairly new to the field, but would this configuration help reduce the cost of logging security events from multiple zones/regions/providers to a colocated cluster?
- raffraffraff 2y ago
  Not really. On AWS, you're always going to pay an egress cost to get those logs out of AWS to your colo. If you were to ship your security logs to S3 and host your security log indexing and search services on EC2 within the same AWS region as the S3 bucket, you wouldn't have to worry about egress.
FridgeSeal 2y ago
I was on a team once where we ran agent-mode Prometheus into a Mimir cluster and it was endless pain and suffering. Parts of it would time out and blow up, one of the dozen components (slight hyperbole) they have you run would go down and half the cluster would go with it. It often had to be nursed back to health by hand, it was expensive to run, queries ate not even that fast. Absolutely would not repeat the experience. We cheered the afternoon we landed the PR to dump it.
- raffraffraff 2y ago
  I definitely think that running the microservices deployment of Mimir (and Loki) looks hairy. But the monolithic deployments can handle pretty large volumes.

dantillberg 2y ago

This excessive inter-AZ data transfer pricing is distorting engineering best practices. It _should_ be cheap to operate HA systems across 2-3 AZs, but because of this price distortion on inter-AZ traffic charges, we lean towards designs that either silo data within an AZ, or that leverage S3 or other hosted solutions as a sort of accounting workaround (i.e. there are no data transfer charges to read/write an S3 bucket from any AZ in the same region). While AWS egress pricing gets a lot of attention, I think that the high cost of inter-AZ traffic is much less defensible. This is transfer on short fat pipes completely owned by Amazon. And at $0.01/GB, that's 2~10X what smaller providers charge for _internet_ egress.

hipadev23 2y ago
I assume it’s to discourage people from architecting designs that abuse the network. A good example would be collecting every single metric, every second, across every instance for no real business reason.
- thayne 2y ago
  Or maybe it is price discrimination. A way to extract more money from customers that need higher availability and probably have higher budgets.
  - hipadev23 2y ago
    Price discrimination is when you charge different amounts for the same thing to different customers. And usually the difference in those prices are not made apparent. Like when travel websites quote iOS users more than Android because they generally can afford to pay more. This is just regular ole pricing.
    - thayne 2y ago
      So what is the correct term for "charge an extremely high markup for a feature that some, but not all, of your customers need"?

thayne 2y ago

> while it’s tempting to use the infinitely-scalable object storage (like S3), the good old block storage is just cheaper and more performant How is it cheaper? Object storage is cheaper per GB. Does using s3 have another component that is more expensive, maybe a caching layer? Is the storage format significantly less efficient? Are you not using a vpc enpoint to avoid egress charges?

jdreaver 2y ago
You are correct that storage is cheaper in S3, but S3 charges per request to GET, LIST, POST, COPY, etc objects in your bucket. Block storage can be cheaper when you are frequently modifying or querying your data.
- thayne 2y ago
  That's a lot of requests.
  - hansvm 2y ago
    It is, but it's not _that_ many. AWS pricing is complicated, but for fairly standard services and assuming bulk discounts at ~100TB level, your break-even points for requests/network vs storage happens at: 1. (modifications) 4200 requests per GB stored per month 2. (bandwidth) Updating each byte more than once every 70 days You'll hit the break-even sooner, typically, since you incur both bandwidth and request charges. That might sound like a lot, but updating some byte in each 250KB chunk of your data once a month isn't that hard to imagine. Say each user has 1KB of data, 1% are active each month, and you record login data. You'll have 2.5x the break-even request count and pay 2.5x more for requests than storage, and that's only considering the mutations, not the accesses. You can reduce request costs (not bandwidth though) if you can batch them, but that's not even slightly tenable till a certain scale because of latency, and even when it is you might find that user satisfaction and retention are more expensive than the extra requests you're trying to avoid. Batching is a tool to reduce costs for offline workloads.
    - 2y ago

nathan_jr 2y ago

Is it possible to query cloud watch to calculate the current cost attributed to inter AZ traffic ?

sgarland 2y ago

My only hope is that as more and more companies find stuff like this out, there will be a larger shift towards on-prem / colo. Time is a circle.

brunkerhart 2y ago
Running on-prem/colo across two sites linked with 1 digit latency link won’t be free of charge either

remram 2y ago

(bit of a tangent) I like the new generation of metric database, but are there systems that allow for true distributed deployments? E.g. where some machines might be offline for a few days, and need to sync (send some/receive some) metrics with other machines when back online?

jiggawatts 2y ago

Azure inter-AZ traffic has been free for a while. AWS is not the only public cloud.