3 ms·

I'd love to see a comparison with Mimir. Some of the problems that this article describes with Prometheus are also solved by Mimir. I'm running it in single bin

I'd love to see a comparison with Mimir. Some of the problems that this article describes with Prometheus are also solved by Mimir. I'm running it in single binary mode, and everything is stored in S3. I'm deploying Prometheus in agent mode so it just scrapes and remote writes to Mimir, but doesn't store anything. The helm chart is a bit hairy because I have to use a fork for single binary mode, but it has actually been extremely stable and cheap to run. The same AZ cost saving rules apply, but my traffic is low enough right now for it not to matter. But I suppose I could also run ingesters per AZ to eliminate cross-AZ traffic.

thelittleone 2y ago
Interesting. I'm fairly new to the field, but would this configuration help reduce the cost of logging security events from multiple zones/regions/providers to a colocated cluster?
- raffraffraff 2y ago
  Not really. On AWS, you're always going to pay an egress cost to get those logs out of AWS to your colo. If you were to ship your security logs to S3 and host your security log indexing and search services on EC2 within the same AWS region as the S3 bucket, you wouldn't have to worry about egress.
FridgeSeal 2y ago
I was on a team once where we ran agent-mode Prometheus into a Mimir cluster and it was endless pain and suffering. Parts of it would time out and blow up, one of the dozen components (slight hyperbole) they have you run would go down and half the cluster would go with it. It often had to be nursed back to health by hand, it was expensive to run, queries ate not even that fast. Absolutely would not repeat the experience. We cheered the afternoon we landed the PR to dump it.
- raffraffraff 2y ago
  I definitely think that running the microservices deployment of Mimir (and Loki) looks hairy. But the monolithic deployments can handle pretty large volumes.
aramattamara 2y ago
Performance benchmark: https://victoriametrics.com/blog/mimir-benchmark/ https://victoriametrics.com/blog/mimir-benchmark/

thelittleone 2y ago

Interesting. I'm fairly new to the field, but would this configuration help reduce the cost of logging security events from multiple zones/regions/providers to a colocated cluster?

raffraffraff 2y ago
Not really. On AWS, you're always going to pay an egress cost to get those logs out of AWS to your colo. If you were to ship your security logs to S3 and host your security log indexing and search services on EC2 within the same AWS region as the S3 bucket, you wouldn't have to worry about egress.

FridgeSeal 2y ago

I was on a team once where we ran agent-mode Prometheus into a Mimir cluster and it was endless pain and suffering. Parts of it would time out and blow up, one of the dozen components (slight hyperbole) they have you run would go down and half the cluster would go with it. It often had to be nursed back to health by hand, it was expensive to run, queries ate not even that fast. Absolutely would not repeat the experience. We cheered the afternoon we landed the PR to dump it.

raffraffraff 2y ago
I definitely think that running the microservices deployment of Mimir (and Loki) looks hairy. But the monolithic deployments can handle pretty large volumes.

aramattamara 2y ago

Performance benchmark: https://victoriametrics.com/blog/mimir-benchmark/ https://victoriametrics.com/blog/mimir-benchmark/