4 ms·

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of

by agentgt 10y ago

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k messages/s when embarrassingly had debug turned on in production and RabbitMQ did not fail but Elastic Search and sadly Flume did as well (I tried to get rid of flume with a custom Rust AMQP to Elastic Search client but at the time had some bugs with the libraries.. Maybe I will recheck out Mozilla Heka someday). There is this sort of beating of the developer chest with a lot of tech companies.. that hey listen we are ultra important and we are dealing with ridiculously traffic and we need ultra high performance. Please tell/show me these numbers.... Or maybe stop logging crap you don't need to log. Or maybe I'm wrong and we should log absolutely everything and Auth0 made the right choice given their needs (lets assume they have millions of messages a second), I still think I could make a sharded RabbitMQ go pretty far. This goes with other technology as well. You don't need to pick hot glamorous NoSQL when Postgresql or MySQL and a tiny bit of engineering will get the job done just fine particularly when mature solutions give you such many things free out of the box (RabbitMQ gives you a ton of stuff like a cool admin UI and routing that you would have to build in ZeroMQ).

brightball 10y ago
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
- agentgt 10y ago
  Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
  - brightball 10y ago
    No apologies needed, lol. I went to Clemson. Just saw it and was curious.
packetized 10y ago
We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.
- agentgt 10y ago
  I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace. BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it. In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).
  - trimbo 10y ago
    > cluster failing with some network issues we had with Rackspace Were you using Zen Discovery at the time? I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.
  - gerakinis 10y ago
    You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)
- trimbo 10y ago
  > We run an average of 14k logs/sec through a two-node RMQ cluster How many MB/s are you indexing?
- benmccann 10y ago
  What are the new options that help improve indexing rate?
gerakinis 10y ago
Heka is really stupid to configure, really fast, and now deprecated... I've used it for log tailing and metrics forwarding extensively and can't recommend it enough if you need to use amqps out. If you don't need amqps out there are more modern, better supported projects.
smetj 10y ago
100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s? I'd be really interested to hear how you can achieve such a thoughput with rabbitmq
- agentgt 10y ago
  I'm not sure how much I can help because I didn't setup the RMQ cluster so I don't know the configuration details but I know it is fairly powerful (its also partly why I can't entirely be critical of Auth0 because ZMQ is probably far less expensive infrastructure wise). I do know we use multiple queues and even exchanges (and I did not know about the one core to queue). A simple googling shows though folks have achieved far greater throughput[1] than 100k (and by the way this wasn't sustained.. it was spikes). [1]: https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-m...

by agentgt 10y ago

brightball 10y ago

Agree with everything you said, just curious is the GT in your name for Georgia Tech?

agentgt 10y ago
Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
- brightball 10y ago
  No apologies needed, lol. I went to Clemson. Just saw it and was curious.

packetized 10y ago

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

agentgt 10y ago
I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace. BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it. In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).
- trimbo 10y ago
  > cluster failing with some network issues we had with Rackspace Were you using Zen Discovery at the time? I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.
- gerakinis 10y ago
  You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)
trimbo 10y ago
> We run an average of 14k logs/sec through a two-node RMQ cluster How many MB/s are you indexing?
benmccann 10y ago
What are the new options that help improve indexing rate?

gerakinis 10y ago

Heka is really stupid to configure, really fast, and now deprecated... I've used it for log tailing and metrics forwarding extensively and can't recommend it enough if you need to use amqps out. If you don't need amqps out there are more modern, better supported projects.

smetj 10y ago

100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s? I'd be really interested to hear how you can achieve such a thoughput with rabbitmq

agentgt 10y ago
I'm not sure how much I can help because I didn't setup the RMQ cluster so I don't know the configuration details but I know it is fairly powerful (its also partly why I can't entirely be critical of Auth0 because ZMQ is probably far less expensive infrastructure wise). I do know we use multiple queues and even exchanges (and I did not know about the one core to queue). A simple googling shows though folks have achieved far greater throughput[1] than 100k (and by the way this wasn't sustained.. it was spikes). [1]: https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-m...