4 ms·

I use redis with rspamd and have systems that scan millions of mails per day and redis is a bottle neck for some of the modules. I’ve been impressed with dragon

by quags 3y ago

I use redis with rspamd and have systems that scan millions of mails per day and redis is a bottle neck for some of the modules. I’ve been impressed with dragonfly for the reputation set which has stopped the timeouts I was seeing and was able to replace a multi redis set up using rspamd internal hash/sharding to a single dragonflydb. I plan to replace more over but need stream support which is coming. The dev is also responsive on their discord channel.

dangoodmanUT 3y ago
1M req/day is 12 req/s. If you can't pull event 1k req/s out of it I'd assume it's being used incorrectly.
- viraptor 3y ago
  A single email doesn't have to mean a single redis query.
  - qeternity 3y ago
    Presume the upper bound of single digit millions (9) for just over 100 emails/second. Also presume a rather pedestrian single instance (single core) Redis limit of 100k/ops/sec. Are people really running software that is making 1k Redis calls/message?
    - quags 3y ago
      First let me say redis has served me well for years. Before rspamd I used spamasassassin - another perfectly fine piece of software. Having more options is a good thing. For redis I started with single instances and moved to master (for writes), slaves (for reads), or sharding/hashing depending on what rspamd uses. I moved one section so far to dragonflydb called reputation module. This is performing better for my usage. As for email, if email was broken down and nicely distributed over the same period I likely would have no issues. However, there are times where there is much more scanning going on, and a good 4 hours of peak time. This is where issues can crop up. If rspamd scan can't get a return fast enough, that test will be skipped. It still works, but I'd rather have all tests go through. So lets then take a single email and where redis is used *Ratelimit checking * Replies plugin * Some multi map are stored in redis *mx check module * neural module (a bit like bayes but automatic and short term) * fuzzy module used from spam traps * history module - for redis admin, stats So a single email is doing more than a single redis query for each email. The mail server (for smtp), which is not rspamd, is also using redis on its own. Now imagine a spike of emails, perhaps a large spamming activity that is being stopped. Needing to scan 100 emails in a second isn't impossible at peak. The scanning is also for incoming and outgoing email.
      - qeternity 3y ago
        Yeah I appreciate there is a lot going on. But is it really 1k ops/message? Even at 100 ops/message you should be able to do > 1k messages/sec without breaking a sweat.
- Alifatisk 3y ago
  Parent said "millions of mails per day", so > 1M req / day I assume.

by quags 3y ago

dangoodmanUT 3y ago

1M req/day is 12 req/s. If you can't pull event 1k req/s out of it I'd assume it's being used incorrectly.

viraptor 3y ago
A single email doesn't have to mean a single redis query.
- qeternity 3y ago
  Presume the upper bound of single digit millions (9) for just over 100 emails/second. Also presume a rather pedestrian single instance (single core) Redis limit of 100k/ops/sec. Are people really running software that is making 1k Redis calls/message?
  - quags 3y ago
    First let me say redis has served me well for years. Before rspamd I used spamasassassin - another perfectly fine piece of software. Having more options is a good thing. For redis I started with single instances and moved to master (for writes), slaves (for reads), or sharding/hashing depending on what rspamd uses. I moved one section so far to dragonflydb called reputation module. This is performing better for my usage. As for email, if email was broken down and nicely distributed over the same period I likely would have no issues. However, there are times where there is much more scanning going on, and a good 4 hours of peak time. This is where issues can crop up. If rspamd scan can't get a return fast enough, that test will be skipped. It still works, but I'd rather have all tests go through. So lets then take a single email and where redis is used *Ratelimit checking * Replies plugin * Some multi map are stored in redis *mx check module * neural module (a bit like bayes but automatic and short term) * fuzzy module used from spam traps * history module - for redis admin, stats So a single email is doing more than a single redis query for each email. The mail server (for smtp), which is not rspamd, is also using redis on its own. Now imagine a spike of emails, perhaps a large spamming activity that is being stopped. Needing to scan 100 emails in a second isn't impossible at peak. The scanning is also for incoming and outgoing email.
    - qeternity 3y ago
      Yeah I appreciate there is a lot going on. But is it really 1k ops/message? Even at 100 ops/message you should be able to do > 1k messages/sec without breaking a sweat.
Alifatisk 3y ago
Parent said "millions of mails per day", so > 1M req / day I assume.