4 ms·
I've personally written a (somewhat hacky but simple) setup for multiplexing high-throughput job queues for a Rails app across multiple Redis instances because
by sethhochberg 7y ago
I've personally written a (somewhat hacky but simple) setup for multiplexing high-throughput job queues for a Rails app across multiple Redis instances because our chosen job framework at the time relied a lot on Lua operations on large sets in Redis, and these could easily result in 100% CPU utilization for the Redis process.
We picked the job queue framework long before we started moving tens of thousands of jobs a second through it with a later feature, and probably exceeded its design constraints - in that instance we effectively were using Redis + jobs as a pauseable and throttleable write buffer for MySQL.
With major tooling changes we likely could have come up with something a lot more elegant, I'm not longer with that company, but our plan was always to get rid of the need for that system to hit MySQL at all, which would eliminate a lot of our need to control throughput with our Redis buffer. Of course, doing that would have taken a lot more time and effort, and in our case doing "the simplest possible thing that would work reliably and serve our customers" meant that we probably could have used a faster Redis instead of multiplexing requests and workers across multiple Redis instances on the same hardware.
Sometimes an "improved" version of a tool you're already using can be really useful, if you find yourself in a bind and the alternative is major architectural changes.