5 ms·

Optimizing Sidekiq

43 points by nateberkopec 11y ago

sandGorgon 11y ago
How do you guys deploy Sidekiq. The big problem is when I deploy Sidekiq through Foreman, then I always end up with hung processes on restart. Somehow, the TERM signal does not work very well. I have now started using supervisor for deploying Sidekiq, but I would have preferred a Foreman like tool (so that development is also nice and simple)
- eropple 11y ago
  I just kick off the application with a bash script for both dev and prod and push it through supervisor. (Remember to set `stopasgroup` to true, though, or else you can easily orphan processes and you will be sad.)
- JonnieCache 11y ago
  I've always just done `bundle exec sidekiqctl stop tmp/blah_worker.pid` at deploy-time, and then let the process monitoring system bring them back up. It works fine. I've never had a problem with orphans in production, but our apps dont rely on sidekiq for their core functionality.
diminish 11y ago
GC Disabled, Memory required to process 10,000 jobs reduced from 1257 MB to 151 MB . So it may mean (unless there's a general overhead) that, 15KB of memory is used vs. 125.7KB previously. assuming the job still itself leaks the same memory it means sidekiq itself was leaking 125.7KB per job previously, am I right?
- schneems 11y ago
  Leaking is the wrong word. A leak implies that the memory was retained, RAM usage will go up forever and the process will eventually crash as a result. This is memory that sidekiq needs to do it's job, allocating strings, hashes, arrays etc. that will all be cleaned up eventually. Larger, more complex objects take up more memory. I.e. strings take up less memory than a hash. What he has done is eliminate the need to allocate so many large objects. This means less GC is needed, and overall performance is increased. See http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html for more info.
unknownzero 11y ago
I know what Sidekiq is, but that doesn't stop me from always thinking these articles are about hacking the old Sidekick phone https://en.wikipedia.org/wiki/Danger_Hiptop https://en.wikipedia.org/wiki/Danger_Hiptop and getting weirdly nostalgic. Words are strange.

43 points by nateberkopec 11y ago

How do you guys deploy Sidekiq. The big problem is when I deploy Sidekiq through Foreman, then I always end up with hung processes on restart. Somehow, the TERM signal does not work very well. I have now started using supervisor for deploying Sidekiq, but I would have preferred a Foreman like tool (so that development is also nice and simple)

eropple 11y ago
I just kick off the application with a bash script for both dev and prod and push it through supervisor. (Remember to set `stopasgroup` to true, though, or else you can easily orphan processes and you will be sad.)
JonnieCache 11y ago
I've always just done `bundle exec sidekiqctl stop tmp/blah_worker.pid` at deploy-time, and then let the process monitoring system bring them back up. It works fine. I've never had a problem with orphans in production, but our apps dont rely on sidekiq for their core functionality.

diminish 11y ago

GC Disabled, Memory required to process 10,000 jobs reduced from 1257 MB to 151 MB . So it may mean (unless there's a general overhead) that, 15KB of memory is used vs. 125.7KB previously. assuming the job still itself leaks the same memory it means sidekiq itself was leaking 125.7KB per job previously, am I right?

schneems 11y ago
Leaking is the wrong word. A leak implies that the memory was retained, RAM usage will go up forever and the process will eventually crash as a result. This is memory that sidekiq needs to do it's job, allocating strings, hashes, arrays etc. that will all be cleaned up eventually. Larger, more complex objects take up more memory. I.e. strings take up less memory than a hash. What he has done is eliminate the need to allocate so many large objects. This means less GC is needed, and overall performance is increased. See http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html for more info.

unknownzero 11y ago

I know what Sidekiq is, but that doesn't stop me from always thinking these articles are about hacking the old Sidekick phone https://en.wikipedia.org/wiki/Danger_Hiptop https://en.wikipedia.org/wiki/Danger_Hiptop and getting weirdly nostalgic. Words are strange.