3 ms·
I took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby? Have you also tested the performance of this with 40
by AznHisoka 13y ago
I took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby?
Have you also tested the performance of this with 40-50 workers doing intensive I/O work such as crawling web pages? One of the main disadvantages of Sidekiq is that the workers just freeze every 15-30 minutes when you have a ton of workers crawling web pages. The only workaround for me is to setup a cron job that restarts the workers if I detect this pattern.
- benmanns 13y agoYes, workers are written in Go, but can be queued from Ruby. I'll put together an example in the goworker-examples [0] to test web crawling. [0] https://github.com/benmanns/goworker-examples https://github.com/benmanns/goworker-examples
- dkhenry 13y agoSo workers written in go are 1000x times faster then workers written in Ruby? It almost seemed from the documentation that the workers were written in Ruby as well.
- benmanns 13y agoThey are typically 10x faster in "raw" speed and 100x faster for concurrency. However, since Go's goroutines take almost no memory, you can run over 1,000 workers in the same amount of memory as one Ruby worker. On the queuing side, you use Ruby. For the examples, I was trying to show what the "equivalent" Ruby worker was so you could get an idea of what kind of workload it was. However, I do see that it is confusing.
- oelmekki 13y agoThat looks great, congrats. I suppose the main use case is working on data loosely coupled with web app, thus ? For example, if we have to do any big processing on active_record objects, we would have to reimplement all model stack in go ?