6 ms·
Show HN: goworker – a faster, Resque-compatible background worker
- benmanns 13y agoThis is a weekend project I started on as a way to learn Go that I quickly realized would be something worth sharing. It lets you queue jobs from Rails that are processed in Go, which can speed things up considerably. I would appreciate all kinds of feedback, and for anyone who wants to try goworker out, if you give me your email at the bottom of the page, I'll send you my contact info for some free one-on-one help getting started.
- atonse 13y agoVery cool - I just interviewed with a company that is using Sidekiq to process up to millions of jobs (at about 2/sec per worker) and I asked them to look into using Go for workers. I can send this to them.
- benmanns 13y agoAwesome! Can you send me your contact info? Mine is in my HN profile.
- sylvinus 13y agoWould be awesome to have RQ compatibility too ;-) http://python-rq.org http://python-rq.org
- benmanns 13y agoHmm. I took a look at this and it could be a possibility, but RQ's message format is much different from Resque and Sidekiq's. See https://gist.github.com/benmanns/6583712 https://gist.github.com/benmanns/6583712 for a comparison.
- hmottestad 13y agoPlease label your graphs. X-axis, Y-axis and title :)
- benmanns 13y agoAh, yes. I'm using Flot and was having trouble getting good looking labels. I'll work on that right away, but in the meantime: X-axis is jobs, Y-axis is seconds to execute.
- adefa 13y agoThere is Java implementation of Resque as well: https://github.com/gresrun/jesque https://github.com/gresrun/jesque I've been using jesque to consume Resque tasks with no problems, and performance is great.
- benmanns 13y agoCool! I'll check Jesque out and add it to the benchmarks.
- mikeni 13y agointerested in your findings regarding goworker, so the worker logic has to be written in go correct? for my rails app if its something simple like sending email, I can use that, but if it involves something where I need activerecord is it still a good option?
- gresrun 13y agoAuthor of Jesque here. Jesque has numerous additional features over Resque, including ability to dynamically change the queues that a worker polls and an efficient pooled worker implementation. Also available is a Java version of resque-web [1] [1] https://github.com/gresrun/jesque-web https://github.com/gresrun/jesque-web
- Titanous 13y agoI have a similar project that we use in production; a Sidekiq clone in Go: https://github.com/cupcake/gokiq https://github.com/cupcake/gokiq I should really document it. It has full type safety, stats, scheduled jobs, error reporting, retries, etc.
- jemeshsu 13y agoInterestingly there is a Sidekiq compatible job queue written in Go that has a similar name: go-workers https://github.com/jrallison/go-workers https://github.com/jrallison/go-workers Wondering how these ruby/rails inspired work queues compare with NSQ.
- jonnii 13y agoI'm using this and it works great. We even managed to get it deploy to heroku, which was pretty magic.
- benmanns 13y agoHow do you handle when Heroku sends a SIGTERM to kill your process? I couldn't find a way to preempt running workers, so everything running on Heroku has to finish within 10 seconds or you can lose it.
- jonnii 13y agoI'm not sure tbh, we haven't had that problem yet.
- jrallison 13y agoHey Ben, Author of go-workers here. On SIGTERM, I stop accepting new work, and wait for all running workers to finish before halting. If workers take longer than the 10 seconds Heroku gives you, go-workers uses reliable queueing (using http://redis.io/commands/brpoplpush http://redis.io/commands/brpoplpush) so the job will run again next time you start up the process.
- benmanns 13y agoOkay, cool. I do the same thing on the polling side, but don't use reliable queueing yet. I think that is probably the best way to handle the failures.
- rargulati 13y ago
- AznHisoka 13y agoI took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby? Have you also tested the performance of this with 40-50 workers doing intensive I/O work such as crawling web pages? One of the main disadvantages of Sidekiq is that the workers just freeze every 15-30 minutes when you have a ton of workers crawling web pages. The only workaround for me is to setup a cron job that restarts the workers if I detect this pattern.
- benmanns 13y agoYes, workers are written in Go, but can be queued from Ruby. I'll put together an example in the goworker-examples [0] to test web crawling. [0] https://github.com/benmanns/goworker-examples https://github.com/benmanns/goworker-examples
- dkhenry 13y agoSo workers written in go are 1000x times faster then workers written in Ruby? It almost seemed from the documentation that the workers were written in Ruby as well.
- benmanns 13y agoThey are typically 10x faster in "raw" speed and 100x faster for concurrency. However, since Go's goroutines take almost no memory, you can run over 1,000 workers in the same amount of memory as one Ruby worker. On the queuing side, you use Ruby. For the examples, I was trying to show what the "equivalent" Ruby worker was so you could get an idea of what kind of workload it was. However, I do see that it is confusing.
- oelmekki 13y agoThat looks great, congrats. I suppose the main use case is working on data loosely coupled with web app, thus ? For example, if we have to do any big processing on active_record objects, we would have to reimplement all model stack in go ?
- skion 13y agoAnd so the age starts where we rewrite in Go for performance all cool things Ruby, Python...
- benmanns 13y agoThat was my thought. However, with goworker you can keep 9/10 workers in Ruby and only your most used or most resource-intensive in Go.
- zellyn 13y agoPerformance, and/or deployabity...
- agentultra 13y agoYou also forget the competition from Javascript as well. Rewrite all of the things in all of the languages. I'd much rather see neat algorithms and libraries being re-written in C with an FFI-friendly API. I think it would reduce the number of "Show HN: I rewrote X in Y" posts. The majority of HN users rarely care about Y. Such posts are just attention-grabbing noise. If it was written in C you could just write bindings to it in Y and nobody but Y developers would need to hear about it. Instead we could read about X on HN. X is interesting.
- agentultra 13y agoCould the down-voters enlighten me as to what was wrong with my comment/criticism?
- iamwil 13y agoActually, I've wondered if there's a docker for libraries in different languages. The closest I've found are http APIs at the application/service level, or libraries mature enough to be run at the command line, so you can use them in conjunction with other unix commands. It doesn't seem right that we can't leverage libraries in other languages without significant effort.
- jrochkind1 13y ago> For simple dequeuing, deserialization, execution, and updating stats (as with the Hello worker above), goworker is 80x faster than Resque and 23x faster than Sidekiq. In real world use cases, how often is that kind of overhead a significant portion of total run time, compared with the actual work? If it's a small portion, would that make the speed up fairly irrelevant, does it actually speed up real world use cases significantly? Of course, your actual work payload may be faster too in go than ruby if you know how to write it well.
- bradleyland 13y ago> In real world use cases, how often is that kind of overhead a significant portion of total run time, compared with the actual work? I wish more people paid attention to questions like this. Be it web servers, app servers, whatever; too many people look at microbenchmarks and decide that a move is imperative when they're really only optimizing a tiny fraction of the total run time. > Of course, your actual work payload may be faster too in go than ruby if you know how to write it well. I think that this is the big bet you're going to make when you choose goworker over a Ruby implementation. What makes this intriguing is that in a web app scenario, background workers are often used for tasks that would take too long to run in the HTTP request/response cycle. This means there is a selection bias toward tasks that may be computationally difficult. That's not to say that all tasks handed off to background workers are CPU bound; many (most, maybe?) are I/O bound, but for the class of problems that are CPU bound, this offers a great solution.
- derefr 13y agoBut for the CPU-bound cases, people are usually calling into a library implemented in C either way (this is why Resque uses a forking model instead of green threads, after all.) The only speed-up you'll get is in the glue, because the task itself will probably use the same C library.
- bradleyland 13y agoIf you're already doing the majority of your work (total run time) in C code, then of course you're not going to benefit from a Golang back-end, but I'd challenge the "usually" portion of your claim. C wrapped in Ruby is not the same as pure C or Golang. The tool-selection argument quickly devolves in to the usual arguments between the benefits of languages like Ruby versus lower level languages, but I'm not sure that's the most productive conversation. That's well traveled ground. What I'm saying is that I'm happy to see this (Golang workers available through a Resque API) as an option. I'm not advocating blindly replacing C implementations (or any other implementation), but I wouldn't be surprised to see a Golang back-end outperform Ruby code, even when that Ruby code is calling in to C libraries. The usual rules apply when making that evaluation. You have to see for yourself.
- g5pw 13y agoA nice log graph would be better than the linear ones IMO, especially when the scale stretches a lot (like in the Database Insert tab)
- benmanns 13y agoTrue, but I thought that that would look disingenuous, because it would look like the worker times scale logarithmically when they are about as linear as you can get.
- kohanz 13y agoIn my side-project, which is my first foray into Rails, I use a Resque background worker to do processing and it takes a significant amount of time. As I've heard more about Go, I've wondered how I could use it to speed up this background process. Here is my answer! Thank you very much, I'm looking forward to trying this out as it fits my needs exactly.
- benmanns 13y agoAwesome! Contact me (info in HN profile), and I'll help you get started.
- steveklabnik 13y agoHello there! Resque maintainer here. This is super neat! I'll make sure to link to goworker in our docs. I have wanted to do something like this, but haven't gotten around to it yet. Please let me know how I can help you stay abreast of what we're doing with Resque 2.
- benmanns 13y agoHello! Thanks for your comment and tweet. I've been watching both Resque and Resque 2 while writing goworker. I would definitely like to stay in contact with you.
- joevandyk 13y agoHow would you write integration tests if you had a rails app that pushed jobs out to goworker? Say a user clicks a link that creates an async job that does a API call somewhere. What's a good way to test that entire process at once? You can easily test them separately, that clicking a link creates an async job, and that the worker can process the job, but it's useful to test the whole system at once.
- benmanns 13y agoThat's a great question that I haven't thought about before. You could possibly write a Go wrapper that executes your workers, getting the job from a command line argument rather than Redis. https://gist.github.com/benmanns/6584142 https://gist.github.com/benmanns/6584142
- joevandyk 13y agoThe test should use redis though, to test the whole stack. Same question goes for any time of queue system, like quque_classic, which stores the jobs in a postgresql table.