5 ms·
How does Sidekiq really work?
- vlz 3y agoAn interesting deep dive with code into how sidekiq job processing in ruby works. Articles like these are a great way of reading (or skimming?) code and absorbing patterns without having to puzzle everything together yourself. Would also be interesting to compare this to GoodJob or basecamp's Solid Queue which work with relational databases instead of redis (and might be more specialised for use with Rails/ActiveJob).
- jddj 3y agoHangFire on .net also took some inspiration from sidekiq. I believe it works with a relatively broad range of storage technologies
- neerajdotname2 3y agoWe wrote a bit about how "Solid Queue" uses "UPDATE SKIP LOCKED". https://www.bigbinary.com/blog/solid-queue https://www.bigbinary.com/blog/solid-queue
- neilv 3y agoHow Sidekick really works: https://en.wikipedia.org/wiki/Terminate-and-stay-resident_program https://en.wikipedia.org/wiki/Terminate-and-stay-resident_pr...
- drdaeman 3y agoI'm sorry but I fail to find how this is relevant. TSR was an old simple trick in achieving a likeliness of non-preemptive multitasking on an operating system that didn't have any multitasking, but only one single active process. Sidekiq (or Celery or nearly any task runner system) are designed for multitasking operating systems and they all keep the master process running, spawning/forking worker processes (or threads or whatever primitives they use for concurrency) as needed. They don't use CPU interrupts to stay active (IRQs aren't particularly accessible to user space), as most queue systems don't exactly allow it - typically they have a sleep-poll-repeat cycle. Though if the queue system allows it they may sleep waiting for I/O (which, in turn, indeed, could be triggered by an IRQ), but that's the closest resemblance I can find - and it's quite a stretch.
- viraptor 3y ago> configuration file (...) this file will be evaluated before the Rails application or (...) which makes referencing constants defined in the application code impossible That's not strictly true. You can use them just fine if there's a good reason to do it. For example this erb file works as expected: <% require 'socket' %> <%= Socket.gethostname %> Useful if you have a specific provider for the config values.
- dansvetlov 3y agoIt's true that you can write arbitrary code in the template and reference other gems if needed. However, to reference constants located in any directory of your application, such as `app/` or `lib/`, you'd need to either explicitly require these files, overtaking the eager- or auto-loading mechanism of your app framework, or boot the whole application. So, you're right that it's not impossible, but it requires some additional considerations.
- echelon 3y agoIs anybody using Rails / Ruby in production these days? For new projects, not 7+ year old companies. (Earnest / honest question.) If so, why? Do you have trouble hiring? If not, what did you switch to and why?
- fragmede 3y agoWhat would you use instead?
- echelon 3y agoWe're using Rust/Actix, but I assume a lot of shops went to Python/Flask/Django, Go, and Node/Deno, etc. I've also heard of Nim and Swift in use. Wondering what folks are up to.
- danpalmer 3y agoI'd still choose Python/Django as the basis for a startup now. Would probably choose Ruby/Rails if that was where my expertise lied, although I still see lots of Python engineers (in London) and few Ruby engineers. I think the pattern of a monolith that's very easy to rapidly develop on, plus specialist services around the edge that might be high performance/security/etc and are written in languages like Rust is a pretty good strategy. Personally, having seen the speed of development on a Django project, it's hard to justify much else for the boring CRUD core of a service.
- petepete 3y agoThe HN bubble in full effect, where people are leaving Rails en masse to use Nim to build web apps.
- satvikpendem 3y agoThey never implied "en masse." But yes, for what it's worth, those that would use Rails 10 years ago are now increasingly using JS based frameworks, such as NodeJS with express and/or Next.js. In fact, many people who are in web dev these days only know JS/TypeScript, they know zero Ruby at all.
- mberning 3y agoSidekiq is a work of art. I’m so glad Mike Perham was able to turn it into a nice business. Truly well deserved.
- throwaway918274 3y agoSidekiq is incredible. Been using it for almost a decade and it's just been absolutely rock solid. I put it on the same level as Postgres and Redis is my "GOAT-tier" software list. When I saw he created a language agnostic sibling "Faktory", I started recommending it to all my non-Ruby colleagues looking for a background job processor.
- groseje 3y agoIt works like charm! I love sidekiq. It powers Microsoft’s one of the biggest database management services controlplane. It can easily handle hundreds of thousands of operations with a proper deployment.
- Alifatisk 3y ago> It powers Microsoft’s one of the biggest database management services controlplane. Really? Where can I read more about it?
- groseje 3y agoThere are no blog posts on the controlplane side, so I cannot really share any reading materials with you. Thanks to the NDA, I cannot really share the product’s name as well. However, considering the nature of sidekiq with ruby + postgresql, you might guess which of the products is powered by sidekiq :)
- Avicebron 3y agoI think they mean github
- komuW 3y agoNice article. If you ever wondered how the python equivalent(celery) works, I wrote a blogpost about that ~4years ago; https://www.komu.engineer/blogs/07/understand-how-celery-works https://www.komu.engineer/blogs/07/understand-how-celery-wor...
- mperham 3y agoThis is amazing. Dan, if you see this get in touch.
- xcrunner529 3y agoIsn’t sidekiq a big part of the problem of why Mastodon can never scale? Tons of hacking just keeping the larger networks running.
- j_kao 3y agoA bit of an aside, I'm a big fan of how Mike Perham managed to start a business around Sidekiq. I think it really gives us Hacker News folks that there's hope starting a viable bootstrapped business around open-source infrastructure.