4 ms·
Writing A Job Runner (In Elixir) (Again) (10 years later)
- rhgraysonii 1y agoI wrote this up after having written a tutorial on doing this in GenStage around a decade ago, and thought it was interesting to have the two of them side by side to consider. I linked the original in the document linked to here. Overall, I'm wildly impressed at how this Elixir code held up, and it was a joy to revisit this.
- freedomben 1y agoGreat write up, thanks for sharing! Nothing against Oban, but it is nice seeing someone in the Elixir community not just say, "run Oban" and drop the mic
- dangerousnight 1y agoAs someone who has been learning Elixir on and off for over a year, this looks really exciting on first skim through. Excited to give this a deep read this weekend!
- rhgraysonii 1y agoIf you have any feedback or anything is unclear feel free to open an issue. I am thinking I am going to take this and expand the concepts to start as a beginners primer going through the primary concepts in brief akin to Elixir School, and then expand it into building this and a web service that is using it and offering some real time features.
- sbrother 1y agoFYI if anyone is looking for a production ready job runner in Elixir, I strongly recommend Oban. They have a paid pro version but the open source version is still absolutely fantastic.
- victorbjorklund 1y agoJust to add to the parent the Open Source version and the Pro Version just differs in that the Pro Version has a few plugins that the normal version has (for example workflows where you can have multiple workers working together) but it is not something you need for most use-cases.
- pikdum 1y agoIt's more than just a few - even more basic things like rate limiting or concurrency controls are gated behind Pro. It works extremely well, but I've been reluctant to use it in open source projects because there's quite a bit in there I'd need to rebuild.
- throwawaymaths 1y agoim curious about your use case -- it seems weird (to me) to use it in an open source project unless its some kind of turnkey full app -- is there a way to just release it and encourage people to bring their own oban keys? that way it looks good for the elixir ecosystem that it has found a way to support hybrid open source libraries and expands the obam ecosystem
- pikdum 1y agoMy open source projects are generally more applications than libraries, yeah.
- victorbjorklund 1y agoYou are right. Those are not really things I had a need for but yea I guess they can be useful.
- mercer 1y agoAnd they recently open-sourced (among other things?) the Oban Web plugin!
- rhgraysonii 1y ago
- cpursley 1y agoI’d love to see one for Elixir built around the Postgres pgmq and pg_crom extensions similar to what pgflow is doing: https://www.pgflow.dev/ https://www.pgflow.dev/
- rhgraysonii 1y agoThat would be super cool. Pretty much everything Supabase is up to is awesome.
- cpursley 1y agoYeah, their realtime thing is powered by Elixir (listens to Postgres WAL).
- jumski 1y agopgflow can be used with any language/runtime, I just started with TypeScript and Supabase, as that's what I'm using. The worker is stateless and "dumb" by design (currently it runs on serverless functions) - it just calls SQL functions: "poll_for_tasks" to get some tasks from the queue and then either "complete_task" or "fail_task" after executing user code - that's it, nothing more, so it should be relatively easy to adopt it to other runtimes. I have written a small architecture primer on pgflow if anyone is interested in its simple but flexible design https://www.pgflow.dev/concepts/how-pgflow-works/ https://www.pgflow.dev/concepts/how-pgflow-works/
- jamal-kumar 1y agoThe absolute facility with which you can do distributed applications in Elixir is the big thing that sold me in the first place. Nice to see this tutorial!
- hmmokidk 1y agoIncredibly well written. You clearly put a lot of thought and effort into this. Thank you. Also FOR UPDATE SKIP LOCKED is interesting.
- rhgraysonii 1y agoYou also could use an advisory exact lock, which would serialize the order of input and force everyone to wait in line, but the advantage here is that we are allowed shared concurrent reads while maintaining our lock. Thank you for the feedback. It was fun to revisit after 10 years. The initial piece came after a Columbus, OH Ruby meetup that Jose attended. It was quite fun to see him put it together as a whole and I ran with things a bit further than his first ideas with this.
- sarchertech 1y agoSKIP LOCKED is amazing for this kind of thing. I used it to build a transaction outbox a few years ago. One thing worth looking into if you do this in production is adding a way to add partitions such that each partition is single threaded. It’s the only way to guarantee ordering if your jobs are doing anything non-deterministic.
- boruto 1y agoWe have a system where each pod spins up around 30 scheduled job instances of one job each processing a "partition", then transaction outbox is queried with hash of identifier equating it to partition. We increased partition counts on sale days and it works well for us. Couple of gotchas we had were. 1) Using hashtext from postgres is sketchy. 2) Increasing partiton count is an orchestra which requires stopping the partition.
- oxidant 1y agoFOR UPDATE SKIP LOCKED is great, but it needs to be in a transaction. In the example code it won't "do" anything because it selects for update then immediately loses the lock. Claude says you can use a CTE to select and the run your update with the locked rows, but I have only ever used transactions.
- huqedato 1y agoYears ago I built something similar, a Pub/Sub notification system: https://github.com/huqedato/qnotix https://github.com/huqedato/qnotix. It is still running (with small modifications), in production, at my ex-customer. Such a pity the industry (customers) reacts with skepticism every time I propose solutions based in Elixir/Erlang. I always hear: "Elixir, what? We want Java/.Net/Python/php"
- jbverschoor 1y agoIt means they don't trust your judgement, aka you're just there to execute their plans. You're the floor guy, not the architect, owner, or developer of a house.
- huqedato 1y agoNot really. I am usually the one they pay to solve their problems. It's not a matter of trusting my judgement, rather they are trapped in an institutional/corporate mindset (old patterns are most suitable, 'best practice', 'reliable, proven tech' and such)
- stonemetal12 1y ago> I am usually the one they pay to solve their problems. In the context that doesn't really say a lot, they pay the floor guy and the architect to solve problems, just different problems. >they are trapped in an institutional/corporate mindset (old patterns are most suitable, 'best practice', 'reliable, proven tech' and such) While true, it is also correct. From their perspective Elixir is an exotic material, the next "one they pay to solve their problems" is not going to know how to work with it, scrap it, and start over. Therefore the Elixir solution isn't a good solution for them. As a business man once told me, "I never saw a feature I liked so much that I was wiling to pay for it twice."
- icedchai 1y agoThe problem is finding other people to work in less mainstream languages. Probably not an issue right now, but during better times...
- udkl 1y agoJust want to say thank you for the detailed write up. I am enjoying the article a lot!
- rhgraysonii 1y agoThanks!
- udkl 1y ago> The Beauty of Modeling Everything as Events - hen you model work as events, powerful patterns emerge Wouldn't you say these patterns aren't unique to event based systems since functional composition can also lead to these patterns? For eg: see the Streams API in Java ========= - Where would push event systems make sense over pull based systems?
- udkl 1y agoWould this be the right model to think about what is presented in the article? https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1&title=Untitled%20Diagram.drawio#R7Vpbd5s4EP41fkwOF0Psx%2FrSNHvcbXa9e7bdNxlkUAOICuHL%2FvqOjAQI0Tqbxo6b3eacBn26IL5vZjQDGbjTdHfLUB6%2FpyFOBo4V7gbubOA4tmc58Esg%2BwoZe%2BMKiBgJ5aAGWJJ%2FsAQtiZYkxIU2kFOacJLrYECzDAdcwxBjdKsPW9NEv2uOImwAywAlJvoXCXlcoSPPavB3mESxurNtyZ4UqcESKGIU0m0LcucDd8oo5dVVupviRJCneKnmvf1Gb70xhjP%2BmAmzDVlv%2BfTq73GB8HK7%2FzIt0yupzgYlpXxguVm%2BVwzAvnNxGewTkoWYuQN3so0Jx8scBaJjC8oDFvM0gZYNlytawshwsaoBFDxETKAfSg7LYIkXld62B9fm88hH3GDG8a4Fyee7xTTFnO1hiOodSq6lsdmubG8b6XwJxS3VRhJD0liieuWGT7iQlP4Lel2D3nvMClJw8YyCZcqE%2FQm7ZPD%2FL3RVwH3mG%2BguDCGABK7zjBISZUIaGI8ZAIIqAsb7RnakJAzF9AnDQDVaHZayoJ1TIm4B63qTgTcTa5WcKjmEMpzRBzylCWzMnWVUSDZZkyTpQM%2Bgmq%2BLNjY1q6NBWzT3VKINTdEYDcsAGO5KwipDl6Qe8YrnMHBb56qPLNvpIcs%2FFVmewQkOIYDKJmU8phHNUDJv0InOWjNmQWkuufqMOd%2FL00BYps4kEMj2H8X8a081P8nlDo3ZTmvtZeubChS0ZAE%2BHic5YhHmx21HcPBdPRlOECcb%2FZDpE0dOvRf%2B2rKDm44ddPWtNipndSSut%2FF01X3DRfIySSBkWTQThz1OURb2WsYCrSBDOFkUMxX%2BntUajlfnEfIm2lHd55BX1rXlaVIoU3mqwmoIXa8LfBLxbgzxpjQryvQS4luHzGHPAX7e%2BDZ6FfHtaNy6OVPc8nV9Xa%2BjWxWIjbh1dKGhdd4AOP7JA%2BDovxwAVRL1Sqoe79KqHts2%2BP2%2F7DFkG15a3WObLwMupvDxLq7wsd1XkRkcrXwU8UdTCGU%2Bp84hvBeufWxVPFxg%2Ftx1E89%2FcTfxX8RNdoS3vARan1o9jY%2BIxjO4yGPfDtijM7lIN80ePzHN7vra0PXP6mtOzwvUsogB%2BXzIGS42v1Zm%2F%2BMJNuTXvn%2BjqSAXutwEu%2F7k8drPxtFjHd97bsf%2FsaCsyr%2FGrX4rMVwaVdAD5kGsJKmKmGn9pU2Aa5pxlRsPHBd%2B3oqNTCKGQoKbPpk36%2Fl1e0I7yQZ8ffgHeIiKuLYK5bgHJ7%2BHxJ0TKhx4RTmnaY9nc2E7ZgBoWY14AGlSEMdlWzIgbomKvHraNdmJfUxUZZjuIvHJ8xpti%2BE1BI6DsdwFYj8ijlRX%2BqgvB5JPUvQN7ccVfSc76B2z6Jsv7j7e%2FQ6r3c5%2FXf7x5nYO3e8%2FzOYLw86OZEambThB0GdNob%2FyPf807yKdl06l1FuK9mH45%2FLdT0Wqdz5Sodl8XK9OtuZPFNz5Vw%3D%3D https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&ed...
- 1oooqooq 1y agopure HN comedy that the article is about the disgust of running acme tools that cram all buzzwords just for the lulz, and most comments are pushing even worse tools with even more buzzwords ("still the torment Nexus, but in elixir!") sigh. anyway. i had the same reaction to acme. but I'm lazier. i just ran ngix and the acme client in qemu and picked the certs out. i applaud the author for her tenacity.
- abrookewood 1y agoMinor criticism, you can't claim this as a problem "Unfair distribution - Fast workers might grab all the work" and then turn around and claim it as a benefit in your own solution "Natural load balancing - Fast consumers get more work".
- rhgraysonii 1y agoFair point. Can you think of a better way to weave that into what I’m trying to illustrate as a mental model? I struggled with that. Happy to read a PR that tries to capture it better.
- throwawaymaths 1y agoit's fine. grabbing "all" the work is not the same as getting "more" work.
- 29athrowaway 1y agoI have not seen a single Elixir project that doesn't have a spaghetti mess made out of macros. You can rant all you want about the disadvantages of OOP and other paradigms. But after your rant, do not come up with an even more absurd type system that gives you even more work to do without any upside. Building a project out of constructs such as macros does not scale with project size or team size. The only aspect that scales is the resulting size of the unmanageable spaghetti. Tolerating having that as a widely used feature in your language and then praise it as being fantastic truly makes one wonder what are you comparing the Elixir experience with? Elixir is fantastic compared to... fighting 100 gorillas blindfolded. There are good choices built into the language, like railroad oriented programming. But those will never undo the infinitely awful and inadequate insanity of having to deal with macros to survive. And to be frank, even if there are use-cases where the language shines and can be excellent tool for the job, most of the time those are not your use-case, and most of the time your team is not familiar with the language enough to leverage its advantages or build something production-worthy. And if you ever go through the pain of mastering Elixir, 99.99% of the time your next job will not use it. If you are at a startup celebrating the virtues of Elixir before having enough revenue, your next non-Elixir job is closer than you think because your company will run out of money. And if your plan is having a LLM help you write Elixir faster before you burn through your budget? Good luck debugging a labyrinth made out of AI generated macros, your on-call rotation will go great. Your stakeholders will be eager to learn more about how great Elixir is when they ask for an explanation of why nothing works, an estimate of when it will be fixed, why is the project late or how can they add more team members to make it go faster.
- victorbjorklund 1y agoI almost never use macros (unless you count that Elixir itself is built with macros or that Phoenix uses macros. But they are well-written and documented. Problem with macro more comes when you got your own DSL that no one knows).
- conradfr 1y agoI have almost never written macros in my elixir projects. Because I have a hard time even understanding them :) I think they are mostly useful if you write libraries.