7 ms·
Bridging Elixir and Python with Oban
- cpursley 8mo agoVery nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?
- elitepleb 8mo agothat's easy with https://hexdocs.pm/pythonx/Pythonx.html https://hexdocs.pm/pythonx/Pythonx.html and https://hexdocs.pm/lua/Lua.html https://hexdocs.pm/lua/Lua.html and works well too
- cpursley 8mo agobtw, a lot of postgres envs are not going to have pgmq, so just use Oban and don't reinvent the wheel like I did ;)
- languagehacker 8mo agoI feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture. This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
- whalesalad 8mo agoWhat leads you to this conclusion
- Arubis 8mo agoIt’s almost as though choosing a single-threaded, GIL-encumbered interpreted scripting language as the primary interface to an ecosystem of extremely parallelized and concurrent high-performance hardware-dependent operations wasn’t quite the right move for our industry.
- markstos 8mo agoHa. The question now is whether the ML industry will change directions or if the momentum of Python is a runaway train. I can't guess. Perl was once the "800-pound gorilla" of web development, but that chapter has long been closed. Python on the other hand has only gained traction since that time.
- geooff_ 8mo agoI disagree, using python for a web-server and something like celery for background work is a pretty common pattern. My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
- languagehacker 8mo agoThat's not the sort of architecture I'm referring to. I'm specifically talking about splitting your application layer between Elixir and Python.
- victorbjorklund 8mo agoWait until you find out about some people not writing pure python apps but also have some code in JavaScript. Crazy to mix more than one language in one machine.
- victorbjorklund 8mo agoStrange opinion. Plenty of apps have more than one language. I might end up using this. Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
- markstos 8mo agoSometimes the "right tool for the job" philosophy leads to breaking down a larger problem into two small problems, each which has a different "right tool". Choosing a single tool that tries to solve every single problem can lead to its own problems.
- ananthakumaran 8mo agoWe have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.
- dnautics 8mo agoI have one vibecoded ml pipeline now and I'm strongly considering just clauding it into Nx so I can ditch the python
- flippant 8mo agoI did exactly this in early 2025 with a small keyword tagging pipeline. You may run into some issues with Docker and native deps once you get to production. Don’t forget to cache the bumblebee files.
- dnautics 8mo agoNo problem. It's an SLM, I have a dedicated on-prem GPU server that I deploy behind tailscale for inference. For training, I reach out to lambdalabs and just get a beefy GPU for a few hours for the cost of a Starbucks coffee.
- markstos 8mo agoIs this part of a web server or some other system where you could end up spawning N python processes instead of 1 at a time?
- rozap 8mo agoI use a similar strategy for python calls from elixir. This is in a web server, usually they're part of a process pool. So we start up N workers and they hang out and answer requests when needed. I just have an rpc abstraction that handles all the fiddly bits. The two sides pass erlang terms back and forth. Pretty simple.
- jongjong 8mo agoI don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set. I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests... The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.
- HorizonXP 8mo agoSo I know these are just benchmarks, but apparently Elixir is one of the best languages to use with AI, despite having a smaller training dataset: https://www.youtube.com/watch?v=iV1EcfZSdCM https://www.youtube.com/watch?v=iV1EcfZSdCM and https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/main https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma... Furthermore, it's actually kind of annoying that the LLMs are not better than us, and still benefit from having code properly typed, well-architected, and split into modules/files. I was lamenting this fact the other day; the only reason we moved away from Assembly and BASIC, using GOTOs in a single huge file was because us humans needed the organization to help us maintain context. Turns out, because of how they're trained, so do the LLMs. So TypeScript types and tests actually do help a lot, simply because they're deterministic guardrails that the LLM can use to check its work and be steered to producing code that actually works.
- dnautics 8mo agoI don't think LLMs benefit from having code properly typed (at the call definition). It's costly to have to check a possibly remote file to check. The LLM should be able to intuit what the types are at the callsite and elixir has ~strong conventions that LLMs probably take advantage of
- baseonmars 8mo agollms benefit greatly from feedback and typing/type errors are one of the fastest and easiest methods of feedback to give to an llm.
- nimbus-hn-test 8mo ago[dead]
- mrcwinn 8mo agoI absolutely love Elixir, but if this is the bridge you need to cross, just write it in Python in the first place.
- dnautics 8mo agoIt's 2026 and the LLMs score high on elixir, just write it in python and patch it over to elixir gradually
- Towaway69 8mo agoOr patch it over to python, I assume LLMs are even better at python.
- dnautics 8mo agoDon't assume. Empirically, they are not. (This post Feb 2026 may change in future yadda yadda) See: autocodebench https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/main https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...
- Towaway69 8mo agoReading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two? Specific studies, as the one quoted, are a long way from original real world problems.
- deleted 8mo ago[deleted]
- rekoros 8mo agoOban is great!
- Kaliboy 8mo agoThis is a similar concept to Faktory, which uses a built in Redis server to manage shared job state. You then implement workers in your language of choice and subscribe to queues. Very interesting though, the article mentioned a few things I hadn't considered before like shared access to one database from multiple (different) apps. I wonder how database schema state is handled in a case like that. And CI/CD.
- pantsforbirds 8mo agoWhen rust was still a fairly new language i remember using capn'n proto to communicate between some rust code and python as a way to experiment with handing off performance critical tasks to a compiled language. I wonder how well a similar approach would work with elixir + python. Elixir obviously has very easy process isolation, but I think you'd be stuck using a NIF approach for Elixir, which probably removes any reason to try capn'n proto over just protobufs?