4 ms·
Postgres LISTEN/NOTIFY was a consistent pain point for Oban (background job processing framework for Elixir) for a while. The payload size limitations and conne
by sorentwo 1y ago
Postgres LISTEN/NOTIFY was a consistent pain point for Oban (background job processing framework for Elixir) for a while. The payload size limitations and connection pooler issues alone would cause subtle breakage.
It was particularly ironic because Elixir has a fantastic distribution and pubsub story thanks to distributed Erlang. That’s much more commonly used in apps now compared to 5 or so years ago when 40-50% of apps didn’t weren’t clustered. Thanks to the rise of platforms like Fly that made it easier, and the decline of Heroku that made it nearly impossible.
- cpursley 1y agoHow did you resolve this? Did you consider listening to the WAL?
- sorentwo 1y agoWe have Postgres based pubsub, but encourage people to use a distributed Erlang based notifier instead whenever possible. Another important change was removing insert triggers, partially for the exact reasons mentioned in this post.
- MuffinFlavored 1y ago> Another important change was removing insert triggers, partially for the exact reasons mentioned in this post. What did you replace them with instead?
- sorentwo 1y agoIn app notifications, which can be disabled. Our triggers were only used to get subsecond job dispatching though.
- parthdesai 1y agoDistributed Erlang if application is clustered, redis if it is not. Source: Dev at one of the companies that hit this issue with Oban
- alberth 1y agoI didn’t realize Oban didn’t use Mnesia (Erlang built-in).
- sorentwo 1y agoVery very few applications use mnsesia. There’s absolutely no way I would recommend it over Postgres.
- arcanemachiner 1y agoI have heard the mnesia is very unreliable, which is a damn shame. I wonder if that is fixable, or just inherent to its design.
- sb8244 1y agoMy understanding is that mnesia is sort of a relic. Really hard to work with and lots of edge / failure cases. I'm not sure if it should be salvaged?
- asg0451 1y agocan you explain why?
- spooneybarger 1y agoMnesia along with clustering was a recipe for split brain disasters a few years ago I assume that hasn't been addressed.
- ahoka 1y agoI have only worked with a product that used it, so no direct experience, but one problem that was often mentioned is split-brains happening very frequently.
- tecleandor 1y agoI think RabbitMQ still uses by default for its metadata storage. Is it problematic?
- nightpool 1y agoWhat about Heroku made Erlang clustering difficult? It's had the same DNS clustering feature that Fly has, and they've had it since 2017: https://devcenter.heroku.com/articles/dyno-dns-service-discovery/ https://devcenter.heroku.com/articles/dyno-dns-service-disco....
- sorentwo 1y agoThe problem was with restrictive connections, not DNS based discovery for clustering. It wasn't possible (as far as I'm aware) to connect directly from one dyno to another through tcp/udp.
- nightpool 1y agoThat is not an issue when using Private Spaces, which have been available since 2015