6 ms·
For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how
by mattbaker 4y ago
For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?)
In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this all depends on how a system’s architected.)
After doing that we were free to use all of Erlang’s cool cluster stuff! In our case we have in-memory caches for a few things, and if a given instance does a lookup because of a cache miss it broadcasts a message to all the other nodes saying “I just looked up $expensive_thing, here’s its value” so they don’t have to do the lookup themselves, they just cache that value, so you end up with a little distributed cache with a few lines of code. In our case, btw, these cache entries are short lived and a little inconsistency does us no harm if one of our instances misses the message, networks are networks, but it’s been great!
Anyway, I think it’s super cool and I’d encourage you to play around if you get the chance.
Also the observer is just amazing. We’ve debugged some pretty weird memory and cpu usage issues with it, I have some internal blog posts, maybe I should see if I could make them public.
- JohnCurran 4y agoCan you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work
- throwawaymaths 4y agoSame. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?
- sprite 4y agoNot sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-in-aws-as-ecs-fargate-using-service-discovery-to-interconnect-91386a48dfa1 https://towardsaws.com/an-elixir-migration-to-microservices-...
- benmmurphy 4y agojust beware service discovery via DNS in AWS will return up to 8 addresses. so if you have more than 8 nodes you will get a random subsample for each request. depending on how your clustering works this may or may not be a problem. you can use the web api if you need to handle more than 8 hosts.
- toast0 4y agoI didn't write this blog, but it looks like it's got the right info: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... You can skip to "Let’s use something else" if you've already got a good grasp of the knobs to tune with epmd.
- mattbaker 4y agoYeah, so we followed the link toast0 shared to get started: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... It took some work to piece together but wasn't too complicated in the end. It's internal code that I can't quickly sanitize or I'd just dump it in a gist :/ Someone on the Elixir Forum might have a template or library handy though.
- kungfooguru 4y agoOther replies are a bit outdated. It got a lot easier in OTP-23 to go without EPMD, https://blog.erlware.org/epmdlessless/ https://blog.erlware.org/epmdlessless/ -- if you run with the same port. If not using the same port across nodes then you may want to look at epmdless https://github.com/tsloughter/epmdless https://github.com/tsloughter/epmdless
- jjtheblunt 4y ago> deployed to $N containers it's funny how 20 years ago "N containers" would be normal, but now everyone can read both
- mattbaker 4y agoI've been in too many shell scripts lately ;)
- jjtheblunt 4y agoSame!