4 ms·

This is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent

This is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent AWS instances. So I'm always intrigued by some of the more BEAM-specific things that folks do, like using `observer` on a remote (production??) node here, or distributed Elixir where the nodes communicate with each other, or "hot" code updates. How do companies deploy Elixir in such a way to take advantage of all those things? Does Sequin talk anywhere about their deploy process and how their infrastructure looks?

lycos 4y ago
Distributed Elixir can be done with Docker containers too, see https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster which by default has some Kubernetes support but you can also have third party (or custom) clustering strategies. I've not done this myself but I've seen articles about this a lot during the past years. Hot code updates for most applications aren't really worth it in my opinion, assuming you do something like blue/green rollover deployments. It's cool that it's possible though. But it requires appup files and afaik Distillery is one of the release tools that has support for it built-in.
ranyefet 4y ago
If you deploy to fly.io it should be very easy to create a cluster of elixir nodes.
cpursley 4y ago
Distributed Elixir on Render is crazy easy. Fly.io also looks neat.
mattbaker 4y ago
For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this all depends on how a system’s architected.) After doing that we were free to use all of Erlang’s cool cluster stuff! In our case we have in-memory caches for a few things, and if a given instance does a lookup because of a cache miss it broadcasts a message to all the other nodes saying “I just looked up $expensive_thing, here’s its value” so they don’t have to do the lookup themselves, they just cache that value, so you end up with a little distributed cache with a few lines of code. In our case, btw, these cache entries are short lived and a little inconsistency does us no harm if one of our instances misses the message, networks are networks, but it’s been great! Anyway, I think it’s super cool and I’d encourage you to play around if you get the chance. Also the observer is just amazing. We’ve debugged some pretty weird memory and cpu usage issues with it, I have some internal blog posts, maybe I should see if I could make them public.
- JohnCurran 4y ago
  Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work
  - throwawaymaths 4y ago
    Same. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?
    - sprite 4y ago
      Not sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-in-aws-as-ecs-fargate-using-service-discovery-to-interconnect-91386a48dfa1 https://towardsaws.com/an-elixir-migration-to-microservices-...
      - benmmurphy 4y ago
        just beware service discovery via DNS in AWS will return up to 8 addresses. so if you have more than 8 nodes you will get a random subsample for each request. depending on how your clustering works this may or may not be a problem. you can use the web api if you need to handle more than 8 hosts.
  - toast0 4y ago
    I didn't write this blog, but it looks like it's got the right info: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... You can skip to "Let’s use something else" if you've already got a good grasp of the knobs to tune with epmd.
  - mattbaker 4y ago
    Yeah, so we followed the link toast0 shared to get started: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... It took some work to piece together but wasn't too complicated in the end. It's internal code that I can't quickly sanitize or I'd just dump it in a gist :/ Someone on the Elixir Forum might have a template or library handy though.

lycos 4y ago

Distributed Elixir can be done with Docker containers too, see https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster which by default has some Kubernetes support but you can also have third party (or custom) clustering strategies. I've not done this myself but I've seen articles about this a lot during the past years. Hot code updates for most applications aren't really worth it in my opinion, assuming you do something like blue/green rollover deployments. It's cool that it's possible though. But it requires appup files and afaik Distillery is one of the release tools that has support for it built-in.

ranyefet 4y ago

If you deploy to fly.io it should be very easy to create a cluster of elixir nodes.

cpursley 4y ago

Distributed Elixir on Render is crazy easy. Fly.io also looks neat.

mattbaker 4y ago

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this all depends on how a system’s architected.) After doing that we were free to use all of Erlang’s cool cluster stuff! In our case we have in-memory caches for a few things, and if a given instance does a lookup because of a cache miss it broadcasts a message to all the other nodes saying “I just looked up $expensive_thing, here’s its value” so they don’t have to do the lookup themselves, they just cache that value, so you end up with a little distributed cache with a few lines of code. In our case, btw, these cache entries are short lived and a little inconsistency does us no harm if one of our instances misses the message, networks are networks, but it’s been great! Anyway, I think it’s super cool and I’d encourage you to play around if you get the chance. Also the observer is just amazing. We’ve debugged some pretty weird memory and cpu usage issues with it, I have some internal blog posts, maybe I should see if I could make them public.

JohnCurran 4y ago
Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work
- throwawaymaths 4y ago
  Same. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?
  - sprite 4y ago
    Not sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-in-aws-as-ecs-fargate-using-service-discovery-to-interconnect-91386a48dfa1 https://towardsaws.com/an-elixir-migration-to-microservices-...
    - benmmurphy 4y ago
      just beware service discovery via DNS in AWS will return up to 8 addresses. so if you have more than 8 nodes you will get a random subsample for each request. depending on how your clustering works this may or may not be a problem. you can use the web api if you need to handle more than 8 hosts.
- toast0 4y ago
  I didn't write this blog, but it looks like it's got the right info: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... You can skip to "Let’s use something else" if you've already got a good grasp of the knobs to tune with epmd.
- mattbaker 4y ago
  Yeah, so we followed the link toast0 shared to get started: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd/ https://www.erlang-solutions.com/blog/erlang-and-elixir-dist... It took some work to piece together but wasn't too complicated in the end. It's internal code that I can't quickly sanitize or I'd just dump it in a gist :/ Someone on the Elixir Forum might have a template or library handy though.