4 ms·

Hi there, Very interesting read! I work for a large news org, The team I'm in primarily uses elixir, which I know the people at fly.io love too! Why did you

Hi there, Very interesting read! I work for a large news org, The team I'm in primarily uses elixir, which I know the people at fly.io love too! Why did you decide not to containerise your own infrastructure? We use some 'chonky' ec2s but are thinking about using containers. Given that the BEAM has quite a large footprint, do you think it still a good candidate for containers, or would that introduces too much overhead?

tptacek 5y ago
We would containerize everything if we could! But our infrastructure components usually live outside the security boundaries our container interface sets up. The proxy has to be able to talk to every app, not just apps in its own organization, and it needs direct access to other infrastructure components. The orchestration code needs to be able to launch Firecracker VMs --- it can't itself be a Firecracker VM. We can design around all these constraints! We just haven't gotten around to it yet. I think there's a general consensus that hiding things inside Firecrackers is a good design, and we'll do it where we can. Elixir runs great inside Firecrackers. The idea behind Firecracker is that it introduces a minimal load, by dint of being ruthlessly simple. When you run an Elixir app on an EC2 instance, you're running hypervised as well (and probably by a hypervisor that's less efficient --- not a knock, Firecracker is Amazon code too). The thing that makes this doable at Fly.io is that we run our own hardware, so we're not trying to nest VMs inside VMs. Of course, that's what EC2 is doing too. So whatever your intuitions are about things that run well on vanilla EC2, they should carry over to us.
- freedomben 5y ago
  What are you using for the host OS on your bare metal?
  - tptacek 5y ago
    Linux. :)
    - freedomben 5y ago
      heh, could you share if it's more of an rpm flavor or more of a deb flavor? :-) (or some third option. no false dichotomies intended. actually it would be badass if you're running Arch :-D)
      - tptacek 5y ago
        At this moment I will go as far as to say that it is not a badass flavor of Linux. :)

tptacek 5y ago

We would containerize everything if we could! But our infrastructure components usually live outside the security boundaries our container interface sets up. The proxy has to be able to talk to every app, not just apps in its own organization, and it needs direct access to other infrastructure components. The orchestration code needs to be able to launch Firecracker VMs --- it can't itself be a Firecracker VM. We can design around all these constraints! We just haven't gotten around to it yet. I think there's a general consensus that hiding things inside Firecrackers is a good design, and we'll do it where we can. Elixir runs great inside Firecrackers. The idea behind Firecracker is that it introduces a minimal load, by dint of being ruthlessly simple. When you run an Elixir app on an EC2 instance, you're running hypervised as well (and probably by a hypervisor that's less efficient --- not a knock, Firecracker is Amazon code too). The thing that makes this doable at Fly.io is that we run our own hardware, so we're not trying to nest VMs inside VMs. Of course, that's what EC2 is doing too. So whatever your intuitions are about things that run well on vanilla EC2, they should carry over to us.

freedomben 5y ago
What are you using for the host OS on your bare metal?
- tptacek 5y ago
  Linux. :)
  - freedomben 5y ago
    heh, could you share if it's more of an rpm flavor or more of a deb flavor? :-) (or some third option. no false dichotomies intended. actually it would be badass if you're running Arch :-D)
    - tptacek 5y ago
      At this moment I will go as far as to say that it is not a badass flavor of Linux. :)