5 ms·

I've noticed there's a section in there about deploying with Docker. I know many other development communities swear by Docker, but a number of the OTP/BEAM dev

I've noticed there's a section in there about deploying with Docker. I know many other development communities swear by Docker, but a number of the OTP/BEAM devs I've spoken to seem unsure about it. I also tend to believe Docker makes little sense, where OTP releases are the alternative. What's the opinion here? This question only pertains to OTP development.

kungfooguru 7y ago
Hi, one of the authors here, the Docker chapter and soon to come Kubernetes chapter hopefully shed some light on this very question. They are not alternatives, docker and k8s complement Erlang. Jose Valim, creator of Elixir, also wrote about this recently http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-erlang-vm-orchestration-on-the-large-and-the-small/ http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...?
- deleted 7y ago
  [deleted]
- StreamBright 7y ago
  Do you have experience with performance optimization of Erlang code running in Docker? Is there a way to make the BEAM schedulers sticky? Does it make sense for latency reasons? I know that some products rely on core stickiness for maximum performance. Maybe Erlang/BEAM is not in that category. https://www.scylladb.com/2018/08/09/cost-containerization-scylla/ https://www.scylladb.com/2018/08/09/cost-containerization-sc...
  - kungfooguru 7y ago
    There are some performance related topics covered in the Docker chapter regarding schedulers. And there are improvements already merged for Erlang 23 to remove the need for as much manual tweaking -- for example in 23 the VM will only start a scheduler per-cpu the cgroup actually allows, so if you are limited to 2 "cpus" by the cgroup there will only by 2 active schedulers. Erlang does have the option to bind schedulers to processors but I don't know of any real world usage of it: Binding Schedulers: http://erlang.org/doc/man/erl.html#+sbt http://erlang.org/doc/man/erl.html#+sbt Custom CPU Topologies: http://erlang.org/doc/man/erl.html#+sct http://erlang.org/doc/man/erl.html#+sct
whalesalad 7y ago
It’s difficult to deploy Erlang apps in Erlang-style using Docker. The style I speak of is zero downtime deploys where the VM is never turned off, but is updated with new software as a living organism. You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos). It can be done. It’s just kinda going against the grain.
- kungfooguru 7y ago
  It is not going against the grain. Using release upgrades (upgrading a release without restarting the VM -- unless the version of erts has changed) should only be used where necessary and they rarely are. They are great to have when you need them but should not be chosen lightly. Part of the point of this booksite is to show integrating Erlang into a modern environment and not being the oddball deploy at the company. It is the lack of libraries for services used for deployment/monitoring and lack of documentation that is the reason for this. Also, there is no abstracting away network partitions.
- dynamite-ready 7y ago
  This is what I've been thinking. At the very least, I know that Docker interferes with the BEAM VM's default behaviour for port management, and node discovery. Docker containers require developers to select which ports containers communicate through. Isn't that right?
  - mononcqc 7y ago
    This is ultimately not very different than what you would get with any firewall or network rules in any business, and it's an issue shared by most deployments regardless of whether you get containerized or not.
- 7y ago
h91wka 7y ago
> I also tend to believe Docker makes little sense, where OTP releases are the alternative. Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)
- GordonS 7y ago
  Was going to say something similar - having a consistent, predictable deployment environment is really valuable from a support perspective, especially if you are shipping a service to several end users (as opposed to deploying an in-house service).
- conradfr 7y ago
  Sometimes I wonder if Linux containers (e.g lxd) would not be more suited than docker for Beam apps, but nobody hardly ever mentions them.
yellowapple 7y ago
It's just a matter of using the right tool for the right job. There's a lot of things Docker can do that an OTP application already handles (and in a way that's better-integrated with your application). Same with an orchestration system (e.g. Kubernetes). You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.). Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain. On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.
- kungfooguru 7y ago
  If you get to choose your infrastructure that may be fine. If you don't and you work in even a medium size company you likely work with what they have, which is increasingly kubernetes. That said, they still do not do similar things. OTP does not take care of orchestration if you must run more than 1 node. You are still benefiting from OTP's supervision and scheduling when using docker or k8s, the difference is in what other features and the size you need. No one is saying always deploy to k8s, but they are not overlapping. Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust, ... If your application is suited to running on a few droplets the language you are using is not likely the deciding factor for that.

kungfooguru 7y ago

Hi, one of the authors here, the Docker chapter and soon to come Kubernetes chapter hopefully shed some light on this very question. They are not alternatives, docker and k8s complement Erlang. Jose Valim, creator of Elixir, also wrote about this recently http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-erlang-vm-orchestration-on-the-large-and-the-small/ http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...?

deleted 7y ago
[deleted]
StreamBright 7y ago
Do you have experience with performance optimization of Erlang code running in Docker? Is there a way to make the BEAM schedulers sticky? Does it make sense for latency reasons? I know that some products rely on core stickiness for maximum performance. Maybe Erlang/BEAM is not in that category. https://www.scylladb.com/2018/08/09/cost-containerization-scylla/ https://www.scylladb.com/2018/08/09/cost-containerization-sc...
- kungfooguru 7y ago
  There are some performance related topics covered in the Docker chapter regarding schedulers. And there are improvements already merged for Erlang 23 to remove the need for as much manual tweaking -- for example in 23 the VM will only start a scheduler per-cpu the cgroup actually allows, so if you are limited to 2 "cpus" by the cgroup there will only by 2 active schedulers. Erlang does have the option to bind schedulers to processors but I don't know of any real world usage of it: Binding Schedulers: http://erlang.org/doc/man/erl.html#+sbt http://erlang.org/doc/man/erl.html#+sbt Custom CPU Topologies: http://erlang.org/doc/man/erl.html#+sct http://erlang.org/doc/man/erl.html#+sct

whalesalad 7y ago

It’s difficult to deploy Erlang apps in Erlang-style using Docker. The style I speak of is zero downtime deploys where the VM is never turned off, but is updated with new software as a living organism. You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos). It can be done. It’s just kinda going against the grain.

kungfooguru 7y ago
It is not going against the grain. Using release upgrades (upgrading a release without restarting the VM -- unless the version of erts has changed) should only be used where necessary and they rarely are. They are great to have when you need them but should not be chosen lightly. Part of the point of this booksite is to show integrating Erlang into a modern environment and not being the oddball deploy at the company. It is the lack of libraries for services used for deployment/monitoring and lack of documentation that is the reason for this. Also, there is no abstracting away network partitions.
dynamite-ready 7y ago
This is what I've been thinking. At the very least, I know that Docker interferes with the BEAM VM's default behaviour for port management, and node discovery. Docker containers require developers to select which ports containers communicate through. Isn't that right?
- mononcqc 7y ago
  This is ultimately not very different than what you would get with any firewall or network rules in any business, and it's an issue shared by most deployments regardless of whether you get containerized or not.
7y ago

h91wka 7y ago

> I also tend to believe Docker makes little sense, where OTP releases are the alternative. Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)

GordonS 7y ago
Was going to say something similar - having a consistent, predictable deployment environment is really valuable from a support perspective, especially if you are shipping a service to several end users (as opposed to deploying an in-house service).
conradfr 7y ago
Sometimes I wonder if Linux containers (e.g lxd) would not be more suited than docker for Beam apps, but nobody hardly ever mentions them.

yellowapple 7y ago

It's just a matter of using the right tool for the right job. There's a lot of things Docker can do that an OTP application already handles (and in a way that's better-integrated with your application). Same with an orchestration system (e.g. Kubernetes). You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.). Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain. On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.

kungfooguru 7y ago
If you get to choose your infrastructure that may be fine. If you don't and you work in even a medium size company you likely work with what they have, which is increasingly kubernetes. That said, they still do not do similar things. OTP does not take care of orchestration if you must run more than 1 node. You are still benefiting from OTP's supervision and scheduling when using docker or k8s, the difference is in what other features and the size you need. No one is saying always deploy to k8s, but they are not overlapping. Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust, ... If your application is suited to running on a few droplets the language you are using is not likely the deciding factor for that.