3 ms·

being unable to do dynamic dispatch seems like a big show-stopper for distributed/meshed apps, which seems to be the only reason for targeting the BEAM to begin

by toolz 6y ago

being unable to do dynamic dispatch seems like a big show-stopper for distributed/meshed apps, which seems to be the only reason for targeting the BEAM to begin with, no? If you don't pass functions around as MFAs in a distributed app you run into trouble with upgrading your cluster as they'll have different versions of a function during a rolling deploy

jolux 6y ago
Different organizations use different amounts of the tools the BEAM provides in building their distributed applications. We containerize everything and don't really use the hot upgrade features.
- toolz 6y ago
  sure, but I'm not talking about hot upgrades, I'm talking about upgrading entire nodes that exist in a cluster. The nodes will not be able to pass around functions as an upgraded node in a cluster will have a different version of that function. So this is an issue without considering hot code reloading, as I don't use that feature either.
  - jolux 6y ago
    I don't quite follow how static types change this situation?
  - yawaramin 6y ago
    So we're not talking about hot code reloading here, we're talking exactly about upgrading nodes in a cluster. Many--I'd say most--Erlang/Elixir deployments today don't do that. They use Kubernetes.
    - toolz 6y ago
      Kubernetes does not solve the issue, it merely has strategies for how to upgrade nodes. The issue is that when you compile new code, even if the function didn't change at all, it will be a new version of the function. If the upgraded node then sends a function to another node to be executed (as is a very common practice in meshed OTP apps) the older node will not be able to execute the function. Now, I understand that most erlang apps still don't utilize clustering and instead are stateless apps and that's fine, but ocaml can already do stateless apps just fine, right? So my assumption here is that they're putting ocaml on the BEAM to utilize OTP and clustered apps, correct? If that's the case I don't see how you could have a real-world app that you can quickly scale up/down (which is something you often want in a clustered app) - instead it seems like you'd be forced to spin up entirely new clusters and then direct traffic to the new cluster to bypass this problem. This of course isn't a viable tradeoff for very large clusters.
      - jolux 6y ago
        As I understand it, there are real questions about the type theory needed to correctly express what the BEAM does, and they have yet to be answered sufficiently. Nonetheless, a lot of people clearly want static types in their BEAM applications, and there are certainly parts of any application that don't cross the network. I'm interested in efforts like Caramel and Gleam mostly because of my interest in BEAM as a general-purpose platform, and because I want to see if they come up with solutions to the type theory problems.
      - derefr 6y ago
        I think what the parent is saying isn't that most production Erlang apps aren't clustered; it's that most production Erlang apps aren't clustered using the distribution protocol. Instead, they're clustered at the application layer, with explicit wire protocols like gRPC. You can upgrade nodes within such clusters just fine, because the ABI of the wire protocol is an explicit part of the application, something that can be stabilized separately from the app itself, rather than being an implicit property of what ERTS version you're using, or about how the application's processes communicate internally. Note that even in the context of Ericsson, Erlang's distribution protocol was never designed for the use-case of horizontal N-node clustering. It was designed for static-role clustering — where each node is like an organ in your body, with a name and a specific function relative to the "system" that is your body. For example, "master" and "warm standby" roles. (By analogy to Kubernetes, the Erlang nodes of a distributed-OTP architecture are closest to being like the sibling containers of a single k8s pod. Except that Erlang's "containers" can be running on separate machines while still being part of the same "pod".) If you want to scale such a system (a "pod" of nodes), you are supposed to spawn N copies of the entire distribution set; and then those N system-copies will coordinate with other such clones not via the distribution protocol, but via explicit coordination protocols. (Sometimes this explicit coordination protocol is designed to use the distribution protocol as a carrier — this is one of the reasons that Erlang supports making manual distribution-protocol connections between nodes, rather than forcing a connected topology on you. But such connections carefully avoid passing arbitrary RPC data across them, instead usually having an explicit "peer server" on both ends, where the servers speak a limited and ABI-stable term protocol to one another.) Yes, the Erlang distribution protocol has been repeatedly taken beyond its design tolerances to accomplish horizontal scaling. CouchDB does this, for example. But many Erlang applications that you might think of as doing this, are actually much closer to static "organs in a body" architecture than you'd think, despite claims of scalability. Riak and Ejabberd, for two examples, both treat their nodes very much like static organs, rather than a fleet.

by toolz 6y ago

jolux 6y ago

Different organizations use different amounts of the tools the BEAM provides in building their distributed applications. We containerize everything and don't really use the hot upgrade features.

toolz 6y ago
sure, but I'm not talking about hot upgrades, I'm talking about upgrading entire nodes that exist in a cluster. The nodes will not be able to pass around functions as an upgraded node in a cluster will have a different version of that function. So this is an issue without considering hot code reloading, as I don't use that feature either.
- jolux 6y ago
  I don't quite follow how static types change this situation?
- yawaramin 6y ago
  So we're not talking about hot code reloading here, we're talking exactly about upgrading nodes in a cluster. Many--I'd say most--Erlang/Elixir deployments today don't do that. They use Kubernetes.
  - toolz 6y ago
    Kubernetes does not solve the issue, it merely has strategies for how to upgrade nodes. The issue is that when you compile new code, even if the function didn't change at all, it will be a new version of the function. If the upgraded node then sends a function to another node to be executed (as is a very common practice in meshed OTP apps) the older node will not be able to execute the function. Now, I understand that most erlang apps still don't utilize clustering and instead are stateless apps and that's fine, but ocaml can already do stateless apps just fine, right? So my assumption here is that they're putting ocaml on the BEAM to utilize OTP and clustered apps, correct? If that's the case I don't see how you could have a real-world app that you can quickly scale up/down (which is something you often want in a clustered app) - instead it seems like you'd be forced to spin up entirely new clusters and then direct traffic to the new cluster to bypass this problem. This of course isn't a viable tradeoff for very large clusters.
    - jolux 6y ago
      As I understand it, there are real questions about the type theory needed to correctly express what the BEAM does, and they have yet to be answered sufficiently. Nonetheless, a lot of people clearly want static types in their BEAM applications, and there are certainly parts of any application that don't cross the network. I'm interested in efforts like Caramel and Gleam mostly because of my interest in BEAM as a general-purpose platform, and because I want to see if they come up with solutions to the type theory problems.
    - derefr 6y ago
      I think what the parent is saying isn't that most production Erlang apps aren't clustered; it's that most production Erlang apps aren't clustered using the distribution protocol. Instead, they're clustered at the application layer, with explicit wire protocols like gRPC. You can upgrade nodes within such clusters just fine, because the ABI of the wire protocol is an explicit part of the application, something that can be stabilized separately from the app itself, rather than being an implicit property of what ERTS version you're using, or about how the application's processes communicate internally. Note that even in the context of Ericsson, Erlang's distribution protocol was never designed for the use-case of horizontal N-node clustering. It was designed for static-role clustering — where each node is like an organ in your body, with a name and a specific function relative to the "system" that is your body. For example, "master" and "warm standby" roles. (By analogy to Kubernetes, the Erlang nodes of a distributed-OTP architecture are closest to being like the sibling containers of a single k8s pod. Except that Erlang's "containers" can be running on separate machines while still being part of the same "pod".) If you want to scale such a system (a "pod" of nodes), you are supposed to spawn N copies of the entire distribution set; and then those N system-copies will coordinate with other such clones not via the distribution protocol, but via explicit coordination protocols. (Sometimes this explicit coordination protocol is designed to use the distribution protocol as a carrier — this is one of the reasons that Erlang supports making manual distribution-protocol connections between nodes, rather than forcing a connected topology on you. But such connections carefully avoid passing arbitrary RPC data across them, instead usually having an explicit "peer server" on both ends, where the servers speak a limited and ABI-stable term protocol to one another.) Yes, the Erlang distribution protocol has been repeatedly taken beyond its design tolerances to accomplish horizontal scaling. CouchDB does this, for example. But many Erlang applications that you might think of as doing this, are actually much closer to static "organs in a body" architecture than you'd think, despite claims of scalability. Riak and Ejabberd, for two examples, both treat their nodes very much like static organs, rather than a fleet.