5 ms·

What are you doing to handle replication among nodes? There's still a great reason to reach for Redis for caching and messaging despite using Elixir: when the

by Dowwie 4y ago

What are you doing to handle replication among nodes? There's still a great reason to reach for Redis for caching and messaging despite using Elixir: when the Application crashes, the state is gone. As for Stripe, some client developers have been code-generating their client with mixed success. ExAws, however, is essential.

liveoneggs 4y ago
replication is built in to mnesia and your "app" crashing really shouldn't be taking down the entire OTP on your local machine, so ETS would persist on a single node
- mplewis 4y ago
  When you’re running in a highly-distributed environment, your instances can go away for reasons other than OTP crashing – say, you’re being rescheduled onto a new node. Am I missing context here?
  - liveoneggs 4y ago
    "crash" in the context of elixir (to me) is about a part of a supervision tree - "let it crash" Obviously running mnesia on an auto-scaling group makes about as much sense as running postgres on one. So the missing context is ERTS, I suppose.
  - toast0 4y ago
    That's not a risk unique to Erlang or Mnesia. There's all sorts of reasons nodes may fail, some where data persisted on that node is temporarily unavailable (network, power, cabling, OS/application fault, etc), some where it is permanently unavailable (storage failure, node reassigned, operator error, confiscation, etc). In my experience, Mnesia works best if you can ensure an orderly sequence of online nodes, but unless you use ram_copies tables, the data is persisted to disc, with journaling, so it's recoverable as long as you have an orderly sequence of nodes in general. By orderly sequence, I mean that you don't lose access to a node thatis the sole holder of data. On hard provisioned systems, two geoseparated pairs of two nodes should be more than sufficient, but don't get them all storage from the same batch and turn on everything at once. You might want to add a fifth node somewhere if you want to pursue a majority transaction setup. If your nodes are more ephemeral, you probably want three (or more?) per location so you have a better chance of data migration within the location. If you can provision network storage with good persistance forecasts, you could look into storing snapshots (and maybe journals) there, but that's not an out of the box experience. As much as possible, you want to put nodes that store redundant data in different failure domains; not in the same rack attached to the same switch and the same PDU. Of course, there's connectivity benefits of being close too, so you have to weigh the risks with the benefits.

by Dowwie 4y ago

liveoneggs 4y ago

replication is built in to mnesia and your "app" crashing really shouldn't be taking down the entire OTP on your local machine, so ETS would persist on a single node

mplewis 4y ago
When you’re running in a highly-distributed environment, your instances can go away for reasons other than OTP crashing – say, you’re being rescheduled onto a new node. Am I missing context here?
- liveoneggs 4y ago
  "crash" in the context of elixir (to me) is about a part of a supervision tree - "let it crash" Obviously running mnesia on an auto-scaling group makes about as much sense as running postgres on one. So the missing context is ERTS, I suppose.
- toast0 4y ago
  That's not a risk unique to Erlang or Mnesia. There's all sorts of reasons nodes may fail, some where data persisted on that node is temporarily unavailable (network, power, cabling, OS/application fault, etc), some where it is permanently unavailable (storage failure, node reassigned, operator error, confiscation, etc). In my experience, Mnesia works best if you can ensure an orderly sequence of online nodes, but unless you use ram_copies tables, the data is persisted to disc, with journaling, so it's recoverable as long as you have an orderly sequence of nodes in general. By orderly sequence, I mean that you don't lose access to a node thatis the sole holder of data. On hard provisioned systems, two geoseparated pairs of two nodes should be more than sufficient, but don't get them all storage from the same batch and turn on everything at once. You might want to add a fifth node somewhere if you want to pursue a majority transaction setup. If your nodes are more ephemeral, you probably want three (or more?) per location so you have a better chance of data migration within the location. If you can provision network storage with good persistance forecasts, you could look into storing snapshots (and maybe journals) there, but that's not an out of the box experience. As much as possible, you want to put nodes that store redundant data in different failure domains; not in the same rack attached to the same switch and the same PDU. Of course, there's connectivity benefits of being close too, so you have to weigh the risks with the benefits.