4 ms·

The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones. Like upgrading when a data structure changes between vers

by gull 11y ago

The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones. Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.

vardump 11y ago
Synchronize to a checkpoint. Serialize all state. Run data structure upgrading code. Deserialize upgraded state. Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!
toast0 11y ago
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and handle_info if you use those). You have to do a little work, but I don't see how you avoid that?
- gull 11y ago
  That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Update_safety http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_manual/code_loading.html#id86424 http://www.erlang.org/doc/reference_manual/code_loading.html...
  - serialx 11y ago
    If you write the code within the `gen_server` guidelines, state migration is supported by `code_change`: http://www.erlang.org/doc/man/gen_server.html#Module:code_change-3 http://www.erlang.org/doc/man/gen_server.html#Module:code_ch... For example: http://stackoverflow.com/questions/1840717/achieving-code-swapping-in-erlangs-gen-server http://stackoverflow.com/questions/1840717/achieving-code-sw... BTW, You can even support downgrade. :)
  - toast0 11y ago
    > Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it. The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
    - gull 11y ago
      You touched on most points, primarily avoiding the situation of reasoning about concurrent old and new code. I didn't know a gen_server manages code loading like this, thank you for that. An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages. An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues. http://pmg.csail.mit.edu/upgrades/ http://pmg.csail.mit.edu/upgrades/
Thaxll 11y ago
What about running websockets?

by gull 11y ago

vardump 11y ago

Synchronize to a checkpoint. Serialize all state. Run data structure upgrading code. Deserialize upgraded state. Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!

toast0 11y ago

In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and handle_info if you use those). You have to do a little work, but I don't see how you avoid that?

gull 11y ago
That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Update_safety http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_manual/code_loading.html#id86424 http://www.erlang.org/doc/reference_manual/code_loading.html...
- serialx 11y ago
  If you write the code within the `gen_server` guidelines, state migration is supported by `code_change`: http://www.erlang.org/doc/man/gen_server.html#Module:code_change-3 http://www.erlang.org/doc/man/gen_server.html#Module:code_ch... For example: http://stackoverflow.com/questions/1840717/achieving-code-swapping-in-erlangs-gen-server http://stackoverflow.com/questions/1840717/achieving-code-sw... BTW, You can even support downgrade. :)
- toast0 11y ago
  > Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it. The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
  - gull 11y ago
    You touched on most points, primarily avoiding the situation of reasoning about concurrent old and new code. I didn't know a gen_server manages code loading like this, thank you for that. An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages. An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues. http://pmg.csail.mit.edu/upgrades/ http://pmg.csail.mit.edu/upgrades/

Thaxll 11y ago

What about running websockets?