4 ms·
While I was at WhatsApp, we used hot loading for most server updates on our Erlang cluster(s). There's tricks and traps, but if you do things right, it basicall
by toast0 5y ago
While I was at WhatsApp, we used hot loading for most server updates on our Erlang cluster(s). There's tricks and traps, but if you do things right, it basically works and you can fix a lot of bugs right away without dropping connections.
The way Erlang works is any module can have two versions loaded; you can call them current and old. A fully qualified function call (Module:Function) calls into the current module, but non-fully qualified calls (like function calls within the same module, and some function references) will stay with the version they were made from, so if you're in what was current code, and a new version is loaded, current becomes old and non-fully qualified calls go to old.
You can only have those two versions though, if you load a third, any processes that are in the old version will be killed before current becomes old and new becomes current. There's ways to check if there are any processes 'lingering' in old code, and design techniques to reduce the chances of that; the easiest is making sure your loop code calls ?MODULE:loop(...) instead of loop(...), and most of the OTP looping constructs (gen_X) do that, or even if they weren't reloadable, they're doing the looping and calling into your code with fully qualified calls anyway. Tail Call Optimization is the easiest way to get the older code off the bottom of your stack, although there are others.
Of course, you can do hot code loading in lots of other languages too. In C, you can do it intentionally with dlopen/dlsym, or if your OS uses mmaped libraries, by editing libraries while they're loaded (whoops! use install instead of cp to avoid doing that by accident).
- roeles 5y agoThanks for providing some insight in how WhatsApp did this with Erlang! Is all of the logic to support runtime upgrading inside Erlang/OTP, or does the software you write have to be written (partly) in mind with runtime upgrades as well? Also, thanks for the hint regarding cp/install. I figured that the kernel would recognize an open descriptor to the old file and keep that old copy until all references are gone.
- toast0 5y agoOTP has some structures to help with upgrading that we didn't use. Mostly, you have to keep upgrading in mind when you make changes. Easy ones are things like if you add another field into a record (like a struct) that you're using for process state, you need to still handle the old version of the record, since it will be loaded into processes with old state. This is similar to what you need to do with APIs (local library or remote calls to other computers), but it's a little unusual to consider it in a running process. For cp/install; cp will generally write over the existing file, but install unlinks and then writes a new file. If you've mmaped the file, you get changes in your process from cp (copy on write doesn't apply to mmap), but not with install (the old file is kept around until it's no longer referenced).