6 ms·

This is great news! The deployment story has progressively improved since I started using Elixir in 2016, but the best way is to just have _one_ community bless

by atonse 7y ago

This is great news! The deployment story has progressively improved since I started using Elixir in 2016, but the best way is to just have _one_ community blessed system. Distillery has helped a lot for sure, so it's nice that it's basically been rolled into the base elixir distro.

AlchemistCamp 7y ago
Based on what I've learned from the Elixir forum, these new releases have "no support for hot code updates / appups / relups". This means that Distillery will remain the best choice for many of us, including myself. I have a very simple deploy process with Edeliver and Gitlab CI/CD that basically just consists of pushing or merging a commit to master that passes the tests. There's zero down-time and it doesn't require containers on the server. I'd also say that the "deployment story" wasn't any worse than Rails or other stacks in the past except in that people have a choice of having a stateful server. This is how I'm doing it now: https://youtu.be/-mm44ADU3kc?t=172 https://youtu.be/-mm44ADU3kc?t=172
- superdisk 7y ago
  Wait, what? I wrote an Elixir/Phoenix webapp around 2015 and one of the major selling points was the ability to hot-upgrade code that was inherited from Erlang. What changed?
  - dmix 7y ago
    OP mentions it's still entirely possible with other deployment libraries (Distillery). Just the one being newly packaged into Elixir 1.9+ by default does not yet support hot-loading...
  - andy_ppp 7y ago
    The problem with hot code updates is they can break your app if your structs change shape and are extremely hard to get right over a period of time. Best to do rolling deployments with Kubenetes or something...
    - gamache 7y ago
      You make it sound like setting up k8s is a walk in the park! I've used hot code upgrades in production for 3+ years, and we've had it blow up in our face perhaps twice. If you do canary deploys, it's not hard to control the blast radius.
    - toast0 7y ago
      Hot code updates can be challenging at times, but the decrease in deployment time and hassle for changes where it makes sense is definitely worth it, and using hot code loading for most updates doesn't prevent you from using a rolling restart for the updates where it's more appropriate. If you're using distribution, there's a good chance you need to deal with messages from both updated and not updated nodes anyway, so handling different shapes from different versions is a skill you already need to have; it's just hot load exposes it at more layers than previously. Since working in Erlang for several years, when I have to work in languages where hot loading is not easy or common, it's always frustrating. Throwing away connection attached state to update code is a hard choice that doesn't need to be.
    - skissane 7y ago
      > The problem with hot code updates is they can break your app if your structs change shape I have no experience with Erlang/Elixir myself, so please excuse me if this is a silly question – but, why can't the language/platform actually detect the difference in struct shape, and refuse to deploy the new version in that case? Or, demand that you provide some sort of conversion function that maps the old struct to the new one? Is this a consequence of lack of static typing?
      - dnautics 7y ago
        I have never actually used this feature but here goes: The language standard library gives you hooks to relup the internal state of all of your processes: https://hexdocs.pm/elixir/GenServer.html#c:code_change/3 https://hexdocs.pm/elixir/GenServer.html#c:code_change/3 What it doesn't do is give you a hook to relup the messages that get passed between processes. If you do it right, you can pattern match against different versions of messages, and do the correct conversion function, but there are liable to be many, many, shapes of messages (including ones that you might not write yourself, e.g. coming from a library) and it might be difficult to catch them all. To do it right, I would want to set up a lot of testing to make sure you can hot code reload safely. There is no specific guideline, and it probably hurts forward progress in developing guidelines, that generally, it's okay to have some downtime in any individual server node as erlang/elixir encourages you to think about failover anyways, and most elixir apps are relatively stateless webservers, so you've probably got robust load balancing and migration scheme in place in your cluster to begin with, making blue/green or rolling updates a "pretty much good enough" thing. > Is this a consequence of lack of static typing No.
  - dudul 7y ago
    I would be curious to know what makes this feature a major selling point. Aren't rolling deploys easier to handle? I'm not saying it's not a great feature, but honestly, it probably stopped being useful since Erlang/OTP was originally designed.
    - gamache 7y ago
      Hot code upgrades are one of the Erlang features that helps me keep headcount down on our operations team. Appcues is three years into running Elixir in prod, and we still have not had to set up blue/green deploys because the built-in upgrade and rollback feature is so robust. For the first year and a half of that, we had a single platform developer (hi!) supporting dozens of servers and hundreds of customers. Hot upgrades are very, very useful.
      - wstuartcl 7y ago
        yes, but they do come with a lot of cognitive load and sharp edges. For many apps they are simply not worth it.
        gamache 7y ago
        The cognitive load is quite low when deploying changes which only involve your app code (i.e., no dependency upgrades). My app has a few tens of millions of open websockets at a time, and it's worth it to me to avoid mass reconnects. I'm not everyone, but my use case isn't totally unique either.
        wstuartcl 7y ago
        I am not sure about your specific use case but it owuld be hard to imagine that graceful reconnect on websocket + a more traditional deploy would not be much more effective resourcing vs that hurdles you have to go through to properly appup. Is it really an advantage to you or has it slowed your ability to get to market for something that seems like it "should" be an advantage. Its not like you don't need graceful reconnect handling logic for the websockets anyways -- net-splits and endpoint failures are a thing.
  - dpeck 7y ago
    It was talked about a lot by people who were fans of elixir/erlang but hadn't really done much work with it (these folks are increasingly common in tech, confusingly so sometimes but good). Digging deeper the line from folks using erlang/elixir "in anger" was always that it was a supported feature but the reality is that most people shouldn't do it and wouldn't need to for the hassle it has.
  - wstuartcl 7y ago
    I compare it to unwrapping a handfull of utility razors chucking them up in the air and saying "free razors!". You CAN do hot upgrades, but many times the complexity of doing so far outweighs the benefits. For any non trivial app it makes updates/deploys to the app non trivial as well.
  - yellowapple 7y ago
    Nothing changed; hot code reloading is still possible through the mechanisms that have always existed. What's being stated here is that the new Elixir support for "releases" does not include support for this, so you would have to use a separate mechanism to perform the hot code reload. That "separate mechanism" could be as simple as a plug that tells Mix to recompile and reload. Example (designed for - and part of - the Sugar framework, but should theoretically work for any Plug-based app, including Phoenix-based ones): https://github.com/sugar-framework/plugs/blob/master/lib/sugar/plugs/hot_code_reload.ex https://github.com/sugar-framework/plugs/blob/master/lib/sug... Hot upgrades like this are not foolproof (which is likely the reason why the above example is gated to :dev environments); there are other concerns like database migrations and other internal and external variances that make this inappropriate for most production situations. That said, these same concerns often exist for other high-availability situations as well, so if you know that you want zero-downtime code upgrades, figuring out a way to do it cleanly within the application is likely valuable as a way to avoid the hell on Earth that is trying to do this with, say, a bunch of load-balanced Docker comtainers.
    - yellowapple 7y ago
      As a bit of a correction here to my own comment, it looks like Elixir releases don't ship with Mix, so the above example probably wouldn't work in that particular scenario (unless maybe you figure out some way to include Mix). So the better option would be to figure out a way to compile a new release, get the modules in place where the currently-running release expects them, then write up a plug similar to the above to check for new module versions and load them (or just do it as part of whatever mechanism you'd use to get the updated modules into the running release in the first place).
gamache 7y ago
> Distillery has helped a lot for sure, so it's nice that it's basically been rolled into the base elixir distro. That would be nice if it were true. However, for better or worse, Distillery was Not Invented Here, so what Elixir shipped with today is a reimplementation of the simple half of Distillery.
- atonse 7y ago
  Which is fine for probably most of us using Elixir who do blue-green deployments behind a load balancer. Although we should probably embrace this aspect of the ecosystem, (like with clustering of servers) I bet it's even easier than I can imagine.
- pdimitar 7y ago
  Or, you know, the creators want one official blessed way so as not to fragment the ecosystem, so they integrated part of Distillery upstream in the language itself.
josevalim 7y ago
There are some comments below about Distillery vs Elixir releases, so I will hijack the root comment to add some clarifications. It was clear to both Paul (Distillery author) and the Elixir team that Elixir releases were going to be a subset of what Distillery provides. There is a good chunk of what Distillery does that is well established and that's what we streamlined and brought into Elixir. Some other areas were left out, such as a complete structure for setting up CLI hooks, and most notably, hot code upgrades. The documentation explains some of the complexities behind hot code upgrades: https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-code-upgrades https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-cod... - so I won't go over those. More importantly, during discussions with different teams and library authors, it was clear there is not a golden path for hot code upgrades. There are different approaches, with different gains and pitfalls, and it would be too premature to choose one as blessed. So what is the path forward? While Elixir releases don't do hot code upgrades by default, its whole structure supports it. We use the proper names and structure everywhere. But the front-end to start the upgrade is not there. I have discussed with Paul the possibility of Distillery building on top of what Elixir releases provide with a more complete feature set. It doesn't have to be done by Distillery either. Maybe someone (or multiple people) will provide a smaller package that focuses on hot code upgrades. Then once we gather enough collective knowledge, we can choose if and how to proceed. Still, I believe Elixir releases cover the majority of use cases out there. But we will know for sure over the next months. At least, by making releases part of core, we hope we are easing the learning curve for releases altogether, which we will naturally lead more people to explore hot code upgrades.
- atonse 7y ago
  We've never actually done hot code upgrades yet before, although, like many Elixir/Erlang/OTP things, I feel confident that if I was required to do so, it would be easier than in most other ecosystems. Similarly I suspect many users of elixir are doing the "boring" thing that we're doing – load balancer -> stateless web server(s) running phx -> DB server. And they'd probably be more than happy with this.
- AlchemistCamp 7y ago
  It seems like a pure win as long as there's still a zero downtime solution that doesn't rely on containers. I'm definitely looking forward to what gets built on top of 1.9 releases!

by atonse 7y ago

AlchemistCamp 7y ago

Based on what I've learned from the Elixir forum, these new releases have "no support for hot code updates / appups / relups". This means that Distillery will remain the best choice for many of us, including myself. I have a very simple deploy process with Edeliver and Gitlab CI/CD that basically just consists of pushing or merging a commit to master that passes the tests. There's zero down-time and it doesn't require containers on the server. I'd also say that the "deployment story" wasn't any worse than Rails or other stacks in the past except in that people have a choice of having a stateful server. This is how I'm doing it now: https://youtu.be/-mm44ADU3kc?t=172 https://youtu.be/-mm44ADU3kc?t=172

superdisk 7y ago
Wait, what? I wrote an Elixir/Phoenix webapp around 2015 and one of the major selling points was the ability to hot-upgrade code that was inherited from Erlang. What changed?
- dmix 7y ago
  OP mentions it's still entirely possible with other deployment libraries (Distillery). Just the one being newly packaged into Elixir 1.9+ by default does not yet support hot-loading...
- andy_ppp 7y ago
  The problem with hot code updates is they can break your app if your structs change shape and are extremely hard to get right over a period of time. Best to do rolling deployments with Kubenetes or something...
  - gamache 7y ago
    You make it sound like setting up k8s is a walk in the park! I've used hot code upgrades in production for 3+ years, and we've had it blow up in our face perhaps twice. If you do canary deploys, it's not hard to control the blast radius.
  - toast0 7y ago
    Hot code updates can be challenging at times, but the decrease in deployment time and hassle for changes where it makes sense is definitely worth it, and using hot code loading for most updates doesn't prevent you from using a rolling restart for the updates where it's more appropriate. If you're using distribution, there's a good chance you need to deal with messages from both updated and not updated nodes anyway, so handling different shapes from different versions is a skill you already need to have; it's just hot load exposes it at more layers than previously. Since working in Erlang for several years, when I have to work in languages where hot loading is not easy or common, it's always frustrating. Throwing away connection attached state to update code is a hard choice that doesn't need to be.
  - skissane 7y ago
    > The problem with hot code updates is they can break your app if your structs change shape I have no experience with Erlang/Elixir myself, so please excuse me if this is a silly question – but, why can't the language/platform actually detect the difference in struct shape, and refuse to deploy the new version in that case? Or, demand that you provide some sort of conversion function that maps the old struct to the new one? Is this a consequence of lack of static typing?
    - dnautics 7y ago
      I have never actually used this feature but here goes: The language standard library gives you hooks to relup the internal state of all of your processes: https://hexdocs.pm/elixir/GenServer.html#c:code_change/3 https://hexdocs.pm/elixir/GenServer.html#c:code_change/3 What it doesn't do is give you a hook to relup the messages that get passed between processes. If you do it right, you can pattern match against different versions of messages, and do the correct conversion function, but there are liable to be many, many, shapes of messages (including ones that you might not write yourself, e.g. coming from a library) and it might be difficult to catch them all. To do it right, I would want to set up a lot of testing to make sure you can hot code reload safely. There is no specific guideline, and it probably hurts forward progress in developing guidelines, that generally, it's okay to have some downtime in any individual server node as erlang/elixir encourages you to think about failover anyways, and most elixir apps are relatively stateless webservers, so you've probably got robust load balancing and migration scheme in place in your cluster to begin with, making blue/green or rolling updates a "pretty much good enough" thing. > Is this a consequence of lack of static typing No.
- dudul 7y ago
  I would be curious to know what makes this feature a major selling point. Aren't rolling deploys easier to handle? I'm not saying it's not a great feature, but honestly, it probably stopped being useful since Erlang/OTP was originally designed.
  - gamache 7y ago
    Hot code upgrades are one of the Erlang features that helps me keep headcount down on our operations team. Appcues is three years into running Elixir in prod, and we still have not had to set up blue/green deploys because the built-in upgrade and rollback feature is so robust. For the first year and a half of that, we had a single platform developer (hi!) supporting dozens of servers and hundreds of customers. Hot upgrades are very, very useful.
    - wstuartcl 7y ago
      yes, but they do come with a lot of cognitive load and sharp edges. For many apps they are simply not worth it.
      - gamache 7y ago
        The cognitive load is quite low when deploying changes which only involve your app code (i.e., no dependency upgrades). My app has a few tens of millions of open websockets at a time, and it's worth it to me to avoid mass reconnects. I'm not everyone, but my use case isn't totally unique either.
        wstuartcl 7y ago
        I am not sure about your specific use case but it owuld be hard to imagine that graceful reconnect on websocket + a more traditional deploy would not be much more effective resourcing vs that hurdles you have to go through to properly appup. Is it really an advantage to you or has it slowed your ability to get to market for something that seems like it "should" be an advantage. Its not like you don't need graceful reconnect handling logic for the websockets anyways -- net-splits and endpoint failures are a thing.
- dpeck 7y ago
  It was talked about a lot by people who were fans of elixir/erlang but hadn't really done much work with it (these folks are increasingly common in tech, confusingly so sometimes but good). Digging deeper the line from folks using erlang/elixir "in anger" was always that it was a supported feature but the reality is that most people shouldn't do it and wouldn't need to for the hassle it has.
- wstuartcl 7y ago
  I compare it to unwrapping a handfull of utility razors chucking them up in the air and saying "free razors!". You CAN do hot upgrades, but many times the complexity of doing so far outweighs the benefits. For any non trivial app it makes updates/deploys to the app non trivial as well.
- yellowapple 7y ago
  Nothing changed; hot code reloading is still possible through the mechanisms that have always existed. What's being stated here is that the new Elixir support for "releases" does not include support for this, so you would have to use a separate mechanism to perform the hot code reload. That "separate mechanism" could be as simple as a plug that tells Mix to recompile and reload. Example (designed for - and part of - the Sugar framework, but should theoretically work for any Plug-based app, including Phoenix-based ones): https://github.com/sugar-framework/plugs/blob/master/lib/sugar/plugs/hot_code_reload.ex https://github.com/sugar-framework/plugs/blob/master/lib/sug... Hot upgrades like this are not foolproof (which is likely the reason why the above example is gated to :dev environments); there are other concerns like database migrations and other internal and external variances that make this inappropriate for most production situations. That said, these same concerns often exist for other high-availability situations as well, so if you know that you want zero-downtime code upgrades, figuring out a way to do it cleanly within the application is likely valuable as a way to avoid the hell on Earth that is trying to do this with, say, a bunch of load-balanced Docker comtainers.
  - yellowapple 7y ago
    As a bit of a correction here to my own comment, it looks like Elixir releases don't ship with Mix, so the above example probably wouldn't work in that particular scenario (unless maybe you figure out some way to include Mix). So the better option would be to figure out a way to compile a new release, get the modules in place where the currently-running release expects them, then write up a plug similar to the above to check for new module versions and load them (or just do it as part of whatever mechanism you'd use to get the updated modules into the running release in the first place).

gamache 7y ago

> Distillery has helped a lot for sure, so it's nice that it's basically been rolled into the base elixir distro. That would be nice if it were true. However, for better or worse, Distillery was Not Invented Here, so what Elixir shipped with today is a reimplementation of the simple half of Distillery.

atonse 7y ago
Which is fine for probably most of us using Elixir who do blue-green deployments behind a load balancer. Although we should probably embrace this aspect of the ecosystem, (like with clustering of servers) I bet it's even easier than I can imagine.
pdimitar 7y ago
Or, you know, the creators want one official blessed way so as not to fragment the ecosystem, so they integrated part of Distillery upstream in the language itself.

josevalim 7y ago

There are some comments below about Distillery vs Elixir releases, so I will hijack the root comment to add some clarifications. It was clear to both Paul (Distillery author) and the Elixir team that Elixir releases were going to be a subset of what Distillery provides. There is a good chunk of what Distillery does that is well established and that's what we streamlined and brought into Elixir. Some other areas were left out, such as a complete structure for setting up CLI hooks, and most notably, hot code upgrades. The documentation explains some of the complexities behind hot code upgrades: https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-code-upgrades https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-cod... - so I won't go over those. More importantly, during discussions with different teams and library authors, it was clear there is not a golden path for hot code upgrades. There are different approaches, with different gains and pitfalls, and it would be too premature to choose one as blessed. So what is the path forward? While Elixir releases don't do hot code upgrades by default, its whole structure supports it. We use the proper names and structure everywhere. But the front-end to start the upgrade is not there. I have discussed with Paul the possibility of Distillery building on top of what Elixir releases provide with a more complete feature set. It doesn't have to be done by Distillery either. Maybe someone (or multiple people) will provide a smaller package that focuses on hot code upgrades. Then once we gather enough collective knowledge, we can choose if and how to proceed. Still, I believe Elixir releases cover the majority of use cases out there. But we will know for sure over the next months. At least, by making releases part of core, we hope we are easing the learning curve for releases altogether, which we will naturally lead more people to explore hot code upgrades.

atonse 7y ago
We've never actually done hot code upgrades yet before, although, like many Elixir/Erlang/OTP things, I feel confident that if I was required to do so, it would be easier than in most other ecosystems. Similarly I suspect many users of elixir are doing the "boring" thing that we're doing – load balancer -> stateless web server(s) running phx -> DB server. And they'd probably be more than happy with this.
AlchemistCamp 7y ago
It seems like a pure win as long as there's still a zero downtime solution that doesn't rely on containers. I'm definitely looking forward to what gets built on top of 1.9 releases!