4 ms·

Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? I'm genuinely curious, I kind of had this impr

Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang.

devgutt 13y ago
An interesting example of Erlang hot code reloading in embedded system: http://www.youtube.com/watch?v=96UzSHyp0F8 http://www.youtube.com/watch?v=96UzSHyp0F8
agentultra 13y ago
> Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? Yes. > I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang. It's not. iIRC there's even a version of the JVM with code reloading. What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definitions can be changed and live instances will be updated without stopping the program or reloading anything. There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.
- simula67 13y ago
  > there's even a version of the JVM with code reloading. Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.
  - phamilton 13y ago
    Is there a mechanism to migrate existing data to the new code?
    - simula67 13y ago
      If you have external state stored in say a database, I think you have to control your deployments to migrate the schema before the model objects can be deployed.
      - phamilton 13y ago
        So no. One of the beauties of Erlang's hot code loading is the ability to migrate internal application state. An example: I'm storing a lookup table going from key to value as a tuple {key, value}. With a change going out, I'm going to start doing {key, {value, other-value}}. In Erlang there's an easy pattern in place to do that. Just provide a function that goes from {key, value} to {key, {value, sensible-default}} and as part of the code reloading process that function will be run before the new code starts executing.
        simula67 13y ago
        You may have misunderstood. A file can have multiple, say 'deployment units'. You can have a 'migrate deployment unit' which depend on the 'model deployment unit' which forces the app server to deploy the migrate unit first. You can have code there that migrates the tables and fill with default values before your model entities become available for use. From your comment this seems to be the same as what Erlang does for your example, unless you are claiming that Erlang's implementation is superior which could be the true, since I have never written any non-trivial programs in it.
        bitwalker 13y ago
        I think the difference here is that the migration of internal state happens the moment the module is swapped out in memory. Erlang doesn't have mutable state, so state is instead passed around within a module. When a fix needs to be deployed, the two versions (old and new) run simultaneously, any new processes immediately run the new code, old processes call a `code_change` function which takes the current state within that process, performs any transformations required for the new code, and then the old module code is swapped out for the new. That's not at all a perfect explanation of the process, but it's more or less useful enough to describe the difference between the two platforms.
  - cmsd2 13y ago
    sure, any existing connections continue to be serviced by the old code, and new connections are picked up by the new code. if you need more elaborate lifecycle management of components as they are upgraded, then maybe OSGI could do that. erlang's approach really appeals to me though -- not only does the code get upgraded, but it's in cooperation with its message loop so you don't miss a step, and you also get a callback to migrate your actor state.
- bitwalker 13y ago
  > It's not. iIRC there's even a version of the JVM with code reloading. I've seen some info about the JVM implementation, but I could've sworn it wasn't able to preserve state while hot swapping. I haven't dug in to it though, so it's totally possible it handles that. > There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software. I'm a fan of Lisp in general, though my experience is limited to a tiny bit of Scheme and experimenting with Clojure. I'm glad to see that stuff like this is available in at least one of the Lisp variants! Question for you (if you program in Common Lisp): What are the main selling points? I spend a lot of time already with a couple of languages, Elixir/Erlang (for fun), C#/F# (for work) - what is it that would make me want to switch from one of those to Common Lisp? Thanks!
  - logicchains 13y ago
    As someone who's written a bit of Common Lisp, I can say that the macros are a pretty awesome feature to use. Common Lisp has the advantage over Clojure and Scheme that the SBCL compiler is incredibly fast; it can come close to C#/F# on Mono for some tasks, in spite of Lisp being a dynamic language. It also compiles code and expands macros much faster. It's got more power than the other lisps, but it's also less elegant, so make of that what you will.
    - klibertp 13y ago
      > I can say that the macros are a pretty awesome feature to use Erlang has them in the form of parse transforms and Elixir has them too and makes them easy to use. Not as easy as sexp based languages, but almost.
      - sea6ear 13y ago
        There is also LFE (Lisp Flavored Erlang) [0] which gives lisp style macros and some other goodies to Erlang [0] http://lfe.github.io/ http://lfe.github.io/
  - pron 13y ago
    BTW, a very Erlangish actor system, complete with hot code swapping (while preserving state), is available for Clojure and is called Pulsar (all open-source): http://docs.paralleluniverse.co/pulsar/ http://docs.paralleluniverse.co/pulsar/
- shiro 13y ago
  I use Common Lisp and Scheme at work, and have written and maintained several systems heavily relied on class redefinition and run-time code loading. They're great. OTOH, I feel Common Lisp's concurrent process handling is rather weak, and I envy Erlang for that (Not that CL can't do the trick, but usually what implementations offer is fairly low-level and I had to roll my own higher-level IPC.) For example, how well CL's class redefintion and instance reinitialization work with preemptive threads? It isn't obvious how each implementation address this issue by skimming docs, so I'm curious if anybody with direct experience. When I implemented CLOS/MOP style OO in my Scheme, I ended up using a global lock during class redefinition, for it need to modify lots of places. It's MT-safe now, but I'm still wary to run class redefinition on a system written in my Scheme, which is busy handling requests.
lispm 13y ago
One loads and replaces incrementally in Lisp.

devgutt 13y ago

An interesting example of Erlang hot code reloading in embedded system: http://www.youtube.com/watch?v=96UzSHyp0F8 http://www.youtube.com/watch?v=96UzSHyp0F8

agentultra 13y ago

> Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? Yes. > I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang. It's not. iIRC there's even a version of the JVM with code reloading. What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definitions can be changed and live instances will be updated without stopping the program or reloading anything. There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.

simula67 13y ago
> there's even a version of the JVM with code reloading. Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.
- phamilton 13y ago
  Is there a mechanism to migrate existing data to the new code?
  - simula67 13y ago
    If you have external state stored in say a database, I think you have to control your deployments to migrate the schema before the model objects can be deployed.
    - phamilton 13y ago
      So no. One of the beauties of Erlang's hot code loading is the ability to migrate internal application state. An example: I'm storing a lookup table going from key to value as a tuple {key, value}. With a change going out, I'm going to start doing {key, {value, other-value}}. In Erlang there's an easy pattern in place to do that. Just provide a function that goes from {key, value} to {key, {value, sensible-default}} and as part of the code reloading process that function will be run before the new code starts executing.
      - simula67 13y ago
        You may have misunderstood. A file can have multiple, say 'deployment units'. You can have a 'migrate deployment unit' which depend on the 'model deployment unit' which forces the app server to deploy the migrate unit first. You can have code there that migrates the tables and fill with default values before your model entities become available for use. From your comment this seems to be the same as what Erlang does for your example, unless you are claiming that Erlang's implementation is superior which could be the true, since I have never written any non-trivial programs in it.
        bitwalker 13y ago
        I think the difference here is that the migration of internal state happens the moment the module is swapped out in memory. Erlang doesn't have mutable state, so state is instead passed around within a module. When a fix needs to be deployed, the two versions (old and new) run simultaneously, any new processes immediately run the new code, old processes call a `code_change` function which takes the current state within that process, performs any transformations required for the new code, and then the old module code is swapped out for the new. That's not at all a perfect explanation of the process, but it's more or less useful enough to describe the difference between the two platforms.
- cmsd2 13y ago
  sure, any existing connections continue to be serviced by the old code, and new connections are picked up by the new code. if you need more elaborate lifecycle management of components as they are upgraded, then maybe OSGI could do that. erlang's approach really appeals to me though -- not only does the code get upgraded, but it's in cooperation with its message loop so you don't miss a step, and you also get a callback to migrate your actor state.
bitwalker 13y ago
> It's not. iIRC there's even a version of the JVM with code reloading. I've seen some info about the JVM implementation, but I could've sworn it wasn't able to preserve state while hot swapping. I haven't dug in to it though, so it's totally possible it handles that. > There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software. I'm a fan of Lisp in general, though my experience is limited to a tiny bit of Scheme and experimenting with Clojure. I'm glad to see that stuff like this is available in at least one of the Lisp variants! Question for you (if you program in Common Lisp): What are the main selling points? I spend a lot of time already with a couple of languages, Elixir/Erlang (for fun), C#/F# (for work) - what is it that would make me want to switch from one of those to Common Lisp? Thanks!
- logicchains 13y ago
  As someone who's written a bit of Common Lisp, I can say that the macros are a pretty awesome feature to use. Common Lisp has the advantage over Clojure and Scheme that the SBCL compiler is incredibly fast; it can come close to C#/F# on Mono for some tasks, in spite of Lisp being a dynamic language. It also compiles code and expands macros much faster. It's got more power than the other lisps, but it's also less elegant, so make of that what you will.
  - klibertp 13y ago
    > I can say that the macros are a pretty awesome feature to use Erlang has them in the form of parse transforms and Elixir has them too and makes them easy to use. Not as easy as sexp based languages, but almost.
    - sea6ear 13y ago
      There is also LFE (Lisp Flavored Erlang) [0] which gives lisp style macros and some other goodies to Erlang [0] http://lfe.github.io/ http://lfe.github.io/
- pron 13y ago
  BTW, a very Erlangish actor system, complete with hot code swapping (while preserving state), is available for Clojure and is called Pulsar (all open-source): http://docs.paralleluniverse.co/pulsar/ http://docs.paralleluniverse.co/pulsar/
shiro 13y ago
I use Common Lisp and Scheme at work, and have written and maintained several systems heavily relied on class redefinition and run-time code loading. They're great. OTOH, I feel Common Lisp's concurrent process handling is rather weak, and I envy Erlang for that (Not that CL can't do the trick, but usually what implementations offer is fairly low-level and I had to roll my own higher-level IPC.) For example, how well CL's class redefintion and instance reinitialization work with preemptive threads? It isn't obvious how each implementation address this issue by skimming docs, so I'm curious if anybody with direct experience. When I implemented CLOS/MOP style OO in my Scheme, I ended up using a global lock during class redefinition, for it need to modify lots of places. It's MT-safe now, but I'm still wary to run class redefinition on a system written in my Scheme, which is busy handling requests.

lispm 13y ago

One loads and replaces incrementally in Lisp.