8 ms·

Jurigged – Hot code reloading for Python

141 points by breuleux 6y ago

gloryless 6y ago
Is it pronounced "jew rigged?" I'd go with a different name, cause that's like the go-to neo nazi conspiracy theory, that jews rig everything.
- cstuder 6y ago
  No, it's not. https://en.wikipedia.org/wiki/Jury_rigging https://en.wikipedia.org/wiki/Jury_rigging
- coldtea 6y ago
  Jury rigging is totally different, and the connection to "jews rig everything" is a total stretch (not to mention the tradition phrasing of the conspiracy is that jews RUN everything, not that they rig it).
framecowbird 6y ago
This looks impressive, and equally a recipe for utter confusion. I've always thought that it's a brilliant feature of Python that you can edit code while it's running and not affect the process; unlike, say, bash scripts.
- musingsole 6y ago
  In a proper managed environment (whatever that may mean...), this could be useful as a means for continuous integration. Maybe not worth all the hassle, but maybe there's a usecase where this would be a beloved silver bullet.
  - ohples 6y ago
    Hot code reloading (ala Erlang) is often used in embedded devices for control code where restarting is very expensive. Mostly telco and networking devices.
    - dnautics 6y ago
      we use it in prod to instrument code to check up on some internal data, say, from the database, when some forensic analysis is needed
  - kikos 6y ago
    We use something similar in some of the web crawlers in my company, we dynamically load the module that selects the data from a web page so that if something (website and/or requirement) changes we can just swap the module without downtime.
edem 6y ago
Wait what? I thought Python was an interpreted language. Did I miss something?
- nonameiguess 6y ago
  Imported modules are compiled to bytecode and frozen on import, so changing the code in-place won't change what actually get run when the code is called unless you clear the bytecode cache and re-import. importlib provide a reload method for this purpose: https://docs.python.org/3.8/library/importlib.html#importlib.reload https://docs.python.org/3.8/library/importlib.html#importlib...
  - dfilppi 6y ago
    Interesting. I've done it with no issues on both 27 and 36. Just edit the code directly in site-packages.
- coldtea 6y ago
  So? Doesn't change the fact that once you've intepreted a function in some file, you've interpreted it and now it's in memory, and you're going to use the same as long as your program runs (even if you try to reload the module). (I mean, this is the default case without using something like this project, or other tricks).
- breuleux 6y ago
  Python doesn't re-read a function's source code from the filesystem every time it calls it, and that would be a bad idea in general since you'd inadvertently crash your running programs during development. This watches for changes on the filesystem and injects them into the running program as well as it can, basically keeping the current program state but with new behaviour. This is kind of a footgun and there are many changes you can make that will crash or ruin the process, but if you're aware of the limitations it can be pretty useful.
emj 6y ago
Looks more usable than the reloading library which does a similar thing, going to be interesting to use them together. I found Reloading when it was discussed in 2019: https://news.ycombinator.com/item?id=21268324 https://news.ycombinator.com/item?id=21268324
- julvo 6y ago
  Hey author of reloading here - cool to see this mentioned. Would love to know how reloading could be more usable for you. Since the post on HN in 2019, reloading can also be used as a decorator to load functions from source before execution. For some use cases that may work better than the reloading loop.
  - emj 6y ago
    I still use the version from 2019, and I usually do not have much code in tight loops, but rather need to reload different parts of a much bigger application so I tend to decorate the wrong parts of the code. I have no complaints about Reloading, just admiration using the AST module like that was genious.
sachamps 6y ago
what’s the vscode theme?
- breuleux 6y ago
  Noctis Obscuro
aidos 6y ago
If you use iPython you can enable auto reloading for all your sessions. It works really well - I've been using it for years. You need to know the limitations, but on the whole it give huge performance gains when I'm coding. To enable it add the following to your ipython_config.py: c = get_config() c.InteractiveShellApp.exec_lines = ['%autoreload 2'] c.InteractiveShellApp.extensions = ['autoreload']
- breuleux 6y ago
  You can also run `jurigged -vm IPython`, for what it's worth (or put `import jurigged; jurigged.watch()` in the config). There are a few edge cases where jurigged will work and not IPython's autoreload, and vice versa. Notably, jurigged updates decorated functions a bit more aggressively and only runs the code diff so it may keep some state that IPython won't.
- elcomet 6y ago
  I do this too and it's amazing. I also import common packages and classes that I use regularly, such as numpy, os, math, collections...
rosstex 6y ago
Make sure to turn off auto-save on your editor :)
adinisom 6y ago
Neat! Attempted to run on Windows and it almost works on my machine. - Watchdog's win32 implementation watches directories rather than files. If Jurigged is patched to watch the directory instead, this part works. - When the toplevel code is patched, the thread dispatching Watchdog events stops and won't pick up further filesystem changes. This can happen when changing code at toplevel or sometimes a file update gets picked up as Delete/Add instead which causes Jurigged to patch toplevel. Still trying to track down why that happens.
- adinisom 6y ago
  And the reason Jurigged breaks on changes to toplevel is it's re-running the top level in its own thread... including the infinite loop that is your application. This makes sense: it has to re-run the module's code. As long as you don't touch the module that kicks off your application it'll work fine. How can we make it work in the same module that runs the app? One approach that seems to work is to use the: if __name__ == "__main__" guard on your app code, then modify Jurigged to change __name__ to something else upon reloading. I'm not familiar enough with Python internals to tell if changing __name__ is kosher. Another possibility is to examine the stack and stop executing code when reaching the root of the current stack... although there's no guarantee the code has been running long enough to reach the program's infinite loop such as when the file was modified a split second after the program started.
  - breuleux 6y ago
    I mean, if you watch the example gif in the README, it shows the toplevel module being changed (you can see the if __name__ guard on the screen). It should work. Basically, Jurigged only re-runs the diff: if you don't change the `if __name__ ...`, it shouldn't run it again. If you change a function in the toplevel module (outside of the guard) it should figure that out by comparing the old AST to the new one and only re-execute that specific function. Can I see the code you're using for your tests? There might be some other issue.
    - adinisom 6y ago
jfrunyon 6y ago
It seems like this is just watching a file for changes and then calling reload on it? What am I missing?
- Too 6y ago
  See the How it works and the following Comparision sections of the readme. It's a bit more advanced than simply reloading. Reimporting a module would likely work most of the times and be a simpler solution but i think it does not affect already created instances of classes, which this solution seem to do.
jrmann100 6y ago
I've been using watchgod [0], which seems to work similarly—not sure if jurigged also monitors all included modules, or is cross-platform-friendly. Anyone know the similarities/differences? [0] https://github.com/samuelcolvin/watchgod https://github.com/samuelcolvin/watchgod
- breuleux 6y ago
  I believe watchgod basically restarts the process whenever changes happen. Jurigged does not restart anything: it computes a diff between the old and new code, executes that diff, fudges the code object pointers inside the existing functions, and prays that doesn't break it. The advantage is that it keeps the program state, so you can edit a program that has a slow startup or salvage a process that has been running for hours. It's especially nice with a repl, if you run jurigged with no arguments or jurigged -m IPython, your changes will be automatically included in your interactive session. The disadvantage is that it keeps the program state, so if you make changes to init functions for example, it'll most likely just crash. So it depends on your use case. If your process is lightweight and/or you don't care about keeping the current program state, watchgod is probably better for you (I believe there's a similar alternative called hupper that you might want to check out).

141 points by breuleux 6y ago

Is it pronounced "jew rigged?" I'd go with a different name, cause that's like the go-to neo nazi conspiracy theory, that jews rig everything.

cstuder 6y ago
No, it's not. https://en.wikipedia.org/wiki/Jury_rigging https://en.wikipedia.org/wiki/Jury_rigging
coldtea 6y ago
Jury rigging is totally different, and the connection to "jews rig everything" is a total stretch (not to mention the tradition phrasing of the conspiracy is that jews RUN everything, not that they rig it).

framecowbird 6y ago

This looks impressive, and equally a recipe for utter confusion. I've always thought that it's a brilliant feature of Python that you can edit code while it's running and not affect the process; unlike, say, bash scripts.

musingsole 6y ago
In a proper managed environment (whatever that may mean...), this could be useful as a means for continuous integration. Maybe not worth all the hassle, but maybe there's a usecase where this would be a beloved silver bullet.
- ohples 6y ago
  Hot code reloading (ala Erlang) is often used in embedded devices for control code where restarting is very expensive. Mostly telco and networking devices.
  - dnautics 6y ago
    we use it in prod to instrument code to check up on some internal data, say, from the database, when some forensic analysis is needed
- kikos 6y ago
  We use something similar in some of the web crawlers in my company, we dynamically load the module that selects the data from a web page so that if something (website and/or requirement) changes we can just swap the module without downtime.

edem 6y ago

Wait what? I thought Python was an interpreted language. Did I miss something?

nonameiguess 6y ago
Imported modules are compiled to bytecode and frozen on import, so changing the code in-place won't change what actually get run when the code is called unless you clear the bytecode cache and re-import. importlib provide a reload method for this purpose: https://docs.python.org/3.8/library/importlib.html#importlib.reload https://docs.python.org/3.8/library/importlib.html#importlib...
- dfilppi 6y ago
  Interesting. I've done it with no issues on both 27 and 36. Just edit the code directly in site-packages.
coldtea 6y ago
So? Doesn't change the fact that once you've intepreted a function in some file, you've interpreted it and now it's in memory, and you're going to use the same as long as your program runs (even if you try to reload the module). (I mean, this is the default case without using something like this project, or other tricks).
breuleux 6y ago
Python doesn't re-read a function's source code from the filesystem every time it calls it, and that would be a bad idea in general since you'd inadvertently crash your running programs during development. This watches for changes on the filesystem and injects them into the running program as well as it can, basically keeping the current program state but with new behaviour. This is kind of a footgun and there are many changes you can make that will crash or ruin the process, but if you're aware of the limitations it can be pretty useful.

emj 6y ago

Looks more usable than the reloading library which does a similar thing, going to be interesting to use them together. I found Reloading when it was discussed in 2019: https://news.ycombinator.com/item?id=21268324 https://news.ycombinator.com/item?id=21268324

julvo 6y ago
Hey author of reloading here - cool to see this mentioned. Would love to know how reloading could be more usable for you. Since the post on HN in 2019, reloading can also be used as a decorator to load functions from source before execution. For some use cases that may work better than the reloading loop.
- emj 6y ago
  I still use the version from 2019, and I usually do not have much code in tight loops, but rather need to reload different parts of a much bigger application so I tend to decorate the wrong parts of the code. I have no complaints about Reloading, just admiration using the AST module like that was genious.

sachamps 6y ago

what’s the vscode theme?

breuleux 6y ago
Noctis Obscuro

aidos 6y ago

If you use iPython you can enable auto reloading for all your sessions. It works really well - I've been using it for years. You need to know the limitations, but on the whole it give huge performance gains when I'm coding. To enable it add the following to your ipython_config.py: c = get_config() c.InteractiveShellApp.exec_lines = ['%autoreload 2'] c.InteractiveShellApp.extensions = ['autoreload']

breuleux 6y ago
You can also run `jurigged -vm IPython`, for what it's worth (or put `import jurigged; jurigged.watch()` in the config). There are a few edge cases where jurigged will work and not IPython's autoreload, and vice versa. Notably, jurigged updates decorated functions a bit more aggressively and only runs the code diff so it may keep some state that IPython won't.
elcomet 6y ago
I do this too and it's amazing. I also import common packages and classes that I use regularly, such as numpy, os, math, collections...

rosstex 6y ago

Make sure to turn off auto-save on your editor :)

adinisom 6y ago

Neat! Attempted to run on Windows and it almost works on my machine. - Watchdog's win32 implementation watches directories rather than files. If Jurigged is patched to watch the directory instead, this part works. - When the toplevel code is patched, the thread dispatching Watchdog events stops and won't pick up further filesystem changes. This can happen when changing code at toplevel or sometimes a file update gets picked up as Delete/Add instead which causes Jurigged to patch toplevel. Still trying to track down why that happens.

adinisom 6y ago
And the reason Jurigged breaks on changes to toplevel is it's re-running the top level in its own thread... including the infinite loop that is your application. This makes sense: it has to re-run the module's code. As long as you don't touch the module that kicks off your application it'll work fine. How can we make it work in the same module that runs the app? One approach that seems to work is to use the: if __name__ == "__main__" guard on your app code, then modify Jurigged to change __name__ to something else upon reloading. I'm not familiar enough with Python internals to tell if changing __name__ is kosher. Another possibility is to examine the stack and stop executing code when reaching the root of the current stack... although there's no guarantee the code has been running long enough to reach the program's infinite loop such as when the file was modified a split second after the program started.
- breuleux 6y ago
  I mean, if you watch the example gif in the README, it shows the toplevel module being changed (you can see the if __name__ guard on the screen). It should work. Basically, Jurigged only re-runs the diff: if you don't change the `if __name__ ...`, it shouldn't run it again. If you change a function in the toplevel module (outside of the guard) it should figure that out by comparing the old AST to the new one and only re-execute that specific function. Can I see the code you're using for your tests? There might be some other issue.
  - adinisom 6y ago

jfrunyon 6y ago

It seems like this is just watching a file for changes and then calling reload on it? What am I missing?

Too 6y ago
See the How it works and the following Comparision sections of the readme. It's a bit more advanced than simply reloading. Reimporting a module would likely work most of the times and be a simpler solution but i think it does not affect already created instances of classes, which this solution seem to do.

jrmann100 6y ago

I've been using watchgod [0], which seems to work similarly—not sure if jurigged also monitors all included modules, or is cross-platform-friendly. Anyone know the similarities/differences? [0] https://github.com/samuelcolvin/watchgod https://github.com/samuelcolvin/watchgod

breuleux 6y ago
I believe watchgod basically restarts the process whenever changes happen. Jurigged does not restart anything: it computes a diff between the old and new code, executes that diff, fudges the code object pointers inside the existing functions, and prays that doesn't break it. The advantage is that it keeps the program state, so you can edit a program that has a slow startup or salvage a process that has been running for hours. It's especially nice with a repl, if you run jurigged with no arguments or jurigged -m IPython, your changes will be automatically included in your interactive session. The disadvantage is that it keeps the program state, so if you make changes to init functions for example, it'll most likely just crash. So it depends on your use case. If your process is lightweight and/or you don't care about keeping the current program state, watchgod is probably better for you (I believe there's a similar alternative called hupper that you might want to check out).