4 ms·

I got it working well in a large project but I basically designed auto reloading into deep parts of the framework. It's not something you can safely bolt on, an

I got it working well in a large project but I basically designed auto reloading into deep parts of the framework. It's not something you can safely bolt on, and I'm honestly not sure I recommend the sanity impact of trying to build it properly (e.g. ran into a bunch of memory leaking even once I had it working safely). But there are some kinds of programs you can't really restart on every change. The biggest caveat of these kinds of single file reloaders is they completely fall apart when you import between your modules. For example, if `a` does `from b import obj` and `b` does `from c import obj`, and you modify `c.py`, `a` and `b` will both have a stale reference to obj (which might be a primitive or even interned type like a string or an int, so it's not like you can mutate it as a solution).

orf 6y ago
I updated my comment to say “impossible in the general case” which is more accurate, but I’d be interested in knowing more about that project - how big was it, and how did handle some of the tricky bits? Like, module A creates a class/function that is consumed by module B and C. If you change module A, how did you know to also reload B and C (recursively)?
- lunixbochs 6y ago
  This is for a scriptable desktop accessibility program. It is fairly big, is very actively used by end users, and many users have >80 modules and edit them _constantly_ during daily use. So the reloading system isn't just commonly used, it's a critical path of using the app. (I also have not gotten any complaints related to the module reloading in ~months, when I last fixed a bug in the long dependency chain tracking, and reloading hasn't caused any major problems in general). I slightly break the mold of conventional Python imports - script folders don't have a single entry point, the entire user script directory tree is recursively imported (in alphabetical order) and autoreloaded on change. Anytime I call into a module it is marked as the active module in its thread (a sort of call stack but at the module level and only for certain kinds of events). "Call into" generally means the module is being imported, or I'm calling a callback that was registered by the module. Anytime code creates a global object (such as opening a window, registering new voice commands, or registering a callback for some kind of event), cleanup for the object is registered on the active module. Additionally, all imports are tracked, so I have a graph of which modules import which other modules. The import tracking allows me to reload dependent module chains (even long chains, trees, or import loops) at the correct time. The object tracking allows me to gracefully replace a module without it requiring any explicit cleanup. This graph extends to more than just Python files. There are special functions for opening or reading files that mark those files as a module's dependencies as well, so you can modify something like a CSV and the script that uses it will get reloaded (and any module chains that depend on that script). This all working well is tightly coupled to how the framework behaves. Scripts won't generally spawn threads or spin in long loops, they'll mostly register for event callbacks. For example, instead of a script spawning a thread or sleeping if it wants to run code later, there's a cron system that works something like javascript setTimeout/setInterval. Instead of storing something in a global, you can put it in the persistent storage layer if you want it to survive reloads (and restarts). You can definitely do a few things that aren't reloadable, like spawning your own thread, but most things you want to do end up being reloadable, or have an equivalent or alternative API that is reloadable. There are a few other nice touches and APIs that help glue all of this together. Generally it means you can write your scripts as short, flat, mostly declarative Python (with perhaps a few event handlers and setup code that don't need to go in a __main__ guard) and not worry about common cases for reloading. (There are also subsystems specifically watching out for edge cases, such as a user script running an infinite loop) Honestly the most annoying part of all of this (that I do generally handle now) was reliably monitoring file changes when users put file or directory symlinks at arbitrary places in the tree (which it turns out a lot of users like to do).

orf 6y ago

I updated my comment to say “impossible in the general case” which is more accurate, but I’d be interested in knowing more about that project - how big was it, and how did handle some of the tricky bits? Like, module A creates a class/function that is consumed by module B and C. If you change module A, how did you know to also reload B and C (recursively)?

lunixbochs 6y ago
This is for a scriptable desktop accessibility program. It is fairly big, is very actively used by end users, and many users have >80 modules and edit them _constantly_ during daily use. So the reloading system isn't just commonly used, it's a critical path of using the app. (I also have not gotten any complaints related to the module reloading in ~months, when I last fixed a bug in the long dependency chain tracking, and reloading hasn't caused any major problems in general). I slightly break the mold of conventional Python imports - script folders don't have a single entry point, the entire user script directory tree is recursively imported (in alphabetical order) and autoreloaded on change. Anytime I call into a module it is marked as the active module in its thread (a sort of call stack but at the module level and only for certain kinds of events). "Call into" generally means the module is being imported, or I'm calling a callback that was registered by the module. Anytime code creates a global object (such as opening a window, registering new voice commands, or registering a callback for some kind of event), cleanup for the object is registered on the active module. Additionally, all imports are tracked, so I have a graph of which modules import which other modules. The import tracking allows me to reload dependent module chains (even long chains, trees, or import loops) at the correct time. The object tracking allows me to gracefully replace a module without it requiring any explicit cleanup. This graph extends to more than just Python files. There are special functions for opening or reading files that mark those files as a module's dependencies as well, so you can modify something like a CSV and the script that uses it will get reloaded (and any module chains that depend on that script). This all working well is tightly coupled to how the framework behaves. Scripts won't generally spawn threads or spin in long loops, they'll mostly register for event callbacks. For example, instead of a script spawning a thread or sleeping if it wants to run code later, there's a cron system that works something like javascript setTimeout/setInterval. Instead of storing something in a global, you can put it in the persistent storage layer if you want it to survive reloads (and restarts). You can definitely do a few things that aren't reloadable, like spawning your own thread, but most things you want to do end up being reloadable, or have an equivalent or alternative API that is reloadable. There are a few other nice touches and APIs that help glue all of this together. Generally it means you can write your scripts as short, flat, mostly declarative Python (with perhaps a few event handlers and setup code that don't need to go in a __main__ guard) and not worry about common cases for reloading. (There are also subsystems specifically watching out for edge cases, such as a user script running an infinite loop) Honestly the most annoying part of all of this (that I do generally handle now) was reliably monitoring file changes when users put file or directory symlinks at arbitrary places in the tree (which it turns out a lot of users like to do).