3 ms·
Note that the autoreload features can be very tricky to use safely with Python. For example, at least in some previous versions, Caffe and TensorFlow make inco
by mlthoughts2018 8y ago
Note that the autoreload features can be very tricky to use safely with Python.
For example, at least in some previous versions, Caffe and TensorFlow make incompatible assumptions about the ability to claim all available GPU memory. So there can be situations where you first import Caffe, then later import TensorFlow with restrictions on its GPU policy. If you naively re-import the Caffe code, it can evict TensorFlow from whatever GPUs it had reclaimed, and coming up with a group of settings that reliably prevent this, across possibly different machine where the notebook will be run, is very tricky.
This once led to a huge time sink because someone on my team created a mistaken GitHub issue claiming our TensorFlow model had a bug (since the notebook was producing an error). We spent all this time trying to reproduce it and figure out why it wasn't working, and eventually realized it was because of this hidden auto-reload setting on his specific IPython setup that caused Caffe to evict TensorFlow just for his specific usage pattern, resulting in strange errors because the TensorFlow model was no longer loaded in GPU memory.
There can be other problems too, like auto-reloading modules that have large start-up times (say if they load a very large model into memory). Sometimes you want to re-run a cell without auto-reload, even if you still want selective auto-reload functionality in other parts.
- cbcoutinho 8y agoThanks for explaining the downsides related to using this feature. Like all good config options, there are tradeoffs. Luckily I haven't been bitten by it yet, but I'll remember that if I run into issues