3 ms·

The inputs and outputs of anything that is slow enough to care about caching is so large in an ml context that caching is unreasonable. The one exception would

The inputs and outputs of anything that is slow enough to care about caching is so large in an ml context that caching is unreasonable. The one exception would be people recomputing entire notebooks over and over, but again caching in some generic way that can't tell if underlying data changed or not is going to break all kinds of stuff. Don't use notebooks for real work guys, I know you have a PHD but that's not an excuse to refuse to learn software engineering. We have lots of ways to define data dependencies and conditionally rebuild just the parts that need rebuilt. Look at any build system.

skadamat 3y ago
Notebooks aren't necessarily for production but are a really great way to explore data & models and move at the speed of thought. It's for prototyping, collaboration, and feedback. I sense that most of the frustration teams have with notebooks is when they try to ship notebooks, which is likely a mistake (in many but not all cases) as you're pointing out!
amakelov 3y ago
I see two concerns here: - inputs/outputs being high volume: the inputs/outputs that are large are often also things that don't change over the course of a project (e.g. a dataset or a model). So you don't really need to cache the object itself, just a (typically short string) immutable reference to it. As long as the object can be looked up at runtime, everything's fine; - detecting changes in data: content hashing is the general way in which you can tell if a result changed; using `joblib.dump` and then hashing the resulting string provides a good starting implementation, though certainly there are some corner cases to be aware of. Both of these approaches are available/used in mandala (https://github.com/amakelov/mandala https://github.com/amakelov/mandala; disclosure: I'm the author), which uses content hashing to tell when data (or even code/code dependencies) have changed, and gives you a generic caching decorator for functions which can then look up large objects by reference; this is the way I used it for e.g. my mechanistic interpretability work, which is often of the form one big model + lots of analyses producing tiny artifacts based on it.

skadamat 3y ago

Notebooks aren't necessarily for production but are a really great way to explore data & models and move at the speed of thought. It's for prototyping, collaboration, and feedback. I sense that most of the frustration teams have with notebooks is when they try to ship notebooks, which is likely a mistake (in many but not all cases) as you're pointing out!

amakelov 3y ago

I see two concerns here: - inputs/outputs being high volume: the inputs/outputs that are large are often also things that don't change over the course of a project (e.g. a dataset or a model). So you don't really need to cache the object itself, just a (typically short string) immutable reference to it. As long as the object can be looked up at runtime, everything's fine; - detecting changes in data: content hashing is the general way in which you can tell if a result changed; using `joblib.dump` and then hashing the resulting string provides a good starting implementation, though certainly there are some corner cases to be aware of. Both of these approaches are available/used in mandala (https://github.com/amakelov/mandala https://github.com/amakelov/mandala; disclosure: I'm the author), which uses content hashing to tell when data (or even code/code dependencies) have changed, and gives you a generic caching decorator for functions which can then look up large objects by reference; this is the way I used it for e.g. my mechanistic interpretability work, which is often of the form one big model + lots of analyses producing tiny artifacts based on it.