3 ms·
It may be better to just start with managing caching with code. In the standard library, the @functools.cache and @functools.cached_property function and metho
by westurner 3y ago
It may be better to just start with managing caching with code.
In the standard library, the @functools.cache and @functools.cached_property function and method decorators do LRU caching in RAM only. https://docs.python.org/3/library/functools.html https://docs.python.org/3/library/functools.html
Dask docs > "Automatic Opportunistic Caching": https://docs.dask.org/en/stable/caching.html#automatic-opportunistic-caching https://docs.dask.org/en/stable/caching.html#automatic-oppor... ; dask.cache.Cache(size_bytes:int) ... "Cache tasks, not expressions"
Pickles are not a safe way to deserialize data; there is not a data only pickling protocol.
So caching arbitrary cell objects (or e.g. stack frames) to disk, as pickles at least, creates risk of code injection if the serialized data contains executable code objects.
Similarly, the file permissions on e.g. rr traces. https://en.wikipedia.org/wiki/Rr_(debugging) https://en.wikipedia.org/wiki/Rr_(debugging)
Dataclasses in the standard library helps with object serialization, but not with caching cell outputs containing code objects.
Apache Arrow and Parquet also require schema for efficient serialization.
LRU: Least-Recently Used
MRU: Most-Recently Used; most frequently accessed
Out-of-order execution in notebooks may or may not have wasted cycles of human time and CPU time. If the prompt numbers aren't sequential, what you ran in a notebook is not necessarily what others will get when running that computation graph in order; and so it's best practice to "Restart and Run All" to test the actual control flow before committing or pushing.
There are ways to run and test notebooks in order on or before git commit (and compare their output with the output from the previous run) like software with tests.
- ylow 3y agoThe issue comes when cells take many minutes or even hours to run (intentionally or not). The ideal is indeed sequential, and this helps me a lot with maintaining the sequential ordering as it simplifies and speeds up the “restart and run all” process.
- westurner 3y agoOh I understand the issue. E.g. dask-labextension does not implicitly do dask.cache for you. How are the objects serialized, and are code objects serialized to files on disk, what is the permission umask on such files, and what directory (/var/cache) should they be selinux-labeled in (when it is running code from not memcache instead of the source) because if you can write to those cache files, you control the execution flow of the notebook (which is already unreproducibly out-of-ordered without consideration)
- IanCal 3y agoI agree about being explicit with this, though I will warn about using the python functools caching, because it changes the logic of your program. Because it doesn't copy data, any return that's mutable is risky. It's also not obvious this happens, and even less obvious if you're not keenly aware of this kind of issue being likely.
- westurner 3y agoDoes any cache imply invariance, which DbC like icontract can be used to check for really only within one program execution?
- IanCal 3y agoPlenty of caches are built that have this property. Streamlit for example offers two different caching helpers, one is faster but only safe if you don't mutate the return value and one is slower but safer as it makes a copy. Diskcache, my go-to, appears to be safe with regards to this too.
- westurner 3y agoRegardless of cache implementation, if the notebook input cells are run in a different sequence like 0,1,2,2,2,3 the output with caching there is not necessarily same as without caching and not necessarily the same as 0,1,2,2,3 ; caching alone does not achieve functional idempotency, in fact that's many more partners that can diverge when the interest is parametric reproducibility (and notebooks are already a strange abstraction for computation graphs complex enough to justify e.g. dask.cache.Cache() for, and there are reasons to not always enable dask.cache.Cache() globally (if the argument were whether auto-caching is safe enough to be the default behavior in notebooks))
- theamk 3y agoCan we stop bringing up "pickles are not a safe way to deserialize data" for things that do not matter, like caching on local machine? Yes, pickle can be exploited by someone with ~/ write access. But you know what else someone with that access do? Append to .bashrc, add to desktop startup script, create pth file in local python dist, create usercustomize, append code into any locally installed libraries, add code to config of many editors... I've seen so much effort wasted because people rejected pickle for being insecure in contexts where it doesn't matter at all.
- westurner 3y agoIf there is a privilege escalation or just an OS exec() as the app process user in a container with no interactive shell, no desktop, root-owned site-packages configured at image build time, and/or a pex zipapp or similar with no ~/.local (`type -a python; python -m site`) then it can write() to the on disk files or a database with schema and field length limits that doesn't store code.
- theamk 3y agoI am having trouble parsing your reply, looks like some parts are missing. Can you rephrase? We were talking about jupyter notebooks, which are typically used for exploratory work, and thus are executed either on local desktop under default user account, or in containers with pretty open access controls. My point was that under those circumstances, it is OK to assume any cache data is trusted. Your reply somehow talks about secured containers and privilege escalation but I cannot figure out how is it connected to running Jupyter on local laptop, nor the threat model you are worried about.
- westurner 3y agoI replied to the stated security risks with specific controls. Should the app write additional .py files next to itself and exec those? That's what caching code blocks as pickles on disk would get you there.