4 ms·

Don’t use lru_cache. It stores pointers to the values, so if a mutable value is changed downstream it gets changed in the cache. That’s not proper caching.

SV_BubbleTime 5y ago
This seems like it would have been an easy fix on their end. Was the issue memory and storage?
- jakear 5y ago
  It’d be annoying for every type used in a cache to need to properly implement __deepcopy__(), and it’d pose significant performance impact, especially if the cached objects are large (which there’s a good chance they are, given you’ve felt the need to cache them rather than build them from scratch). Much better off using the same assignment semantics used throughout Python and let people choose to deepcopy() all the objects they read from the cache if they really really want to modify them later; it would even work as a simple decorator to stack onto the existing one for such cases. Aside: If we’re missing “deep coping” anything by default in Python, it’s definitely default parameters :) deep copying caches makes much less sense than that.
  - nerdponx 5y ago
    For anyone wondering what exactly the assignment semantics are, see: https://nedbatchelder.com/text/names1.html https://nedbatchelder.com/text/names1.html and https://www.youtube.com/watch?v=_AEJHKGk9ns https://www.youtube.com/watch?v=_AEJHKGk9ns
    - tialaramex 5y ago
      > An important fact about assignment: assignment never copies data. Is that really what's going on here? I'm in way too deep to be sure what's best for beginner programmers, but I feel like Python must surely optimise... sheep = 1 goats = sheep sheep = sheep + 10 ... by simply copying that 1 into goats, rather than tracking that goats is for now an alias to the same value as sheep and then updating that information when sheep changes on the next line. Now, if we imagine those numbers are way bigger (say 200 digits), Python still just works, whereas the low level languages I spend most time with will object because 200 digit integers don't fit in a machine word. You could imagine that copying isn't cheaper in that case, but I don't think I buy it. The 200 digit integer is a relatively small data structure, still probably cheaper to copy it than mess about with small objects which need garbage collecting.
      - hashmush 5y ago
        > by simply copying that 1 into goats I don't think it does that. You can try it yourself: >>> x = 123456 >>> y = x >>> id(x) 139871830996560 >>> id(y) 139871830996560
      - faho 5y ago
        >Is that really what's going on here? I'm in way too deep to be sure what's best for beginner programmers, but I feel like Python must surely optimise... For these particular numbers, CPython has one optimization I know of: Small integers (from -5 to 256) are pre-initialized and shared. See https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong This is generally invisible to the python programmer, but leads to https://wsvincent.com/python-wat-integer-cache/ https://wsvincent.com/python-wat-integer-cache/
        tialaramex 5y ago
        Thanks! That's a wild ride. On my system these "cached" integers seem to each be 32 byte objects, so, over 8kB of RAM is used by CPython to "cache" the integers -5 through 256 in this way. There's similar craziness over in String town. If I mint the exact same string a dozen times from a constant, those all have the same id, presumably Python has a similar "cache" of such constant strings. But if I assemble the same result string with concatenation, each of the identical strings has a different id (this is in 3.9) So, my model of what's going on in a Python program was completely wrong. But the simple pedagogic model in the article was also wrong, just not in a way that's going to trip up new Python programmers.
        faho 5y ago
        >presumably Python has a similar "cache" of such constant strings. Not really. You're hitting constant folding: https://arpitbhayani.me/blogs/constant-folding-python https://arpitbhayani.me/blogs/constant-folding-python This isn't a pre-made list of certain strings that should be cached, this is the compiler noticing that you mentioned the same constant a bunch of times. Also in general you would see a lot of things with the same id because python uses references all over the place. E.g. assignment never copies.
  - Quekid5 5y ago
    Might be interesting as an option for the @lru_cache decorator to be able to specify a function to call before handing a cached value to the user. Then you could just do @lru_cache(post=deepcopy, ...) to have a new (copied) instance in cases where that's required. Or do whatever else you needed. Maybe you happen to know some details about the returnee that would let you get away with copying less and could call my_copy instead. Something something power of function composition.
goodside 5y ago
This is how I would expect it to work. Caching mutable collections is a legitimate use case, distinct from caching the immutable values those collections contain. `lru_cache` does not wrap your cached values in container types on its own, so if you’re getting bit by mutable values it’s probably because what you cached was a mutable container (like `list`, `dict`, or `set`). If you use primitive values in the cache, or immutable containers (like `tuple`, `namedtuple`, or `frozenset`), you won’t hit this problem. If you were to manually cache values yourself using a `dict`, you’d see similar issues with mutability when storing `list` objects as values. It’s not a problem specific to `lru_cache`.
- pyuser583 5y ago
  Can you show me a caching framework/library written in something other than Python that exhibits this behavior? Edit: Can you show me a caching backend other than local memory (Redis, for example) that implements this behavior?
  - goodside 5y ago
    This isn’t special behavior that lru_cache “implements”, it’s intrinsic to how mutable contained objects work. Imagine you `lru_cache` a zero-argument function that sleeps for a long time, then creates a new temp file with random name and returns a writable handle for it. It should be no surprise that a cached version would have to behave differently: Only one random file would ever be made, and repeated calls to the cached function would return the same handle that was opened originally. If you didn’t expect this, you might complain that writes and seeks suddenly and mysteriously persist across what should be distinct handles, and maybe this `lru_cache` thing isn’t as harmless as advertised. But it’s not the cache’s fault that you cached an object that can mutate state outside the cache, like on a filesystem. Logically, the behavior remains the same if instead of actual file handles we simulate files in memory, or if we use a mutable list of strings representing lines of data. Generalizing, you could cache any mutable Python collection and see that in-place changes you make to it, much like data written to a file, will still be there when you read the cache the next time. The reason you don’t see “frameworks” for this is because tracking references to instantiated Python objects outside of the Python process is pointless — objects are garbage-collected and are not guaranteed to stay at the same memory location from one moment to the next. Further, if the lists themselves are small enough to fit in memory, surely there’s no need for out-of-memory scale to cache simple references to those objects.
Xavdidtheshadow 5y ago
> Don’t use lru_cache I disagree- it works wonders for expensive functions that return strings and ints. Maybe "don't use lru_cache with mutable values" would be more accurate. Thanks for noting that edge case though, I've never thought about that.
- matsemann 5y ago
  Kinda the same as using mutable values (list and dicts maybe being the most common pitfalls) as default arguments to functions. Shared between all function calls.
- pyuser583 5y ago
  This behavior might be justifiable - if it were included in the docs. If you want to reduce resource consumption, you could zip the results, or deduplicate the values using a separate dict. There’s nothing unPythonic about a function returning mutable values. A common use for caching is wrapping a call to a remote servicer. If the response is JSON, it’s likely to be mutable. Nothing weird or edgy about that.
zo1 5y ago
Not to be mean but you can't expect caching to fix a fundamental design issue in your code.
- pyuser583 5y ago
  Imagine if you created a ticket for a dev to create a cache. The backend is Redis, not local memory. The dev creates a PR. The cache is a decorator. It always returns a CachedValue object. The CachedValue object has an API for reading and writing the returned value. If you write to the CachedValue object it will update Redis with the new, altered value. It updated Redis not only for the value derived from that key, but all identical values. This behavior was not included in the ticket. It is not mentioned in the documentation. It only occurs when the value is mutable. What do you say to the developer? Job well done? Or what the f*ck is wrong with you? I’m a nice person so I’d say something nice. I’d ask what they were thinking. But if their reasons was “this is how caches works” I’d take a serious look at their resume and ask whether they might need some mentoring.

SV_BubbleTime 5y ago

This seems like it would have been an easy fix on their end. Was the issue memory and storage?

jakear 5y ago
It’d be annoying for every type used in a cache to need to properly implement __deepcopy__(), and it’d pose significant performance impact, especially if the cached objects are large (which there’s a good chance they are, given you’ve felt the need to cache them rather than build them from scratch). Much better off using the same assignment semantics used throughout Python and let people choose to deepcopy() all the objects they read from the cache if they really really want to modify them later; it would even work as a simple decorator to stack onto the existing one for such cases. Aside: If we’re missing “deep coping” anything by default in Python, it’s definitely default parameters :) deep copying caches makes much less sense than that.
- nerdponx 5y ago
  For anyone wondering what exactly the assignment semantics are, see: https://nedbatchelder.com/text/names1.html https://nedbatchelder.com/text/names1.html and https://www.youtube.com/watch?v=_AEJHKGk9ns https://www.youtube.com/watch?v=_AEJHKGk9ns
  - tialaramex 5y ago
    > An important fact about assignment: assignment never copies data. Is that really what's going on here? I'm in way too deep to be sure what's best for beginner programmers, but I feel like Python must surely optimise... sheep = 1 goats = sheep sheep = sheep + 10 ... by simply copying that 1 into goats, rather than tracking that goats is for now an alias to the same value as sheep and then updating that information when sheep changes on the next line. Now, if we imagine those numbers are way bigger (say 200 digits), Python still just works, whereas the low level languages I spend most time with will object because 200 digit integers don't fit in a machine word. You could imagine that copying isn't cheaper in that case, but I don't think I buy it. The 200 digit integer is a relatively small data structure, still probably cheaper to copy it than mess about with small objects which need garbage collecting.
    - hashmush 5y ago
      > by simply copying that 1 into goats I don't think it does that. You can try it yourself: >>> x = 123456 >>> y = x >>> id(x) 139871830996560 >>> id(y) 139871830996560
    - faho 5y ago
      >Is that really what's going on here? I'm in way too deep to be sure what's best for beginner programmers, but I feel like Python must surely optimise... For these particular numbers, CPython has one optimization I know of: Small integers (from -5 to 256) are pre-initialized and shared. See https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong This is generally invisible to the python programmer, but leads to https://wsvincent.com/python-wat-integer-cache/ https://wsvincent.com/python-wat-integer-cache/
      - tialaramex 5y ago
        Thanks! That's a wild ride. On my system these "cached" integers seem to each be 32 byte objects, so, over 8kB of RAM is used by CPython to "cache" the integers -5 through 256 in this way. There's similar craziness over in String town. If I mint the exact same string a dozen times from a constant, those all have the same id, presumably Python has a similar "cache" of such constant strings. But if I assemble the same result string with concatenation, each of the identical strings has a different id (this is in 3.9) So, my model of what's going on in a Python program was completely wrong. But the simple pedagogic model in the article was also wrong, just not in a way that's going to trip up new Python programmers.
        faho 5y ago
        >presumably Python has a similar "cache" of such constant strings. Not really. You're hitting constant folding: https://arpitbhayani.me/blogs/constant-folding-python https://arpitbhayani.me/blogs/constant-folding-python This isn't a pre-made list of certain strings that should be cached, this is the compiler noticing that you mentioned the same constant a bunch of times. Also in general you would see a lot of things with the same id because python uses references all over the place. E.g. assignment never copies.
- Quekid5 5y ago
  Might be interesting as an option for the @lru_cache decorator to be able to specify a function to call before handing a cached value to the user. Then you could just do @lru_cache(post=deepcopy, ...) to have a new (copied) instance in cases where that's required. Or do whatever else you needed. Maybe you happen to know some details about the returnee that would let you get away with copying less and could call my_copy instead. Something something power of function composition.

goodside 5y ago

This is how I would expect it to work. Caching mutable collections is a legitimate use case, distinct from caching the immutable values those collections contain. `lru_cache` does not wrap your cached values in container types on its own, so if you’re getting bit by mutable values it’s probably because what you cached was a mutable container (like `list`, `dict`, or `set`). If you use primitive values in the cache, or immutable containers (like `tuple`, `namedtuple`, or `frozenset`), you won’t hit this problem. If you were to manually cache values yourself using a `dict`, you’d see similar issues with mutability when storing `list` objects as values. It’s not a problem specific to `lru_cache`.

pyuser583 5y ago
Can you show me a caching framework/library written in something other than Python that exhibits this behavior? Edit: Can you show me a caching backend other than local memory (Redis, for example) that implements this behavior?
- goodside 5y ago
  This isn’t special behavior that lru_cache “implements”, it’s intrinsic to how mutable contained objects work. Imagine you `lru_cache` a zero-argument function that sleeps for a long time, then creates a new temp file with random name and returns a writable handle for it. It should be no surprise that a cached version would have to behave differently: Only one random file would ever be made, and repeated calls to the cached function would return the same handle that was opened originally. If you didn’t expect this, you might complain that writes and seeks suddenly and mysteriously persist across what should be distinct handles, and maybe this `lru_cache` thing isn’t as harmless as advertised. But it’s not the cache’s fault that you cached an object that can mutate state outside the cache, like on a filesystem. Logically, the behavior remains the same if instead of actual file handles we simulate files in memory, or if we use a mutable list of strings representing lines of data. Generalizing, you could cache any mutable Python collection and see that in-place changes you make to it, much like data written to a file, will still be there when you read the cache the next time. The reason you don’t see “frameworks” for this is because tracking references to instantiated Python objects outside of the Python process is pointless — objects are garbage-collected and are not guaranteed to stay at the same memory location from one moment to the next. Further, if the lists themselves are small enough to fit in memory, surely there’s no need for out-of-memory scale to cache simple references to those objects.

Xavdidtheshadow 5y ago

> Don’t use lru_cache I disagree- it works wonders for expensive functions that return strings and ints. Maybe "don't use lru_cache with mutable values" would be more accurate. Thanks for noting that edge case though, I've never thought about that.

matsemann 5y ago
Kinda the same as using mutable values (list and dicts maybe being the most common pitfalls) as default arguments to functions. Shared between all function calls.
pyuser583 5y ago
This behavior might be justifiable - if it were included in the docs. If you want to reduce resource consumption, you could zip the results, or deduplicate the values using a separate dict. There’s nothing unPythonic about a function returning mutable values. A common use for caching is wrapping a call to a remote servicer. If the response is JSON, it’s likely to be mutable. Nothing weird or edgy about that.

zo1 5y ago

Not to be mean but you can't expect caching to fix a fundamental design issue in your code.

pyuser583 5y ago
Imagine if you created a ticket for a dev to create a cache. The backend is Redis, not local memory. The dev creates a PR. The cache is a decorator. It always returns a CachedValue object. The CachedValue object has an API for reading and writing the returned value. If you write to the CachedValue object it will update Redis with the new, altered value. It updated Redis not only for the value derived from that key, but all identical values. This behavior was not included in the ticket. It is not mentioned in the documentation. It only occurs when the value is mutable. What do you say to the developer? Job well done? Or what the f*ck is wrong with you? I’m a nice person so I’d say something nice. I’d ask what they were thinking. But if their reasons was “this is how caches works” I’d take a serious look at their resume and ask whether they might need some mentoring.