3 ms·

Compute a simple checksum over first megabyte or so of the image file (for O(1) complexity) and store it per known file in memory. Store thumbnails using that c

Compute a simple checksum over first megabyte or so of the image file (for O(1) complexity) and store it per known file in memory. Store thumbnails using that checksum as key in global cache. Don't purge because methods discussed in this thread don't.

Someone 6y ago
So, to show thumbnails for a directory with 100 photos, you’re going to open and close 100 files, and read 100 1MB chunks of data? That probably is doable on modern local hardware, but on network disks, it might still be slow. Make that an IMHO not unreasonably large 500 photos and your local disk might be slow, too. Add a virus scanner, and 100 photos might already be annoyingly slow. ⇒ I would try to find a way to store away the checksum somewhere in the file (say in an extended attribute, as a (modification timestamp, checksum) pair)
- loa_in_ 6y ago
  Normally to show thumbnails you need to parse the whole image anyway. Unless that is the format doesn't have downscaled data already (progressive jpeg/PNG)
  - Someone 6y ago
    If you don’t cache thumbnails, that’s true, but that’s too slow, so you cache thumbnails. If you do that, you only have to read the whole image the first time. Once you created the thumbnails, you only need to read them, and they’re a lot smaller, and can be put in a single file. The proposed approach reads a significant part of each file to determine whether the cached thumbnail is still current. That doesn’t give a huge improvement over not caching thumbnails at all (factor of 5 in bytes read/network traffic, as a rough guess, likely less in disk seeks)
    - loa_in_ 6y ago
      You're right. I got a bit ahead of myself it seems.
      - lovelyviking 6y ago
        Thank you for thoughts though. Was interesting to read both of you.
    - lovelyviking 6y ago
      It's a dilemma of memory vs computation. It depends on what we expect/prefer: FM that takes a lot of memory or FM that use computation resources when required only. I tend to think that in general expectation from FM is to be very small in memory demands and take valuable computational resources only if it is absolutely necessary.
      - loa_in_ 6y ago
        My approach would work amazingly in ipfs where hash of contents is an intrinsic property of the file system

Someone 6y ago

So, to show thumbnails for a directory with 100 photos, you’re going to open and close 100 files, and read 100 1MB chunks of data? That probably is doable on modern local hardware, but on network disks, it might still be slow. Make that an IMHO not unreasonably large 500 photos and your local disk might be slow, too. Add a virus scanner, and 100 photos might already be annoyingly slow. ⇒ I would try to find a way to store away the checksum somewhere in the file (say in an extended attribute, as a (modification timestamp, checksum) pair)

loa_in_ 6y ago
Normally to show thumbnails you need to parse the whole image anyway. Unless that is the format doesn't have downscaled data already (progressive jpeg/PNG)
- Someone 6y ago
  If you don’t cache thumbnails, that’s true, but that’s too slow, so you cache thumbnails. If you do that, you only have to read the whole image the first time. Once you created the thumbnails, you only need to read them, and they’re a lot smaller, and can be put in a single file. The proposed approach reads a significant part of each file to determine whether the cached thumbnail is still current. That doesn’t give a huge improvement over not caching thumbnails at all (factor of 5 in bytes read/network traffic, as a rough guess, likely less in disk seeks)
  - loa_in_ 6y ago
    You're right. I got a bit ahead of myself it seems.
    - lovelyviking 6y ago
      Thank you for thoughts though. Was interesting to read both of you.
  - lovelyviking 6y ago
    It's a dilemma of memory vs computation. It depends on what we expect/prefer: FM that takes a lot of memory or FM that use computation resources when required only. I tend to think that in general expectation from FM is to be very small in memory demands and take valuable computational resources only if it is absolutely necessary.
    - loa_in_ 6y ago
      My approach would work amazingly in ipfs where hash of contents is an intrinsic property of the file system