5 ms·

Why is encoding speed irrelevant? I'm currently in the process of scanning and archiving lots of photographs as high-resolution 16-bit PNG's and encoding speed

Why is encoding speed irrelevant? I'm currently in the process of scanning and archiving lots of photographs as high-resolution 16-bit PNG's and encoding speed is definitely a significant influence on how long this process takes.

somishere 4y ago
Can I ask why you chose PNG for your archive format?
- pmoriarty 4y ago
  It's lossless, offers compression, works with 16-bit images, and is well supported in a lot of software on all sorts of platforms.
  - somishere 4y ago
    Interesting, thanks. I'd heard of PNG being used for this before and thought people had lost the plot. For context I started a neg scanning project a few years ago, ~150 rolls on an epson v850 (flatbed), and stuck with the default compressed tif after some surface research. There is little chance I would consider another processing step .. unless it was batched on a server somewhere! Very painstaking process, which has been on indefinite hold for a small while now (waiting for the AI to catch up).
drblast 4y ago
Because it's a one-time cost and you can cheaply throw more CPU power at the problem, or just wait and then you're done. If you're doing this at such a scale that encoding speed matters (like if you're Instagram) the major concern would likely be cost, which you're going to save in bandwidth every time an image is viewed.
- pmoriarty 4y ago
  "you can cheaply throw more CPU power at the problem, or just wait and then you're done" How am I going to "throw more CPU power at the problem" when I'm using a scanner hooked up to my laptop? Should I buy myself a new, more powerful laptop? That doesn't sound cheap. As for "just" waiting... my time is valuable, and I have better things to do with it than wait... and, unfortunately, the process of scanning and encoding needs a lot of babysitting from me, so I can't just dump a bunch of photos in my scanner and come back when it's all over if I want even half-decent results, not to mention archival quality images. In this real-world scenario, a quicker encoding speed would save me a lot of valuable time, and there's no getting around it short of hiring a skilled and trusted person to do it for me.
  - tomovo 4y ago
    Save uncompressed to a big cheap harddrive, batch compress to PNG overnight?
  - Nullinker 4y ago
    Like Tomovo above already said; your workflow would benefit from using an intermediate format that is faster to encode. You can then automate the conversion to the output format to occur unattended at a more convenient time. The use of intermediate formats is a well proven technique in video editing where encoding real time to AVC/HEVC at high quality is not possible. Codecs like ProRes are used that are much easier to encode to at the expense of storage space.
throw10920 4y ago
You missed the "almost". The vast majority of use-cases and users barely care about encoding performance. Your use is in the tiny minority. That isn't to say that you don't have a use-case, just that because of how rarely encoding speed matters, it truly is "almost irrelevant" for 99.99% of users 99.99% of the time. -------------- It definitely would be nice if JPEG-XL also had good encoding performance, though. There's an experimental fast encoder in development currently[1], although before that lands, it would be nice if your scanning tool supported a workflow somewhat like "scan in parallel with image encoding, present encoded images to user, and allow them to asynchronously go back and re-scan/encode/review pages that came out badly". [1] https://github.com/libjxl/libjxl/tree/main/experimental/fast_lossless https://github.com/libjxl/libjxl/tree/main/experimental/fast...
- klodolph 4y ago
  It’s common enough to create an image on the fly and send it to a single user to be viewed once.
Dylan16807 4y ago
How many megabytes per second is your scanner feeding you? I'm very surprised that encoding a PNG is slower than scanning.
- pmoriarty 4y ago
  It's not slower than scanning, but I still spend a significant amount of time waiting for the maximum compression PNG encoding to complete.
  - Dylan16807 4y ago
    If it's not encoding in the background so it can be done within a second or so of the data finishing, then it's just programmed wrong, it's not an encoding speed problem.
  - JyrkiAlakuijala 4y ago
    Don't use maximum compression in this use case. Maximum (Zopfli-based?) is for those who will serve the image hundred times over the Internet while users are waiting for it. It is not cost effective or practical for storage only use.
Melatonic 4y ago
Not a bad choice - all that really matters in those cases anyways is what your scanner is capable of

somishere 4y ago

Can I ask why you chose PNG for your archive format?

pmoriarty 4y ago
It's lossless, offers compression, works with 16-bit images, and is well supported in a lot of software on all sorts of platforms.
- somishere 4y ago
  Interesting, thanks. I'd heard of PNG being used for this before and thought people had lost the plot. For context I started a neg scanning project a few years ago, ~150 rolls on an epson v850 (flatbed), and stuck with the default compressed tif after some surface research. There is little chance I would consider another processing step .. unless it was batched on a server somewhere! Very painstaking process, which has been on indefinite hold for a small while now (waiting for the AI to catch up).

drblast 4y ago

Because it's a one-time cost and you can cheaply throw more CPU power at the problem, or just wait and then you're done. If you're doing this at such a scale that encoding speed matters (like if you're Instagram) the major concern would likely be cost, which you're going to save in bandwidth every time an image is viewed.

pmoriarty 4y ago
"you can cheaply throw more CPU power at the problem, or just wait and then you're done" How am I going to "throw more CPU power at the problem" when I'm using a scanner hooked up to my laptop? Should I buy myself a new, more powerful laptop? That doesn't sound cheap. As for "just" waiting... my time is valuable, and I have better things to do with it than wait... and, unfortunately, the process of scanning and encoding needs a lot of babysitting from me, so I can't just dump a bunch of photos in my scanner and come back when it's all over if I want even half-decent results, not to mention archival quality images. In this real-world scenario, a quicker encoding speed would save me a lot of valuable time, and there's no getting around it short of hiring a skilled and trusted person to do it for me.
- tomovo 4y ago
  Save uncompressed to a big cheap harddrive, batch compress to PNG overnight?
- Nullinker 4y ago
  Like Tomovo above already said; your workflow would benefit from using an intermediate format that is faster to encode. You can then automate the conversion to the output format to occur unattended at a more convenient time. The use of intermediate formats is a well proven technique in video editing where encoding real time to AVC/HEVC at high quality is not possible. Codecs like ProRes are used that are much easier to encode to at the expense of storage space.

throw10920 4y ago

You missed the "almost". The vast majority of use-cases and users barely care about encoding performance. Your use is in the tiny minority. That isn't to say that you don't have a use-case, just that because of how rarely encoding speed matters, it truly is "almost irrelevant" for 99.99% of users 99.99% of the time. -------------- It definitely would be nice if JPEG-XL also had good encoding performance, though. There's an experimental fast encoder in development currently[1], although before that lands, it would be nice if your scanning tool supported a workflow somewhat like "scan in parallel with image encoding, present encoded images to user, and allow them to asynchronously go back and re-scan/encode/review pages that came out badly". [1] https://github.com/libjxl/libjxl/tree/main/experimental/fast_lossless https://github.com/libjxl/libjxl/tree/main/experimental/fast...

klodolph 4y ago
It’s common enough to create an image on the fly and send it to a single user to be viewed once.

Dylan16807 4y ago

How many megabytes per second is your scanner feeding you? I'm very surprised that encoding a PNG is slower than scanning.

pmoriarty 4y ago
It's not slower than scanning, but I still spend a significant amount of time waiting for the maximum compression PNG encoding to complete.
- Dylan16807 4y ago
  If it's not encoding in the background so it can be done within a second or so of the data finishing, then it's just programmed wrong, it's not an encoding speed problem.
- JyrkiAlakuijala 4y ago
  Don't use maximum compression in this use case. Maximum (Zopfli-based?) is for those who will serve the image hundred times over the Internet while users are waiting for it. It is not cost effective or practical for storage only use.

Melatonic 4y ago

Not a bad choice - all that really matters in those cases anyways is what your scanner is capable of