4 ms·
What a great example of how a series of reasonable decisions can add up to something unreasonable when you scale up the inputs. This line is my favorite: “there
by dahart 4y ago
What a great example of how a series of reasonable decisions can add up to something unreasonable when you scale up the inputs. This line is my favorite: “there was an open task for a few years to address it (T73412), and so I did it.”
This situation reminds me a little bit of trying to work with very large images in ImageMagick, like 60k x 60k resolution. A simple resize was taking 6 hours on my mac 10 years ago, due to IM trying to allocate the whole image at once and then swapping non-stop. And then I discovered that the Graphics Magick fork did streamed resizing, and did the task in a couple of minutes. It’s a small and relatively easy change, but someone had to prioritize handling large inputs first. This is one reason software is hard for me; every time I need to process large inputs I wish I had written a streaming mechanism, but every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Maybe it’s the right thing to do to avoid over-engineering, but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
- xcambar 4y ago> every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Absolutely reasonable decision making. Do what you need at the time you write the code, or at best, be slightly ahead of the curve. Don't forecast features you do not need. Humans are unreasonable about success of their own projects. >Maybe it’s the right thing to do to avoid over-engineering It is the only thing to do. You do not know the future use cases. > [...] but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed. If your software is subpar with regards to your needs, congrats, it is successful software! Building the perfect software from the beginning is unreasonable because you might just never release or even use it. Build imperfect things, and be reasonably frustrated BH them so that it gives you the energy to improve them!
- acomjean 4y ago>Building the perfect software from the beginning is unreasonable because you might just never release or even use it. I built a software library for an external device from specification. We eventually got the device to test and it worked when plugged in, but not everything was right.. I was early in my career and disappointed. My boss said "If it works perfectly the first time, you spent too much time on it.".
- xcambar 4y agoThat was a good boss
- tecleandor 4y agoOh yes. I worked for year in medical imaging and we suffered greatly when dealing with sanned pathology slides, that easily have a long side of 30k to 200k pixels, and we had to deal with big volumes of them, go figure. But then we found libvips (that's not only a library but a cli tool too) and it was like night and day in big images. Like 10x faster and 1/10 memory use. I really loved it! https://www.libvips.org/ https://www.libvips.org/
- kaladin-jasnah 4y agoI can second libvips, I had to work on a project that needed to resize GIFs and vips was probably 10 times faster than the ImageMagick or Pillow. I think the only thing that compared to libvips was ffmpeg.
- tecleandor 4y agoIn a quick test I've done resizing to half size a 30k x 26k pixel painting scan [1], vips is half the time and almost a tenth of memory use. Of course, if you're using that much memory (with imagemagick), once you begin to parallelize and do batch conversion, you fill your memory super quickly, you begin to swap, and then the conversion times go through the roof. Image Magick time convert original.jpg -resize 50% im.jpg convert original.jpg -resize 50% im.jpg 15.99s user 4.78s system 131% cpu 15.770 total avg shared (code): 0 KB avg unshared (data/stack): 0 KB total (sum): 0 KB max memory: 1909 MB page faults from disk: 485098 other page faults: 7059 vips time vips resize original.jpg vips.jpg 0.5 vips resize original.jpg vips.jpg 0.5 13.38s user 0.15s system 195% cpu 6.910 total avg shared (code): 0 KB avg unshared (data/stack): 0 KB total (sum): 0 KB max memory: 255 MB page faults from disk: 1 other page faults: 60319 1: https://commons.wikimedia.org/wiki/File:Hans_Holbein_der_J%C3%BCngere_-_Der_Kaufmann_Georg_Gisze_-_Google_Art_Project.jpg https://commons.wikimedia.org/wiki/File:Hans_Holbein_der_J%C...
- tecleandor 4y agoSeems like imagemagick was coredumping and my measurement wasn't correct. So I raised imagemagick memory limits and I made sure both were using the same jpeg quality and downsampling filter and: ImageMagick time convert original.jpg -filter Lanczos -set option:filter:lobes 3 -resize 50% -quality 75% im.jpg 91.43s user 3.65s system 424% cpu 22.396 total avg shared (code): 0 KB avg unshared (data/stack): 0 KB total (sum): 0 KB max memory: 10553 MB page faults from disk: 0 other page faults: 2993653 vips time vips resize original.jpg "vips.jpg[Q=75]" 0.5 --kernel lanczos3 13.00s user 0.15s system 194% cpu 6.771 total avg shared (code): 0 KB avg unshared (data/stack): 0 KB total (sum): 0 KB max memory: 255 MB page faults from disk: 11 other page faults: 60951 So Imagemagick does: 40x the memory, almost 4x the time, almost 9x the CPU time. Versions and computer: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz (8 cores) 16GB RAM Debian 11 Kernel 5.18.8-xanmod1-x64v2 (software not rebuilt for x64v2 instructions) ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org vips-8.12.2-Mon Feb 28 21:28:00 UTC 2022 Options for zsh time: TIMEFMT='%U user %S system %P cpu %*E total'$'\n'\ 'avg shared (code): %X KB'$'\n'\ 'avg unshared (data/stack): %D KB'$'\n'\ 'total (sum): %K KB'$'\n'\ 'max memory: %M 'MB''$'\n'\ 'page faults from disk: %F'$'\n'\ 'other page faults: %R'