6 ms·

Resizing images in Rust, now with EXIF orientation support

78 points by ingve 1y ago

vic1102 1y ago
I ran into the same issue when working with image processing in Go using libvips. By default, libvips ignores EXIF orientation too, so I had to explicitly read and apply the orientation tag before doing any resize or crop. Otherwise users ended up with upside-down processed images from photos. Glad to see this is getting better in Rust.
- jcupitt 1y ago
  Hello, libvips author here, the JPEG loader has a flag to do this for you. You can write: VipsImage *image = vips_image_new_from_file("something.jpg", "autorotate", TRUE, NULL); and it'll flip it upright for you and remove any EXIF orientation tag. It isn't the default since it has (obviously) a large memory and cpu cost. For simple resize and crop it's much better to use the exif tag to adjust the resize and crop parameters (the vipsthumbnail command does this).
  - vic1102 1y ago
    Thanks so much for the clarification and for the great work on libvips! In my Go project I’m using the Golang library github.com/davidbyttow/govips/v2. Images are loaded via `vips.NewImageFromFile()`, which internally calls `vips_image_new_from_file()`. However, `NewImageFromFile()` doesn't support any flags or options beyond the image path, so for JPEGs I manually call `AutoRotate()` before resizing, which wraps `vips_autorot()` — and that works well. Thanks again!
    - jcupitt 1y ago
      Ah no problem! I'm glad it's useful. `govips` was a pretty early binding and wasn't really done the libvips way. It doesn't expose all the operations or options, it's mostly done by hand, and there are a number of leaks and misfeatures. It's been replaced by `vipsgen`: https://github.com/cshum/vipsgen https://github.com/cshum/vipsgen Which is an automatically generated 100% binding. It should have the complete API, it should be very stable and leak-free, and it should be simple to maintain. `autorot` is pretty expensive. You'll see much better performance if you flip x and y in your crop instead.
      - vic1102 1y ago
        Thanks a lot! I really appreciate the detailed reply and the recommendation! I wasn’t aware of `vipsgen`, but it looks great — full coverage and auto-generated bindings are exactly what I’ll be missing in `govips` if the project grows (I hope it will). I’ll definitely give it a try and see how it works. Also, great tip on avoiding `autorot` by flipping crop coordinates — I hadn’t thought of that! Thanks again for your help!
esperent 1y ago
I'm sure there's very good reasoning but I've never been able to convince myself of it - why does EXIF transform metadata exist? If you're taking a selfie, why doesn't the camera itself perform a simple rotation or mirror of the actual image data rather than using EXIF?
- tinco 1y ago
  I think it does this after the photo has already been stored as JPEG. I think users would expect the image be persisted the moment you capture it, so any transformations after that are better done in metadata. When the image has been saved as JPEG the original image data is discarded so you can't transform without deteriorating the image.
  - dividuum 1y ago
    Pretty sure you can rotate JPEG images lossless. But it’s still simpler to just modify metadata.
    - perching_aix 1y ago
      A quick search suggests to me that it's only a lossless process if the image dimensions are a clean integer multiple of 8 or 16 (as the blocks can be 8x8, 8x16, 16x8, or 16x16), otherwise the edges must be reencoded. Never written a JPEG codec though, so happy to be proven wrong.
      - addaon 1y ago
        This is true, and others have mentioned it, but I think people are underestimating just how universal sensor dimensions being a multiple of 16 is — I really can’t think of any exceptions.
- Doohickey-d 1y ago
  One reason I can think of: when you rotate an image in the windows photo viewer (and probably lots of other apps), it stores that in the EXIF metadata. So the rotation is lossless, by not having to re-compress the JPEG. I could also imagine that the earliest digital cameras wouldn't have had the processing power, or RAM to store the entire image in memory after reading it from the sensor, in order to do such a rotation. Hence EXIF rotation as a cheap alternative.
jcynix 1y ago
You can transfer EXIF information, including the rotation info, with either exiftool or ImageMagick into a thumbnail, to let other software know how to display it correctly. If you want to rotate a jpeg yourself (which is easily done with "magic in.jpg rotate 90 out.jpg" and, of course, other software) but lossless, there's https://jpegclub.org/jpegtran/ https://jpegclub.org/jpegtran/ which accomplishes it. When generating rotated thumbnails (without caring for "lossless") the best strategy in my experience is to first rotate the large original image and then generate the thumbnail afterwards, because image resizing will most often hide any rotation artifacts anyways.
- kevincox 1y ago
  But it seems that if you are re-encoding the image anyways it is best to just generate it in the "correct" orientation and skip this metadata. Emitting the metadata seems mostly useful if you are trying to avoid re-encoding the image. Or is there a benefit to generating the thumbnail in the original rotation then preserving the metadata?
  - jcynix 1y ago
    Sure, generating the "correct" format would be a good decision. When you change pixel orientation, you'll need to either update or clear the EXIF orientation info to reflect the new pixel data. Rotation metadata might be helpful in (larger) thumbnails when images are displayed on mobile devices and responsive web pages. Metadata is easy to transfer, at least when scripting with Exiftool or ImageMagick.
- radicality 1y ago
  Is jpeg the only format which can be “rotated losslessly” in this way, ie without exif metadata (and excluding any obvious formats like bitmaps or whatever can be trivially rotated) ?
  - jcynix 1y ago
    Both PNG and Webp exist in lossless formats which can be easily rotated without loss of quality. Jpeg is different, because its internal structure consists of 8x8 blocks (which, BTW, can be rotated lossless if both sides of the image are multiples of 8, iirc. For photos all these assumptions are rather theoretical IMO, because modern cameras create images in sizes which almost always have to be reduced in size for viewing, unless one wants to pixel-peep.
ginko 1y ago
Wouldn't it make more sense if `image::open()` automatically applied the orientation by default?
- jcupitt 1y ago
  Applying rotation can have a large performance penalty (both cpu and memory), so you want to be able to avoid it if possible.
- hn92726819 1y ago
  Probably for a higher-level crate, (like the one the author wrote), but it adds computational complexity, and you don't want that by default with a low level crate like `image`. For example, if you wanted to blur or grayscale an image, do you really want to rotate it? When you re-save it, you'll have to remove the exif rotation metadata too.
pornel 1y ago
The rexif crate supports editing, so you can apply rotation when resizing, and then remove the rotation tag from the EXIF data. Keeping EXIF isn't necessary for small thumbnails, but could be desirable for larger versions of the image.
ddahlen 1y ago
One good reason to keep the raw data and orientation separate is hardware calibrations. I see some discussion here along the lines of "why not just flip the data before saving it", and one counter-argument is that you should store the flip metadata anyways, as it important to trace which hardware pixel corresponds to which stored pixel. I realize this information is not vital for everyone, but it is super useful in fields where you need to characterize the hardware performance at the pixel level (for example astronomy).
mentalgear 1y ago
a wasm port available?
- timeinput 1y ago
  If it's all in rust you can maybe probably compile it to wasm with out much effort. Kinda depends what libraries they used / how they factored their code etc.

78 points by ingve 1y ago

I ran into the same issue when working with image processing in Go using libvips. By default, libvips ignores EXIF orientation too, so I had to explicitly read and apply the orientation tag before doing any resize or crop. Otherwise users ended up with upside-down processed images from photos. Glad to see this is getting better in Rust.

jcupitt 1y ago
Hello, libvips author here, the JPEG loader has a flag to do this for you. You can write: VipsImage *image = vips_image_new_from_file("something.jpg", "autorotate", TRUE, NULL); and it'll flip it upright for you and remove any EXIF orientation tag. It isn't the default since it has (obviously) a large memory and cpu cost. For simple resize and crop it's much better to use the exif tag to adjust the resize and crop parameters (the vipsthumbnail command does this).
- vic1102 1y ago
  Thanks so much for the clarification and for the great work on libvips! In my Go project I’m using the Golang library github.com/davidbyttow/govips/v2. Images are loaded via `vips.NewImageFromFile()`, which internally calls `vips_image_new_from_file()`. However, `NewImageFromFile()` doesn't support any flags or options beyond the image path, so for JPEGs I manually call `AutoRotate()` before resizing, which wraps `vips_autorot()` — and that works well. Thanks again!
  - jcupitt 1y ago
    Ah no problem! I'm glad it's useful. `govips` was a pretty early binding and wasn't really done the libvips way. It doesn't expose all the operations or options, it's mostly done by hand, and there are a number of leaks and misfeatures. It's been replaced by `vipsgen`: https://github.com/cshum/vipsgen https://github.com/cshum/vipsgen Which is an automatically generated 100% binding. It should have the complete API, it should be very stable and leak-free, and it should be simple to maintain. `autorot` is pretty expensive. You'll see much better performance if you flip x and y in your crop instead.
    - vic1102 1y ago
      Thanks a lot! I really appreciate the detailed reply and the recommendation! I wasn’t aware of `vipsgen`, but it looks great — full coverage and auto-generated bindings are exactly what I’ll be missing in `govips` if the project grows (I hope it will). I’ll definitely give it a try and see how it works. Also, great tip on avoiding `autorot` by flipping crop coordinates — I hadn’t thought of that! Thanks again for your help!

esperent 1y ago

I'm sure there's very good reasoning but I've never been able to convince myself of it - why does EXIF transform metadata exist? If you're taking a selfie, why doesn't the camera itself perform a simple rotation or mirror of the actual image data rather than using EXIF?

tinco 1y ago
I think it does this after the photo has already been stored as JPEG. I think users would expect the image be persisted the moment you capture it, so any transformations after that are better done in metadata. When the image has been saved as JPEG the original image data is discarded so you can't transform without deteriorating the image.
- dividuum 1y ago
  Pretty sure you can rotate JPEG images lossless. But it’s still simpler to just modify metadata.
  - perching_aix 1y ago
    A quick search suggests to me that it's only a lossless process if the image dimensions are a clean integer multiple of 8 or 16 (as the blocks can be 8x8, 8x16, 16x8, or 16x16), otherwise the edges must be reencoded. Never written a JPEG codec though, so happy to be proven wrong.
    - addaon 1y ago
      This is true, and others have mentioned it, but I think people are underestimating just how universal sensor dimensions being a multiple of 16 is — I really can’t think of any exceptions.
Doohickey-d 1y ago
One reason I can think of: when you rotate an image in the windows photo viewer (and probably lots of other apps), it stores that in the EXIF metadata. So the rotation is lossless, by not having to re-compress the JPEG. I could also imagine that the earliest digital cameras wouldn't have had the processing power, or RAM to store the entire image in memory after reading it from the sensor, in order to do such a rotation. Hence EXIF rotation as a cheap alternative.

jcynix 1y ago

You can transfer EXIF information, including the rotation info, with either exiftool or ImageMagick into a thumbnail, to let other software know how to display it correctly. If you want to rotate a jpeg yourself (which is easily done with "magic in.jpg rotate 90 out.jpg" and, of course, other software) but lossless, there's https://jpegclub.org/jpegtran/ https://jpegclub.org/jpegtran/ which accomplishes it. When generating rotated thumbnails (without caring for "lossless") the best strategy in my experience is to first rotate the large original image and then generate the thumbnail afterwards, because image resizing will most often hide any rotation artifacts anyways.

kevincox 1y ago
But it seems that if you are re-encoding the image anyways it is best to just generate it in the "correct" orientation and skip this metadata. Emitting the metadata seems mostly useful if you are trying to avoid re-encoding the image. Or is there a benefit to generating the thumbnail in the original rotation then preserving the metadata?
- jcynix 1y ago
  Sure, generating the "correct" format would be a good decision. When you change pixel orientation, you'll need to either update or clear the EXIF orientation info to reflect the new pixel data. Rotation metadata might be helpful in (larger) thumbnails when images are displayed on mobile devices and responsive web pages. Metadata is easy to transfer, at least when scripting with Exiftool or ImageMagick.
radicality 1y ago
Is jpeg the only format which can be “rotated losslessly” in this way, ie without exif metadata (and excluding any obvious formats like bitmaps or whatever can be trivially rotated) ?
- jcynix 1y ago
  Both PNG and Webp exist in lossless formats which can be easily rotated without loss of quality. Jpeg is different, because its internal structure consists of 8x8 blocks (which, BTW, can be rotated lossless if both sides of the image are multiples of 8, iirc. For photos all these assumptions are rather theoretical IMO, because modern cameras create images in sizes which almost always have to be reduced in size for viewing, unless one wants to pixel-peep.

ginko 1y ago

Wouldn't it make more sense if `image::open()` automatically applied the orientation by default?

jcupitt 1y ago
Applying rotation can have a large performance penalty (both cpu and memory), so you want to be able to avoid it if possible.
hn92726819 1y ago
Probably for a higher-level crate, (like the one the author wrote), but it adds computational complexity, and you don't want that by default with a low level crate like `image`. For example, if you wanted to blur or grayscale an image, do you really want to rotate it? When you re-save it, you'll have to remove the exif rotation metadata too.

pornel 1y ago

The rexif crate supports editing, so you can apply rotation when resizing, and then remove the rotation tag from the EXIF data. Keeping EXIF isn't necessary for small thumbnails, but could be desirable for larger versions of the image.

ddahlen 1y ago

One good reason to keep the raw data and orientation separate is hardware calibrations. I see some discussion here along the lines of "why not just flip the data before saving it", and one counter-argument is that you should store the flip metadata anyways, as it important to trace which hardware pixel corresponds to which stored pixel. I realize this information is not vital for everyone, but it is super useful in fields where you need to characterize the hardware performance at the pixel level (for example astronomy).

mentalgear 1y ago

a wasm port available?

timeinput 1y ago
If it's all in rust you can maybe probably compile it to wasm with out much effort. Kinda depends what libraries they used / how they factored their code etc.