14 ms·
A dumb reason computer vision apps aren’t working: Exif Orientation
- layoutIfNeeded 7y agoWhy don’t you train your classifiers to be orientation-agnostic? My brain certainly had no problem recognizing the geese on the sideways image...
- 6gvONxR4sf7o 7y agoWhy make it harder than it has to be?
- nightfly 7y agoA goose is still a goose, even if not viewed from the "normal" orientation.
- variaga 7y agoI'd say you're not making the problem "harder" - rather, requiring that object detection be orietation-agnostic makes the problem exactly as hard as the problem actually is. Allowing the network to train only on images with a known, fixed, (correct) orientation makes the object detection too _easy_, so the network results will likely fail if you feed it any real-world data. i.e. you should be training your image application with skew/stretch/shrink/rotation/color-pallete-shift/contrast-adjust/noise-addition/etc. applied to all training images if you want it to be useful for anything other than getting a high top-N score on the validation set.
- klyrs 7y agoYeah... this article really looks like blame-shifting to me. I'm imagining a future wherein we have bipedal murderbots, but we're still training AI with "properly oriented" images... a bot trips over a rock, and starts blasting away at mid-identified objects.
- TeMPOraL 7y agoWell, even humans find it more difficult to process upside-down faces or objects. There's power in assumptions that are correct 99% of the time. Regardless, the article isn't really shifting blame, in so much as explaining what's happening in the real world, with the real tools. The tools don't care about EXIF. Consumer software uses EXIF to abstract over reality. A lot of people playing with ML don't know about either.
- princeb 7y agoyou may not know what face it is, but you do know that it is a face right? it's like saying you know someone is in front of you if he were standing in up, in your clear view, but the moment he's lying on a couch in centre of your vision you can't tell with certainty if the thing on the couch is a human being?
- TeMPOraL 7y agoI could tell, but I'm under impression that my 4 months old kid still has problems with that, or at least had when she was 2 months old. I think this is closer to the performance we should expect of current neural models - a few months old child, not an adult. NNs may be good at doing stuff similar to what some of our sight apparatus does, but they're missing additional layers of "higher-level" processing humans employ. That, and a whole lot of training data.
- gwern 7y agoData augmentation can have unwanted consequences. For example, horizontal or vertical flipping, what could be more harmless? You can still recognize stuff when it's upside-down, can't you? It's a great data augmentation... Unless, of course, your dataset happens to involve text or symbols in any way, such as the numbers '6' and '9', or left/right arrows in traffic signs, or such obscure letters as 'd' and 'b'.
- thaumasiotes 7y ago> Unless, of course, your dataset happens to involve text or symbols in any way, such as the numbers '6' and '9', or left/right arrows in traffic signs, or such obscure letters as 'd' and 'b'. If your dataset consists of nothing but isolated 'd's and 'p's in unknown orientation, you won't be able to classify them correctly because that is an impossible task. But it would be more common for your dataset to consist of text rather than isolated letters, and in the context of surrounding text it's easy to determine the correct orientation, and therefore to discriminate 'd' from 'p'.
- gwern 7y agoSo it's not a problem, except when it is. Good to know. Incidentally, how does that work for mirroring, when all that surrounding text gets mirrored too? (Consider the real example of the lolcats generated by Nvidia's StyleGAN, where the text captions are completely wrong, and will always be wrong, because it looks like Cyrillic - due to the horizontal dataflipping StyleGAN has enabled by default...)
- bacon_waffle 7y agoInterestingly, your brain actually isn't orientation agnostic: http://nancysbraintalks.mit.edu/video/what-you-can-learn-studying-behavior http://nancysbraintalks.mit.edu/video/what-you-can-learn-stu...
- pgeorgi 7y agoWhat should the classifier identify on https://www.illusionsindex.org/i/duck-rabbit https://www.illusionsindex.org/i/duck-rabbit? [edit: https://www.reddit.com/r/dataisbeautiful/comments/aydqig/is_it_a_duck_or_a_rabbit_for_google_cloud_vision/ https://www.reddit.com/r/dataisbeautiful/comments/aydqig/is_... tried it out]
- jeroenhd 7y agoI'm not sure how good your algorithm is if it can only work in a specific orientation. A slightly tilted image can already cause problems for such an algorithm and many people have a hard time getting a picture with only a few degrees of rotation. It would probably be better to deal with this in an elegant way, e.g. set up the algorithm to work regardless of orientation. This seems like a (mostly) solved problem: https://d4nst.github.io/2017/01/12/image-orientation/ https://d4nst.github.io/2017/01/12/image-orientation/
- H8crilA 7y agoForcing rotation in the training phase would be a good way to regularize the network. But of course that will lower the accuracy.
- gugagore 7y agoYou might not want rotation invariance in your algorithm. Our own perceptual system is not rotation-invariant, most obviously for faces in this illusion: https://en.wikipedia.org/wiki/Thatcher_effect https://en.wikipedia.org/wiki/Thatcher_effect
- cookingrobot 7y agoThat’s not an option if you’re trying to detect left-arrows vs right-arrows, on street signs for ex.
- jeroenhd 7y agoDepends. Based on the linked article, the picture can be completely upside down if it was taken in landscape mode. The article I linked is capable of rotating it upright regardless. After rotating to the right orientation, left and right are suddenly perfectly workable. Of course using exif data for such rotations is easier, but a tilted picture of a tilted sign can create a lot of tilt that human vision copes with fine but an orientation dependent network cannot.
- enlyth 7y agoI ran into this headache while writing a web app. I made an image upload widget that provided a preview, and when users selected the "take a picture" option on their phones, I showed the preview with a blob link and CSS background-image property. The images were showing sideways on some phones. I looked at the EXIF data of those photos and of course Orientation: 90 showed up. It was easy to fix on the backend when processing the images but I struggled to do it in a performant way on the front-end. One solution involved reading the EXIF data with JS and rotating with canvas and .toBlob(), but proved too slow for large 10MB photos as it blocks the main UI thread. One thing I thought of is just reading the orientation and the using CSS transforms to rotate the preview, but I never got around to trying it.
- Waterluvian 7y agoTry web workers! I've been doing all kinds of crazy expensive spatial data validation in many parallel threads. It's amazing. I think off screen canvas in workers is gaining browser support recently.
- machinelearning 7y agoAlso known as "programmer error".
- lugg 7y agoAka PBKAC
- acdha 7y agoWhat value do you think this comment contributed? The entire article clearly describes it as a commonly mistake and is trying to raise awareness so people stop repeating it.
- machinelearning 7y agoI think it contributed the warning that the title is clickbait. Its very clear based on the discussion in the thread that people who've just read the title are discussion computer vision and machine learning strategies to solve this problem when the article describes an almost elementary programming issue.
- secraetomani 7y agoSurely someone can create a neural network that reorients the image, even if the Exif orientation is wrong (JPEG lets you can do this without re-encoding). Sounds like it should be a very simple problem for the vast majority of "regular" images.
- machinelearning 7y agoOr how about just use the damn exif information to orient it correctly as the article outlines? The actual article is far less interesting than what the title seems to imply. While the title suggests a problem with computer vision. This is more of a programmer logic error.
- secraetomani 7y agoI've encountered plenty of pictures with incorrect EXIF orientation, mostly caused by holding the phone at a weird angle. For example, if I hold the phone parallel with the ground (aimed at the floor), what is the correct EXIF orientation?
- magicalhippo 7y agoThere are many times my phone detects I'm holding it horizontally while taking a picture, but I'm actually trying to take a vertical picture, and vice versa. For those images, the exif information is technically correct, but actually wrong.
- machinelearning 7y agoIf exif information is incorrect, that's just programmer error at the firmware/library level.
- SiVal 7y agoEXIF info is often missing or wrong. Photos that have been passed around, going through various services, apps, scanners, screenshots, etc. along the way, frequently have their EXIF info stripped (e.g., for privacy when exported or uploaded), written incorrectly originally (e.g., you scan an old B&W print, put it in sideways to fit in the scanner, and the scanner invisibly includes EXIF data assuming it was correctly oriented), or rewritten incorrectly (e.g., stripped EXIF is replaced with default orientation).
- codesushi42 7y agoNope. It is not working because of your own dumb reasoning of not augmenting the training data to produce different orientations. Dummies.
- brlewis 7y agoAs an alternative to the python code in the article, there's this command-line tool: https://linux.die.net/man/1/exiftran https://linux.die.net/man/1/exiftran
- 6gvONxR4sf7o 7y agoOne of my pet peeves in ML/stats/data science is people who hardly look at their data. Unless there are privacy reasons not to, then you really need to look at some data. You'll learn so much more from looking at a few hundred samples than you will from different metrics. You'll get a feel for how complex the problem is, or whether something simple will do. Check your assumptions. You might even realize that your images are sideways.
- anoncareer0212 7y agoThe article explains why it's practically impossible to see that (check your assumptions! :P )
- m12k 7y agoWell, after you've read in your data, read them back out and display them on the screen.
- toxik 7y agoThis actually is generally true for any computer programming debugging. You need to look at the data as it appears to your program, to catch bugs in your preprocessing pipelines.
- BubRoss 7y agoI've tried working like that and it makes a massive difference. To be able to visualize all the intermediate stages is valuable because they are basically never right until you are able to isolate and refine them individually. https://github.com/LiveAsynchronousVisualizedArchitecture/lava https://github.com/LiveAsynchronousVisualizedArchitecture/la...
- ken 7y agoIt’s easy to print out the dimensions of the image when it’s loaded. I always log the size (pixels, rows, etc) of any data I load.
- NeoBasilisk 7y agoWould it really be too much of a burden for phones to just save photos at the correct orientation now? I understand the hardware limitations that were present in 2007, but surely these can't still be a factor?
- ryandrake 7y agoIs there any good reason to save a photo in an orientation that does not match the orientation that the device is being held in? Shouldn’t up be up? If that results in a NxM photo, save it NxM. If it results in a MxN photo save it that way! The only edge case is a camera pointed straight up or straight down. Or a camera in space.
- TeMPOraL 7y agoCorrect orientation is relative to the photographer, not to the ground. Most of the times the photographer is in the usual, vertical orientation but sometimes you really want to take a shot at an angle, and you have to fight with your phone to do it correctly. I really dislike these kinds of "convenience" optimizations.
- uoaei 7y agoComing back to ancient software design principles: > The user is always right Let them have a setting. It's really that easy.
- anchpop 7y ago> Or a camera in space. Or freefall
- kllrnohj 7y agoWould it really be too much of a burden for python image libraries to decode jpegs to the correct orientation? I understand lack of knowledge about exif in the 2000s, but surely these days that can't still be a problem?
- Kenji 7y agoI hold the - apparently controversial - view that if your ML algorithm cannot detect a clean picture of a goose when you rotate it by 90°, your ML algorithm is garbage. How is this of any use when it's so easily fooled?
- octorian 7y agoI should add that this very problem also exists with video. What makes it worse is that "smartphone" video apps started regularly using video orientation metadata years before desktop video apps even seemed to acknowledge that it was a thing.
- BEEdwards 7y agoIt's the simplest way to do it and has so few downsides that it's not going to change until someone invents a googly eye camera that rotates with the phone.
- deleted 7y ago[deleted]
- rkagerer 7y agoEven outside of machine vision this issue has caused me all sorts of headaches. Especially when switching between tools that honor or ignore the tag. "But the tricky part is that your camera doesn’t actually rotate the image data inside the file that it saves to disk." Cameraphones are so powerful these days. I don't understand why the app can't simply include a behavior setting to always flip the original pixels.
- not2b 7y agoThe phone is accurately recording the image, as well as the orientation. The bug happens when the EXIF information is stripped. Sure, phone apps could add an option to physically rotate the image, as a bug workaround, but it's not surprising that they don't do this.
- uoaei 7y agoIt is surprising, because it seems like such an obvious fix to the problem of stripped EXIF data. Couldn't be that hard to implement a user setting which tells the app to rotate the file itself before saving. This is a major oversight for the companies who develop camera apps. The major ones even have whole teams dedicated to that single app.
- cookingrobot 7y agoIt would be trivial to rotate a bitmap with no loss of quality, but you can’t rotate an already compressed jpg without changing the image itself, which reduces quality.
- mrob 7y agoAssuming the image size is a multiple of the block size (8x8, 16x8, or 16x16 depending on the chroma subsampling), lossless rotation of JPEGs is possible, e.g. with jpegtran: http://jpegclub.org/jpegtran/ http://jpegclub.org/jpegtran/
- 7y ago
- m3kw9 7y agoTrain a model that recognizes L,R, or U side way images to reorient them all. Run it once.
- LeoPanthera 7y agoImageMagick will "apply" the exif rotation to the image data for you. convert input -auto-orient output Or in place: mogrify -auto-orient *
- jonshariat 7y agoIsn't it better to train the model that way though? In fact it might be good to load every image in different orientations to add to the training data.
- mark-r 7y agoThere's a big photography site I use that treats the EXIF inconsistently. I rotate the image in my editor to work on it, then save and upload. In some contexts it looks OK, but in other contexts the site rotates the image again and it's wrong. I don't want to strip the EXIF because it has interesting information such as the camera model and lens, and the exposure settings. My editor doesn't correct the EXIF rotation setting, so I have no choice but to use a utility to strip that single value from the file before I upload it.
- mceachen 7y agoGetting image and video orientation correct, which should be trivial, is decidedly not. In writing PhotoStructure [1], I discovered 1) Firefox respects the Orientation tag using a standard [2] CSS tag, but Chrome doesn't [3] 2) That videos sometimes use "Rotation", not "Orientation" (and Rotation is encoded in degrees, not the crazy EXIF rotation enum). Oh, and some manufacturers use "CameraOrientation" instead, just for the LOLs [4] 3) That the embedded images in JPEGs, RAW, and videos sometimes are rotated correctly, and sometimes they are not, depending on the Make and Model of the camera that produced the file. If Orientation or Rotation say anything other than "not rotated," you can't trust what's in that bag of bits. [1] https://blog.photostructure.com/introducing-photostructure/ https://blog.photostructure.com/introducing-photostructure/ [2] https://www.w3.org/TR/css-images-4/#image-notation https://www.w3.org/TR/css-images-4/#image-notation [3] https://bugs.chromium.org/p/chromium/issues/detail?id=158753 https://bugs.chromium.org/p/chromium/issues/detail?id=158753 [4] https://exiftool-vendored.js.org/interfaces/makernotestags.html#rotation https://exiftool-vendored.js.org/interfaces/makernotestags.h... (caution, large page!)
- jcims 7y agoThey should just put a gravity vector in the image.
- chmullig 7y agoI’m pretty sure iPhones do put it in. You should exif dump an iPhone XS image. INSANE what’s in there.
- alanh 7y agoGo on...
- microcolonel 7y agoMight as well put in acceleration and rotation while we're at it.
- 7y ago
- egfx 7y agoNonsense. Image recognition engines are very capable of detection in even at the most extreme angles. Sure it won’t rotate the image for you but it will certainly tell you there is a goose in it. What tricks image recognition technology more then anything is lighting.
- egfx 7y agoLet me clarify and say googles IR engine is capable. I actually ran the first set of tests developed around image recognition technology. I worked at Neven Vision which got acquired by google in 2006.
- faceshapeapp 7y agoI ran into this issue while developing https://www.faceshapeapp.com https://www.faceshapeapp.com where user uploads their photo and face detector is run on top of the picture. Shortly after launching about 10% of users complained having rotated images, after some debugging, I discovered that it was due to exif rotation. You could parse the exif and rotate the image using canvas, but thankfully, there's already a JS library which does it for you: https://github.com/blueimp/JavaScript-Load-Image https://github.com/blueimp/JavaScript-Load-Image.
- Fission 7y agoThis is functionally a coordinate system problem. Thankfully, it's pretty easy here. Just wait until we start getting more models for 3D data. I worked with 3D data for the longest time, and it was incredibly painful — different libraries can use wildly different coordinate systems (e.g. I've seen the up-direction be +z, -z, +y, and -y). At that point, it's nontrivially difficult to even figure out what the right way to convert between coordinate systems is.
- MrStonedOne 7y agoThis article is all wrong. At the core, there is something that is converting this jpeg or misc encoded data into raw encoded data, and this process MUST account for the orientation. Either the app is reading the image, converting it before passing it to the CV/ML/AI library, and this conversion step needs to respect this tag, and either transfer the tag or apply it to the transformed object; OR, the CV/ML/AI library is getting in encoded image data, and it needs to check for this tag. Those are the two options, either the CV/ML/AI library sees the tag, and should consider it, or it doesn't, and the library that is stripping it away shouldn't be doing that
- laughinghan 7y agoThis article helps people use the libraries they have, and it does so correctly. If you want to fix the libraries, submit a PR, don't complain about this article.
- yellowapple 7y agoSounds like the training sets should include more rotated samples.
- lolc 7y agoOnly goes to show how far we have to go still when computers don't even tilt their heads automatically when given rotated input.
- fake_satire 7y ago"Computer Science Expert". lololol. :)
- eggie5 7y agoCNNs are translation and scale invariant thanks mostly to the pooling operation. Good data augmentation (rotating images for example) would have build a model more robust to this effect.
- unnouinceput 7y agoHi Adam, if you read this, you have a small mistake. Your quote: "This tells the image viewer program that the image needs to be rotated 90 degrees counter-clockwise before being displayed on screen" should be: "This tells the image viewer program that the image needs to be rotated 90 degrees clockwise before being displayed on screen". Cheers.
- ageitgey 7y agoThanks!
- moonbug 7y agothat or train models to be rotationally invariant.
- rhizome 7y agoSure, that would reduce the problem space by one, but that ain't the reason CV doesn't work.
- spullara 7y agoWhen I built a consumer site that processed tons of photos I ran into this all the time. Ended up doing it all myself by parsing the exif data and doing the rotation. Also ended up writing some pretty extensive resize code that worked much better than what was built-in and more like the great scaling effects you see in Preview.app.
- kccqzy 7y agoIt always bothers me that many libraries still use nearest-neighbor to resize images instead of some kind of Lanczos filtering.
- peterhj 7y agoBonus round: (1) Find the ImageNet (ILSVRC2012) images possessing EXIF orientation metadata. (2) Of those images, find which ones have the "correct" EXIF orientation.
- yeldarb 7y agoHas anyone done this yet? I'd be curious to know if their dataset is "cleaned" or not.
- peterhj 7y agoSpoiler, ish: The last time I measured ImageNet JPEGs with EXIF orientation metadata, the number of affected images was actually quite small (< 100, out of a dataset of 1.28M). There are also some duplicates, but altogether it seems fairly "clean."
- irascible 7y agoWhy isn't a "detect and fix orientation" NN, the first stage in the machine vision pipeline?
- deleted 7y ago[deleted]
- sandos 7y agoIf the algoritm fails with a rotated image, then I would claim its a bad algorithm that is over-fit and not at all generalising what it has learnt. What about slight rotation? Where does it start failing, and would a human? Also, making an algorithm for detecting rotated images should be easy if it affect the results so much.
- DougBTX 7y agoYea, there’s some truth to that, but then we wouldn’t get these gems: Is it a duck or is it a rabbit? https://www.reddit.com/r/dataisbeautiful/comments/aydqig/is_it_a_duck_or_a_rabbit_for_google_cloud_vision/ https://www.reddit.com/r/dataisbeautiful/comments/aydqig/is_...
- mister_hn 7y agoWell, if you are using the input Image as JPEG. The information is stored in a tag, you can remove it with any kind of language. There's also a C++ library for it: https://github.com/mayanklahiri/easyexif https://github.com/mayanklahiri/easyexif
- deleted 7y ago[deleted]
- quietbritishjim 7y agoOne of the common image processing libraries does take into account EXIF rotation: OpenCV [1] I would tend to use that over manually rotating using the code from the article. Although beware that OpenCV cannot open quite as many formats as Pillow, most notably it cannot open GIFs due to patent issues. You can get a prebuilt wheel of OpenCV by pip installing the package opencv-python [2] [1] https://docs.opencv.org/trunk/d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56 https://docs.opencv.org/trunk/d4/da8/group__imgcodecs.html#g... [2] https://pypi.org/project/opencv-python/ https://pypi.org/project/opencv-python/
- anilakar 7y agoSome websites also strip EXIF data from user submitted content but do not rotate the image to match the now-missing information about which side is up.
- planis 7y agoTopic of image orientation always reminds me about this article: https://www.daveperrett.com/articles/2012/07/28/exif-orientation-handling-is-a-ghetto/ https://www.daveperrett.com/articles/2012/07/28/exif-orienta...
- logicallee 7y agoit's kind of funny, but makes sense. if you think about it the gold standard for computer vision is humans but people are pretty bad at reading upside down text (if you're on mobile you can try it but don't forget to lock orientation first!). the same is true of upside down images as the famous "Thatcher effect" illusion shows: http://thatchereffect.com/ http://thatchereffect.com/ shows it pretty clearly.
- ensiferum 7y agoMaybe the first step before doing image classification or object detection images is to do a image rotation detector ;)
- raveenb 7y agosome of the object detection ai systems do actually rotate the images in 4 to 8 ways like in fast.ai. when trained with such input data, the ai detects the object in almost all orientations.
- UglycupRawky 7y agoWho thought Exif Orientation was a good idea? It's a needless complication of the file format. It's not like iPhones (or any camera) lacks the computing power to rotate the image before saving it.
- bradfa 7y agoWikipedia says EXIF was released in 1995 (https://en.wikipedia.org/wiki/Exif https://en.wikipedia.org/wiki/Exif). If you were shooting with a DSLR, say 6 Mega pixel with 8 bits per pixel, the raw output would be 18MB in size (https://en.wikipedia.org/wiki/Kodak_DCS_400_series https://en.wikipedia.org/wiki/Kodak_DCS_400_series). In order to rotate this raw 6Mp image you would need 36MB of RAM (input and output buffers of the same size, non-overlapping). Then, after the rotation you could perform JPEG, so that the rotation is lossless. Finally, you could store the JPEG image to disk. 36MB of RAM just for raw image buffers would have been quite expensive in 1995. Simply tagging some extra data onto the image to say which orientation it should be presented in takes almost no extra memory or processing within the camera, some big desktop PC could easily rotate the uncompressed JPEG to perform a "lossless" rotation after the fact (ie: uncompress JPEG in wrong orientation, rotate, present to user).
- bradfa 7y agoTechnically, you wouldn't need a full 18MB for the output buffer so long as you perform the JPEG in-line with the rotation and are willing to deal with slicing the image into swaths. So in theory you could get away with like a 1MB output buffer but then your rotation time would depend on your JPEG timing and you couldn't take another picture with the main raw buffer until rotation and JPEG were both complete. It's a tradeoff, time versus memory.
- theon144 7y ago>Most Python libraries for working with image data like numpy, scipy, TensorFlow, Keras, etc, think of themselves as scientific tools for serious people who work with generic arrays of data. They don’t concern themselves with consumer-level problems like automatic image rotation — even though basically every image in the world captured with a modern camera needs it. What a snotty attitude. The tools are already complex enough to take on responsibilities of parsing the plethora of ways a JPEG can be "rotated". This thread is a testament to the non-triviality of the issue and I certainly don't want a matrix manipulation or machine learning library to bloat up and have opinions on how to load JPEGs just so someone careless out there can save a couple lines.
- 08027470333 7y ago82福岡県飯塚市川津176-202
- 08027470333 7y ago福岡県飯塚市川津176-202
- 08027470333 7y ago08027470333
- 08027470333 7y agoiPhone8
- kwhitefoot 7y agoIf your computer vision application can't even recognize which way up a picture like the ones in the article is then surely you have bigger problems then EXIF orientation. As others have said: what about pictures that are simply not aligned with the horizon?
- sicariusnoctis 7y agoAll the top architectures are not rotation-invariant. I think some of the blame lies in how successful CNNs are (as Hinton claims).
- pequalsnp 7y agohttps://docs.fast.ai/vision.transform.html https://docs.fast.ai/vision.transform.html
- superjan 7y agoI just could not stop laughing reading this. It does remind me, that my professor once showed us a slide of the coastline of africa, which none of us recognized until he rotated it to the correct orientation.