12 ms·
Sharing learnings about our image cropping algorithm
- natpat 5y ago> One of our conclusions is that not everything on Twitter is a good candidate for an algorithm, and in this case, how to crop an image is a decision best made by people. This seems like it should have been a foregone conclusion. What was the driving force in the first place to think cropping images with an AI model was desirable? Seems like ML was a solution looking for a problem here, and I'm glad they've realised that.
- klodolph 5y agoIt seems obvious in retrospect. Calling it a foregone conclusion is too harsh. Twitter crops photos to fit their preview formats. It seems like an obvious improvement to show people's faces when cropping, etc.
- the_gastropod 5y agoRight but... we've been cropping images in web applications since... y'know, pretty much ever. Using ML to do this was always pretty ridiculous overkill. Give the users an image cropper, and be done with it.
- klodolph 5y agoI can't see why this is overkill. You're eliminating a step from the image posting process, and making it so users don't have to crop an image twice (once for the full image, and a second time for the preview). That makes sense when you're writing a CMS or blogging platform like Wordpress, but for Twitter it adds some friction. So, previously, the preview was just cropped in the center. But this made some images look funny, since people's faces would get chopped off. Coming up with a workable solution to this with ML is not especially hard. You can get things like face detection off the shelf, maybe just tell your autocropper, "crop closer to the face" and have a demo within a couple days (and then much more effort to productionize it). From there, you can start introducing ML models to improve on your basic face detection. (I'm not counting face detection as ML.) This is not a case where some massive ML model is being brought in to save two seconds of your time. This is a very natural and obvious application of ML, at a company which already does ML at scale, in a way that sounds like it has a good chance at improving the appearance of the site without introducing additional friction. Instragram gets around this by encouraging everyone to take square photos.
- eyelidlessness 5y agoI don’t think anyone is saying “I will always prefer to crop every photo and everyone else should too”. I think the point is closer to, if I may borrow a Simpsons line, “I liked your half-assed underparenting a lot more than your half-assed overparenting”. It’s actually impressive that Twitter didn’t object “but I was using my whole ass”, which is basically their default trope when they address user complaints.
- klodolph 5y ago> I don’t think anyone is saying “I will always prefer to crop every photo and everyone else should too”. The parent comments are basically saying that, just not in such an exaggerated way. "Give the users an image cropper, and be done with it."
- deleted 5y ago[deleted]
- Thorrez 5y agoLook at the before and after pictures of when they released the ML crop: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... All those examples show large improvement. Of course they might cherrypick images with large improvement for their blog advertising the feature. But still, it illustrates why people would think it's a good idea. Of course they don't seem to consider the idea of not cropping at all.
- amelius 5y agoI fear this is still horribly incomplete. E.g. if a picture shows brand A next to brand B, which brand will be cropped?
- e-clinton 5y agoDidn’t sound like the model had that info, so I’d imagine it would select one based on physical attributes of the brand logo, not the “brand” itself.
- gipp 5y agoI'm more forgiving about corporate jargon than most. A lot of it really does help optimize communication for the situations you encounter in corporate work. But "learnings" is literally, exactly, just a synonym for "lessons." Can we not?
- bloak 5y agoAlso the "sharing". Is the article about "sharing"? Perhaps a normal human would have used the title "An evaluation of our image cropping algorithm".
- dylan604 5y agoMaybe there's a bias to sharing since it is a social media platform where sharing is the thing?
- alanbernstein 5y agoI disagree, "Sharing lessons..." would mean "here is an educational resource that we have created, as teachers, for an audience of students". I think "Lessons learned..." is closer to what you mean to suggest, and "Learnings..." is more concise (this is from Twitter, after all).
- robotresearcher 5y agoGoogle 'lessons from' and you'll see that one usage is synonymous with 'learnings from'. The suggested completions are perfectly idiomatic.
- kzrdude 5y ago"Can we not?" is a pretty modern colloquialism. Is it better or worse than learnings?
- NovemberWhiskey 5y agoIt's a neologism, or possibly the resurrection of a long unused form - I don't know exactly how we came about it but I agree completely that one of the meanings of "lesson" is "a thing which has been learned". In my experience, there's a tendency toward folksiness in certain varieties of corpspeak that causes rejection of "formal-sounding" terms and repurposing of "plainer" forms to create new words, hence lessons = learnings, protégé = mentee, and so on.
- rwmj 5y agoBias aside, the saliency algorithm doesn't work well either. This twitter feed (SFW) https://twitter.com/punhubonline https://twitter.com/punhubonline often shows the punchlines in the preview, spoiling the joke.
- alanbernstein 5y agoAsking for an image analysis algorithm to both read and understand jokes might be expecting a bit much...
- charrondev 5y ago… who asked for the image analysis algorithm to be the only thing to crop the photo? You don’t even have to add extra required steps, generate the initial thumbnail automatically, then allow the user to move the focal point or zoom it. Sites that allow uploading profile photos have had this type of functionality for more than a decade.
- alanbernstein 5y agoI agree that a manual option should have always been available. I'm just being pedantic: the context here is a "saliency algorithm." In the context of computer vision, the term saliency has a specific meaning. It's about visual perception of images, which doesn't generally include understanding letters, words, sentences, or humor.
- rwmj 5y agoCan't a neural network be trained so it recognizes cartoon panels and only shows the first ones? Seems at least plausible.
- alanbernstein 5y agoSure, if it's designed for cropping cartoon images to show the first panel. This algorithm is designed to work for any image shared on twitter. What should it do for cartoons where the last panel is a much more interesting visual image? Single-panel cartoons that include both the setup and the punchline in one frame (https://twitter.com/PunHubOnline/status/1392509471482683399/photo/1 https://twitter.com/PunHubOnline/status/1392509471482683399/...)? How about before-and-after comparison images?
- grenoire 5y agoImagine the world where images aren't algorithmically cropped. It's easy if you try. No AI below us, above us only thumbnails.
- goldenkey 5y agoSomething something, John Lennon, crop out Ringo.
- eyelidlessness 5y agoPoor George can’t even get enough attention to be the butt of the joke :( Edit: I realize I missed an opportunity for a funnier way to make this observation, which is my curse.
- cmckn 5y agoSo, I can choose to see only un-cropped images on my TL, and the author can see a preview of the algorithm's crop before they tweet -- but a glaring omission is simply exposing a crop tool to the author. The model works by choosing a point on which to center the crop. Why can't you give user's a UI to do the same? "Tap a focal point in the image, or let our robot decide!" The blog post mentions several times how ML might not be the right choice for cropping; but their conclusion was...to keep using ML for cropping. I hope someone got a nice bonus for building the model!
- deleted 5y ago[deleted]
- jawns 5y agoI can't really see any down side, besides maybe a little bit of developer time, to allowing users to see a preview of the crop and optionally override. It's done all the time in other places.
- mattacular 5y agoIt's probably a bit harder at Twitter's unique scale. They have an incredibly high throughput of new posts and a large portion of these posts include between 1-4 images that need cropping.
- kixiQu 5y agoI dunno, that just means it's a pain for the user, not for Twitter
- bavell 5y agoIsn't this just a UI change and wiring that up to the backend? The cropping happens either way, it might actually be faster considering that if the user providers the crop info, Twitter does't have to burn CPU cycles on figuring out the "optimal" crop dimensions.
- 5y ago
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- codeulike 5y agoSome context: They dont mention it directly but I think this refers back to this thread last september https://twitter.com/colinmadland/status/1307111816250748933 https://twitter.com/colinmadland/status/1307111816250748933 (Note the thread displays differently now because Twitter have changed their cropping algorithm) Originally @colinmadland was trying to post examples of how Zoom virtual background had removed his black colleagues head, however when he posted the side-by-side images (with heads) on Twitter, twitter always cropped out his colleague and just showed him, even if he horizontally swapped the image. So, while trying to talk about an apparently racist algorithm in Zoom, he was scuppered by an apparently racist algorithim in Twitter. It was widely covered in the press at the time https://www.theguardian.com/technology/2020/sep/21/twitter-apologises-for-racist-image-cropping-algorithm https://www.theguardian.com/technology/2020/sep/21/twitter-a...
- SiempreViernes 5y agoHere's an example that still works: https://twitter.com/bascule/status/1307440596668182528?s=20 https://twitter.com/bascule/status/1307440596668182528?s=20
- fshbbdssbbgdd 5y agoThe web version shows Mitch, but the app shows a blank white (which is at the center of the image, meaning it didn’t try to crop to one of the faces). I’m on iOS.
- yxhuvud 5y agoThat example is from last September, so it doesn't say anything on if it is improved or not. They probably generate the cropping once, on posting the tweet.
- cbsks 5y agoYet another reason why the Nitter UI is better… https://nitter.cc/bascule/status/1307440596668182528?s=20 https://nitter.cc/bascule/status/1307440596668182528?s=20
- nyberg 5y ago
- jedberg 5y agoImage cropping algorithms are hard. When we made our first one for reddit, it used this algorithm: Find the larger dimension of the image. Remove either the first or last row/column of pixels, based on which had less entropy. Keep repeating until the image was a square. The most notable "bias" of this algorithm was the male gaze problem identified in the article. Women's breasts tended to have more entropy than their face, so the algorithm focused on that since it was optimized for entropy. To solve the problem, we added software that allowed the user to choose their thumbnail, but not a lot of users used it or even realized they could. I assume they've since upgraded it to use more AI with actual face detection and so on, but at the time, doing face detection on every image was computational infeasible.
- erichahn 5y agoHow is entropy related to "male gaze". This approach seems to be unsupervised, I don't see the problem.
- colllectorof 5y agoI see that you didn't get the memo. If an algorithm or a mathematical definition is made by a white man, it's by definition racist and sexist because it was made by a white man. Entropy was defined by Claude Shannon who was a white man, so entropy is racist and sexist, because it absorbed all of Shannon's biases.
- oceanghost 5y agoI honestly can't tell if you're joking or not.
- bigfudge 5y agoSadly he’s too angry to make the joke funny. As always there probably is a grain of truth to the idea that postmodern feminist critiques occasionally disappear up their own fundamental, and so there’s a joke to be made here. But this isn’t it!
- Areibman 5y ago"We began testing a new way to display standard aspect ratio photos... without the saliency algorithm crop. The goal of this was to give people more control over how their images appear while also improving the experience of people seeing the images in their timeline. After getting positive feedback on this experience, we launched this feature to everyone." So the solution all along was to give users the ability to crop their own photos. Why wasn't this the original way of doing things? Instead of forcing a complicated algorithm into the Twitter experience, it seems to me that the solution all along was just to let users do what they do best-- make tweets for themselves. This incident strikes me as a major failing of AI: We are so eager to shoehorn AI/ML into our products that we lose sight of what actually makes users happy.
- ggggtez 5y ago> Why wasn't this the original way of doing things? Someone wanted to do a feature so they could get promoted. Probably with some mumbo jumbo about how it reduces the number of clicks to create a tweet and thus increases revenue.
- cblconfederate 5y agoNext they'll tell us that chronological mode is better! (it is for me in any case)
- eyelidlessness 5y agoI like it better too. And fortunately the setting doesn’t seem to be reverted at random as frequently as it used to.
- bobbylarrybobby 5y agoWhat’s really remarkable is that giving users the ability to manually crop would be an amazing way to gather data on optimal cropping, which they could have used to train their model down the road. I can only imagine how much more time and money went into gathering eye tracking data.
- 5y ago
- cratermoon 5y agoNews flash: ML sucks. Most of the time.
- deleted 5y ago[deleted]
- boulos 5y ago/rant but I feel like talking about percentage points of difference is always hard for humans. For example: > In comparisons of men and women, there was an 8% difference from demographic parity in favor of women. would have been clearer (and more correct) as "an 8 percentage-point difference from demographic parity". That 8 pp difference though is a 16% "relative" difference (58/50), or more starkly "The algorithm chose the woman almost 40% more often" (58/42 => 1.38). That said, the diagram in the post [1] is much easier for humans to parse and say "wow, that looks pretty far off!". tl;dr: A number like 8% sounds like "no big deal", but 8 percentage points (on each side) is a big deal! [1] https://cdn.cms-twdigitalassets.com/content/dam/blog-twitter/engineering/en_us/insights/2021/imagecropping/newimagecropchart.jpg.img.fullhd.medium.jpg https://cdn.cms-twdigitalassets.com/content/dam/blog-twitter...
- MauranKilom 5y agoOr, summed up: https://xkcd.com/985/ https://xkcd.com/985/
- LinAGKar 5y agoHow about letting people disable this cropping crap altogether
- tjkrusinski 5y agoPeople abuse tall aspect ratio images to take up more space in the UI to get more attention on their tweets.
- refactor_master 5y agoJust letterbox them.
- ironmagma 5y agoThat’s definitely an engineer’s solution. It works, but looks ugly by most designers’ eyes.
- Nick87633 5y agoHow about not calling this the male gaze? If we're trying to remove bias FFS.
- krapp 5y agoThe article says the algorithm was trained on eye tracking data, and if the predominance of data came from men then referring to the male gaze as a possible source of unintended bias seems valid.
- Nick87633 5y agoThe article puts "male gaze" in quotes and there is a Wikipedia page about the meaning it has in common usage about gender discussion. It's disappointing that Twitter is trying to deflect blame for their failures onto "men". [Edit - this is inaccurate]
- dragonwriter 5y ago> It's disappointing that Twitter is trying to deflect blame for their failures onto "men". They...aren’t. Leaving aside the question fo whether identifying the “male gaze” as a cobtributor to problems with the algorithm’s output would be blame passing when the entire context is taking responsibility for and addressing problems resulting from the choice of algorithm (hint: no, it wouldn’t), they initially identified that as a potential thing the saliency algorithm might* be biased for, constructed a test, and determined it was not a factor, so they are saying “male gaze” wasn’t a problem. Kneejerk reactions to the use of the phrase without considering anything about what it is being used to say aren't helpful.
- Nick87633 5y agoYou're right, I went back and reread the post. I retract my statement about deflecting responsibility.
- fwip 5y ago> In comparisons of black and white individuals, there was a 4% difference from demographic parity in favor of white individuals. It's hard to believe that the bias was only 4% - there were a lot of people testing with images that they sourced themselves, and the preference for white people seemed much closer to 80-20. The paper authors mention that their training data is from Wikidata (pictures of celebrities). I wonder if the types of photos in that dataset are meaningfully representative of the kinds of photos that people usually post to Twitter.
- gwern 5y ago> It's hard to believe that the bias was only 4% - there were a lot of people testing with images that they sourced themselves, and the preference for white people seemed much closer to 80-20. It's very easy to believe the bias was near-zero given you are citing highly motivated people on Twitter cherrypicking from thousands of examples and a little baffling you find that to be more credible than controlled systematic experimentation; note, for example, the extremely striking fact that the fuss completely missed the other bias they found which was several times larger - showing how totally useless people on social media are for testing these things and how they can conjure up "80-20" biases which don't exist.
- eyelidlessness 5y ago> highly motivated people on Twitter cherrypicking One of the reference threads that identifies the issue happened on it by accident highlighting a surprising experience in another product (Zoom). Believe it or not, people who care about this stuff are not looking for stuff to complain about, we’re tired and overwhelmed. And I would hope that people who, upon discovering a vulnerability find and catalogue the ways it can be exploited, would be celebrated here.
- fwip 5y agoI admit that it wasn't especially rigorous testing, but I personally tested this along with other people I knew. I used real photos from my camera and my wife's (we are of different races), featuring photos of ourselves, friends, and family. I of course hope that the systems I use aren't racist against my loved ones. I am motivated to confirm whether or not they are, but I didn't go on to parlay my findings into an essay for clout. I gained nothing from doing this, except the knowledge that Twitter was suckier than I knew.
- umvi 5y agoPossibly a political question but why is the word "equitably" more popular now than "equally"? I'm not sure when I first noticed this but it seems pretty recent that "equity" became more used than "equality" when referring to diversity and inclusivity
- eyelidlessness 5y agoBecause sometimes addressing inequality requires addressing situations unequally. As a (hopefully) completely uncontroversial example: if you were a governor or president during Civil Rights era desegregation, evaluating whether to deploy National Guard soldiers to enforce desegregation of schools, it would be absurd to apply that equally to schools which are segregated and schools which are integrated.
- seneca 5y agoIt rose along with the recent rise in popularity of socialism.
- Thorrez 5y agoCould you elaborate? Correlation doesn't mean causation.
- Thorrez 5y agoThis image illustrates the idea behind equality vs equity: https://images2.minutemediacdn.com/image/upload/c_crop,h_1351,w_2400,x_0,y_172/f_auto,q_auto,w_1100/v1591824036/shape/mentalfloss/625404-iisc_equalityequity_300ppi.jpg https://images2.minutemediacdn.com/image/upload/c_crop,h_135...
- dmitrybrant 5y agoThis is a very cynical take (I'm cranky from my second vaccine dose), but: Imagine how much work, how much energy and effort, went into building an ML-based image cropping feature, just because an up-and-coming Designer decreed that a certain specific crop ratio was the most aesthetically pleasing to the human eye... ...so that years later, after countless hours of additional user research, they would just remove the feature because it doesn't work, and award themselves a medal for doing it.
- paxys 5y agoIt’s often the opposite at “hot” tech companies going through a massive hiring/growth phase (which I imagine Twitter was back when this feature was implemented). You hire one too many highly paid experts and then realize there’s no real practical work for them which couldn’t be done by a new college grad, so the end result is needless over-engineering.
- snowwrestler 5y agoThey address this directly in the article. The previous horizontal crop was not an arbitrary design decision, it was put in place to improve the browsing of tweets by fitting more of them into the viewport. Their intent was to improve the user experience. The ML-powered auto-crop was the engineering solution for how to deliver the best version of that. They don’t say it in this article but I suspect the need to fit more tweets into the viewport was obviated by smart phone UI. The screen is tall and it is super easy to scroll quickly with the flick gesture. No need for short images anymore.
- Causality1 5y agoThose results are quite interesting. The bias is much smaller than I would have expected given how we've seen systems like facial recognition and face unlock behave.
- ALittleLight 5y agoShouldn't they also check to see how frequently humans crop pictures to favor whites versus blacks, male versus female, and whether or not humans exhibit "male gaze" in their cropping decisions? Going by the numbers they report all of the biases seemed relatively small. Slight favor for women over men and white over black and no evidence of male gaze - but single digit percentages in each case. I wouldn't be surprised if humans were more biased than machines given the results I saw.
- URSpider94 5y agoThat’s not the meaningful comparison, because Twitter has no control over what people do on their own, whereas their algorithm was introducing bias into photos with no recourse for the people posting the images. I would argue that a 7% preference for White women over Black women is pretty darned significant. But beyond that, cropping a photo in a way that disrespects the intent of the poster is not the right approach. Fortunately, Twitter realized that it goes beyond the numbers.
- ALittleLight 5y agoHow is that not a meaningful comparison? If your intent is to reduce bias in image cropping this research doesn't even suggest that changing to manual cropping would do that. What if humans are even more biased than the algorithm? It seems like Twitter, and you, are saying they should try to minimize responsibility for bias in image cropping. Right now Twitter is responsible for it because their algorithm does it. If they foist the responsibility to users, then users will be responsible, but is there more or less bias in image cropping? No idea. I think, if bias in image cropping is a problem, then they should check to see whether their change increases or reduces the problem. If bias in image cropping isn't a problem, then who cares. If their problem is that they don't want to be responsible for bias in image cropping then of course making the users responsible is the right way to go.
- URSpider94 5y agoIt's not meaningful because the question here is not whether the algorithm reduces overall bias. The question is whether the algorithm introduces unwanted bias into the presentation of people's posts, with no ability for them to opt out. The bar would be different, in my mind, if we let people choose whether to use this algorithm or opt out. Also, as Twitter finally came to at the end of their post, the much bigger issue is that this algorithm doesn't respect people's desire to curate their own public image. Taking out the bias aspect, if I post a group photo that includes my best friend, and the algorithm crops them out of the thumbnail entirely, then that's going to be really frustrating to me. If I want to center a person of color or a woman (or an LGBTQ individual, which we don't know how the algorithm would rate - we don't have the data), and the algorithm makes a different choice, then my message is damaged.