5 ms·

Until this bias controversy came about I had no idea this much effort was put into image cropping. I assumed thumbnails were just shrunk versions of the whole i

Until this bias controversy came about I had no idea this much effort was put into image cropping. I assumed thumbnails were just shrunk versions of the whole image. If you're going to analyze images with AI before cropping, why not just tell it to keep all the human faces? What kind of tortured spreadsheet sorcery was used to justify trying to create a system that decided which people in a photo were important, and how could that possibly generate enough profit to be worth it?

haneefmubarak 5y ago
IIRC it purely computed the area of the image that had the highest level of contrast and then used that. I think the contrast was intended to be a proxy for visual detail, but this ended up having unintended consequences that made some folks angry so they threw it all away.
- kevingadd 5y ago
  This whole time what many people wanted was direct control over cropping, and it's always seemed absurd to me that Twitter invested huge amounts of time into AI cropping when letting users crop their own content probably would have produced good results (or better)
  - krisoft 5y ago
    It is really not a huge amount of time. This is not some cutting edge machine learning algorithm. In fact i would bet that a UI which lets you crop on all the platforms tweets can be posted from will be more work. Why do i think so? It is based on how many different teams has to cooperate to roll the feature out. I’m not working at twitter so thr following will be based on guesses. I bet that there is a part of their infra which processes uploaded images. It recodes them to the prefered format, strips metadata, generates various thumbnail sizes. I also bet that there is a team/sub-team who maintains this pipeline. That team can add a new processing step independent of the whole company which figure out a cropping and puts that info alongside with the rest of the image metadata. They own the backend, so they can do this easily as long as they don’t require extra user input from the frontend. Then the frontend teams who want to use the new cropped image format can update their tweet display code to utilize the new metadata. This doesn’t have to all happen at once on every platforms. The ones who care about being able to show more tweets in a fixed amount of vertical space can do so, and for everyone else things stay business as usual. So with this design the feature can be rolled out by only touching two teams, and even those two teams can act somewhat independently of each other with minimal amount of coordination. If you need user input then every twitter image upload interface needs to be updated. Probably the 3 big ones are the web, ios and android, but probably there are other more exotic stuff too. You cannot really assume that the cropping metadata is available until the last of these upload streams are updated. That involves cooperation from a lot of lot of teams. And also because you now have a new piece of UI strings needs to be localised. Also i would bet that twitter cares very much about their engagement numbers. If the new cropping step after an update makes people tweet photos less they will yank the feature back in no time. Not saying that the UI-less cropping was a good decision. It was clearly not, but I can see why someone might go with that implementation, given all the considerations described above.
    - omegalulw 5y ago
      There is no reason to wait for all UI teams. They could just implement it on one platform where the crop coordinates come from the user and keep using the current setup for crops from remaining platforms.
IfOnlyYouKnew 5y ago
My theory is that cropping is intended to require interested users to click on the image to reveal it, thus generating engagement data.
- rgoulter 5y ago
  More likely to avoid the UX of 1x5000px sized images or so, I'd think.
  - jazzyjackson 5y ago
    Well you have to have a limit, and people do post very tall comic strip images, there might be something to the fact that Reddit Twitter and Facebook all do some auto crop on the timeline - I think it’s because they don’t want the extra whitespace in the timeline, but pumping engagement numbers by making users click to see the whole meme wouldn’t surprise me.
FooBarBizBazz 5y ago
Twitter does the kind of stuff Twitter users are into. This approach might also be better for engagement.

haneefmubarak 5y ago

IIRC it purely computed the area of the image that had the highest level of contrast and then used that. I think the contrast was intended to be a proxy for visual detail, but this ended up having unintended consequences that made some folks angry so they threw it all away.

kevingadd 5y ago
This whole time what many people wanted was direct control over cropping, and it's always seemed absurd to me that Twitter invested huge amounts of time into AI cropping when letting users crop their own content probably would have produced good results (or better)
- krisoft 5y ago
  It is really not a huge amount of time. This is not some cutting edge machine learning algorithm. In fact i would bet that a UI which lets you crop on all the platforms tweets can be posted from will be more work. Why do i think so? It is based on how many different teams has to cooperate to roll the feature out. I’m not working at twitter so thr following will be based on guesses. I bet that there is a part of their infra which processes uploaded images. It recodes them to the prefered format, strips metadata, generates various thumbnail sizes. I also bet that there is a team/sub-team who maintains this pipeline. That team can add a new processing step independent of the whole company which figure out a cropping and puts that info alongside with the rest of the image metadata. They own the backend, so they can do this easily as long as they don’t require extra user input from the frontend. Then the frontend teams who want to use the new cropped image format can update their tweet display code to utilize the new metadata. This doesn’t have to all happen at once on every platforms. The ones who care about being able to show more tweets in a fixed amount of vertical space can do so, and for everyone else things stay business as usual. So with this design the feature can be rolled out by only touching two teams, and even those two teams can act somewhat independently of each other with minimal amount of coordination. If you need user input then every twitter image upload interface needs to be updated. Probably the 3 big ones are the web, ios and android, but probably there are other more exotic stuff too. You cannot really assume that the cropping metadata is available until the last of these upload streams are updated. That involves cooperation from a lot of lot of teams. And also because you now have a new piece of UI strings needs to be localised. Also i would bet that twitter cares very much about their engagement numbers. If the new cropping step after an update makes people tweet photos less they will yank the feature back in no time. Not saying that the UI-less cropping was a good decision. It was clearly not, but I can see why someone might go with that implementation, given all the considerations described above.
  - omegalulw 5y ago
    There is no reason to wait for all UI teams. They could just implement it on one platform where the crop coordinates come from the user and keep using the current setup for crops from remaining platforms.

IfOnlyYouKnew 5y ago

My theory is that cropping is intended to require interested users to click on the image to reveal it, thus generating engagement data.

rgoulter 5y ago
More likely to avoid the UX of 1x5000px sized images or so, I'd think.
- jazzyjackson 5y ago
  Well you have to have a limit, and people do post very tall comic strip images, there might be something to the fact that Reddit Twitter and Facebook all do some auto crop on the timeline - I think it’s because they don’t want the extra whitespace in the timeline, but pumping engagement numbers by making users click to see the whole meme wouldn’t surprise me.

FooBarBizBazz 5y ago

Twitter does the kind of stuff Twitter users are into. This approach might also be better for engagement.