10 ms·
Content-aware image resize library
- rhema 9y agoLooks really interesting. I'd love to see an animated version that shrinks from large to small.
- sheraz 9y agoLike this one? https://github.com/vivianhylee/seam-carving https://github.com/vivianhylee/seam-carving
- asoto 9y agoI wrote a version of this in javascript with a live demo you can play with at https://alexander.soto.io/seam-carving https://alexander.soto.io/seam-carving
- degenerate 9y agoThat's _extremely_ cool! Thanks for sharing
- peterburkimsher 9y agoYour JavaScript version is very easy to use, and I'm so glad I can upload my own images instead of just using examples! The algorithm itself looks like it needs face (and body) detection to be really useful, though. Example: My girlfriend and I on holiday in Australia. Original: http://peterburk.free.fr/seamCarving/lighthouse.jpg http://peterburk.free.fr/seamCarving/lighthouse.jpg Shrunk: http://peterburk.free.fr/seamCarving/lighthouse_seamCarving.jpg http://peterburk.free.fr/seamCarving/lighthouse_seamCarving....
- lysium 9y agoIn all fairness, the read me says “Todo: face detection”
- peterburkimsher 9y agoThat's true, and I agree with the author! It will be more complicated than it seems, though, because I just tried the GUI version from gabeiscoding above, and got the following result: http://peterburk.free.fr/seamCarving/lighthouse_seamCarvingGUI.jpg http://peterburk.free.fr/seamCarving/lighthouse_seamCarvingG...
- overload119 9y agoAm I the only one getting a laugh out of this? I also tried it on myself and the result was funny.
- khedoros1 9y agoWith a little change to the algorithm, you can mark areas as "must not change" and "must change", basically manually marking areas as "high energy" and "low energy".
- lph 9y agoYeah, but the lighthouse looks great!
- CoryG89 9y agoSuper awesome!
- tangoalpha 9y agoUnable to resize on mobile. Consider giving some margin or padding on the right of the image so one can drag the line to resize.
- elsurudo 9y agoThat's really neat! With my tongue somewhat-in-cheek, I wanted to point you that you may want to be careful with that sample pic of the Eiffel Tower at night! See: https://petapixel.com/2017/10/14/photos-eiffel-tower-night-illegal/ https://petapixel.com/2017/10/14/photos-eiffel-tower-night-i...
- jonathanhefner 9y agoTry this short video from SIGGRAPH 2007: https://www.youtube.com/watch?v=vIFCV2spKtg https://www.youtube.com/watch?v=vIFCV2spKtg
- wtsui 9y agoFlash implementation from 2007 here: http://rsizr.com http://rsizr.com
- squared9 9y agoHere you go: https://www.youtube.com/watch?v=_JAmnloIDnI https://www.youtube.com/watch?v=_JAmnloIDnI
- AgalmicVentures 9y agoWould this still be covered under the original patent until 2029 [1]? [1] https://www.google.com/patents/US8213745 https://www.google.com/patents/US8213745
- jimrandomh 9y agoThat patent was filed in 2009, and appears to be a copy of a 2007 paper http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.570.6321&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.570... with no novel contributions, and no overlap in authorship.
- aaronblohowiak 9y agoThis one has one of the original authors and was filed in 2008: https://www.google.com/patents/US8280191 https://www.google.com/patents/US8280191 not sure about differences in claims. IANAPA
- vanderZwan 9y ago2008: Original Assignee Abode Systems Incorporated 2009: Original Assignee Eastman Kodak Company Perhaps the Kodak patent covers a hardware-implementation for their cameras?
- dotancohen 9y ago_Eastman_ Kodak manufactures and distributes cinema film. The camera division is the _other_ Kodak.
- deleted 9y ago[deleted]
- djsumdog 9y agoNot in New Zealand, France or a host of other countries without software patents. Where is the creator located?
- ttoinou 9y agoCould we use any direction for pixel removal, rather than only vertical or horizontal ?
- vanderZwan 9y agoWell, think about what it would mean to remove a seam diagonally: the edges of the picture would no longer line up. The only way to avoid this is to always remove a seam along one of the axes at a time.
- jontro 9y agoJust run the algorithm twice?
- ianjsikes 9y agoBetter yet, scale the image proportionally down to the largest of the two target sides, then run the algorithm once in the direction that needs to shrink.
- bitL 9y agoYou either shrink picture horizontally by one pixel or vertically by one pixel; seams must be 8-connected and always advance by 1 pixel in the opposite direction to the one being removed. It's a heuristics and by making it that way you can guarantee exactness of 1px removal. These days it's a popular algorithm to be implemented as a midterm project in computational photography courses at top schools.
- aaronblohowiak 9y agoIs anyone aware of applying a similar technique to computer models ie: fake parametric design?
- vanderZwan 9y agoSeam carving is fascinating, and especially the first version of the algorithm really really simple to implement. However, that algorithm introduces certain artefacts - the boat in this picture[0] makes it pretty obvious. The cause of these artefacts are mainly that the original algorithm did not look at energy that was introduced by removing a seam: each time this happens, the pixels adjacent to this seam become neighbours, which creates a new energy gradient. So sometimes removing the least-energy seam would produce a net increase in energy. The original authors of the seam carving paper realised this[1], which lead to the obvious fix for it: instead of only looking at the current energy of a picture, and removing the least-energy seam, look how much of a net energy difference the removal of a seam would make, aka "forward energy". A few years later another paper came out that used "forward gradient difference maps" which supposedly work even better, but to be honest the formulas described in that paper are too complex for me to understand[2]. Conceptually though, I think they just extended the original energy function (a simple Sobel operator[3]) with a few other ones that include orientation and others: > The energy function measures the curvature inconsistency between the pixels that become adjacent after seam removal, and involves the difference of gradient orientation and magnitude of the pixels. Our objective is to minimize the differences induced by the removed seam, and the optimization is performed by dynamic programming based on multiple cumulative energy maps, each of which corresponds to the seam pattern associated with a pixel. The proposed technique preserves straight lines and regular shapes better than the original and improved seam carving, and can be easily combined with other types of energy functions within the seam carving framework With that in mind, it shouldn't be that complicated to come up with more improvements to seam-carving: just stack on different energy functions (either forward or current energy), and compute the ideal seam based on the (weighted) sum of them. [0] https://user-images.githubusercontent.com/883386/35498498-3c0f182c-04d7-11e8-9af8-695bc071e0f1.jpg https://user-images.githubusercontent.com/883386/35498498-3c... [1] http://www.eng.tau.ac.il/~avidan/papers/vidret.pdf http://www.eng.tau.ac.il/~avidan/papers/vidret.pdf [2] http://cvlab.postech.ac.kr/~hyeonwoonoh/acmmm2012.pdf http://cvlab.postech.ac.kr/~hyeonwoonoh/acmmm2012.pdf [3] https://en.wikipedia.org/wiki/Sobel_operator https://en.wikipedia.org/wiki/Sobel_operator
- bitL 9y agoSeam carving is also slow and doesn't work well for stretching images beyond around 50%. Moreover, computing optimal sequence of carving steps using dynamic programming for large images is not practical.
- mstade 9y agoI remember first seeing content aware image scaling at Adobe MAX in Barcelona I think it was. We were completely dumbfounded by what was surely magic happening on stage. When they showed removing objects we just lost it, jaws on the floor and all. That was a fun conference.
- nojvek 9y agoYeah Adobe sure had a lot of magic to show in that conference.
- esimov 9y agoI'm the author of this library. The algorithm permits to remove an object from the image, only it needs somehow to be localized. After localization it can be applied to that area a higher energy map, which means that part will be avoided by the seam carver. I will implement a face detection algorithm to automatically exclude faces to get not altered by the carver.
- rcarmo 9y agoThere are a few interesting things to explore in that field, with different twists: https://algorithmia.com/algorithms/opencv/SmartThumbnail https://algorithmia.com/algorithms/opencv/SmartThumbnail https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- prokoudine 9y agoOut of curiosity, liblqr was not good enough for you? :)
- amelius 9y agoSee also the Patch-Match algorithm: http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/index.php http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/index.php (Licensed by Adobe for noncommercial research use only.)
- nothis 9y ago
- mholt 9y agoSeam carving is cool and has a lot of potential, especially for certain kinds of images. I can't wait until we see something like it in pro photo editors like Affinity Photo.
- bitL 9y agoPhotoshop has it for ages; even bought the original research.
- airstrike 9y agoTime to write an iOS app and make millions?
- gabeiscoding 9y agoI wrote a GUI for another seam carving library back in 2009[1], and it looks like although in archive mode, you can still access the source as well as the windows / mac binaries. Just tested it and it still works! Not as fancy as photoshop I'm sure, but does have the ability to paint a mask of regions to keep / remove to aid the algorithm and get the desired result. Multi-threaded too! [1] https://code.google.com/archive/p/seam-carving-gui/ https://code.google.com/archive/p/seam-carving-gui/
- peterburkimsher 9y agoThat's cool! I tried it, and the app is really nice to use, but the picture still came out looking weird. See my other comment below.
- oh_sigh 9y agoThanks for that. I used it way back when(mid 2010) to make it look like a a 30' cliff jump was a 60' one. Really impressed my friends with it.
- MaxBarraclough 9y agoHigh praise indeed - you know the software is good when you can use it to lie to your friends.
- oh_sigh 9y agoNow that this thread is essentially tially dead, I can let out my little secret - I'm not the bad boy you think I am. I told them thw truth eventually -about 30 seconds after showing them the doctored one. I hope this doesn't change your opinion of me.
- justinator 9y agoI think they had this in Photoshop circa 1999, or I def. remember seeing a demo of it by Adobe around that time. Anyways, neat! Nice contribution.
- bringtheaction 9y agoIt was much more recent than that. Content Aware Fill was introduced in Photoshop CS5 which was released in 2010. Perhaps you were thinking of the Spot Healing Brush of Photoshop which is somewhat similar but much more manual and limited.
- justinator 9y agoI distinctly remember the tech being demo'd in 1999-2000. Huge amounts of "wow's" being echo'd in the auditorium. This was around when PS 4.5/5 was out. I guess it was a feature to be slated for PS 6?
- pcurve 9y agoIt looks promising, but the current result isn't great. Use cases for something like this is very specific... perhaps mass conversion of thousands of photos from 4:3 into 16:9...
- Jack000 9y agois this still the best approach to this problem? I would think that a GAN or similar would be better for keeping the correct features for things like faces etc.
- Grue3 9y agoJust saying, but GIMP had this for years with Liquid Rescale plugin (I think before even Photoshop): http://liquidrescale.wikidot.com/ http://liquidrescale.wikidot.com/
- deleted 9y ago[deleted]
- saurik 9y agoNeither GIMP nor Photoshop are libraries ("just sayin'"). It sounds like there were other previous libraries: that would be more relevant of a callback. http://liblqr.wikidot.com/ http://liblqr.wikidot.com/
- visarga 9y agoHow well can it remove objects from images? Can it remove an object in a cluttered environment? I'm thinking of image augmentation for deep CNN's. Normally, augmentation can help with invariability to small rotations, translations and flips, but if we could remove objects from images, it might be useful for creating many examples from one image, for image based reasoning.
- trumpeta 9y agoI read the paper long time ago, but if I remember correctly, it does not remove objects, the way it works is it iteratively finds a "seam" - a pixel-level path that goes perpendicular to the direction of resizing - that has the lowest "energy" and removes it. So objects are preserved and "empty space" is reduced.
- esimov 9y agoThe algorithm has the ability to remove image parts based on the energy level of each object calculated by a simple sobel operator. So in order to remove an object you have to either localize the object and give a higher energy value or to train a CNN to localize some specific object type, then again increase the energy value. Right now i'm working to implement the face detection part. So if anyone is interested in extending the library even further and to integrate a CNN, it's more than welcome.
- utkarshsinha 9y agoSurprisingly, imagemagick already supports this.
- cooper12 9y agoLink: http://www.imagemagick.org/Usage/resize/#liquid-rescale http://www.imagemagick.org/Usage/resize/#liquid-rescale. Seems like it was added as an experimental feature in v6.3.8-4. It requires compiling with the option `--with-liblqr`.
- rudi-c 9y agoHappy to see this here! Seam carving is one of those neat little algorithms that hit a sweet spot, being a sum of a few moderately complex algorithms, comes from a readable paper, and gives very satisfying result. I had a lot of fun implementing it in undergrad. One thing that I always wondered is how Photoshop managed to make it so fast that you can resize in real-time. If n is the width or height of the image, then the dynamic programming part is O(n^2) and needs to be recomputed after every seam removed. Since every seam is a single pixel wide, resizing the image by a non-trivial amount (say half) is O(n^3). There are other papers that remove multiple seams at a time but the quality isn't as good. GPU acceleration perhaps? Another thing I learned while testing seam carving extensively is that it works nicely in certain scenes/situations, but tends to break down most of the time. The two most common scenarios are: 1) lines that are off by the horizontal by more than a few degrees get cutoff 2) objects loose their proportions and even when the manipulation is not directly obvious, it tends to feel off (in the uncanny valley sense). I expect some interesting work to use deep learning for content-aware resizing, since neural nets could theoretically be more semantically and holistically aware of objects in the image.
- johndough 9y agoI also thought about augmenting this algorithm with the semantic information output of a neural network, but couldn't come up with a good way to generate training data for semantic segmentation which wouldn't distort objects. For example, if you consider an image of a roof, the roof pixels will all be semantically similar, but if you remove any of them just based on that, the regular structure will be distorted. Do you have an idea how to solve this?
- rudi-c 9y agoThere's one paper I found that explores this approach: https://arxiv.org/pdf/1708.02731.pdf https://arxiv.org/pdf/1708.02731.pdf I know little about deep learning though, so I can't really comment on it. For situations like the roof problem you describe, I think PatchMatch is more likely to work out well. The downside is that it's considerably harder to implement correctly than seam carving. http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/patchmatch.pdf http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/patchmatch....
- Terretta 9y agoCame for the “seam carving”... okay. Didn’t expect image expanding... was blown away. Mix in temporal energy plus a ML objects DB, put that in a chip, and turn decades of 4:3 TV into widescreen!
- serpix 9y agoI'm already quite interested in resizing wide images to square images for Convolutional Neural Networks training
- madshiva 9y agoSure it's a complex and very nice algorithm but I prefer an image that's not manipulated.
- justkez 9y agoLoosely tangental; can anyone recommend an image library management system they've had success with? Looking for uploading, thumb-nailing, search metadata, access levels and ideally some on-the-fly transformations like Cloudinary offer
- igordebatur 9y agoI would suggest having a look at Uploadcare, https://stackshare.io/uploadcare/how-uploadcare-built-a-stack-that-handles-350m-file-api-requests-per-day https://stackshare.io/uploadcare/how-uploadcare-built-a-stac... (though I am biased as a CEO)
- barrystaes 9y agoNice implementation. Like face detection also text-detection would be nice, to avoid distorting / aliasing text (like in advertisements).
- michaelthiessen 9y agoI am always blown away by this stuff, but it still applies some distortion to the image in a way that feels slightly unnatural. It's similar to the effect that changing focal length on a camera has. eg. Wider angle lenses seem to squish far away objects and enlarge closer objects. However, this affects the entire image in a consistent way whereas changing focal length does not. I wonder if this could be used to simulate focal length changes?
- microcolonel 9y agoNot to downplay the cool factor of a new implementation, Liquid Rescale (long-time GIMP plugin) also has protective masks, which prevent or delay the laying of seams in some parts of the image. Caire seems to intend to do a more automagical subset of that, with the "Face detection" todo, but the better short term solution is to accept masks directly. http://liquidrescale.wikidot.com/en:examples http://liquidrescale.wikidot.com/en:examples
- anandkunwar 9y agoLooks cool, will check out soon. I remember I created an image resize web application using seam carving during college for my software engineering project as I was fascinated by the algorithm. Not sure if in working state currently. https://github.com/pixation/pixation https://github.com/pixation/pixation
- mfbx9da4 9y agoI actually also implemented seam carving in JS a while ago with a live demo in the browser http://davidalbertoadler.com/projects/seam-carving-js/ http://davidalbertoadler.com/projects/seam-carving-js/ https://github.com/mfbx9da4/seam-carving-js https://github.com/mfbx9da4/seam-carving-js