3 ms·

I have done something similar at a little scale. At small load, it is great. The larger you get, real problems start to arise. The bottleneck with this impleme

I have done something similar at a little scale. At small load, it is great. The larger you get, real problems start to arise. The bottleneck with this implementation is ImageMagick. ImageMagick leaks a lot of memory and is generally very inefficient with resizing operations. GraphicsMagick is not much better. Under high load, this will crush your CPUs and max out your Nginx threads much faster than it is worth. You will almost definitely need to use something like OpenCV on the GPU for this to scale. Although caching is referenced briefly in this article, it is crucial for this system to work. A good CDN with fast invalidation and a low eviction rate would be ideal.

bithive123 13y ago
VIPS can be a great replacement for gd and magick; it's not as user friendly but it's ridiculously fast. http://www.vips.ecs.soton.ac.uk/index.php?title=Libvips http://www.vips.ecs.soton.ac.uk/index.php?title=Libvips
bochi 13y ago
With OpenCV you can also do face detection and feature recognition for smarter crops. Thumbor does that and is used in a large scale at globo.com: https://github.com/globocom/thumbor https://github.com/globocom/thumbor.
hierro 13y ago
I have to disagree with that. We've been using graphicsmagick via cgo on memecrunch.com (alexa 30K, 8K in US) for 9 months and we're serving around 500K images per day on a single server without any issues. In fact, I've just logged in and the app is currently using 0.3% of the memory (out of 64GB) after running for a couple of weeks, since the last update I pushed to production. I don't believe graphicsmagick leaks memory in any significant way. Of course, you need to cache the thumbnails for a reasonable time once they're generated, otherwise the CPU usage will skyrocket, but that's pretty easy to achieve with nginx's proxy_cache directive.
patrickg_zill 13y ago
Figure 1 second CPU execution time; 12 cores (2x 6core CPU). You can handle 12x60 = 720 images per minute, or, more than 40,000 images per hour.

bithive123 13y ago

VIPS can be a great replacement for gd and magick; it's not as user friendly but it's ridiculously fast. http://www.vips.ecs.soton.ac.uk/index.php?title=Libvips http://www.vips.ecs.soton.ac.uk/index.php?title=Libvips

bochi 13y ago

With OpenCV you can also do face detection and feature recognition for smarter crops. Thumbor does that and is used in a large scale at globo.com: https://github.com/globocom/thumbor https://github.com/globocom/thumbor.

hierro 13y ago

I have to disagree with that. We've been using graphicsmagick via cgo on memecrunch.com (alexa 30K, 8K in US) for 9 months and we're serving around 500K images per day on a single server without any issues. In fact, I've just logged in and the app is currently using 0.3% of the memory (out of 64GB) after running for a couple of weeks, since the last update I pushed to production. I don't believe graphicsmagick leaks memory in any significant way. Of course, you need to cache the thumbnails for a reasonable time once they're generated, otherwise the CPU usage will skyrocket, but that's pretty easy to achieve with nginx's proxy_cache directive.

patrickg_zill 13y ago

Figure 1 second CPU execution time; 12 cores (2x 6core CPU). You can handle 12x60 = 720 images per minute, or, more than 40,000 images per hour.