6 ms·
At 5 seconds an image using a tool like ImageMagick (which might be optimistic) Are you sure ImageMagick would take that long? I haven't timed it, but I don't
by orr94 12y ago
At 5 seconds an image using a tool like ImageMagick (which might be optimistic)
Are you sure ImageMagick would take that long? I haven't timed it, but I don't recall thumbnail creation with ImageMagick taking that long.
Also, if you can parallelize it, it won't take 115 days.
- phprecovery 12y agoThe files are scanned TIFFs in the 25-30 MB range so we're thinking 5 seconds might be on the optimistic side. Yes, definitely planning to parallelize but we're likely limited to four simultaneous instances so that will still take ~20-30 days.
- Igglyboo 12y agoAre cloud providers an option? You could use AWS of Google Compute Engine, setup 150 instances and do it in ~1 day.
- bilalhusain 12y agoAs jenkstom pointed out, there might be bandwidth issues. I guess it'll require uploading few dozen TB data.
- snowwrestler 12y agoAWS has a "ship a hard drive" upload option.
- meryn 12y agoI'm not sure what resolution these images are, but if it's very high-res, it may make sense to make intermediate-sized versions first. I expect that would make recreating thumbnails (i.e. in a different size - or multiple sizes) much quicker. Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.
- snowwrestler 12y agoI guess it's worth asking, why not just take 30 days to do it? Just work on something else until it's done.
- orr94 12y agoFWIW, some benchmarks say GraphicsMagick is faster than ImageMagick. http://www.compsci.wm.edu/SciClone/documentation/software/graphics/gm-1.3.7/benchmarks.html http://www.compsci.wm.edu/SciClone/documentation/software/gr... I suggest actually timing some of this stuff before writing off the idea. If an image can be processed in 1 second, and you can parallelize it to 4 cores, some quick and dirty (and probably useless) math says 2 million images could be done in just under 6 days. I don't think that's too shabby for 2 million images.
- Igglyboo 12y agoI'd assume that the bottle neck would be I/O if these images are ~30mb as the OP stated in the thread. I don't think you'd see 4x speedups from parallelizing.