3 ms·
The files are scanned TIFFs in the 25-30 MB range so we're thinking 5 seconds might be on the optimistic side. Yes, definitely planning to parallelize but we'r
by phprecovery 12y ago
The files are scanned TIFFs in the 25-30 MB range so we're thinking 5 seconds might be on the optimistic side.
Yes, definitely planning to parallelize but we're likely limited to four simultaneous instances so that will still take ~20-30 days.
- Igglyboo 12y agoAre cloud providers an option? You could use AWS of Google Compute Engine, setup 150 instances and do it in ~1 day.
- bilalhusain 12y agoAs jenkstom pointed out, there might be bandwidth issues. I guess it'll require uploading few dozen TB data.
- snowwrestler 12y agoAWS has a "ship a hard drive" upload option.
- meryn 12y agoI'm not sure what resolution these images are, but if it's very high-res, it may make sense to make intermediate-sized versions first. I expect that would make recreating thumbnails (i.e. in a different size - or multiple sizes) much quicker. Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.
- snowwrestler 12y agoI guess it's worth asking, why not just take 30 days to do it? Just work on something else until it's done.