4 ms·
While it's great to make code run as fast as possible, not every scientific problem is bounded by code execution. I've recently been writing satellite image pr
by porker 11y ago
While it's great to make code run as fast as possible, not every scientific problem is bounded by code execution.
I've recently been writing satellite image processing code, and profiled it thinking the algorithm was the problem. It turned out that even on a SSD nearly 90% of the program time (~30 minutes) was reading from & writing to disk intermediate image files.
More could be kept in RAM, but high-memory cloud machines aren't cheap, and our local development machines only have 16-32GB.
- miahi 11y agoIt still is an algorithm problem if you don't compress the intermediate files (or you compress them with an algorithm that does not help your processing).
- porker 11y agoThat's true. We've tried enabling GDAL's compression (written in C++ IIRC) on our TIFF files, but the combined compress+write/read+decompress is slower than writing/reading the uncompressed file to/from disk.
- sgt101 11y agoYou can get some amazing rack mounted machines now, ok not for cheap, but for relatively cheap given the value that they bring. I'm buying an analytics "box" at the moment with 20k gpu cores, 386GB ram and a 45k iops ssd, ok its $40k but shared across 20 engineers the productivity boost more than costs in really fast. It sounds to me that you need to take an investment case to your boss.
- porker 11y ago> It sounds to me that you need to take an investment case to your boss. Welcome to academia...
- sgt101 11y agowell, if you want the publications you need the kit.
- dagw 11y agoOr if it's just the occasional batch job, use AWS EC2 spot instances. A machine with 122 GB of RAM can often be had for $0.25-0.50 pr hour and sometimes for a lot less.