4 ms·
Oh man, a sorting error of files due to filesystem differences.. A very similar issue happened to me while doing astronomy during undergrad. This was a software
by saboot 7y ago
Oh man, a sorting error of files due to filesystem differences.. A very similar issue happened to me while doing astronomy during undergrad. This was a software pipeline for a specific telescope and I followed the guide they provided. Essentially there were two directories, one for images, and another for error estimates. Files in both directories had the same filename. The software needed text documents of the images and errors. The guide listed to do a basic 'ls images/ > images_list.txt` `ls errors/ > errors_list.txt`.
HOWEVER HYPERTHREADING WASN'T ACCOUNTED FOR. Between the images_list.txt and errors_list.txt about every 20 or 30 lines the file order would be swapped. Invalidating the analysis and producing poor images.
It only took this undergrad two months to learn what was happening.
- liability 7y ago'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option. Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.
- kardos 7y agoI don't think this is one size fits all advice. Surely once your results are measured in TBs this will fall part.