This is still a problem. I attended a talk a few weeks ago by a member of staff on the Square Kilometer Array Science Data Processor project, and they were talk

This is still a problem. I attended a talk a few weeks ago by a member of staff on the Square Kilometer Array Science Data Processor project, and they were talking about data volumes on the order of an exabyte... per day. Obviously, an awful lot of pre-processing has to be done to get this down to a manageable data volume for analysis (100s of TBs), but if something gets lost in processing in the early stages, it's gone forever. Even managing and archiving the processed datasets will be a challenge. Oh, and the second stage of the project was going to add 100x to those data volume numbers.