3 ms·
> academic scientists don't have the time, money, or expertise to manage large datasets I may be cynical about general expertise, as a support person, but larg
by gnufx 5y ago
> academic scientists don't have the time, money, or expertise to manage large datasets
I may be cynical about general expertise, as a support person, but large datasets have long been stock in trade of areas I'm more or less familiar with, whether "large" is TBs or PBs like CERN experiments. (When I were a lad, it was what you could push past the tape interface in a few days -- data big in cubic feet...)
- dekhn 5y agoTape is worthless except for archival purposes (and it's not particularly good). it should not be the constraint on the dataset (IE, any important dataset should already be in live serving with replication). Very few players wrangle petabytes effectively. Many players have petabytes, but they're just piles of disorganized data that couldn't be used for training ML. Moving petabytes is still a huge pain and few folks have proficiency in giving ML algorithms high performance access to the data.
- pas 5y agoThe linked GH repo's readme says that it needs to download ~430GB and it takes ~2.5TB unzipped.