A lot of machine learning datasets only include metadata with a link to the original source (e.g., a full resolution image which might be too wasteful to includ

A lot of machine learning datasets only include metadata with a link to the original source (e.g., a full resolution image which might be too wasteful to include with the dataset). So the first step is to write a crawler and async is a really fast way to do that because everything is network IO bounded.