4 ms·
If you had ever compiled some of these datasets you would realize that it was impossible to get the datasets without extremely complicated scrappers that genera
by rburhum 9y ago
If you had ever compiled some of these datasets you would realize that it was impossible to get the datasets without extremely complicated scrappers that generated jobs, waited for a tiny little portion to be extracted (like 0.1% of it), and download the result from an ftp location that had non-standardized names... or how you would end up just giving up, calling people on the phone to ultimately get the person in charge to return the call to you, and buy pre-filled HDs from them that would get mailed to you eventually.
This is far more than a glorified mirror.
- joelhaasnoot 9y agoI realize yes some of these sources have small budgets, others have million dollar budgets, yet haven't solved the "cheap storage & distribution" in a way Amazon has. It is a case of each to his own, Amazon is able to help in their way. My comment was more a warning to not expect this to be any easier to access than the original data in the way of manipulating data.