3 ms·
On a related subject, does anyone know of a simple technique to download and save large quantities of all publicly available research articles?
by gzer0 4y ago
On a related subject, does anyone know of a simple technique to download and save large quantities of all publicly available research articles?
- colechristensen 4y agoWrite a scraping program with appropriate throttling to be nice to providers with beautifulsoup on python. I have done this fairly successfully.
- bee_rider 4y agoSince it is available for free online, why not just download what you need? Their site has nice suggestions for related works anyway -- managing a big heap o' papers seems like a pain.
- ksr 4y agoSimple technique and large quantities are often not compatible... Check out the work of Internet Archive Scholar at https://scholar.archive.org https://scholar.archive.org. Their code is open source https://github.com/internetarchive/fatcat-scholar https://github.com/internetarchive/fatcat-scholar