3 ms·
It seems so intentionally "broken", I can only guess it is to prevent scraping? Since searching for generic-ish search terms and sorting by date is a common scr
by crazygringo 2y ago
It seems so intentionally "broken", I can only guess it is to prevent scraping? Since searching for generic-ish search terms and sorting by date is a common scraping strategy.
Still, you'd think they'd do a cutoff of e.g. 500 or 1,000 items rather than filter by the past year.
So I can't help but wonder if it's a contractual limitation insisted on by publishers? Since the publishers also don't want all their papers being spidered via Scholar? It feels kind of like a limitation a lawyer came up with.
- eesmith 2y agoUnlikely, since the easy work-around for scrapers is to search by date range and grab things that way. That's what I do now manually.
- asdff 2y agopubmed is literally built for academic scraping. It even has a command line interface to access it. If publishers were worried about scraping they'd target that, but they don't. In fact when papers go on pubmed after a year they are rehosted by pubmed central and made freely available to anyone in the world.