4 ms·

> I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publisher

by kanzure 12y ago

> I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publishers with no modification (because they silently follow undocumented standards like the HighWire metadata). We've got a growing community of volunteers who will keep the definitions up to date for hundreds or thousands of journals. If you also use our scraper definitions for your metadata you'll get all the publishers for free. My approach has been to use the Zotero translators, since their 200,000 users have been alright at responding to publisher site changes. Unfortunately they are trapped in the Firefox ecosystem until someone converts their translators to generic js. Then Zotero could be a downstream consumer of the same scrapers, but also maybe maintain them as well. zotero's scrapers: https://github.com/zotero/translators https://github.com/zotero/translators which for example I use for an IRC bot, https://github.com/kanzure/paperbot https://github.com/kanzure/paperbot Are these yours or are there more somewhere? https://github.com/ContentMine/journal-scrapers https://github.com/ContentMine/journal-scrapers I dunno about a majority following HighWire.. here's a corpus dump of what I've seen (just random debug data from paperbot): http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip (Only 333 of the 1218 samples have "citation_pdf_url". But this collection is extremely biased towards things that I am reading, rather than a sample of the entire academic spectrum.)

Blahah 12y ago
I started out with the Zotero translators, but they are really messy and not standardised. Our ultimate goal is to make it trivial for non-programmers to define and maintain journal scrapers. That was going to be extremely hard with the Zotero system. We started over by building a generic declarative scraping system. I also aim to get Zotero to eventually adopt our scraper system and collection. The stuff in that repo is a proof of principle - we will be growing the collection massively before we demo in mid July. Thanks for the corpus dump, taking a look now. edit: I'm not suggesting a majority use HighWire, but that we can have far fewer definitions than publishers. If we include Prism and DC along with some obvious sets of CSS class names, that will already get us pretty far.
juretriglav 12y ago
Thanks for the highly relevant links. It does make sense to try and outsource the maintenance of scraper rules, especially if there are projects focusing solely on that part.

by kanzure 12y ago

Blahah 12y ago

I started out with the Zotero translators, but they are really messy and not standardised. Our ultimate goal is to make it trivial for non-programmers to define and maintain journal scrapers. That was going to be extremely hard with the Zotero system. We started over by building a generic declarative scraping system. I also aim to get Zotero to eventually adopt our scraper system and collection. The stuff in that repo is a proof of principle - we will be growing the collection massively before we demo in mid July. Thanks for the corpus dump, taking a look now. edit: I'm not suggesting a majority use HighWire, but that we can have far fewer definitions than publishers. If we include Prism and DC along with some obvious sets of CSS class names, that will already get us pretty far.

juretriglav 12y ago

Thanks for the highly relevant links. It does make sense to try and outsource the maintenance of scraper rules, especially if there are projects focusing solely on that part.