4 ms·

Jure, your projects never cease to impress me. Really looking forward to talking in depth at OKfest. This idea is so close to what we've been doing that it's a

by Blahah 12y ago

Jure, your projects never cease to impress me. Really looking forward to talking in depth at OKfest. This idea is so close to what we've been doing that it's a real shame we didn't talk earlier, but the parts of what you're doing that are unique are also truly awesome. At ContentMine we're doing something totally complementary to this. Some of the tools will overlap and we should be sharing what we're doing. For example, I've been working on a standardised declarative JSON-XPath scraper definition format and a subset of it for academic journal scraping. I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publishers with no modification (because they silently follow undocumented standards like the HighWire metadata). We've got a growing community of volunteers who will keep the definitions up to date for hundreds or thousands of journals. If you also use our scraper definitions for your metadata you'll get all the publishers for free. Our goal initially is to scrape the entire literature (we have TOCs for 23,000 journals) as it is published every day. We then use natural language and image processing tools to extract uncopyrightable facts from the full texts, and republish those facts in open streams. For example we can capture all phylogenetic trees, reverse engineer the newick format from images, and submit them to the Tree Of Life. Or we can find all new mentions of endangered species and submit updates to the IUCN Red List. There's a ton of other interesting stuff downstream (e.g. automatic fraud detection, data streams for any conceivable subject of interest in the scientific literature). I have a question. Why are you saying you'll never do full texts? You could index all CC-BY and better full texts completely legally, and this would greatly expand the literature search power.

kanzure 12y ago
> I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publishers with no modification (because they silently follow undocumented standards like the HighWire metadata). We've got a growing community of volunteers who will keep the definitions up to date for hundreds or thousands of journals. If you also use our scraper definitions for your metadata you'll get all the publishers for free. My approach has been to use the Zotero translators, since their 200,000 users have been alright at responding to publisher site changes. Unfortunately they are trapped in the Firefox ecosystem until someone converts their translators to generic js. Then Zotero could be a downstream consumer of the same scrapers, but also maybe maintain them as well. zotero's scrapers: https://github.com/zotero/translators https://github.com/zotero/translators which for example I use for an IRC bot, https://github.com/kanzure/paperbot https://github.com/kanzure/paperbot Are these yours or are there more somewhere? https://github.com/ContentMine/journal-scrapers https://github.com/ContentMine/journal-scrapers I dunno about a majority following HighWire.. here's a corpus dump of what I've seen (just random debug data from paperbot): http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip (Only 333 of the 1218 samples have "citation_pdf_url". But this collection is extremely biased towards things that I am reading, rather than a sample of the entire academic spectrum.)
- Blahah 12y ago
  I started out with the Zotero translators, but they are really messy and not standardised. Our ultimate goal is to make it trivial for non-programmers to define and maintain journal scrapers. That was going to be extremely hard with the Zotero system. We started over by building a generic declarative scraping system. I also aim to get Zotero to eventually adopt our scraper system and collection. The stuff in that repo is a proof of principle - we will be growing the collection massively before we demo in mid July. Thanks for the corpus dump, taking a look now. edit: I'm not suggesting a majority use HighWire, but that we can have far fewer definitions than publishers. If we include Prism and DC along with some obvious sets of CSS class names, that will already get us pretty far.
- juretriglav 12y ago
  Thanks for the highly relevant links. It does make sense to try and outsource the maintenance of scraper rules, especially if there are projects focusing solely on that part.
juretriglav 12y ago
Thanks Richard! Open Knowledge festival is going to be off the hook :) I realized you and your team (Peter et al.) have been working on a project in a similar space, and was hoping there would some beneficial overlap. It looks like there is! The ScraperJSON definitions for publishers sounds like exactly what Scholar Ninja needs. Let's chat soon :) Thanks you for the quick description of ContentMine, the more awareness about this projects, the better. I've talked to Peter Murray-Rust on a few occasions and have to say that what you are doing with ContentMine is phenomenal and I wish you all the best. I hope you'll see that our projects are complimentary rather than competitive. About indexing full texts: We do generate keyword indexes based on full texts, but we do not add this full text to the network. I hope you can see what I'm saying: you can still search through full text, but you can't access full text, it doesn't exist on the network, only a keyword: [entry1, entry2,...] index exists. One improvement to Scholar Ninja for open access articles would be the ability to show snippets, like Google Scholar does; it's on the mental TODO.

by Blahah 12y ago

kanzure 12y ago

> I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publishers with no modification (because they silently follow undocumented standards like the HighWire metadata). We've got a growing community of volunteers who will keep the definitions up to date for hundreds or thousands of journals. If you also use our scraper definitions for your metadata you'll get all the publishers for free. My approach has been to use the Zotero translators, since their 200,000 users have been alright at responding to publisher site changes. Unfortunately they are trapped in the Firefox ecosystem until someone converts their translators to generic js. Then Zotero could be a downstream consumer of the same scrapers, but also maybe maintain them as well. zotero's scrapers: https://github.com/zotero/translators https://github.com/zotero/translators which for example I use for an IRC bot, https://github.com/kanzure/paperbot https://github.com/kanzure/paperbot Are these yours or are there more somewhere? https://github.com/ContentMine/journal-scrapers https://github.com/ContentMine/journal-scrapers I dunno about a majority following HighWire.. here's a corpus dump of what I've seen (just random debug data from paperbot): http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip http://diyhpl.us/~bryan/papers2/paperbot/publisherhtml.zip (Only 333 of the 1218 samples have "citation_pdf_url". But this collection is extremely biased towards things that I am reading, rather than a sample of the entire academic spectrum.)

Blahah 12y ago
I started out with the Zotero translators, but they are really messy and not standardised. Our ultimate goal is to make it trivial for non-programmers to define and maintain journal scrapers. That was going to be extremely hard with the Zotero system. We started over by building a generic declarative scraping system. I also aim to get Zotero to eventually adopt our scraper system and collection. The stuff in that repo is a proof of principle - we will be growing the collection massively before we demo in mid July. Thanks for the corpus dump, taking a look now. edit: I'm not suggesting a majority use HighWire, but that we can have far fewer definitions than publishers. If we include Prism and DC along with some obvious sets of CSS class names, that will already get us pretty far.
juretriglav 12y ago
Thanks for the highly relevant links. It does make sense to try and outsource the maintenance of scraper rules, especially if there are projects focusing solely on that part.

juretriglav 12y ago

Thanks Richard! Open Knowledge festival is going to be off the hook :) I realized you and your team (Peter et al.) have been working on a project in a similar space, and was hoping there would some beneficial overlap. It looks like there is! The ScraperJSON definitions for publishers sounds like exactly what Scholar Ninja needs. Let's chat soon :) Thanks you for the quick description of ContentMine, the more awareness about this projects, the better. I've talked to Peter Murray-Rust on a few occasions and have to say that what you are doing with ContentMine is phenomenal and I wish you all the best. I hope you'll see that our projects are complimentary rather than competitive. About indexing full texts: We do generate keyword indexes based on full texts, but we do not add this full text to the network. I hope you can see what I'm saying: you can still search through full text, but you can't access full text, it doesn't exist on the network, only a keyword: [entry1, entry2,...] index exists. One improvement to Scholar Ninja for open access articles would be the ability to show snippets, like Google Scholar does; it's on the mental TODO.