6 ms·

If I weren't too timid to risk doing so, I would do the following (read I hope someone else does this). Process the pdf's with an OCR program to extract as muc

If I weren't too timid to risk doing so, I would do the following (read I hope someone else does this). Process the pdf's with an OCR program to extract as much text from each document as possible. The extraction should be done page by page, so the extracted text can be referenced to a PDF page#. Then, provide a searchable/browse-able directory of the extracted content. Each page of text has link to the original PDF page so you can easily open up the PDF to the page the text was extracted from. I'd also make all text user editable wiki style. Combined with the inline PDF page references it would be super easy for any user to fix up translation errors from the OCR process. Tie in a karma system to the users profile so that edits can be thanked/kudos on a job well done to help with automating moderation of user edits by rating the user's current karma to decide if the edit should be accepted automatically or provided as an alternate version other users can check and rate up if they think it should replace current version. Maybe mash in an image cropping service so diagrams can be cropped from the PDF and inserted inline with the translated text. Provide simple wiki formatting markup to allow users to format the articles. Use ad revenue/donations to alleviate/cover hosting costs. 1, 2, 3, go.

crocowhile 15y ago
That is what jstor is about actually: full text search of a number of articles.
- brettnak 15y ago
  I haven't used it seriously in quite a while, but I remember the search being fairly primitive, though faceted.
  - nullc 15y ago
    They don't do full text search on these articles.
drewda 15y ago
DocumentCloud [1] provides all the pieces necessary. It's all very nicely documented, too. [1] http://www.documentcloud.org http://www.documentcloud.org
ristretto 15y ago
Regarding this, since the archives are in the public domain, wouldn't it be possible for any third party (e.g. google books) to ask and obtain permission to photograph the originals ? Anyway the ethical thing to do here is to lobby jstor (a non-profit organization) to make reprints free.
qqqqrrrr 15y ago
As someone who's published in journals that are on JSTOR, I hope that whoever will do this will go to jail and get raped on a daily basis. The law protects my copyrights for a reason, and no Robin Hood wannabe douchebag should be allowed to take them away.
- dantheman 15y ago
  The fact you would wish rape upon anyone is deplorable.
- mjdwitt 15y ago
  Raped on a daily basis? Really? Your opinion would have more weight if you left out the histrionics.
- JustinHoffman 15y ago
  Shut up. The documents should be free. Thank god someone is standing up to these outdated, lazy, and stupid industries.
- pw 15y ago
  Yeah, and this is HN. Those sort of sentiments (violence towards others) aren't welcome here.
  - deleted 15y ago
    [deleted]
  - deleted 15y ago
    [deleted]
    - masterzora 15y ago
      There's no violence in forced rape? News to me.
      - Andrew_Quentin 15y ago
        Sorry, the forced rape part must have gone past my eyes. Lets see if I can delete my comment.
anigbrowl 15y ago
Well, most of this already exists in Google books, where I'm reading some of these particular scientific journals right now, can switch to OCRed text at any time (albeit with ftrange contemporarie fpelling), or download a facsimile of any public domain work in PDF format. The only thing they don't have in place (and should add) is the wiki-style crowd editing. So the guy has basically built a 33gb torrent of stuff that was already freely available to the public, just from a different source.
- deleted 15y ago
  [deleted]
prosody 15y ago
There's already a system somewhat like that in place and it was actually referenced in the article. Wikisource (http://wikisource.org http://wikisource.org) is run by the same nonprofit as Wikipedia, and has a pretty robust system for pairing up side by side transcribed texts and scans. Someone's already started the germ of a Phil. Trans. library (http://en.wikisource.org/wiki/Philosophical_Transactions http://en.wikisource.org/wiki/Philosophical_Transactions). At present the first volume is done and about forty volumes (http://en.wikisource.org/wiki/Index:Philosophical_Transactions_-_Volume_001.djvu http://en.wikisource.org/wiki/Index:Philosophical_Transactio...) are uploaded and ready for users to transcribe.
mkrecny 15y ago
I used OCR to do some interesting things with a similar dataset: http://news.ycombinator.com/item?id=2791611 http://news.ycombinator.com/item?id=2791611
bnewbold 15y ago
If you're curious about what these papers are, here is a hack at exposing the metadata in an easily searchable format. It would not be hard to scrape the PDFs to searchable text using unix tools (doesn't come out very readable) and render them as SVG files with inkscape for easy browser viewing: http://robocracy.org/royal-search/ http://robocracy.org/royal-search/ https://github.com/bnewbold/royal-search https://github.com/bnewbold/royal-search Lots of good old stuff, such as "On the Double Organs of Generation of the Lamprey, the Conger Eel, the Common Eel, the Barnacle, and Earth Worm, Which Impregnate Themselves; Though the Last from Copulating, Appear Mutually to Impregnate One Another" This is all available elsewhere, i'm just trying to demonstrate how low the barrier to search is with OSS tools; I hope all these papers end up on archive.org with full text search!
dunham 15y ago
Distributed Proofreaders already has some infrastructure set up for proofreading and editing OCR'd documents. It's not quite the same as your proposal, but they do track stats and have levels of participants based on skill and experience. Might be worth checking out. http://www.pgdp.net/c/
qqqqrrrr 15y ago
This is very disappointing. A comment that's advocating theft is voted up to the top of the thread, whereas my comment explaining that this is THEFT is deleted. Shame on you. And I love how the hivemind has used the opportunity to pile up on me. I just want to clarify: I am a publishing academic (a grad student), and I received none of your tax money to publish. In fact, I'm not even American. Just to reiterate: please stop making up excuses for what is essentially STEALING.

crocowhile 15y ago

That is what jstor is about actually: full text search of a number of articles.

brettnak 15y ago
I haven't used it seriously in quite a while, but I remember the search being fairly primitive, though faceted.
- nullc 15y ago
  They don't do full text search on these articles.

drewda 15y ago

DocumentCloud [1] provides all the pieces necessary. It's all very nicely documented, too. [1] http://www.documentcloud.org http://www.documentcloud.org

ristretto 15y ago

Regarding this, since the archives are in the public domain, wouldn't it be possible for any third party (e.g. google books) to ask and obtain permission to photograph the originals ? Anyway the ethical thing to do here is to lobby jstor (a non-profit organization) to make reprints free.

qqqqrrrr 15y ago

As someone who's published in journals that are on JSTOR, I hope that whoever will do this will go to jail and get raped on a daily basis. The law protects my copyrights for a reason, and no Robin Hood wannabe douchebag should be allowed to take them away.

dantheman 15y ago
The fact you would wish rape upon anyone is deplorable.
mjdwitt 15y ago
Raped on a daily basis? Really? Your opinion would have more weight if you left out the histrionics.
JustinHoffman 15y ago
Shut up. The documents should be free. Thank god someone is standing up to these outdated, lazy, and stupid industries.
pw 15y ago
Yeah, and this is HN. Those sort of sentiments (violence towards others) aren't welcome here.
- deleted 15y ago
  [deleted]
- deleted 15y ago
  [deleted]
  - masterzora 15y ago
    There's no violence in forced rape? News to me.
    - Andrew_Quentin 15y ago
      Sorry, the forced rape part must have gone past my eyes. Lets see if I can delete my comment.

anigbrowl 15y ago

Well, most of this already exists in Google books, where I'm reading some of these particular scientific journals right now, can switch to OCRed text at any time (albeit with ftrange contemporarie fpelling), or download a facsimile of any public domain work in PDF format. The only thing they don't have in place (and should add) is the wiki-style crowd editing. So the guy has basically built a 33gb torrent of stuff that was already freely available to the public, just from a different source.

deleted 15y ago
[deleted]

prosody 15y ago

There's already a system somewhat like that in place and it was actually referenced in the article. Wikisource (http://wikisource.org http://wikisource.org) is run by the same nonprofit as Wikipedia, and has a pretty robust system for pairing up side by side transcribed texts and scans. Someone's already started the germ of a Phil. Trans. library (http://en.wikisource.org/wiki/Philosophical_Transactions http://en.wikisource.org/wiki/Philosophical_Transactions). At present the first volume is done and about forty volumes (http://en.wikisource.org/wiki/Index:Philosophical_Transactions_-_Volume_001.djvu http://en.wikisource.org/wiki/Index:Philosophical_Transactio...) are uploaded and ready for users to transcribe.

mkrecny 15y ago

I used OCR to do some interesting things with a similar dataset: http://news.ycombinator.com/item?id=2791611 http://news.ycombinator.com/item?id=2791611

bnewbold 15y ago

If you're curious about what these papers are, here is a hack at exposing the metadata in an easily searchable format. It would not be hard to scrape the PDFs to searchable text using unix tools (doesn't come out very readable) and render them as SVG files with inkscape for easy browser viewing: http://robocracy.org/royal-search/ http://robocracy.org/royal-search/ https://github.com/bnewbold/royal-search https://github.com/bnewbold/royal-search Lots of good old stuff, such as "On the Double Organs of Generation of the Lamprey, the Conger Eel, the Common Eel, the Barnacle, and Earth Worm, Which Impregnate Themselves; Though the Last from Copulating, Appear Mutually to Impregnate One Another" This is all available elsewhere, i'm just trying to demonstrate how low the barrier to search is with OSS tools; I hope all these papers end up on archive.org with full text search!

dunham 15y ago

Distributed Proofreaders already has some infrastructure set up for proofreading and editing OCR'd documents. It's not quite the same as your proposal, but they do track stats and have levels of participants based on skill and experience. Might be worth checking out. http://www.pgdp.net/c/

qqqqrrrr 15y ago

This is very disappointing. A comment that's advocating theft is voted up to the top of the thread, whereas my comment explaining that this is THEFT is deleted. Shame on you. And I love how the hivemind has used the opportunity to pile up on me. I just want to clarify: I am a publishing academic (a grad student), and I received none of your tax money to publish. In fact, I'm not even American. Just to reiterate: please stop making up excuses for what is essentially STEALING.