5 ms·

I miss it as well, as the author points out it isn't large and is easily contained on a thumbnail flash drive (several times over!). I've started creating an o

I miss it as well, as the author points out it isn't large and is easily contained on a thumbnail flash drive (several times over!). I've started creating an offline reference library[1] because I see that search engines will be over whelmed with crap and frankly I don't want to have to find cell service to ask my handheld device to locate a factoid. If you've read that now more than 50% of Google desktop searches don't result in a click, it is because Google is getting better at 'one boxing' the answer so you don't need to click through to read it. [1] That project started by digitizing my referenced by less often used text books. Which lead to me digitizing text books for others as well. After Google's win on digitizing books is fair use for air cover much of the work applies NLP and a bit of machine learning to pull out facts which are not, in themselves copyrightable. [2] "Google Wins Copyright Suit" --- https://www.washingtonpost.com/news/the-switch/wp/2016/04/18/google-books-just-won-a-decade-long-copyright-fight/?noredirect=on https://www.washingtonpost.com/news/the-switch/wp/2016/04/18...

dman 7y ago
How do I get a peek into this world according to Chuck slice of cyberspace?
- ChuckMcM 7y ago
  An interesting question that I have yet to have a good answer for. The fair use exemption was pretty good air cover for using the digitized data, but text book publishers are rather litigious bunch. Not a bear I want to poke just yet.
  - dman 7y ago
    When you figure it out, please have my name at top of the beta users list.
shortformblog 7y ago
This sounds amazing. I know it’s a personal project, but you should seriously share it with more people at some point.
- ChuckMcM 7y ago
  The idea came from the fact that I wanted to play with the NLP work and wanted more useful data sets to start from (parsing Shakespeare gets old :-). And the fact that my fairly simple search engine for PDFs was basically just a classic n-tuple index into the word clouds minus stop words and gave good recall (like the old AltaVista) but not good precision. Another comment in this thread about having a tablet to do that suggests an interesting way to deploy something like this on inexpensive table hardware with a somewhat more bespoke (aka performant) OS and application architecture for appliance type operation. (think running a unikernel rather than a general purpose kernel in order to not spend cycles on things you won't ever need)
floren 7y ago
What technique are you using to digitize these books? I'd be interested in digitizing a few books I have.
- ChuckMcM 7y ago
  I have a Fujitsu Scansnap 1500 and a heavy duty paper guillotine (basically a very sharp, very tall blade, kind of like this one on Amazon: https://www.amazon.com/HFS-Heavy-Guillotine-Paper-Cutter/dp/B00NO7PYYI https://www.amazon.com/HFS-Heavy-Guillotine-Paper-Cutter/dp/...). The ScanSnap came with a license to Abbyy's OCR software (which is good for words but it makes for interesting jumbles when you have tables and of course no math recognition). And the Foxit PDF editor version. Since most of the text books I've scanned are hard covers I cut off the covers, scan those on flatbed scanner, then disassemble the binding to remove the pages. Cut off the margins to get to something closer to 8.5" wide (more modern text books have been "experimenting" with page sizes). Disassembly takes about 20 minutes, scanning takes another, 10 minutes depending on how many sets I need to scan (the paper feeder for the ScanSnap is nominally 75 pages, but if it gets overloaded can misfeed). Occasionally I'll get a misfeed (two pages feed at the same time) and I generally clean the paper handling belts every 10 or 15 books because they get accumulation that make them less effective at gripping the paper.
  - floren 7y ago
    Ah, I should have guessed it would be a destructive process. Still something I'll keep in mind but I have a particular weakness about destroying old books, even if intellectually I understand 1) it's not a valuable book and 2) there are other copies out there and 3) it's worth more to me in PDF form.
    - kragen 7y ago
      The process used by the Internet Archive and Google for digitizing books is nondestructive; the Archive open-sourced the plans for their "Scribe" book-scanning machine, and others have made downloadable plans for laser-cuttable versions. It's slower than running a destroyed book through a sheeet-fed scanner (typically you get on the order of ten two-page scans per minute), but it's suitable even for rare books.

dman 7y ago

How do I get a peek into this world according to Chuck slice of cyberspace?

ChuckMcM 7y ago
An interesting question that I have yet to have a good answer for. The fair use exemption was pretty good air cover for using the digitized data, but text book publishers are rather litigious bunch. Not a bear I want to poke just yet.
- dman 7y ago
  When you figure it out, please have my name at top of the beta users list.

shortformblog 7y ago

This sounds amazing. I know it’s a personal project, but you should seriously share it with more people at some point.

ChuckMcM 7y ago
The idea came from the fact that I wanted to play with the NLP work and wanted more useful data sets to start from (parsing Shakespeare gets old :-). And the fact that my fairly simple search engine for PDFs was basically just a classic n-tuple index into the word clouds minus stop words and gave good recall (like the old AltaVista) but not good precision. Another comment in this thread about having a tablet to do that suggests an interesting way to deploy something like this on inexpensive table hardware with a somewhat more bespoke (aka performant) OS and application architecture for appliance type operation. (think running a unikernel rather than a general purpose kernel in order to not spend cycles on things you won't ever need)

floren 7y ago

What technique are you using to digitize these books? I'd be interested in digitizing a few books I have.

ChuckMcM 7y ago
I have a Fujitsu Scansnap 1500 and a heavy duty paper guillotine (basically a very sharp, very tall blade, kind of like this one on Amazon: https://www.amazon.com/HFS-Heavy-Guillotine-Paper-Cutter/dp/B00NO7PYYI https://www.amazon.com/HFS-Heavy-Guillotine-Paper-Cutter/dp/...). The ScanSnap came with a license to Abbyy's OCR software (which is good for words but it makes for interesting jumbles when you have tables and of course no math recognition). And the Foxit PDF editor version. Since most of the text books I've scanned are hard covers I cut off the covers, scan those on flatbed scanner, then disassemble the binding to remove the pages. Cut off the margins to get to something closer to 8.5" wide (more modern text books have been "experimenting" with page sizes). Disassembly takes about 20 minutes, scanning takes another, 10 minutes depending on how many sets I need to scan (the paper feeder for the ScanSnap is nominally 75 pages, but if it gets overloaded can misfeed). Occasionally I'll get a misfeed (two pages feed at the same time) and I generally clean the paper handling belts every 10 or 15 books because they get accumulation that make them less effective at gripping the paper.
- floren 7y ago
  Ah, I should have guessed it would be a destructive process. Still something I'll keep in mind but I have a particular weakness about destroying old books, even if intellectually I understand 1) it's not a valuable book and 2) there are other copies out there and 3) it's worth more to me in PDF form.
  - kragen 7y ago
    The process used by the Internet Archive and Google for digitizing books is nondestructive; the Archive open-sourced the plans for their "Scribe" book-scanning machine, and others have made downloadable plans for laser-cuttable versions. It's slower than running a destroyed book through a sheeet-fed scanner (typically you get on the order of ten two-page scans per minute), but it's suitable even for rare books.