8 ms·

Nice to provide hardware hints and designs but geez that is almost the least of it. Cleverest hardware still only gets you a thumb drive full of page images. No

by fernly 5y ago

Nice to provide hardware hints and designs but geez that is almost the least of it. Cleverest hardware still only gets you a thumb drive full of page images. Now what? There needs to be a software workflow ending with a readable book in PDF, EBOOK or MOBI format, and there are many, many choices to be made along that path. Edit: "Finishing a book" is discussed at a very superficial level here: https://vimeo.com/user33752051 https://vimeo.com/user33752051 at about 1:00: "In order to turn these raw images into an ebook, the very minimum you need to do is A, you need to rotate them, B you need to crop them down to use the page [?], and C you need to combine them into one document like a PDF... You can do OCR to make it searchable ... color correction... de-skewing, de-warping ..."

walrus01 5y ago
The simplest of which would be to turn the images into a multi page raster PDF, using freely licensed linux based command line tools for PDF generation. Which will of course result in a rather large file size vs doing OCR, but might be the best preservation method for books with illustrations, unusual fonts, catalogs, mixed text and photos, etc. I am not clear on to what extent the existing workflow does a de-skew of the camera images to deal with page curvature towards the spine. I think I recall the Internet Archive having an open source design for something similar to this? And other projects which accomplish generally the same idea.
BeetleB 5y ago
Back in 2012, there was a guy who started an open source project that did exactly this - he wrote it specifically for the DIY Book scanner. It had a local Django project as the interface. I don't remember the details, but it did a decent job of taking the images, OCR'ing them and creating an output PDF. I believe he abandoned the project some years later as life got busy and he never found enough volunteers to help him. Would have to go through my email records to find the name of the project.
- fernly 5y ago
  Not the same as Scan Tailor[1,2] ? Which was referenced from the Instructables link cited earlier. That apparently was a comprehensive toolkit in C++ and Qt, now archived. [1] https://scantailor.org/ https://scantailor.org/ [2] https://web.archive.org/web/20210304015939/https://github.com/scantailor/scantailor https://web.archive.org/web/20210304015939/https://github.co...
  - jccalhoun 5y ago
    There are a couple folks that forked scantailor. I'm not sure the status of those. Here are a couple: https://github.com/4lex4/scantailor-advanced https://github.com/4lex4/scantailor-advanced https://github.com/trufanov-nok/scantailor-universal https://github.com/trufanov-nok/scantailor-universal
- markvdb 5y ago
  Spreads[0] is probably what you are referring to. Some backstory: I saw the diybookscanner community - which at that point mostly had Daniel Reetz [1] as its active contributor- struggle with mechanical contraptions for triggering cameras and very little software experience. I built a simple proof of concept to reliably trigger cheap consumer cameras using software. I built it on CHDK[2], the Canon Hack Development Kit, alternative firmware for cheap consumer cameras. The proof of concept worked. I then had a fairly large number of book scanner kits built and shipped mostly around the EU [3]. More of a work of love than a business really, even if it was formally under an llc umbrella. Johannes initially was just a customer. He wanted to build a better software solution, and within the spirit of the project did so as free software. I tried to support him at this as well as I could, setting up build infrastructure, trying to reel in more people, getting him some cameras to test, get the amazing CHDK people to port to new camera models, ... Then real life intervened indeed. Johannes, if you read this, I'm still grateful for the experience of having worked with a great developer like you! [EDIT] And of course, I should also mention Dan Reetz' incredibly inspiring work bootstrapping an incredible open hardware project! Hats off! [0] https://github.com/DIYBookScanner/spreads https://github.com/DIYBookScanner/spreads [1] https://danreetz.com/ https://danreetz.com/ [2] https://chdk.fandom.com/wiki/CHDK https://chdk.fandom.com/wiki/CHDK [3] http://diybookscanner.eu http://diybookscanner.eu
  - BeetleB 5y ago
    Actually, it was a project called Paper Upgrade. Here is an old archive link: http://web.archive.org/web/20140101000000*/http://www.paperupgrade.org/ http://web.archive.org/web/20140101000000*/http://www.paperu... I don't know if you can find the code through there, but I'm pretty sure he had made it free. I think spreads is a bit newer. Edit: Found some more info. It did indeed use Scantailor in the backend. His SW was more of a Web based frontend to all the parts. You can see a video demo of it here: https://www.youtube.com/watch?v=Ad7aFYdbDos https://www.youtube.com/watch?v=Ad7aFYdbDos Start at about 4:40. The source is here: https://code.google.com/archive/p/diy-ebook-creator/ https://code.google.com/archive/p/diy-ebook-creator/
    - daniel_reetz 5y ago
rahimnathwani 5y ago
https://www.instructables.com/Bargain-Price-Book-Scanner-From-A-Cardboard-Box/ https://www.instructables.com/Bargain-Price-Book-Scanner-Fro... "Step 10 - Post-processing" has some steps
jccalhoun 5y ago
scantailor will get you most of the way there. the original project is dead but there are a few forks on github. It has been a while since I did any serious scanning so I can't remember which version I used. https://github.com/4lex4/scantailor-advanced https://github.com/4lex4/scantailor-advanced https://github.com/trufanov-nok/scantailor-universal https://github.com/trufanov-nok/scantailor-universal
- Mediterraneo10 5y ago
  I scan heavily from academic libraries in order to contribute to LibGen, but even with Scantailor it is very time-consuming. For example, if you are scanning scientific literature from the Eastern Bloc, it was often printed on low-quality, speckled paper, which means Scantailor often identifies too much of the scan as the page block, and then you have to manually tweak the rectangle.
sandeep1338 5y ago
Video looks interesting, I'll check it out!
deleted 5y ago
[deleted]
vixen99 5y ago
Just page images? No, Czur software with its OCR generates searchable pdfs and Word or Excel files with no further input. With careful attention to the scanned area, it's easy to get .xlsx files needing zero or minimal editing. The other advantage of the Czur is the automatic correction for curvature when scanning books with narrow margins on either side of the spine. No, I have no connection with Czur - just an enthusiastic user!
hackeyed 5y ago
Right, step 1 -> get page images, step 2 -> author images into book file. While OCR is obviously useful for search, a rotated phone screen will let you comfortably read a pdf book just fine unless you are talking about something like a textbook, in which case you probably wanted a tablet anyway. I wrote up a guide on the authoring process using FOSS tools for some Digital Humanities folks a couple years ago: https://github.com/wikey/bookscan https://github.com/wikey/bookscan It gives some background on the problem and covers a Scantailor (page crop, rotate, deskew), pdfbeads (compression, book metadata) authoring workflow, with pdftk for some general odds and ends.
ajot 5y ago
This is an old article, so maybe some software isn't the best option nowadays, but you can get the idea of postprocessing: http://natecraun.net/articles/linux-guide-to-book-scanning.html http://natecraun.net/articles/linux-guide-to-book-scanning.h...

by fernly 5y ago

walrus01 5y ago

The simplest of which would be to turn the images into a multi page raster PDF, using freely licensed linux based command line tools for PDF generation. Which will of course result in a rather large file size vs doing OCR, but might be the best preservation method for books with illustrations, unusual fonts, catalogs, mixed text and photos, etc. I am not clear on to what extent the existing workflow does a de-skew of the camera images to deal with page curvature towards the spine. I think I recall the Internet Archive having an open source design for something similar to this? And other projects which accomplish generally the same idea.

BeetleB 5y ago

Back in 2012, there was a guy who started an open source project that did exactly this - he wrote it specifically for the DIY Book scanner. It had a local Django project as the interface. I don't remember the details, but it did a decent job of taking the images, OCR'ing them and creating an output PDF. I believe he abandoned the project some years later as life got busy and he never found enough volunteers to help him. Would have to go through my email records to find the name of the project.

fernly 5y ago
Not the same as Scan Tailor[1,2] ? Which was referenced from the Instructables link cited earlier. That apparently was a comprehensive toolkit in C++ and Qt, now archived. [1] https://scantailor.org/ https://scantailor.org/ [2] https://web.archive.org/web/20210304015939/https://github.com/scantailor/scantailor https://web.archive.org/web/20210304015939/https://github.co...
- jccalhoun 5y ago
  There are a couple folks that forked scantailor. I'm not sure the status of those. Here are a couple: https://github.com/4lex4/scantailor-advanced https://github.com/4lex4/scantailor-advanced https://github.com/trufanov-nok/scantailor-universal https://github.com/trufanov-nok/scantailor-universal
markvdb 5y ago
Spreads[0] is probably what you are referring to. Some backstory: I saw the diybookscanner community - which at that point mostly had Daniel Reetz [1] as its active contributor- struggle with mechanical contraptions for triggering cameras and very little software experience. I built a simple proof of concept to reliably trigger cheap consumer cameras using software. I built it on CHDK[2], the Canon Hack Development Kit, alternative firmware for cheap consumer cameras. The proof of concept worked. I then had a fairly large number of book scanner kits built and shipped mostly around the EU [3]. More of a work of love than a business really, even if it was formally under an llc umbrella. Johannes initially was just a customer. He wanted to build a better software solution, and within the spirit of the project did so as free software. I tried to support him at this as well as I could, setting up build infrastructure, trying to reel in more people, getting him some cameras to test, get the amazing CHDK people to port to new camera models, ... Then real life intervened indeed. Johannes, if you read this, I'm still grateful for the experience of having worked with a great developer like you! [EDIT] And of course, I should also mention Dan Reetz' incredibly inspiring work bootstrapping an incredible open hardware project! Hats off! [0] https://github.com/DIYBookScanner/spreads https://github.com/DIYBookScanner/spreads [1] https://danreetz.com/ https://danreetz.com/ [2] https://chdk.fandom.com/wiki/CHDK https://chdk.fandom.com/wiki/CHDK [3] http://diybookscanner.eu http://diybookscanner.eu
- BeetleB 5y ago
  Actually, it was a project called Paper Upgrade. Here is an old archive link: http://web.archive.org/web/20140101000000*/http://www.paperupgrade.org/ http://web.archive.org/web/20140101000000*/http://www.paperu... I don't know if you can find the code through there, but I'm pretty sure he had made it free. I think spreads is a bit newer. Edit: Found some more info. It did indeed use Scantailor in the backend. His SW was more of a Web based frontend to all the parts. You can see a video demo of it here: https://www.youtube.com/watch?v=Ad7aFYdbDos https://www.youtube.com/watch?v=Ad7aFYdbDos Start at about 4:40. The source is here: https://code.google.com/archive/p/diy-ebook-creator/ https://code.google.com/archive/p/diy-ebook-creator/
  - daniel_reetz 5y ago

rahimnathwani 5y ago

https://www.instructables.com/Bargain-Price-Book-Scanner-From-A-Cardboard-Box/ https://www.instructables.com/Bargain-Price-Book-Scanner-Fro... "Step 10 - Post-processing" has some steps

jccalhoun 5y ago

scantailor will get you most of the way there. the original project is dead but there are a few forks on github. It has been a while since I did any serious scanning so I can't remember which version I used. https://github.com/4lex4/scantailor-advanced https://github.com/4lex4/scantailor-advanced https://github.com/trufanov-nok/scantailor-universal https://github.com/trufanov-nok/scantailor-universal

Mediterraneo10 5y ago
I scan heavily from academic libraries in order to contribute to LibGen, but even with Scantailor it is very time-consuming. For example, if you are scanning scientific literature from the Eastern Bloc, it was often printed on low-quality, speckled paper, which means Scantailor often identifies too much of the scan as the page block, and then you have to manually tweak the rectangle.

sandeep1338 5y ago

Video looks interesting, I'll check it out!

deleted 5y ago

[deleted]

vixen99 5y ago

Just page images? No, Czur software with its OCR generates searchable pdfs and Word or Excel files with no further input. With careful attention to the scanned area, it's easy to get .xlsx files needing zero or minimal editing. The other advantage of the Czur is the automatic correction for curvature when scanning books with narrow margins on either side of the spine. No, I have no connection with Czur - just an enthusiastic user!

hackeyed 5y ago

Right, step 1 -> get page images, step 2 -> author images into book file. While OCR is obviously useful for search, a rotated phone screen will let you comfortably read a pdf book just fine unless you are talking about something like a textbook, in which case you probably wanted a tablet anyway. I wrote up a guide on the authoring process using FOSS tools for some Digital Humanities folks a couple years ago: https://github.com/wikey/bookscan https://github.com/wikey/bookscan It gives some background on the problem and covers a Scantailor (page crop, rotate, deskew), pdfbeads (compression, book metadata) authoring workflow, with pdftk for some general odds and ends.

ajot 5y ago

This is an old article, so maybe some software isn't the best option nowadays, but you can get the idea of postprocessing: http://natecraun.net/articles/linux-guide-to-book-scanning.html http://natecraun.net/articles/linux-guide-to-book-scanning.h...