11 ms·

How to digitize your lab notebooks

118 points by sohkamyung 6y ago

MaxBarraclough 6y ago
Can anyone recommend a Free and Open Source tool good for this sort of digitizing work? Presumably transforming photos of documents into PDF files with OCR annotations, though I don't see that it has to be PDF. Decent handling of formulae would be a plus. Also, the website's cookie-consent dark-pattern leaves a bad impression.
- atrettel 6y ago
  If you are fine with the command line, then nothing beats some combination of Ghostscript, ImageMagick, and PDFtk. OCR is much more complicated but I have used Tesseract when OCR was needed. The results were mixed at best, sometimes being excellent and sometimes being terrible. It largely seems to be a function of the quality of the scan (how flat the text is, how horizontal the lines are, ...), though even seemingly minor deviations sometimes result in unreadable OCR text.
- M5x7wI3CmbEem10 6y ago
  https://zettel.hackstream.dev https://zettel.hackstream.dev
  - MaxBarraclough 6y ago
    That seems to be a note-taking tool, not a tool for digitising written notes.
- ajot 6y ago
  This guide [0] is intended for digitizing books, but it can lead you to some places. If you can get your scans to a PDF (e.g. with a scanner or a smartphone app), you can always try OCRmyPDF [1] to perform the OCR for you.[2] has some tips, but try to look for an older version in Wayback Machine, as I think it has been condenses for the worse. [0] https://www.natecraun.net/articles/linux-guide-to-book-scanning.html https://www.natecraun.net/articles/linux-guide-to-book-scann... [1] https://ocrmypdf.readthedocs.io/en/latest/ https://ocrmypdf.readthedocs.io/en/latest/ [2] https://colinpurrington.com/tips/lab-notebooks/ https://colinpurrington.com/tips/lab-notebooks/
- davisr 6y ago
  I've written about the tools I use here: http://www.davisr.me/projects/books/ http://www.davisr.me/projects/books/
  - MaxBarraclough 6y ago
    Great write-up, thanks. I'm finding that ctrl-f isn't working though. When I search for Since Apple it doesn't find the match on page 21. I've tried this on both Firefox and Edge (Chromium version). In an ideal world, it would also 'clean up' the text, replacing the image with 'live' text in the appropriate typeface, but that's more a wishlist than a real problem.
- nekopa 6y ago
  I found this gent's articles quite helpful: http://blog.konradvoelkel.de/2010/01/linux-ocr-and-pdf-problem-solved/ http://blog.konradvoelkel.de/2010/01/linux-ocr-and-pdf-probl... http://www.konradvoelkel.com/2013/03/scan-to-pdfa/ http://www.konradvoelkel.com/2013/03/scan-to-pdfa/
avsteele 6y ago
This is an interesting article but I'm as much or more interested in the bigger picture of what to use going forward. I haven't found the killer app for lab notebooks. Any ideas form the community? Requirements: * simple formatting (ala markdown) * ability to share with colleagues * able to simply paste in images of graphs etc.. * versioning, ideally with ability to timestamp * tables * code snippets (with formatting) * links, tags, backlinks versioning is useful to be able to see how knowledge on a subject was updated over time. There is a second level, important to some users though not necessarily me, where physical notebooks use for patents/IP to establish the date of an innovation. OneNote works on most of the above, but: * its search is terrible * its formatting of code is terrible * easy for people to pickup and use I've thought about a note-taking app like Obsidian, together with git. But it isn't easy to share notes. its handling of inserted/pasted images is a little jank.
- dcolkitt 6y ago
  Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?
- andylynch 6y ago
  Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.
  - sleavey 6y ago
    This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.
    - analog31 6y ago
      Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes. Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion. I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook. Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.
peignoir 6y ago
The new ios14 / ipados14 has OCR on notes, works relatively well on my side. I just need to write patiently enough instead of my normal scribble.
lordnacho 6y ago
I don't know how researchers feel about spending over a grand on it, but an iPad pro lets you write in free form, draw diagrams, etc. You can take pictures with it too. You can even connect it to Google Jamboard and then all your stuff is online and shareable. I'd definitely have preferred to have one while I was studying.
- M5x7wI3CmbEem10 6y ago
  thinking about an e-ink tablet to avoid long-term eye damage like supernote, onyx, remarkable
  - criddell 6y ago
    What kind of eye damage are you talking about?
    - M5x7wI3CmbEem10 6y ago
      blue light and front light strain
- simlan 6y ago
  I tried for a while with a convertible laptop and a stylus that goes with it. In the end it was too bulky and when i spill i might kill it. I am back to the paper notebook for bench work. All data and documentation of results goes into Confluence. This hybrid served me well lately.
- criddell 6y ago
  My daughter started at university this fall and she needed a computer. She could have picked anything (within reason) but what she wanted was a big iPad Pro with the keyboard. She's in a life sciences program and so far it's worked perfectly for her.
  - melling 6y ago
    Which note taking app do people use on the iPad? I like the idea of writing notes with the Apple Pencil and converting some of them to text for readability. The converted text also allows me to fit more on a page. Anything allow embedded Latex?
aimor 6y ago
Here's what I did, which works much better than phone apps or scanners. Mount a digital camera on a tripod and bring a couple lamps over for strong uniform lighting. Turn the page, take a picture, turn the page, take a picture. You'll go through a 100 page notebook in 10 minutes (not an exaggeration). There are better quality ways to digitize documents, but I don't know of anything cheaper and faster.
- ericcholis 6y ago
  Even some of the best quality scanners mimic this technique
- greggyb 6y ago
  I don't have a link handy, but this is what Google did when digitizing books for Google Books back in the day. They had a hardware setup that I believe had foot pedals to snap the picture, but human hands are (were?) still the best page turning technology they could muster. There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open.
  - sogen 6y ago
    yeah! I just remembered there were some news articles about hands appearing in some of the scans
  - jhayward 6y ago
    > There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open Yes, they projected a grid (using red light, I believe) on the page so they could compute the curvature to re-project the image as "flattened".
  - rckoepke 6y ago
    Interesting. I had been led to believe that Google used a linear book scanner towards the end of the big Google Books library scanning project. It looks like this may have been an internal prototype that wasn't necessarily used for large-scale scanning opert https://www.youtube.com/watch?v=4JuoOaL11bw&feature=emb_title https://www.youtube.com/watch?v=4JuoOaL11bw&feature=emb_titl... https://hackaday.com/2012/11/16/google-books-team-open-sources-their-book-scanner/ https://hackaday.com/2012/11/16/google-books-team-open-sourc... https://code.google.com/archive/p/linear-book-scanner/ https://code.google.com/archive/p/linear-book-scanner/ https://linearbookscanner.org/ https://linearbookscanner.org/
kxyvr 6y ago
Though it doesn't help with someone trying to keep their notes in a bound notebook, I'm a huge fan of scanning notes and papers and here's my setup: 1. Scan with a Fujitsu fi-6130. This scanner is out of print and I believe the successors are the 7160 and 7180. When I purchased mine, it was somewhere between $900-1000 and I've used it actively for about a decade with zero issues. 2. Use xsane for scanning 3. (Optional) Use scantailor-advanced (https://github.com/4lex4/scantailor-advanced) for any cleanup 4. (Optional) Use tesseract for any ocr (does not work on handwritten notes, but works great for scans of papers, bills, etc.) Generally speaking, I take hand notes on research topics and the collect them till I need to travel or get too many. Then, I scan each small collection into its own file. To make this easier to back walk, all notes are dated, numbered, and given a general topic/title as I write them. Once they are scanned, they go into their own pdf file with a long file name that contains the date, title, and keywords that I want to use to find it. If there's a long thread of thought that needs to cross reference multiple files, I've added that information directly to the hand written note with a comment like (these notes use the following results from this date). While not perfect, I've thousands of pages of notes and I've always been able to find what I've been looking for with little effort. I'm not sure anyone else could look at what I've done and search it as effectively because I kind of know how I think about problems and label things, but it works for me. As a couple of other small tidbits: - It takes too long to individually fix each scan using scantailor. A good scanner with a straight scan is *much* better than fixing things by hand. I will do it if I have to, but I'm disappointed in myself when I do. - Slow scanners will kill this methodology and make things too burdensome. Don't use a flatbed scanner. Get a fast scanner with an autofeeder. They are expensive, so expect about $1000. That said, I've scanned tens of thousands of documents on mine and it's never failed, so a good scanner is worth the money in my opinion. - I have thought about using a tablet like remarkable to take notes. My issue is that since I already own the scanner and paper notes are so incredibly robust especially when I'm on travel, I've been unable to justify the money. Though, annotating pdfs may be nice and a scanner doesn't help with that unless we want to print, markup, and then rescan, which is a little wasteful.
- davisr 6y ago
  Hi, You'd probably like my site. I discuss both book scanning (using a Fujitsu fi-4120c with SANE)[1] and reMarkable hacking.[2][3] I've also written a popular GUI client for transmitting reMarkable documents without the manufacturer's cloud (sold under GPLv3+) that many people find useful in their workflows.[4] I used to take notes on paper and scan them, but after getting a reMarkable, it became just so much more convenient. I now use it for 100% of my reading and writing. [1]: http://www.davisr.me/projects/books/ http://www.davisr.me/projects/books/ [2]: http://www.davisr.me/projects/remarkable-microsd/ http://www.davisr.me/projects/remarkable-microsd/ [3]: http://www.davisr.me/projects/parabola-rm/ http://www.davisr.me/projects/parabola-rm/ [4]: http://www.davisr.me/projects/rcu/ http://www.davisr.me/projects/rcu/
_understood_ 6y ago
Curious if there has been more adoption of these in the lab environment as well: https://youtu.be/-djt0IA4s0w https://youtu.be/-djt0IA4s0w
johannesboyne 6y ago
One super interesting startup in this sector: https://www.labtwin.com/ https://www.labtwin.com/ they completely automated lab reporting and made it voice-first

118 points by sohkamyung 6y ago

Can anyone recommend a Free and Open Source tool good for this sort of digitizing work? Presumably transforming photos of documents into PDF files with OCR annotations, though I don't see that it has to be PDF. Decent handling of formulae would be a plus. Also, the website's cookie-consent dark-pattern leaves a bad impression.

atrettel 6y ago
If you are fine with the command line, then nothing beats some combination of Ghostscript, ImageMagick, and PDFtk. OCR is much more complicated but I have used Tesseract when OCR was needed. The results were mixed at best, sometimes being excellent and sometimes being terrible. It largely seems to be a function of the quality of the scan (how flat the text is, how horizontal the lines are, ...), though even seemingly minor deviations sometimes result in unreadable OCR text.
M5x7wI3CmbEem10 6y ago
https://zettel.hackstream.dev https://zettel.hackstream.dev
- MaxBarraclough 6y ago
  That seems to be a note-taking tool, not a tool for digitising written notes.
ajot 6y ago
This guide [0] is intended for digitizing books, but it can lead you to some places. If you can get your scans to a PDF (e.g. with a scanner or a smartphone app), you can always try OCRmyPDF [1] to perform the OCR for you.[2] has some tips, but try to look for an older version in Wayback Machine, as I think it has been condenses for the worse. [0] https://www.natecraun.net/articles/linux-guide-to-book-scanning.html https://www.natecraun.net/articles/linux-guide-to-book-scann... [1] https://ocrmypdf.readthedocs.io/en/latest/ https://ocrmypdf.readthedocs.io/en/latest/ [2] https://colinpurrington.com/tips/lab-notebooks/ https://colinpurrington.com/tips/lab-notebooks/
davisr 6y ago
I've written about the tools I use here: http://www.davisr.me/projects/books/ http://www.davisr.me/projects/books/
- MaxBarraclough 6y ago
  Great write-up, thanks. I'm finding that ctrl-f isn't working though. When I search for Since Apple it doesn't find the match on page 21. I've tried this on both Firefox and Edge (Chromium version). In an ideal world, it would also 'clean up' the text, replacing the image with 'live' text in the appropriate typeface, but that's more a wishlist than a real problem.
nekopa 6y ago
I found this gent's articles quite helpful: http://blog.konradvoelkel.de/2010/01/linux-ocr-and-pdf-problem-solved/ http://blog.konradvoelkel.de/2010/01/linux-ocr-and-pdf-probl... http://www.konradvoelkel.com/2013/03/scan-to-pdfa/ http://www.konradvoelkel.com/2013/03/scan-to-pdfa/

avsteele 6y ago

This is an interesting article but I'm as much or more interested in the bigger picture of what to use going forward. I haven't found the killer app for lab notebooks. Any ideas form the community? Requirements: * simple formatting (ala markdown) * ability to share with colleagues * able to simply paste in images of graphs etc.. * versioning, ideally with ability to timestamp * tables * code snippets (with formatting) * links, tags, backlinks versioning is useful to be able to see how knowledge on a subject was updated over time. There is a second level, important to some users though not necessarily me, where physical notebooks use for patents/IP to establish the date of an innovation. OneNote works on most of the above, but: * its search is terrible * its formatting of code is terrible * easy for people to pickup and use I've thought about a note-taking app like Obsidian, together with git. But it isn't easy to share notes. its handling of inserted/pasted images is a little jank.

dcolkitt 6y ago
Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?
andylynch 6y ago
Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.
- sleavey 6y ago
  This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.
  - analog31 6y ago
    Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes. Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion. I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook. Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.

peignoir 6y ago

The new ios14 / ipados14 has OCR on notes, works relatively well on my side. I just need to write patiently enough instead of my normal scribble.

lordnacho 6y ago

I don't know how researchers feel about spending over a grand on it, but an iPad pro lets you write in free form, draw diagrams, etc. You can take pictures with it too. You can even connect it to Google Jamboard and then all your stuff is online and shareable. I'd definitely have preferred to have one while I was studying.

M5x7wI3CmbEem10 6y ago
thinking about an e-ink tablet to avoid long-term eye damage like supernote, onyx, remarkable
- criddell 6y ago
  What kind of eye damage are you talking about?
  - M5x7wI3CmbEem10 6y ago
    blue light and front light strain
simlan 6y ago
I tried for a while with a convertible laptop and a stylus that goes with it. In the end it was too bulky and when i spill i might kill it. I am back to the paper notebook for bench work. All data and documentation of results goes into Confluence. This hybrid served me well lately.
criddell 6y ago
My daughter started at university this fall and she needed a computer. She could have picked anything (within reason) but what she wanted was a big iPad Pro with the keyboard. She's in a life sciences program and so far it's worked perfectly for her.
- melling 6y ago
  Which note taking app do people use on the iPad? I like the idea of writing notes with the Apple Pencil and converting some of them to text for readability. The converted text also allows me to fit more on a page. Anything allow embedded Latex?

aimor 6y ago

Here's what I did, which works much better than phone apps or scanners. Mount a digital camera on a tripod and bring a couple lamps over for strong uniform lighting. Turn the page, take a picture, turn the page, take a picture. You'll go through a 100 page notebook in 10 minutes (not an exaggeration). There are better quality ways to digitize documents, but I don't know of anything cheaper and faster.

ericcholis 6y ago
Even some of the best quality scanners mimic this technique
greggyb 6y ago
I don't have a link handy, but this is what Google did when digitizing books for Google Books back in the day. They had a hardware setup that I believe had foot pedals to snap the picture, but human hands are (were?) still the best page turning technology they could muster. There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open.
- sogen 6y ago
  yeah! I just remembered there were some news articles about hands appearing in some of the scans
- jhayward 6y ago
  > There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open Yes, they projected a grid (using red light, I believe) on the page so they could compute the curvature to re-project the image as "flattened".
- rckoepke 6y ago
  Interesting. I had been led to believe that Google used a linear book scanner towards the end of the big Google Books library scanning project. It looks like this may have been an internal prototype that wasn't necessarily used for large-scale scanning opert https://www.youtube.com/watch?v=4JuoOaL11bw&feature=emb_title https://www.youtube.com/watch?v=4JuoOaL11bw&feature=emb_titl... https://hackaday.com/2012/11/16/google-books-team-open-sources-their-book-scanner/ https://hackaday.com/2012/11/16/google-books-team-open-sourc... https://code.google.com/archive/p/linear-book-scanner/ https://code.google.com/archive/p/linear-book-scanner/ https://linearbookscanner.org/ https://linearbookscanner.org/

kxyvr 6y ago

Though it doesn't help with someone trying to keep their notes in a bound notebook, I'm a huge fan of scanning notes and papers and here's my setup: 1. Scan with a Fujitsu fi-6130. This scanner is out of print and I believe the successors are the 7160 and 7180. When I purchased mine, it was somewhere between $900-1000 and I've used it actively for about a decade with zero issues. 2. Use xsane for scanning 3. (Optional) Use scantailor-advanced (https://github.com/4lex4/scantailor-advanced) for any cleanup 4. (Optional) Use tesseract for any ocr (does not work on handwritten notes, but works great for scans of papers, bills, etc.) Generally speaking, I take hand notes on research topics and the collect them till I need to travel or get too many. Then, I scan each small collection into its own file. To make this easier to back walk, all notes are dated, numbered, and given a general topic/title as I write them. Once they are scanned, they go into their own pdf file with a long file name that contains the date, title, and keywords that I want to use to find it. If there's a long thread of thought that needs to cross reference multiple files, I've added that information directly to the hand written note with a comment like (these notes use the following results from this date). While not perfect, I've thousands of pages of notes and I've always been able to find what I've been looking for with little effort. I'm not sure anyone else could look at what I've done and search it as effectively because I kind of know how I think about problems and label things, but it works for me. As a couple of other small tidbits: - It takes too long to individually fix each scan using scantailor. A good scanner with a straight scan is *much* better than fixing things by hand. I will do it if I have to, but I'm disappointed in myself when I do. - Slow scanners will kill this methodology and make things too burdensome. Don't use a flatbed scanner. Get a fast scanner with an autofeeder. They are expensive, so expect about $1000. That said, I've scanned tens of thousands of documents on mine and it's never failed, so a good scanner is worth the money in my opinion. - I have thought about using a tablet like remarkable to take notes. My issue is that since I already own the scanner and paper notes are so incredibly robust especially when I'm on travel, I've been unable to justify the money. Though, annotating pdfs may be nice and a scanner doesn't help with that unless we want to print, markup, and then rescan, which is a little wasteful.

davisr 6y ago
Hi, You'd probably like my site. I discuss both book scanning (using a Fujitsu fi-4120c with SANE)[1] and reMarkable hacking.[2][3] I've also written a popular GUI client for transmitting reMarkable documents without the manufacturer's cloud (sold under GPLv3+) that many people find useful in their workflows.[4] I used to take notes on paper and scan them, but after getting a reMarkable, it became just so much more convenient. I now use it for 100% of my reading and writing. [1]: http://www.davisr.me/projects/books/ http://www.davisr.me/projects/books/ [2]: http://www.davisr.me/projects/remarkable-microsd/ http://www.davisr.me/projects/remarkable-microsd/ [3]: http://www.davisr.me/projects/parabola-rm/ http://www.davisr.me/projects/parabola-rm/ [4]: http://www.davisr.me/projects/rcu/ http://www.davisr.me/projects/rcu/

_understood_ 6y ago

Curious if there has been more adoption of these in the lab environment as well: https://youtu.be/-djt0IA4s0w https://youtu.be/-djt0IA4s0w

johannesboyne 6y ago

One super interesting startup in this sector: https://www.labtwin.com/ https://www.labtwin.com/ they completely automated lab reporting and made it voice-first