5 ms·

This is an interesting article but I'm as much or more interested in the bigger picture of what to use going forward. I haven't found the killer app for lab no

This is an interesting article but I'm as much or more interested in the bigger picture of what to use going forward. I haven't found the killer app for lab notebooks. Any ideas form the community? Requirements: * simple formatting (ala markdown) * ability to share with colleagues * able to simply paste in images of graphs etc.. * versioning, ideally with ability to timestamp * tables * code snippets (with formatting) * links, tags, backlinks versioning is useful to be able to see how knowledge on a subject was updated over time. There is a second level, important to some users though not necessarily me, where physical notebooks use for patents/IP to establish the date of an innovation. OneNote works on most of the above, but: * its search is terrible * its formatting of code is terrible * easy for people to pickup and use I've thought about a note-taking app like Obsidian, together with git. But it isn't easy to share notes. its handling of inserted/pasted images is a little jank.

dcolkitt 6y ago
Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?
andylynch 6y ago
Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.
- sleavey 6y ago
  This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.
  - analog31 6y ago
    Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes. Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion. I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook. Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.
- easton 6y ago
  Wouldn’t git with signed commits work just as well? You’d have to find a good app that saved stuff as text (not Word, but probably not LaTeX either since that takes time to get the formatting right), but once you figured that out, you’d have a full history of all changes made.
  - andylynch 6y ago
    Yup - that's probably a good equivalent.
  - dkarl 6y ago
    Signed commits still let you falsify your own work after the fact. You need an external trusted service, a digital notary essentially, that would provide a signed record that you submitted a commit on a certain date with certain set of ancestor commits. Each time you added an entry, you would submit the hashes of all the entries in the book to the digital notary and get a signed record that you submitted those commits on that date. Inability to produce this record would be similar to having erasures or pages torn out in a lab book.
    - homarp 6y ago
      a public blockchain would do for the digital notary? or hash posted on twitter?
    - dmurray 6y ago
      You can do this in a pretty cheap and low-tech way by publishing the hash of your current state somewhere that will keep a permanent record. E.g. the Bitcoin blockchain or a classified ad in a local newspaper.
    - yodon 6y ago
      This. True lab notebooks are not just personal notes, they are a legal record. Not every page of every lab notebook will matter, but it absolutely happens and is one of the many real use cases for keeping a lab notebook. If you invent something or discover something of significance, you will want this. Paper notebooks have pages stitched together and ink that can be run through chemical analysis to see if all the text on the page was written in a single session by the same pen. Forensic examiners have about 100 years of experience assessing paper documents. Digital systems can provide legal grade archival support but a sweet front end editor with copy paste and markdown support does not imply anything about traceability of the contents.
- Wohlf 6y ago
  Interesting, what about the fact people can just not write something in it?
doctorwhat 6y ago
I believe eLabFTW (https://github.com/elabftw/elabftw https://github.com/elabftw/elabftw) would fit ALL your requirements.
sleavey 6y ago
Over the past few years I've developed a WordPress plugin that provides most (all, I think) of what you want [1]. I also a while back did a comparison of some other softwares' pros and cons [2]. At my work we've used my system for about 2 years now and there is much more engagement with the system than the one it replaced (a custom Lotus Notes application... go figure). People like that they can open the logbook from the lab, write some stuff, save it and edit/publish it later. We also heavily encourage to document anything and everything and use the categories and tags as much as possible to make searching easier. [1] https://github.com/Academic-Labbook/alp https://github.com/Academic-Labbook/alp [2] https://alp.attackllama.com/2019/02/05/labbook-software-shootout/ https://alp.attackllama.com/2019/02/05/labbook-software-shoo...
willis936 6y ago
My university has supplied labs with labarchives (and has guaranteed if it ever migrates away from labarchives it will provide an equivalent solution and handle migration). It ticks all of those boxes. My only gripes are minor. Things like no embedding of vector graphics and no good embedded spreadsheet solution. They have some MS office browser based solution that handles version control. That is the official best spreadsheet solution right now. I generated an html spreadsheet for run logs because I wanted an embedded setup. I ran into the limitation of 64K characters per entry. It’s handled by trimming html off the end then autofixing broken stuff. It works so well that it took me a while to figure out why the number of rows was decreasing as more rows were filled out. It’s all very tidy once enough people are on board. Whole notebook (and subsections) can be exported to html or PDF. There is even a way to have some robotic file uploading a la directory monitoring.
KyleOS 6y ago
Have you ever considered Kyso (https://kyso.io https://kyso.io) (disclaimer - I'm a co-founder)? I could be way off, and it would be a corner case for us right now, but it seems we'd fit all your requirements: * We sit on top of git, which remains the single source of truth for your code - versioning, reproducibility, etc. * Python & R notebooks or simple markdown (also on the web app itself). * Show or hide code. * Simple links for sharing reports. * Tagging. * Comprehensive search - by tag, keyword and content (soon). Would be really interesting to hear your thoughts on the fit.
ofou 6y ago
Markdown Enhanced for VSCode[1] + Git for versioning. \ [1]: https://shd101wyy.github.io/markdown-preview-enhanced/ https://shd101wyy.github.io/markdown-preview-enhanced/
guiambros 6y ago
If you don't mind relying on a third-party, Notion [1] has pretty much all of your requirements. Formatting is a pleasure (including formulas and LaTeX), tables are decent, you can easily share notes, etc. Code formatting can improve a lot (only basic language semantics/operators/strings); search is decent. Above all: it's incredibly simple to use, and very customizable. My three big missing features: 1) full API, so you can develop extensions that provide the missing functionality, 2) end-to-end encryption, 3) 2FA. For now I'm using only for study lab notes, so encryption is not a big deal. But I wouldn't feel comfortable adding anything confidential without a more decent security model. [1] https://www.notion.so/ https://www.notion.so/
- M5x7wI3CmbEem10 6y ago
  but it's a proprietary format and not stored locally, unlike obsidian
  - guiambros 6y ago
    Re: not storing locally, yes, hence my first comment on relying on a 3rd party. It depends on what is your intended use, given Notion doesn't have e2e encryption yet. Re: proprietary, it exports natively in Markdown (including a zip with all images), or pdf, or html. It's not perfect (there's a bug when exporting some LaTeX formulas in .md), but for the most part it works well.
MereInterest 6y ago
When I was in the sciences, I used a LaTeX document as my labbook * Ability to share with colleagues - Easy to send pdfs or the raw LaTeX to others. * Able to simply past in images - I had a folder with dated images, and a macro to insert a graph at the current location (better than floats for labbooks). * Versioning - Each section had a date in it. Sequential versions were stored in git * Tables - `\begin{tabular}` * Code snippets - `\begin{lstlisting}` * Links, tags, backlinks - `\ref{section_name}` It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".
- privong 6y ago
  > It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple". One possible way of harnessing the power of LaTeX while keeping a simple formatting would be to use use `pandoc` and pandoc-flavored markdown. That keeps the formatting simple, while also making it easy to include references, equations, etc. and easy output to LaTeX/PDF/etc.
vnxli 6y ago
Why are so many people interested in markdown for notes? I've never used it before but HN seems really interested in note taking and in using markdown for note formatting. What does it offer that OneNote or Word doesnt?
- sq_ 6y ago
  I think the difference is probably that markdown is a pretty transparent and versatile way to format things, while still being simpler than many other markup/formatting languages. With Word, you might insert an image or try to add bullets to something and end up with your whole document looking completely different, whereas, with markdown, you know that if you put in a ">" for a quote, all that's going to change is that one spot.
  - dkdk8283 6y ago
    I think it’s formatting as code vs WYSIWYG.
    - sq_ 6y ago
      Good point. Looking back at my comment, that's basically what I meant. I just used a lot of extra words to say it, I guess...
amelius 6y ago
I'd like a speech interface, as I often have my hands full with actual lab stuff.
unkl 6y ago
I think this is a problem not just for the lab community but for digital tools in general. There really isn't a good single app for doing many different styles of thinking and representation (text, drawings, formulas, tables, photos), organizing those things, and sharing them. Many apps get part of that problem right, but not all of them.
sogen 6y ago
Wordpress? Don't know if Blogspot or other blogging platforms offer it, but I'd say Wordpress covers most/all of them. Plus, easy to import/export data.
asdff 6y ago
You can do all of this in rmarkdown and some grep. git integrates well with rstudio too. plus an rmarkdown file or notebook is a plaintext file so it's future proof. It's what I use and I love it.
gnopgnip 6y ago
What specific version of one note are you using? The search in OneNote included with office is noticeably better than the OneNote for Windows 10 that is preinstalled. There is an addin, NoteHighlight2016 that adds code syntax highlighting. OCR for text within images is another helpful feature that is missing from many alternatives.

dcolkitt 6y ago

Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?

andylynch 6y ago

Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.

sleavey 6y ago
This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.
- analog31 6y ago
  Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes. Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion. I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook. Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.
easton 6y ago
Wouldn’t git with signed commits work just as well? You’d have to find a good app that saved stuff as text (not Word, but probably not LaTeX either since that takes time to get the formatting right), but once you figured that out, you’d have a full history of all changes made.
- andylynch 6y ago
  Yup - that's probably a good equivalent.
- dkarl 6y ago
  Signed commits still let you falsify your own work after the fact. You need an external trusted service, a digital notary essentially, that would provide a signed record that you submitted a commit on a certain date with certain set of ancestor commits. Each time you added an entry, you would submit the hashes of all the entries in the book to the digital notary and get a signed record that you submitted those commits on that date. Inability to produce this record would be similar to having erasures or pages torn out in a lab book.
  - homarp 6y ago
    a public blockchain would do for the digital notary? or hash posted on twitter?
  - dmurray 6y ago
    You can do this in a pretty cheap and low-tech way by publishing the hash of your current state somewhere that will keep a permanent record. E.g. the Bitcoin blockchain or a classified ad in a local newspaper.
  - yodon 6y ago
    This. True lab notebooks are not just personal notes, they are a legal record. Not every page of every lab notebook will matter, but it absolutely happens and is one of the many real use cases for keeping a lab notebook. If you invent something or discover something of significance, you will want this. Paper notebooks have pages stitched together and ink that can be run through chemical analysis to see if all the text on the page was written in a single session by the same pen. Forensic examiners have about 100 years of experience assessing paper documents. Digital systems can provide legal grade archival support but a sweet front end editor with copy paste and markdown support does not imply anything about traceability of the contents.
Wohlf 6y ago
Interesting, what about the fact people can just not write something in it?

doctorwhat 6y ago

I believe eLabFTW (https://github.com/elabftw/elabftw https://github.com/elabftw/elabftw) would fit ALL your requirements.

sleavey 6y ago

Over the past few years I've developed a WordPress plugin that provides most (all, I think) of what you want [1]. I also a while back did a comparison of some other softwares' pros and cons [2]. At my work we've used my system for about 2 years now and there is much more engagement with the system than the one it replaced (a custom Lotus Notes application... go figure). People like that they can open the logbook from the lab, write some stuff, save it and edit/publish it later. We also heavily encourage to document anything and everything and use the categories and tags as much as possible to make searching easier. [1] https://github.com/Academic-Labbook/alp https://github.com/Academic-Labbook/alp [2] https://alp.attackllama.com/2019/02/05/labbook-software-shootout/ https://alp.attackllama.com/2019/02/05/labbook-software-shoo...

willis936 6y ago

My university has supplied labs with labarchives (and has guaranteed if it ever migrates away from labarchives it will provide an equivalent solution and handle migration). It ticks all of those boxes. My only gripes are minor. Things like no embedding of vector graphics and no good embedded spreadsheet solution. They have some MS office browser based solution that handles version control. That is the official best spreadsheet solution right now. I generated an html spreadsheet for run logs because I wanted an embedded setup. I ran into the limitation of 64K characters per entry. It’s handled by trimming html off the end then autofixing broken stuff. It works so well that it took me a while to figure out why the number of rows was decreasing as more rows were filled out. It’s all very tidy once enough people are on board. Whole notebook (and subsections) can be exported to html or PDF. There is even a way to have some robotic file uploading a la directory monitoring.

KyleOS 6y ago

Have you ever considered Kyso (https://kyso.io https://kyso.io) (disclaimer - I'm a co-founder)? I could be way off, and it would be a corner case for us right now, but it seems we'd fit all your requirements: * We sit on top of git, which remains the single source of truth for your code - versioning, reproducibility, etc. * Python & R notebooks or simple markdown (also on the web app itself). * Show or hide code. * Simple links for sharing reports. * Tagging. * Comprehensive search - by tag, keyword and content (soon). Would be really interesting to hear your thoughts on the fit.

ofou 6y ago

Markdown Enhanced for VSCode[1] + Git for versioning. \ [1]: https://shd101wyy.github.io/markdown-preview-enhanced/ https://shd101wyy.github.io/markdown-preview-enhanced/

guiambros 6y ago

If you don't mind relying on a third-party, Notion [1] has pretty much all of your requirements. Formatting is a pleasure (including formulas and LaTeX), tables are decent, you can easily share notes, etc. Code formatting can improve a lot (only basic language semantics/operators/strings); search is decent. Above all: it's incredibly simple to use, and very customizable. My three big missing features: 1) full API, so you can develop extensions that provide the missing functionality, 2) end-to-end encryption, 3) 2FA. For now I'm using only for study lab notes, so encryption is not a big deal. But I wouldn't feel comfortable adding anything confidential without a more decent security model. [1] https://www.notion.so/ https://www.notion.so/

M5x7wI3CmbEem10 6y ago
but it's a proprietary format and not stored locally, unlike obsidian
- guiambros 6y ago
  Re: not storing locally, yes, hence my first comment on relying on a 3rd party. It depends on what is your intended use, given Notion doesn't have e2e encryption yet. Re: proprietary, it exports natively in Markdown (including a zip with all images), or pdf, or html. It's not perfect (there's a bug when exporting some LaTeX formulas in .md), but for the most part it works well.

MereInterest 6y ago

When I was in the sciences, I used a LaTeX document as my labbook * Ability to share with colleagues - Easy to send pdfs or the raw LaTeX to others. * Able to simply past in images - I had a folder with dated images, and a macro to insert a graph at the current location (better than floats for labbooks). * Versioning - Each section had a date in it. Sequential versions were stored in git * Tables - `\begin{tabular}` * Code snippets - `\begin{lstlisting}` * Links, tags, backlinks - `\ref{section_name}` It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".

privong 6y ago
> It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple". One possible way of harnessing the power of LaTeX while keeping a simple formatting would be to use use `pandoc` and pandoc-flavored markdown. That keeps the formatting simple, while also making it easy to include references, equations, etc. and easy output to LaTeX/PDF/etc.

vnxli 6y ago

Why are so many people interested in markdown for notes? I've never used it before but HN seems really interested in note taking and in using markdown for note formatting. What does it offer that OneNote or Word doesnt?

sq_ 6y ago
I think the difference is probably that markdown is a pretty transparent and versatile way to format things, while still being simpler than many other markup/formatting languages. With Word, you might insert an image or try to add bullets to something and end up with your whole document looking completely different, whereas, with markdown, you know that if you put in a ">" for a quote, all that's going to change is that one spot.
- dkdk8283 6y ago
  I think it’s formatting as code vs WYSIWYG.
  - sq_ 6y ago
    Good point. Looking back at my comment, that's basically what I meant. I just used a lot of extra words to say it, I guess...

amelius 6y ago

I'd like a speech interface, as I often have my hands full with actual lab stuff.

unkl 6y ago

I think this is a problem not just for the lab community but for digital tools in general. There really isn't a good single app for doing many different styles of thinking and representation (text, drawings, formulas, tables, photos), organizing those things, and sharing them. Many apps get part of that problem right, but not all of them.

sogen 6y ago

Wordpress? Don't know if Blogspot or other blogging platforms offer it, but I'd say Wordpress covers most/all of them. Plus, easy to import/export data.

asdff 6y ago

You can do all of this in rmarkdown and some grep. git integrates well with rstudio too. plus an rmarkdown file or notebook is a plaintext file so it's future proof. It's what I use and I love it.

gnopgnip 6y ago

What specific version of one note are you using? The search in OneNote included with office is noticeably better than the OneNote for Windows 10 that is preinstalled. There is an addin, NoteHighlight2016 that adds code syntax highlighting. OCR for text within images is another helpful feature that is missing from many alternatives.