3 ms·

We are still quite far from being able to implement these kinds of things at scale. For the literature review piece the key problem is that LLMs are exquisitel

by tgbugs 3y ago

We are still quite far from being able to implement these kinds of things at scale. For the literature review piece the key problem is that LLMs are exquisitely bad at working with even the simplest kind of scientific evidence: citations [1, 2]. They will get better, but it is not clear that LLMs can deal effectively with the very sparse kind of evidence that appears in the literature. Also, generating hypotheses isn't exactly the rate limiting step, the bigger issue tends to be when you get people with pet projects/hypotheses in positions of power that dictate funding priorities (e.g. the decades long Alzheimer's Aβ disaster). For automation and instrumentation of labs the vision is on point and there is interest, and active work, if not large amounts of funding, to bring that vision to reality [3, 4, 5]. However, we simply don't have the tooling needed to be able to express the full complexity of experimental protocols in a way that can be verified. Sure you can write a python script to control a robot, but it is exceptionally difficult to extract the scientific meaning from that. My PhD work was to develop a formal language for scientific protocols, and I'll be continuing to develop it, but there is still a long way to go. 1. https://doi.org/10.7759/cureus.39238 https://doi.org/10.7759/cureus.39238 2. https://doi.org/10.1016/j.mcpdig.2023.05.004 https://doi.org/10.1016/j.mcpdig.2023.05.004 3. https://www.youtube.com/watch?v=_gXiVOmaVSo&t=865s https://www.youtube.com/watch?v=_gXiVOmaVSo&t=865s 4. https://doi.org/10.1109/JIOT.2020.2995323 https://doi.org/10.1109/JIOT.2020.2995323 5. https://ccc.ucsf.edu/sites/ccc.ucsf.edu/files/Marshall_W_CCC_Annual_Report_Extracts_fy02_2018.pdf https://ccc.ucsf.edu/sites/ccc.ucsf.edu/files/Marshall_W_CCC...

pocw 3y ago
It's actually possible to use LLMs to assist literature reviews. We built a product that works smashingly. The key is to keyword extract, use a vector database and do search based generation. Our key insight is that the process of citation needs to be handled outside the LLM. They're good for text processing and summarization but as you said, the LLM itself is poor at citation. https://studyrecon.ai https://studyrecon.ai
- tgbugs 3y ago
  > Our key insight is that the process of citation needs to be handled outside the LLM. I can imagine taking the citation tree and using the LLM to compact the hypotheses, results, etc. for each node in the tree and sticking that in the vector database could get you pretty far.

by tgbugs 3y ago

pocw 3y ago

It's actually possible to use LLMs to assist literature reviews. We built a product that works smashingly. The key is to keyword extract, use a vector database and do search based generation. Our key insight is that the process of citation needs to be handled outside the LLM. They're good for text processing and summarization but as you said, the LLM itself is poor at citation. https://studyrecon.ai https://studyrecon.ai

tgbugs 3y ago
> Our key insight is that the process of citation needs to be handled outside the LLM. I can imagine taking the citation tree and using the LLM to compact the hypotheses, results, etc. for each node in the tree and sticking that in the vector database could get you pretty far.