11 ms·

The business of extracting knowledge from academic publications

285 points by kevin_hu 5y ago

woliveirajr 5y ago
> This post is about the issues with semantic intelligence platforms that predominantly leverage the published academic literature. I was happy to see apost that clearly states its purpose. edit: misspelling
bryanph_ 5y ago
One thing that strikes me about most academic knowledge tools is that they seem to focus on parsing the current set of academic literature and producing supposedly interesting insights out of them (which quickly tends to snowball into wanting some kind of generalized model for knowledge as a whole). What I think is much more interesting is creating tools that help people create better academic writing in the first place (thinking tools if you will). This is however much more a UX problem rather than it being a pure engineering problem. That is why I think we see many more tools in the knowledge extraction space as most academics thinking about these kind of things probably have an engineering background. That combined with the the fact that we seemingly all want to throw machine learning at any problem we encounter.
- grlass 5y ago
  I recall seeing a Show HN post a while back about a research focussed web browser that helps as a thinking tool: https://news.ycombinator.com/item?id=28446147 https://news.ycombinator.com/item?id=28446147
  - totetsu 5y ago
    It's amazing what you miss on HN when you skip a day
  - Hard_Space 5y ago
    A sign-in/sign-up necessary just to see the browser in action? Hard pass.
    - beauzero 5y ago
      https://www.loom.com/share/93c7c0012f514c37b58a42fa65badc88 https://www.loom.com/share/93c7c0012f514c37b58a42fa65badc88
- civilized 5y ago
  To your point but even more general: the ML/AI space is far too focused on replacing people rather than helping people. There is a suffocating cultural conceit that we are on the verge of general AI and oh my gosh what will the humans do, we better institute universal basic income right away, etc. What a joke. Try to help humans think better first. If you succeed at that, you might be on the right track towards developing cold fusion, er, general AI.
  - urthor 5y ago
    Unfortunately you'll run into the rapid fact that in the ML/AI space you get almost zero points for building something. You get a whole lot of points for discovering something, designing something, or a proof. But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. Mostly because everyone quietly understands all the extraordinarily complicated mathematics is actually extraordinarily complicated. Hence the ROI isn't worthwhile.
    - geoduck14 5y ago
      >But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. I can't speak to each and every person working on ML, but I thought I would share a fun use case I ran across the other day. There is a business in some foreign country that is similar to Uber Eats: customer goes to an app, browses for food from various restaurants, orders, it gets delivered. The business was using ML to help the restaurants: the restaurants upload a pic of the dishes, a title, and a description (usually all from an existing menu). The business would parse the description to guess at what was in the dish. Scan the picture to guess at the quantity of food (entre, side, desert, etc). Compare ingredients against publicly available nutrition info. Now the end consumer can do things like: search for gluten free, vegetarian, pork free, <300 calories, desert, etc. Almost all of this was "possible" before, but it would have required enormous effort from the restaurants inputting the data or customers reading each item. Now it is "easy", and it actually helps the end customers - and the restaurants.
- eurasiantiger 5y ago
  It likely wouldn’t take much to craft an ”arXiv Copilot” out of GitHub Copilot.
  - TOMDM 5y ago
    "It's well understood how to" And "It likely wouldn't take much to" Are worlds apart in this case, training and deploying models on that scale is a huge investment, even if you already had all the code and cleaned training data.
    - Mezzie 5y ago
      Can confirm: This is my main tech interest at the moment and if I consider how long it's going to take, I want to die.
- a_bonobo 5y ago
  As an ECR with English as a second language, the paid version of Grammarly has clarified my writing quite a bit. I think there's more unexplored value in this space.
- Mezzie 5y ago
  I agree. I'm an academic librarian, and they're completely different ways of working: When I do academic work, I (ideally) have to take my time and I'm not supposed to present my work until it's developed enough that I'm confident it presents a substantial improvement; I have to prove that it's worth a colleague's time to engage with by meeting certain requirements. Coding/developing, on the other hand, requires a lot more back and forth, a lot more "I don't know", and is more immediate in a way I find very satisfying. I would LOVE to see more back and forth between engineers and academics in terms of ways of working; I think there's a lot of benefit to be gained there: Tech tends to not consider the future as much as they should, but the academics could really benefit from doing what you mentioned and improve the system they work in rather than accepting it. One of the things I'm trying to do is get better at/learn some ML so I can play around with turning the things I learned in grad school into useful tools, but I'm a single journeyman dev doing this in my spare time so odds of anything actually useful coming out of it is small.
- Quanttek 5y ago
  Exactly! Assist me in my process of researching and writing. and use what I have already done in e.g. putting and classifying paper in Endnote. It's interesting that the author seemed to have a similar idea for a brief second but then tossed it away: > similar: an app that pops up serendipitous connections between a corpus (previous writings, saved articles, bookmarks ...) and the active writing session or paragraph. The corpus, preferably your own, could be from folders, text files, blog archive, a Roam Research graph or a Notion/Evernote database.
- KevinKallmes 5y ago
  This is entirely correct. Automation is a flash in the pan, making it easier for an expert to gather, process, and present information is much higher value. Unfortunately for the startup ecosystem, it has apparently become rote knowledge among investors that workflow = low-value, AI = 100x value, so the money is pushing entrepreneurs away from the more boring task of making tools for people to process knowledge while chasing fairy tales of turning millions of PDFs into an automatic gold mine...
civilized 5y ago
Interesting but I don't know how to make sense of it. How can it be that "close to nothing of what makes science actually work is published as text on the web"? - Is the information that makes science actually work mostly in images that the machines don't yet understand? - Was the information paywalled or in private databases and inaccessible to this researcher? - Are the papers mostly just advertisements for researchers to come gab with each other at conferences and doodle on cocktail napkins, and that's where all the "real science" happens? - (From the comments) is the information needed to make sense of papers communicated privately or orally from PI's to postdocs and grad students, or within industrial research labs? Something is missing from my mental picture here. Don't real scientists mostly learn how to think about their fields by reading textbooks and papers? (This is a genuine question.) If so, isn't it likely that our tools just aren't advanced enough to learn like humans do? If not, what do humans use to learn how to think about science that's missing from textbooks and papers?
- hoseja 5y ago
  As the article states, papers are mostly career advancement tools and scientists are incentivized to put the least amount of useful information into them they can get away with. Real scientists mostly learn from their instructors who possess all the jealously guarded institutional knowledge. Yes, it is very broken.
  - jokteur 5y ago
    It is always difficult to try to understand and implement the theory explained in papers which seem fine on the surface, but when you look more closely, you find a bunch of mistakes, there are giant holes in the details, and you end up trying to redo the whole paper. There should be journals/websites/blogs dedicated to trying to reexplain / implement papers.
    - bluGill 5y ago
      Doing a good review is hard. I've been reviewing a few C++ papers (things proposed to C++23) lately. Many of them are over my head and all I can find are a few spelling errors. The ones I've understood took me 3 readings before I found some giant holes in the details (which I pointed out to the authors, the next revision corrected them). In one case I actually started implementing the feature myself, and only then did I realize there was something ambiguous in the paper (after talking with the author we decided to document an implementors choices as it doesn't matter for the 99% use case, and the rest could go either way depending on hardware and data details so better to allow choice) The vast majority if papers I'm far too laze to go into that level of detail on. I just assume the authors is a smart person and so I trustingly let them go. It may well be if I understood the paper I'd be horrified, ask me in 2043 when we have 20 years of hindsight... I have to believe that peer review is the same - many reviewers are just reading and looking for something obvious but not really understanding details.
i000 5y ago
When I was in grad school, I joined a startup incubator and build a prototype which combined two of the tools mentioned in the article: "a query builder (by demonstration)" and "A paper recommender system", a simple companion which would help scientist to not miss relevant research to them. This was 10 years ago, before Google Scholar has similar features. The incubator introduced me to advisors with business experience in this field. And I got told in no uncertain terms what is the gist of this article: The value lies in the molecular and clinical data. In 2021 I would add digital pathology / imaging data.
- geoduck14 5y ago
  >And I got told in no uncertain terms what is the gist of this article: The value lies in the molecular and clinical data. In 2021 I would add digital pathology / imaging data. I feel like you are trying to tell me something REALLY valuable, but I don't quite understand it. Can you please elaborate?
  - potatoman22 5y ago
    My take: answering questions using clinical data > answering questions with papers
    - i000 5y ago
      There is immense value in clinical data (all the information captured and siloed through EHR). Pharma companies pay for access to it to gather real-world evidence (RWE) how, for example, their drug performs. Molecular information is increasingly valuable too for research, biomarker development, patient cohort identification etc. The imaging data and pathology data are valuable because they are typically expertly annotated and can be used to train computer-vision algorithms etc. to solve medical problems - like diagnosis.
shusaku 5y ago
Previous discussion: https://news.ycombinator.com/item?id=29445715 https://news.ycombinator.com/item?id=29445715
anyfactor 5y ago
> Why purchase access to a 3rd party AI reading engine or a knowledge graph when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary) I really like jobs people think AI can do in theory but can't really do them effectively irl. Where do I get a part-time gig like that if I think I am capable of reviewing and creating summary of non-STEM papers? Except for homeworks and assignments of course.
- Mezzie 5y ago
  Yeah, you can't outsource that to Hyderabad. You'd need to know subject knowledge + very specific English and possibly other languages depending on the field (not saying Indians can't do this, but I've studied enough languages to know that doing high level/academic work in a non-native language is hell even when the language is pitched to students). And you'd have to know enough about the process and authors to know what makes papers relevant. The metadata matters as much as the data.
  - anyfactor 5y ago
    All good points. But you do have to recognize the tradeoff. Has AI come so far that it could perform better than industry specific human intelligence? You have to consider that maybe some Indian researchers could review the papers as they are doing that job as part time gig. You have to test out both solution. And as these jobs are treated as contracts there is no significant commitment for choosing one over the other. We can't be certain if one method is better than the other without trying both of them out without prejudice. I, for one am agnostic about either choice. Because AI is overhyped yet it has spillover benefits as a marketing-sales point but offshore human intelligence has a bad rep but could be effective if you have proper documentation, supervision and review framework.
    - Mezzie 5y ago
      Oh yeah, I was just thinking currently. In five to ten years once AI/ML/etc. trickle out of tech/theory spaces and starts to be combined with subject expertise, I think we'll see really interesting things. The other matter is that an Indian who could review papers that well would also cost more than 6k/year and would not be easily replaceable, which eliminates the main benefit of outsourcing for a company trying to operate in such a way in 2021. In 2030? I'd say the odds are if somebody in Hyderabad can do that then they can start their OWN company rather than bother with us at all. Honestly, given India's role in pharmaceutical manufacture, I'd be shocked if things like that don't start popping up.
cousin_it 5y ago
I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding. Reading a forum of synth hobbyists is a hundred times more useful.
- jakub_g 5y ago
  I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive: In opensource the attitude is "See bug? Send a PR!" Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.
  - remram 5y ago
    The biggest difference IMHO is when comparing to something like Wikipedia or Stackoverflow. I wish the fabric of scholarly communication similarly allowed for browsing reviews, updating papers, commenting with new references, etc.
    - kwertyoowiyop 5y ago
      Now THAT might be a realistic technical goal & business opportunity.
      - Mezzie 5y ago
        The legal costs make this a non-starter unless it's done by a giant company. Who would, in my opinion, ruin it, and the odds of enough academics complying with a big tech company are small imo. It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them. Building this would be my dream job, but hahaha no.
javajosh 5y ago
Don't know much about this industry but yes, it feels like one of those industries that sprang up because one person with money said, "Hmm, sounds like a good idea," and then other people with money and FOMO joined in. When this happens past a certain level you get a miniature innovation bubble (MIB)! (At least MIBs are rather harmless, at least in the long run, and can actually yield some benefit: innovative people are drawn to these types of industries and inevitably create cool things as a by-product of their work.)
acomjean 5y ago
A lot of papers are hand or computer assisted annotated. For medical papers Mesh terms: https://www.nlm.nih.gov/mesh/meshhome.html https://www.nlm.nih.gov/mesh/meshhome.html Gene information is extracted by flybase/ worm base … It’s time consuming, expensive probably not perfect but for certain types of papers it makes searching better.
_Wintermute 5y ago
I think if the author had listened to pretty much any post-doc/technician or senior researcher in the field who has had to review a number of publications they would have been told these things straight away.
Kydlaw 5y ago
Interesting insights. Particularly on the business aspect. But I am not surprise by the outcome, as the author said: nobody want to pay for what its proposed in academia. Everybody is already more or less struggling with funding, so nobody want to add extra fat in their funding requests. Coming from CS, something I would really like to see though is a tool that would summarize a scientific area/domain. Something that would kill literature reviews and/or would provide an overview of the hot topics/open questions in different areas. Edit: corrections
PaulHoule 5y ago
When you count all the ways to be wrong, the median scientific paper is wrong. In biomedical fields they dismiss more than half of papers out of hand when they do a Cochrane meta analysis. It begs the question of why such papers (which aren’t fit to extract knowledge from) are published or funded at all. I got a PhD in theoretical physics and routinely found that something was wrong on page 23 of a 50 page calculation and nobody published anything about it in 30 years. Possibly the whole body of work on string theory since 1980 [most of hep-th] is a pipe dream at best. Because young physicists have to spend the first third of their career in a squid game fight for survival not to fathom the secrets of the universe but to please their elders we get situations like that absurd idea of Steve Hawking that information gets lost in a black hole. (E.g. if you believe that you aren’t even going to try quantum gravity.)
PaulHoule 5y ago
This paper touches one aspect of it, which is that the source material is bad, but it doesn’t even start on the fact that the tools aren’t good enough and that many of the fashionable ideas (Word embeddings) are dead ends.
- gardenfelder 5y ago
  The very idea that "word embededings are dead ends" requires citations, evidence.
tomlue 5y ago
Knowledge extraction is weird. Just because I extracted some knowledge doesn't mean that I now 'have' that knowledge. The better use case for this is teaching, not creating knowledge bases that nobody will use.
- gardenfelder 5y ago
  I would argue "learning" - personal learning is a great topic.
deleted 5y ago
[deleted]
toss1 5y ago
And the bottom line is: >>... nothing of it will go anywhere. >>Don’t take that as a challenge. Take it as a red flag and run. Run towards better problems. Wow, speaking of the value of negative results, that is hugely valuable! Could easily save person-decades of work & funds for more productive results. The insights that the most relevant knowledge is not written into the publications (for a variety of reasons), and that the few that are are of limited use to the target audience, and even when it is useful it is a small part of the workload (i. e., not a real pain point), are key to seeing that the entire category of projects to extract & encode such knowledge is doomed.
PaulHoule 5y ago
1. The real value is in operational documents such as clinical notes, maintenance records, soldier and police notebooks, etc. this info is proprietary to a organization and its partners and is directly linked to how it produces and pays for value. 2. Superhuman accuracy at limited tasks is not good enough. For instance transcribing audio at 95% word-level accuracy would be good for a human but it means every other sentence is garbled. People communicate despite this because they ask questions. A useful text-to-structured information tool has to exert back pressure on bullshit, give feedback about what it understands and push the author to tell a story that makes sense and has adequate detail.
Vetch 5y ago
The article's core claims are: > Extracting, structuring or synthesizing "insights" from academic publications (papers) or building knowledge bases from a domain corpus of literature has negligible value in industry. > Most knowledge necessary to make scientific progress is not online and not encoded. > Close to nothing of what makes science actually work is published as text on the web > The tech is not there to make fact checking work reliably, even in constrained domains. > Accurately and programmatically transforming an entire piece of literature into a computer-interpretable, complete and actionable knowledge artifact remains a pipe dream. It also states existing old school "biomedical knowledge bases, databases, ontologies that are updated regularly", with Expert Entry cutting through the noise in a way that NLP cannot. Although I disagree with its conclusions, much of this jives with my experience. From the perspective of research, modern NLP and transformers are appropriately hyped but from the perspective of real world application, they are over-hyped. Transformers have deeper understanding than anything prior, they can figure out patterns in their context with a flexibility that goes way beyond regurgitation. They are also prone to hallucinating text, quoting misleading snippets, require lots of resources for inference and enjoy being confidently wrong at a rate that makes industrial use nearly unworkable. They're powerful but you should think hard about whether you actually need them. Most of the time their true advantage is not leveraged. ----- My disagreements are with its advice. > For recommendations, the suggestion is "follow the best institutions and ~50 top individuals". But this just creates a rich get richer effect and retards science since most are reluctant to go against those with a lot of clout. > Why purchase access to a 3rd party AI reading engine...when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary). Would you invest in automation if you have billions of disposable income and access to cheap labor? After talking with employees of huge companies like GSK, AZ and Medscape the answer is a clear no. This reminds me of responses to questions of the sort: "Why didnt't X (where X might be Ottomans or Chinese) get to the industrial revolution first?". Article also warns against working on ideas such as "...semantic search, interoperable protocols and structured data, serendipitous discovery apps, knowledge organization." A lot such apps are solutions chasing after a problem but could work if designed to solve a specific real world problem. On the other hand, an outsider trying to start a generalized VC backed business targeting industry is bound to fail. In fact, this seems a major sticking point in the author's endeavor. Industry is jaded and set in their ways, startups focus on summarization and recommendations and retrieval which are low value in scientific enterprise and academia is focused on automation which turns out brittle. Still, this line of research is needed. Knowledge production is growing rapidly while humans are not getting any smarter. Specialization has meant increases in redundant information, loss of context and a stall in theory production (hence "much less logic and deduction happening"). While the published literature is sorely lacking, humans can with effort extract and or triangulate value from it. Tooling needs to augment that process.
JackFr 5y ago
It's interesting that OP did seemingy little research with respect to existing work in the field. https://www.nlm.nih.gov/medline/medline_overview.html https://www.nlm.nih.gov/medline/medline_overview.html Medline, a searchable online directory of medical research papers has existed for 50 years. The National Library of Medicine for many years was a leader in document search and retrieval before there was a web. In the 80's they were doing vector cosine document similarity, document clustering and automated classifcation. They were also doing so great stuff like indexing papers based on proteins and gene sequences - so a paper which might be in a field completely different than yours might pop up if a similar protein or sequence was mentioned. (Disclosure - I worked at the National Library of Medicine in the 90's) That being said, in the past 30 years search and retrieval exploded to say nothing of ML, but its crazy to ignore the stuff which has come before, AND it's tough to compete with a national lab whose mandate is to basically give the stuff away.
- inlitro 5y ago
  100% It also felt like a long apology/explanation for Emergent Ventures rather than a true deep analysis. Pretty strong (and often false) statements for only what seems like half a year of total, somewhat vague work.
- stevenbedrick 5y ago
  The best thing about the NLM's work in this space is how deeply it has been informed by the needs and workflows of the biomedical researchers, which is a perspective that has been sorely lacking in work coming from outsiders. I did think that the author did a good job of outlining (some of) the basic structural issues that make this a tough field to monetize, but even setting those aside, there's no substitute for actually knowing your users and what they need, and that's something the NLM is amazing at. (Disclosure, my PhD was funded by an NLM training grant, some of my research is funded extramurally by the NLM, and I have a lot of NLM colleagues, so I'm maybe a little bit biased)
- Ldorigo 5y ago
  Medline is great on all levels but what does it have to do with the author's work/product/topic? It's a database of biomedical articles. It's searchable using usual text search ("concept A" AND "concept B"), and has some additional features like MESH terms and metadata. The author is talking about extracting entity-level information from those article and building knowledge bases. Medline provides access to the raw data, but it does nothing like what the article describes (well - there's https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR.html https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR.html , so not entirely true. But it's hardly a final answer to the problem or even close to being usable enough that it makes commercial solutions pointless)
markusstrasser 5y ago
Hey, author here. Great discussion so far. Will update the post with some of the comments and critiques.
Nalta 5y ago
For anyone interested, my whole PhD was in biomedical hypothesis generation! I think the most "serious" attempts at building these systems have been focused around providing assistance to scientists, and not just coming up with new ideas on their own. here's an actual medical paper that my first system, Moliere, was able to help discover: https://link.springer.com/article/10.1007/s11481-019-09885-8 https://link.springer.com/article/10.1007/s11481-019-09885-8
- julienchastang 5y ago
  Is your PhD thesis online anywhere?
  - Nalta 5y ago
    https://sybrandt.com/documents/dissertation.pdf https://sybrandt.com/documents/dissertation.pdf
    - gardenfelder 5y ago
      https://github.com/JSybrandt/MOLIERE https://github.com/JSybrandt/MOLIERE
- gardenfelder 5y ago
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8048136/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8048136/
- chaxor 5y ago
  I also am working within this field in academia, and I must say that I have really enjoyed reading your work. I am focused a bit more on combining biological data sources with text knowledge graphs, but in my literature review of the field, I have found that we both have very similar aspirations for career path. It seems that one major bias that the author of the post's blog has is their heavy conflation of 'worth' with money. Many of us probably realize that this is not true, as academia clearly shows. Furthermore, It's not that crazy that they had trouble making money off of software, as I wouldn't expect many startups at all to be able to get customers from software solutions alone. They also seem to try to compare their success with that of essentially 'other biotech ml companies'; for which I would expect there to be quite a bit of tangible resources that these 'other companies' provide. For example, a startup looking to provide a service of detecting diseases or conditions from DNA methylation data would likely perform the sequencing required before doing an analysis (in order to have good control over experimental conditions). The materials alone in that case could cost quite a bit - so charging a bit more for the analysis isn't that problematic, since the transaction of some currency is already required for covering material costs. Anyway, as you mentioned - it seems it's important to recognize that these systems aren't necessarily meant to generate revenue for a startup, but rather are much more useful as a tool in academia.
holub008 5y ago
> Close to nothing of what makes science actually work is published as text on the web Unless there's some nuance I missed, I immensely disagree with this statement. I'm currently in the biomedical literature review space, and I appreciate the detailed insights. I wonder if the author considered that literature review is used in a wide variety of domains outside pharma/drug discovery (where I perceived their efforts were focused). Regulatory monitoring/reporting, hospital guideline generation, etc. This is a billion dollar industry, and I couldn't agree more that it's technologically underdeveloped. I do not agree that AI-based extraction is the solution, at least in the near-term. The formal methodologies used by reviewers/meta-analysts: search strategy generation, lit search, screening, extraction, critical appraisal, synthesis/statistical analysis, are IMO more nuanced than an AI can capture. They require human input or review. My business is betting on this premise :)
Throwaway197401 5y ago
[dead]
throwaway984393 5y ago
Science is not about innovation. Science is about tiny little results that by themselves have no immediate benefit, but slowly improve our overall understanding, and eventually lead to an unexpected benefit. Science is not developed in order to solve a business problem - it is purely an advancement in overall knowledge of the world (the traditional aim of natural philosophy). In this sense, science is not compatible with business interests.
pezzana 5y ago
> My biggest mistake was that I didn't have experience as a biotech researcher or postdoc working in a lab. That is is big problem - good to recognize it as such. I can tell because the article, though lengthy, never seems to state an explicit problem to be solved. Rather, various ways to apply technology to a field are discussed. This is a recipe for failure. You need 3 things: 1. a problem to be solved 2. a customer who has that problem 3. money in the customer's pocket waiting to be transferred to yours The article never even gets to (1). Regarding (2), if academic groups are the target customer, you're going to have a bad time. They have little money and they tend to be all to happy to build something that sort-of replicates the commercial product you've created for them. This leaves scientific for-profit companies. They have lots of problems (and these days money), but these problems tend to be quite difficult to discover and solve because of the extensive domain and industry knowledge required.
- tvhahn 5y ago
  Yes. I wonder if the author had first worked on the patent side (I'd be interested to hear more about this idea). Perhaps working on patents first would be a path to get experience (and product-market-fit). From there, one could branch out into other domains (e.g. bio).
  - markusstrasser 5y ago
    Patents were the more obvious go to but we didn't want to work with obfuscated patent descriptions for the next years
plaidfuji 5y ago
Having invested quite a bit of my own time into various aspects of the scientific knowledge extraction morass, I’d say the author is largely on point, but there’s a significant, and potentially valuable distinction to be made between extracting research outputs and research inputs. At least in the field of materials science, papers are by and large a record of research outputs. We made material X and it achieved performance Y - here are a bunch of measurements to prove that this is in fact what we made and that it truly achieved performance Y at relevant conditions, etc. In this sense, papers really function as an advertisement: look at what we achieved. What papers do not do is rigorously document inputs, or provide a step-by-step guide to reproduce said results, for obvious reasons. My current take on this topic is that it would be both feasible and valuable to build a knowledge extraction system to compile and compare outputs across a specified field. Think the big “chart of all verified solar cell efficiencies over time” [1], but generated automatically. This would at least immediately orient researchers to the distribution of state of the art results, and help ensure that they don’t omit relevant references in their reviews. But extracting and making sense of inputs (methods), or even “knowledge”? Forget about it. [1] https://www.nrel.gov/pv/cell-efficiency.html https://www.nrel.gov/pv/cell-efficiency.html
bigdict 5y ago
Any article on this topic should mention Tshitoyan et al. https://www.nature.com/articles/s41586-019-1335-8 https://www.nature.com/articles/s41586-019-1335-8
aurizon 5y ago
Business? It is not a business - it is a racket...
jcrben 5y ago
If someone published a promising in vitro study, the mere fact that it was done and even the barest hint of a conclusion can dramatically impact one's decision to pursue that line of research. So I'm a bit skeptical that there's negligible value in biomedical research papers.

285 points by kevin_hu 5y ago

> This post is about the issues with semantic intelligence platforms that predominantly leverage the published academic literature. I was happy to see apost that clearly states its purpose. edit: misspelling

bryanph_ 5y ago

One thing that strikes me about most academic knowledge tools is that they seem to focus on parsing the current set of academic literature and producing supposedly interesting insights out of them (which quickly tends to snowball into wanting some kind of generalized model for knowledge as a whole). What I think is much more interesting is creating tools that help people create better academic writing in the first place (thinking tools if you will). This is however much more a UX problem rather than it being a pure engineering problem. That is why I think we see many more tools in the knowledge extraction space as most academics thinking about these kind of things probably have an engineering background. That combined with the the fact that we seemingly all want to throw machine learning at any problem we encounter.

grlass 5y ago
I recall seeing a Show HN post a while back about a research focussed web browser that helps as a thinking tool: https://news.ycombinator.com/item?id=28446147 https://news.ycombinator.com/item?id=28446147
- totetsu 5y ago
  It's amazing what you miss on HN when you skip a day
- Hard_Space 5y ago
  A sign-in/sign-up necessary just to see the browser in action? Hard pass.
  - beauzero 5y ago
    https://www.loom.com/share/93c7c0012f514c37b58a42fa65badc88 https://www.loom.com/share/93c7c0012f514c37b58a42fa65badc88
civilized 5y ago
To your point but even more general: the ML/AI space is far too focused on replacing people rather than helping people. There is a suffocating cultural conceit that we are on the verge of general AI and oh my gosh what will the humans do, we better institute universal basic income right away, etc. What a joke. Try to help humans think better first. If you succeed at that, you might be on the right track towards developing cold fusion, er, general AI.
- urthor 5y ago
  Unfortunately you'll run into the rapid fact that in the ML/AI space you get almost zero points for building something. You get a whole lot of points for discovering something, designing something, or a proof. But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. Mostly because everyone quietly understands all the extraordinarily complicated mathematics is actually extraordinarily complicated. Hence the ROI isn't worthwhile.
  - geoduck14 5y ago
    >But there's a very large amount of people focused entirely on aims that are very, very distant from actually making human lives genuinely better. I can't speak to each and every person working on ML, but I thought I would share a fun use case I ran across the other day. There is a business in some foreign country that is similar to Uber Eats: customer goes to an app, browses for food from various restaurants, orders, it gets delivered. The business was using ML to help the restaurants: the restaurants upload a pic of the dishes, a title, and a description (usually all from an existing menu). The business would parse the description to guess at what was in the dish. Scan the picture to guess at the quantity of food (entre, side, desert, etc). Compare ingredients against publicly available nutrition info. Now the end consumer can do things like: search for gluten free, vegetarian, pork free, <300 calories, desert, etc. Almost all of this was "possible" before, but it would have required enormous effort from the restaurants inputting the data or customers reading each item. Now it is "easy", and it actually helps the end customers - and the restaurants.
eurasiantiger 5y ago
It likely wouldn’t take much to craft an ”arXiv Copilot” out of GitHub Copilot.
- TOMDM 5y ago
  "It's well understood how to" And "It likely wouldn't take much to" Are worlds apart in this case, training and deploying models on that scale is a huge investment, even if you already had all the code and cleaned training data.
  - Mezzie 5y ago
    Can confirm: This is my main tech interest at the moment and if I consider how long it's going to take, I want to die.
a_bonobo 5y ago
As an ECR with English as a second language, the paid version of Grammarly has clarified my writing quite a bit. I think there's more unexplored value in this space.
Mezzie 5y ago
I agree. I'm an academic librarian, and they're completely different ways of working: When I do academic work, I (ideally) have to take my time and I'm not supposed to present my work until it's developed enough that I'm confident it presents a substantial improvement; I have to prove that it's worth a colleague's time to engage with by meeting certain requirements. Coding/developing, on the other hand, requires a lot more back and forth, a lot more "I don't know", and is more immediate in a way I find very satisfying. I would LOVE to see more back and forth between engineers and academics in terms of ways of working; I think there's a lot of benefit to be gained there: Tech tends to not consider the future as much as they should, but the academics could really benefit from doing what you mentioned and improve the system they work in rather than accepting it. One of the things I'm trying to do is get better at/learn some ML so I can play around with turning the things I learned in grad school into useful tools, but I'm a single journeyman dev doing this in my spare time so odds of anything actually useful coming out of it is small.
Quanttek 5y ago
Exactly! Assist me in my process of researching and writing. and use what I have already done in e.g. putting and classifying paper in Endnote. It's interesting that the author seemed to have a similar idea for a brief second but then tossed it away: > similar: an app that pops up serendipitous connections between a corpus (previous writings, saved articles, bookmarks ...) and the active writing session or paragraph. The corpus, preferably your own, could be from folders, text files, blog archive, a Roam Research graph or a Notion/Evernote database.
KevinKallmes 5y ago
This is entirely correct. Automation is a flash in the pan, making it easier for an expert to gather, process, and present information is much higher value. Unfortunately for the startup ecosystem, it has apparently become rote knowledge among investors that workflow = low-value, AI = 100x value, so the money is pushing entrepreneurs away from the more boring task of making tools for people to process knowledge while chasing fairy tales of turning millions of PDFs into an automatic gold mine...

civilized 5y ago

Interesting but I don't know how to make sense of it. How can it be that "close to nothing of what makes science actually work is published as text on the web"? - Is the information that makes science actually work mostly in images that the machines don't yet understand? - Was the information paywalled or in private databases and inaccessible to this researcher? - Are the papers mostly just advertisements for researchers to come gab with each other at conferences and doodle on cocktail napkins, and that's where all the "real science" happens? - (From the comments) is the information needed to make sense of papers communicated privately or orally from PI's to postdocs and grad students, or within industrial research labs? Something is missing from my mental picture here. Don't real scientists mostly learn how to think about their fields by reading textbooks and papers? (This is a genuine question.) If so, isn't it likely that our tools just aren't advanced enough to learn like humans do? If not, what do humans use to learn how to think about science that's missing from textbooks and papers?

hoseja 5y ago
As the article states, papers are mostly career advancement tools and scientists are incentivized to put the least amount of useful information into them they can get away with. Real scientists mostly learn from their instructors who possess all the jealously guarded institutional knowledge. Yes, it is very broken.
- jokteur 5y ago
  It is always difficult to try to understand and implement the theory explained in papers which seem fine on the surface, but when you look more closely, you find a bunch of mistakes, there are giant holes in the details, and you end up trying to redo the whole paper. There should be journals/websites/blogs dedicated to trying to reexplain / implement papers.
  - bluGill 5y ago
    Doing a good review is hard. I've been reviewing a few C++ papers (things proposed to C++23) lately. Many of them are over my head and all I can find are a few spelling errors. The ones I've understood took me 3 readings before I found some giant holes in the details (which I pointed out to the authors, the next revision corrected them). In one case I actually started implementing the feature myself, and only then did I realize there was something ambiguous in the paper (after talking with the author we decided to document an implementors choices as it doesn't matter for the 99% use case, and the rest could go either way depending on hardware and data details so better to allow choice) The vast majority if papers I'm far too laze to go into that level of detail on. I just assume the authors is a smart person and so I trustingly let them go. It may well be if I understood the paper I'd be horrified, ask me in 2043 when we have 20 years of hindsight... I have to believe that peer review is the same - many reviewers are just reading and looking for something obvious but not really understanding details.

i000 5y ago

When I was in grad school, I joined a startup incubator and build a prototype which combined two of the tools mentioned in the article: "a query builder (by demonstration)" and "A paper recommender system", a simple companion which would help scientist to not miss relevant research to them. This was 10 years ago, before Google Scholar has similar features. The incubator introduced me to advisors with business experience in this field. And I got told in no uncertain terms what is the gist of this article: The value lies in the molecular and clinical data. In 2021 I would add digital pathology / imaging data.

geoduck14 5y ago
>And I got told in no uncertain terms what is the gist of this article: The value lies in the molecular and clinical data. In 2021 I would add digital pathology / imaging data. I feel like you are trying to tell me something REALLY valuable, but I don't quite understand it. Can you please elaborate?
- potatoman22 5y ago
  My take: answering questions using clinical data > answering questions with papers
  - i000 5y ago
    There is immense value in clinical data (all the information captured and siloed through EHR). Pharma companies pay for access to it to gather real-world evidence (RWE) how, for example, their drug performs. Molecular information is increasingly valuable too for research, biomarker development, patient cohort identification etc. The imaging data and pathology data are valuable because they are typically expertly annotated and can be used to train computer-vision algorithms etc. to solve medical problems - like diagnosis.

shusaku 5y ago

Previous discussion: https://news.ycombinator.com/item?id=29445715 https://news.ycombinator.com/item?id=29445715

anyfactor 5y ago

> Why purchase access to a 3rd party AI reading engine or a knowledge graph when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary) I really like jobs people think AI can do in theory but can't really do them effectively irl. Where do I get a part-time gig like that if I think I am capable of reviewing and creating summary of non-STEM papers? Except for homeworks and assignments of course.

Mezzie 5y ago
Yeah, you can't outsource that to Hyderabad. You'd need to know subject knowledge + very specific English and possibly other languages depending on the field (not saying Indians can't do this, but I've studied enough languages to know that doing high level/academic work in a non-native language is hell even when the language is pitched to students). And you'd have to know enough about the process and authors to know what makes papers relevant. The metadata matters as much as the data.
- anyfactor 5y ago
  All good points. But you do have to recognize the tradeoff. Has AI come so far that it could perform better than industry specific human intelligence? You have to consider that maybe some Indian researchers could review the papers as they are doing that job as part time gig. You have to test out both solution. And as these jobs are treated as contracts there is no significant commitment for choosing one over the other. We can't be certain if one method is better than the other without trying both of them out without prejudice. I, for one am agnostic about either choice. Because AI is overhyped yet it has spillover benefits as a marketing-sales point but offshore human intelligence has a bad rep but could be effective if you have proper documentation, supervision and review framework.
  - Mezzie 5y ago
    Oh yeah, I was just thinking currently. In five to ten years once AI/ML/etc. trickle out of tech/theory spaces and starts to be combined with subject expertise, I think we'll see really interesting things. The other matter is that an Indian who could review papers that well would also cost more than 6k/year and would not be easily replaceable, which eliminates the main benefit of outsourcing for a company trying to operate in such a way in 2021. In 2030? I'd say the odds are if somebody in Hyderabad can do that then they can start their OWN company rather than bother with us at all. Honestly, given India's role in pharmaceutical manufacture, I'd be shocked if things like that don't start popping up.

cousin_it 5y ago

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding. Reading a forum of synth hobbyists is a hundred times more useful.

jakub_g 5y ago
I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive: In opensource the attitude is "See bug? Send a PR!" Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.
- remram 5y ago
  The biggest difference IMHO is when comparing to something like Wikipedia or Stackoverflow. I wish the fabric of scholarly communication similarly allowed for browsing reviews, updating papers, commenting with new references, etc.
  - kwertyoowiyop 5y ago
    Now THAT might be a realistic technical goal & business opportunity.
    - Mezzie 5y ago
      The legal costs make this a non-starter unless it's done by a giant company. Who would, in my opinion, ruin it, and the odds of enough academics complying with a big tech company are small imo. It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them. Building this would be my dream job, but hahaha no.

javajosh 5y ago

Don't know much about this industry but yes, it feels like one of those industries that sprang up because one person with money said, "Hmm, sounds like a good idea," and then other people with money and FOMO joined in. When this happens past a certain level you get a miniature innovation bubble (MIB)! (At least MIBs are rather harmless, at least in the long run, and can actually yield some benefit: innovative people are drawn to these types of industries and inevitably create cool things as a by-product of their work.)

acomjean 5y ago

A lot of papers are hand or computer assisted annotated. For medical papers Mesh terms: https://www.nlm.nih.gov/mesh/meshhome.html https://www.nlm.nih.gov/mesh/meshhome.html Gene information is extracted by flybase/ worm base … It’s time consuming, expensive probably not perfect but for certain types of papers it makes searching better.

_Wintermute 5y ago

I think if the author had listened to pretty much any post-doc/technician or senior researcher in the field who has had to review a number of publications they would have been told these things straight away.

Kydlaw 5y ago

Interesting insights. Particularly on the business aspect. But I am not surprise by the outcome, as the author said: nobody want to pay for what its proposed in academia. Everybody is already more or less struggling with funding, so nobody want to add extra fat in their funding requests. Coming from CS, something I would really like to see though is a tool that would summarize a scientific area/domain. Something that would kill literature reviews and/or would provide an overview of the hot topics/open questions in different areas. Edit: corrections

PaulHoule 5y ago

When you count all the ways to be wrong, the median scientific paper is wrong. In biomedical fields they dismiss more than half of papers out of hand when they do a Cochrane meta analysis. It begs the question of why such papers (which aren’t fit to extract knowledge from) are published or funded at all. I got a PhD in theoretical physics and routinely found that something was wrong on page 23 of a 50 page calculation and nobody published anything about it in 30 years. Possibly the whole body of work on string theory since 1980 [most of hep-th] is a pipe dream at best. Because young physicists have to spend the first third of their career in a squid game fight for survival not to fathom the secrets of the universe but to please their elders we get situations like that absurd idea of Steve Hawking that information gets lost in a black hole. (E.g. if you believe that you aren’t even going to try quantum gravity.)

PaulHoule 5y ago

This paper touches one aspect of it, which is that the source material is bad, but it doesn’t even start on the fact that the tools aren’t good enough and that many of the fashionable ideas (Word embeddings) are dead ends.

gardenfelder 5y ago
The very idea that "word embededings are dead ends" requires citations, evidence.

tomlue 5y ago

Knowledge extraction is weird. Just because I extracted some knowledge doesn't mean that I now 'have' that knowledge. The better use case for this is teaching, not creating knowledge bases that nobody will use.

gardenfelder 5y ago
I would argue "learning" - personal learning is a great topic.

deleted 5y ago

[deleted]

toss1 5y ago

And the bottom line is: >>... nothing of it will go anywhere. >>Don’t take that as a challenge. Take it as a red flag and run. Run towards better problems. Wow, speaking of the value of negative results, that is hugely valuable! Could easily save person-decades of work & funds for more productive results. The insights that the most relevant knowledge is not written into the publications (for a variety of reasons), and that the few that are are of limited use to the target audience, and even when it is useful it is a small part of the workload (i. e., not a real pain point), are key to seeing that the entire category of projects to extract & encode such knowledge is doomed.

PaulHoule 5y ago

1. The real value is in operational documents such as clinical notes, maintenance records, soldier and police notebooks, etc. this info is proprietary to a organization and its partners and is directly linked to how it produces and pays for value. 2. Superhuman accuracy at limited tasks is not good enough. For instance transcribing audio at 95% word-level accuracy would be good for a human but it means every other sentence is garbled. People communicate despite this because they ask questions. A useful text-to-structured information tool has to exert back pressure on bullshit, give feedback about what it understands and push the author to tell a story that makes sense and has adequate detail.

Vetch 5y ago

The article's core claims are: > Extracting, structuring or synthesizing "insights" from academic publications (papers) or building knowledge bases from a domain corpus of literature has negligible value in industry. > Most knowledge necessary to make scientific progress is not online and not encoded. > Close to nothing of what makes science actually work is published as text on the web > The tech is not there to make fact checking work reliably, even in constrained domains. > Accurately and programmatically transforming an entire piece of literature into a computer-interpretable, complete and actionable knowledge artifact remains a pipe dream. It also states existing old school "biomedical knowledge bases, databases, ontologies that are updated regularly", with Expert Entry cutting through the noise in a way that NLP cannot. Although I disagree with its conclusions, much of this jives with my experience. From the perspective of research, modern NLP and transformers are appropriately hyped but from the perspective of real world application, they are over-hyped. Transformers have deeper understanding than anything prior, they can figure out patterns in their context with a flexibility that goes way beyond regurgitation. They are also prone to hallucinating text, quoting misleading snippets, require lots of resources for inference and enjoy being confidently wrong at a rate that makes industrial use nearly unworkable. They're powerful but you should think hard about whether you actually need them. Most of the time their true advantage is not leveraged. ----- My disagreements are with its advice. > For recommendations, the suggestion is "follow the best institutions and ~50 top individuals". But this just creates a rich get richer effect and retards science since most are reluctant to go against those with a lot of clout. > Why purchase access to a 3rd party AI reading engine...when you can just hire hundreds of postdocs in Hyderabad to parse papers into JSON? (at a $6,000 yearly salary). Would you invest in automation if you have billions of disposable income and access to cheap labor? After talking with employees of huge companies like GSK, AZ and Medscape the answer is a clear no. This reminds me of responses to questions of the sort: "Why didnt't X (where X might be Ottomans or Chinese) get to the industrial revolution first?". Article also warns against working on ideas such as "...semantic search, interoperable protocols and structured data, serendipitous discovery apps, knowledge organization." A lot such apps are solutions chasing after a problem but could work if designed to solve a specific real world problem. On the other hand, an outsider trying to start a generalized VC backed business targeting industry is bound to fail. In fact, this seems a major sticking point in the author's endeavor. Industry is jaded and set in their ways, startups focus on summarization and recommendations and retrieval which are low value in scientific enterprise and academia is focused on automation which turns out brittle. Still, this line of research is needed. Knowledge production is growing rapidly while humans are not getting any smarter. Specialization has meant increases in redundant information, loss of context and a stall in theory production (hence "much less logic and deduction happening"). While the published literature is sorely lacking, humans can with effort extract and or triangulate value from it. Tooling needs to augment that process.

JackFr 5y ago

It's interesting that OP did seemingy little research with respect to existing work in the field. https://www.nlm.nih.gov/medline/medline_overview.html https://www.nlm.nih.gov/medline/medline_overview.html Medline, a searchable online directory of medical research papers has existed for 50 years. The National Library of Medicine for many years was a leader in document search and retrieval before there was a web. In the 80's they were doing vector cosine document similarity, document clustering and automated classifcation. They were also doing so great stuff like indexing papers based on proteins and gene sequences - so a paper which might be in a field completely different than yours might pop up if a similar protein or sequence was mentioned. (Disclosure - I worked at the National Library of Medicine in the 90's) That being said, in the past 30 years search and retrieval exploded to say nothing of ML, but its crazy to ignore the stuff which has come before, AND it's tough to compete with a national lab whose mandate is to basically give the stuff away.

inlitro 5y ago
100% It also felt like a long apology/explanation for Emergent Ventures rather than a true deep analysis. Pretty strong (and often false) statements for only what seems like half a year of total, somewhat vague work.
stevenbedrick 5y ago
The best thing about the NLM's work in this space is how deeply it has been informed by the needs and workflows of the biomedical researchers, which is a perspective that has been sorely lacking in work coming from outsiders. I did think that the author did a good job of outlining (some of) the basic structural issues that make this a tough field to monetize, but even setting those aside, there's no substitute for actually knowing your users and what they need, and that's something the NLM is amazing at. (Disclosure, my PhD was funded by an NLM training grant, some of my research is funded extramurally by the NLM, and I have a lot of NLM colleagues, so I'm maybe a little bit biased)
Ldorigo 5y ago
Medline is great on all levels but what does it have to do with the author's work/product/topic? It's a database of biomedical articles. It's searchable using usual text search ("concept A" AND "concept B"), and has some additional features like MESH terms and metadata. The author is talking about extracting entity-level information from those article and building knowledge bases. Medline provides access to the raw data, but it does nothing like what the article describes (well - there's https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR.html https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR.html , so not entirely true. But it's hardly a final answer to the problem or even close to being usable enough that it makes commercial solutions pointless)

markusstrasser 5y ago

Hey, author here. Great discussion so far. Will update the post with some of the comments and critiques.

Nalta 5y ago

For anyone interested, my whole PhD was in biomedical hypothesis generation! I think the most "serious" attempts at building these systems have been focused around providing assistance to scientists, and not just coming up with new ideas on their own. here's an actual medical paper that my first system, Moliere, was able to help discover: https://link.springer.com/article/10.1007/s11481-019-09885-8 https://link.springer.com/article/10.1007/s11481-019-09885-8

julienchastang 5y ago
Is your PhD thesis online anywhere?
- Nalta 5y ago
  https://sybrandt.com/documents/dissertation.pdf https://sybrandt.com/documents/dissertation.pdf
  - gardenfelder 5y ago
    https://github.com/JSybrandt/MOLIERE https://github.com/JSybrandt/MOLIERE
gardenfelder 5y ago
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8048136/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8048136/
chaxor 5y ago
I also am working within this field in academia, and I must say that I have really enjoyed reading your work. I am focused a bit more on combining biological data sources with text knowledge graphs, but in my literature review of the field, I have found that we both have very similar aspirations for career path. It seems that one major bias that the author of the post's blog has is their heavy conflation of 'worth' with money. Many of us probably realize that this is not true, as academia clearly shows. Furthermore, It's not that crazy that they had trouble making money off of software, as I wouldn't expect many startups at all to be able to get customers from software solutions alone. They also seem to try to compare their success with that of essentially 'other biotech ml companies'; for which I would expect there to be quite a bit of tangible resources that these 'other companies' provide. For example, a startup looking to provide a service of detecting diseases or conditions from DNA methylation data would likely perform the sequencing required before doing an analysis (in order to have good control over experimental conditions). The materials alone in that case could cost quite a bit - so charging a bit more for the analysis isn't that problematic, since the transaction of some currency is already required for covering material costs. Anyway, as you mentioned - it seems it's important to recognize that these systems aren't necessarily meant to generate revenue for a startup, but rather are much more useful as a tool in academia.

holub008 5y ago

> Close to nothing of what makes science actually work is published as text on the web Unless there's some nuance I missed, I immensely disagree with this statement. I'm currently in the biomedical literature review space, and I appreciate the detailed insights. I wonder if the author considered that literature review is used in a wide variety of domains outside pharma/drug discovery (where I perceived their efforts were focused). Regulatory monitoring/reporting, hospital guideline generation, etc. This is a billion dollar industry, and I couldn't agree more that it's technologically underdeveloped. I do not agree that AI-based extraction is the solution, at least in the near-term. The formal methodologies used by reviewers/meta-analysts: search strategy generation, lit search, screening, extraction, critical appraisal, synthesis/statistical analysis, are IMO more nuanced than an AI can capture. They require human input or review. My business is betting on this premise :)

Throwaway197401 5y ago

[dead]

throwaway984393 5y ago

Science is not about innovation. Science is about tiny little results that by themselves have no immediate benefit, but slowly improve our overall understanding, and eventually lead to an unexpected benefit. Science is not developed in order to solve a business problem - it is purely an advancement in overall knowledge of the world (the traditional aim of natural philosophy). In this sense, science is not compatible with business interests.

pezzana 5y ago

> My biggest mistake was that I didn't have experience as a biotech researcher or postdoc working in a lab. That is is big problem - good to recognize it as such. I can tell because the article, though lengthy, never seems to state an explicit problem to be solved. Rather, various ways to apply technology to a field are discussed. This is a recipe for failure. You need 3 things: 1. a problem to be solved 2. a customer who has that problem 3. money in the customer's pocket waiting to be transferred to yours The article never even gets to (1). Regarding (2), if academic groups are the target customer, you're going to have a bad time. They have little money and they tend to be all to happy to build something that sort-of replicates the commercial product you've created for them. This leaves scientific for-profit companies. They have lots of problems (and these days money), but these problems tend to be quite difficult to discover and solve because of the extensive domain and industry knowledge required.

tvhahn 5y ago
Yes. I wonder if the author had first worked on the patent side (I'd be interested to hear more about this idea). Perhaps working on patents first would be a path to get experience (and product-market-fit). From there, one could branch out into other domains (e.g. bio).
- markusstrasser 5y ago
  Patents were the more obvious go to but we didn't want to work with obfuscated patent descriptions for the next years

plaidfuji 5y ago

Having invested quite a bit of my own time into various aspects of the scientific knowledge extraction morass, I’d say the author is largely on point, but there’s a significant, and potentially valuable distinction to be made between extracting research outputs and research inputs. At least in the field of materials science, papers are by and large a record of research outputs. We made material X and it achieved performance Y - here are a bunch of measurements to prove that this is in fact what we made and that it truly achieved performance Y at relevant conditions, etc. In this sense, papers really function as an advertisement: look at what we achieved. What papers do not do is rigorously document inputs, or provide a step-by-step guide to reproduce said results, for obvious reasons. My current take on this topic is that it would be both feasible and valuable to build a knowledge extraction system to compile and compare outputs across a specified field. Think the big “chart of all verified solar cell efficiencies over time” [1], but generated automatically. This would at least immediately orient researchers to the distribution of state of the art results, and help ensure that they don’t omit relevant references in their reviews. But extracting and making sense of inputs (methods), or even “knowledge”? Forget about it. [1] https://www.nrel.gov/pv/cell-efficiency.html https://www.nrel.gov/pv/cell-efficiency.html

bigdict 5y ago

Any article on this topic should mention Tshitoyan et al. https://www.nature.com/articles/s41586-019-1335-8 https://www.nature.com/articles/s41586-019-1335-8

aurizon 5y ago

Business? It is not a business - it is a racket...

jcrben 5y ago

If someone published a promising in vitro study, the mere fact that it was done and even the barest hint of a conclusion can dramatically impact one's decision to pursue that line of research. So I'm a bit skeptical that there's negligible value in biomedical research papers.