5 ms·

This is fantastic. I’d love to see a system that uses an LLM to generate knowledge graphs from academic papers to make them machine readable. Some kind of prom

This is fantastic. I’d love to see a system that uses an LLM to generate knowledge graphs from academic papers to make them machine readable. Some kind of prompt like “does paper P contain idea A and does it suggest that A is true.” Then you could automatically categorise citations by whether they agree/disagree with the cited paper. Sometimes I see papers with 2,000 citations and I wonder: how many of those are dis/agreeing with the paper.

throwaway290 3y ago
By that logic why don't you just directly ask the LLM on whether a citation agrees or not, you are already trusting it to be correct with that graph in the first place...
- brutusborn 3y ago
  You are correct, I wrote that in a rush and mixed up examples in my head. I don’t think you need to trust the LLMs for this kind of thing to be very useful. The LLM could generate the KG with every node labelled as “autogenerated.” When you use the graph for research, you are still going to read the papers you are interested in so you can then update the relevant citation node with the label “human checked.” If a research group uses the same graph over time, the nodes will gradually become “trustworthy” (ie verified by humans). Maybe even get reviewers to update a papers graph during review and publish that for other groups to add to their graphs.
  - visarga 3y ago
    That's a great idea. It would be easier to know if anyone done what you want, get a better overview of the current knowledge.
  - throwaway290 3y ago
    Then why not just author knowledge as a graph in the first place, as humans?
    - brutusborn 3y ago
      That would be great but if we can’t even convince most researchers to use open science principles then good luck convincing them to spend the time to convert their papers into graphs.
      - throwaway290 3y ago
        I reckon researchers would be on board if the tooling is there.
- 3abiton 3y ago
  I already feel synthesis is becoming a useless skill for humans.
anentropic 3y ago
There are already models which are specialised to this task, e.g. https://huggingface.co/Babelscape/rebel-large https://huggingface.co/Babelscape/rebel-large (if I understood you correctly) Though with LLM and sufficient context length you could probably just use that prompt directly on the academic paper without ever generating a knowledge graph
nvm0n2 3y ago
This has already been studied. Negative citations are vanishingly rare. So virtually all of them will be either neutral or positive.
- photonthug 3y ago
  Applications to case law might be interesting since establishing precedent is somewhat more nonbinary
- jll29 3y ago
  > Sometimes I see papers with 2,000 citations and I wonder: how many of those are dis/agreeing with the paper. One example of an author that is very influential, despite causing a lot of disagreement (even in more than one discipline) is Noam Chomsky, who is also the most cited person alive, and the second most cited person in recorded history after Aristotle. His views about generative grammar are in part revolutionary, in part plain wrong; your assessment of his views about the Palestine conflict and U.S. foreign politics will largely depend on your political leanings; and his contribution to formal language theory is fundamental regardless of your leanings (Chomsky hierarchy; Chomsky Normal Form). > This has already been studied. Negative citations are vanishingly rare. So virtually all of them will be either neutral or positive. Might be a difference between science/engineering (where true) and humanities (where a larger amount is negative).
  - melagonster 3y ago
    sounds like biologists cites Darwin. if you find something he did not find, cool; if your region is similar to him, cooler :) I guess most of conditions is because when biologists debate, they only have one opponent.
- brutusborn 3y ago
  Know any good papers on this?
  - nvm0n2 3y ago
    https://www.pnas.org/doi/10.1073/pnas.1502280112 https://www.pnas.org/doi/10.1073/pnas.1502280112 Out of 762,355 citations from 15,731 articles in the Journal of Immunology (1998–2007), we identified 18,304 as negative (about 2.4% of the total).

throwaway290 3y ago

By that logic why don't you just directly ask the LLM on whether a citation agrees or not, you are already trusting it to be correct with that graph in the first place...

brutusborn 3y ago
You are correct, I wrote that in a rush and mixed up examples in my head. I don’t think you need to trust the LLMs for this kind of thing to be very useful. The LLM could generate the KG with every node labelled as “autogenerated.” When you use the graph for research, you are still going to read the papers you are interested in so you can then update the relevant citation node with the label “human checked.” If a research group uses the same graph over time, the nodes will gradually become “trustworthy” (ie verified by humans). Maybe even get reviewers to update a papers graph during review and publish that for other groups to add to their graphs.
- visarga 3y ago
  That's a great idea. It would be easier to know if anyone done what you want, get a better overview of the current knowledge.
- throwaway290 3y ago
  Then why not just author knowledge as a graph in the first place, as humans?
  - brutusborn 3y ago
    That would be great but if we can’t even convince most researchers to use open science principles then good luck convincing them to spend the time to convert their papers into graphs.
    - throwaway290 3y ago
      I reckon researchers would be on board if the tooling is there.
3abiton 3y ago
I already feel synthesis is becoming a useless skill for humans.

anentropic 3y ago

There are already models which are specialised to this task, e.g. https://huggingface.co/Babelscape/rebel-large https://huggingface.co/Babelscape/rebel-large (if I understood you correctly) Though with LLM and sufficient context length you could probably just use that prompt directly on the academic paper without ever generating a knowledge graph

nvm0n2 3y ago

This has already been studied. Negative citations are vanishingly rare. So virtually all of them will be either neutral or positive.

photonthug 3y ago
Applications to case law might be interesting since establishing precedent is somewhat more nonbinary
jll29 3y ago
> Sometimes I see papers with 2,000 citations and I wonder: how many of those are dis/agreeing with the paper. One example of an author that is very influential, despite causing a lot of disagreement (even in more than one discipline) is Noam Chomsky, who is also the most cited person alive, and the second most cited person in recorded history after Aristotle. His views about generative grammar are in part revolutionary, in part plain wrong; your assessment of his views about the Palestine conflict and U.S. foreign politics will largely depend on your political leanings; and his contribution to formal language theory is fundamental regardless of your leanings (Chomsky hierarchy; Chomsky Normal Form). > This has already been studied. Negative citations are vanishingly rare. So virtually all of them will be either neutral or positive. Might be a difference between science/engineering (where true) and humanities (where a larger amount is negative).
- melagonster 3y ago
  sounds like biologists cites Darwin. if you find something he did not find, cool; if your region is similar to him, cooler :) I guess most of conditions is because when biologists debate, they only have one opponent.
brutusborn 3y ago
Know any good papers on this?
- nvm0n2 3y ago
  https://www.pnas.org/doi/10.1073/pnas.1502280112 https://www.pnas.org/doi/10.1073/pnas.1502280112 Out of 762,355 citations from 15,731 articles in the Journal of Immunology (1998–2007), we identified 18,304 as negative (about 2.4% of the total).