3 ms·

What they are and how they can accelerate LLM training

isoprophlex 3y ago
"Training" appears once on the page, in the following paragraph: Generative AI models such as large language models typically leverage huge amounts of unstructured data during their initial training and fine-tuning. Knowledge graphs can be leveraged to identify the structure of the underlying data which can then improve the amount of signal which is extracted from the data during the training. Which is too vague to be of interest, generally uninformative, and furthermore to me reeks of LLM-generated babble instead of something a human wrote.
- krallistic 3y ago
  That point is also a bit of a stretch. While there are a few papers on using KG/Ontologies to enhance training, this is really far from the mainstream, and i would be surprised if it would be used anywhere (outside of a research paper)
throwaway346434 3y ago
Statements and axioms. Merge two knowledge graphs, what can you reason about them? IE people and obesity, people and mortality. Can you find correlations between the two? The next layer up is to prove the trust in the data and your insights. Also you are a decade late, google already bought freebase and made schema.org
- dumb1224 3y ago
  Yes there was once a hype about these graphs and standards so called Resource Description Framework (RDF). It never took off. The knowledge itself requires hefty curation though, if it's in a field that requires low false positive rate such as pharmacology.
  - dvas 3y ago
    Not my area at all, just passing by and wondering to the extent and how they use knowledge graphs for drug discovery. Some time back I had a peek at AstraZeneca's GitHub [0] and got me curious. I know in genomics they try to use custom hardware to accelerate the process using FPGAs and others [1]. Curious if anyone can shed light on knowledge graph use at scale is being accelerated. [0] AstraZeneca; Awesome Drug Discovery Knowledge Graphs https://github.com/AstraZeneca/awesome-drug-discovery-knowledge-graphs https://github.com/AstraZeneca/awesome-drug-discovery-knowle... [1] Gene sequencing accelerates with custom hardware https://www.mewburn.com/news-insights/gene-sequencing-accelerates-with-custom-hardware https://www.mewburn.com/news-insights/gene-sequencing-accele...
    - dumb1224 3y ago
      I only know in my area you can infer gene and protein interaction network using knowledge graphs to some degree. In drug discovery I've seen graphs of chemical knowledges and what can be drugged and what not. https://pubs.acs.org/doi/10.1021/acs.jpca.2c06408 https://pubs.acs.org/doi/10.1021/acs.jpca.2c06408 There might be more publications on mining literature and building such graphs but I'm not following it much since deep learning took over.

"Training" appears once on the page, in the following paragraph: Generative AI models such as large language models typically leverage huge amounts of unstructured data during their initial training and fine-tuning. Knowledge graphs can be leveraged to identify the structure of the underlying data which can then improve the amount of signal which is extracted from the data during the training. Which is too vague to be of interest, generally uninformative, and furthermore to me reeks of LLM-generated babble instead of something a human wrote.

krallistic 3y ago
That point is also a bit of a stretch. While there are a few papers on using KG/Ontologies to enhance training, this is really far from the mainstream, and i would be surprised if it would be used anywhere (outside of a research paper)

throwaway346434 3y ago

Statements and axioms. Merge two knowledge graphs, what can you reason about them? IE people and obesity, people and mortality. Can you find correlations between the two? The next layer up is to prove the trust in the data and your insights. Also you are a decade late, google already bought freebase and made schema.org

dumb1224 3y ago
Yes there was once a hype about these graphs and standards so called Resource Description Framework (RDF). It never took off. The knowledge itself requires hefty curation though, if it's in a field that requires low false positive rate such as pharmacology.
- dvas 3y ago
  Not my area at all, just passing by and wondering to the extent and how they use knowledge graphs for drug discovery. Some time back I had a peek at AstraZeneca's GitHub [0] and got me curious. I know in genomics they try to use custom hardware to accelerate the process using FPGAs and others [1]. Curious if anyone can shed light on knowledge graph use at scale is being accelerated. [0] AstraZeneca; Awesome Drug Discovery Knowledge Graphs https://github.com/AstraZeneca/awesome-drug-discovery-knowledge-graphs https://github.com/AstraZeneca/awesome-drug-discovery-knowle... [1] Gene sequencing accelerates with custom hardware https://www.mewburn.com/news-insights/gene-sequencing-accelerates-with-custom-hardware https://www.mewburn.com/news-insights/gene-sequencing-accele...
  - dumb1224 3y ago
    I only know in my area you can infer gene and protein interaction network using knowledge graphs to some degree. In drug discovery I've seen graphs of chemical knowledges and what can be drugged and what not. https://pubs.acs.org/doi/10.1021/acs.jpca.2c06408 https://pubs.acs.org/doi/10.1021/acs.jpca.2c06408 There might be more publications on mining literature and building such graphs but I'm not following it much since deep learning took over.