5 ms·

You know you could probably get really far just training an LLM on wikipedia and all linked citations, and nothing else The whole problem of wikipedia only bei

You know you could probably get really far just training an LLM on wikipedia and all linked citations, and nothing else The whole problem of wikipedia only being an "inch deep" on any given topic is basically solved if the LLM also has to read every cited work in full And maybe citation counts could affect exposure to that work during training

jabbany 3y ago
A lot of full text for research (outside CS) is still locked up behind subscription paywalls. Plus, often times PDFs are not the best format to extract text out of. Interesting suggestion but probably a lot of practical limitations.
- sam0x17 3y ago
  That's also a blessing in disguise though, we don't want an LLM trained on closed source stuff, so ignoring sources it can't access is probably a great thing
wizofaus 3y ago
The ability to predict the expected answer to a given question isn't something I could see naturally falling out of those sources though, unlike an LLM trained on text from online forums and the like.
- sam0x17 3y ago
  AFAIK the training works by just selecting random words in sentences to have the LLM "fill in" until it generalizes to being able to write entire sentences. The training data can be completely unstructured.
slg 3y ago
> and all linked citations I wonder what percentage of Wikipedia citations are actually currently available on the internet. For example, here is today's featured article[1]. The majority of references on that pages are books, journals, magazines, television, and unlinked news articles that can't be easily accessed. Plus on more niche topics, it is common for the externally linked references to disappear over time. [1] - https://en.wikipedia.org/wiki/David_Kelly_(weapons_expert) https://en.wikipedia.org/wiki/David_Kelly_(weapons_expert)
- harshreality 3y ago
  archive dot org and dot is, sci-hub and λιβgen / zλιβ will cover a lot of those text sources. Aren't bots largely what's responsible for links being updated to point to archived sources? I've noticed archive links a lot lately. Someone doing serious AI training will mirror sci-hub and λιβgen first, so they'll already have a fair amount of the (good quality) referenced papers and educational books. Wikipedia (and citation count on google scholar for papers) could be used as a filter for which books and papers to train on first.
awb 3y ago
That’s basically Google’s PageRank algorithm with Wikipedia as the 10/10 ranked source of truth.
kernal 3y ago
[flagged]
- flangola7 3y ago
  That's a strong accusation. What evidence do you have? What are the "ideological views" and why do you think they are baseless?

jabbany 3y ago

A lot of full text for research (outside CS) is still locked up behind subscription paywalls. Plus, often times PDFs are not the best format to extract text out of. Interesting suggestion but probably a lot of practical limitations.

sam0x17 3y ago
That's also a blessing in disguise though, we don't want an LLM trained on closed source stuff, so ignoring sources it can't access is probably a great thing

wizofaus 3y ago

The ability to predict the expected answer to a given question isn't something I could see naturally falling out of those sources though, unlike an LLM trained on text from online forums and the like.

sam0x17 3y ago
AFAIK the training works by just selecting random words in sentences to have the LLM "fill in" until it generalizes to being able to write entire sentences. The training data can be completely unstructured.

slg 3y ago

> and all linked citations I wonder what percentage of Wikipedia citations are actually currently available on the internet. For example, here is today's featured article[1]. The majority of references on that pages are books, journals, magazines, television, and unlinked news articles that can't be easily accessed. Plus on more niche topics, it is common for the externally linked references to disappear over time. [1] - https://en.wikipedia.org/wiki/David_Kelly_(weapons_expert) https://en.wikipedia.org/wiki/David_Kelly_(weapons_expert)

harshreality 3y ago
archive dot org and dot is, sci-hub and λιβgen / zλιβ will cover a lot of those text sources. Aren't bots largely what's responsible for links being updated to point to archived sources? I've noticed archive links a lot lately. Someone doing serious AI training will mirror sci-hub and λιβgen first, so they'll already have a fair amount of the (good quality) referenced papers and educational books. Wikipedia (and citation count on google scholar for papers) could be used as a filter for which books and papers to train on first.

awb 3y ago

That’s basically Google’s PageRank algorithm with Wikipedia as the 10/10 ranked source of truth.

kernal 3y ago

[flagged]

flangola7 3y ago
That's a strong accusation. What evidence do you have? What are the "ideological views" and why do you think they are baseless?