5 ms·

I’m late to the game about this. So I’ll ask a stupid question. As a contrived example, what happens if you feed the LoTR books, the Hobbit, the Silmarillion,

I’m late to the game about this. So I’ll ask a stupid question. As a contrived example, what happens if you feed the LoTR books, the Hobbit, the Silmarillion, and whatever else is germane, into an LLM? Is there a base, empty, “ignorant” LLM that is used as a seed? Do you end up with a Middle Earth savant? Just how does all this work?

LeoPanthera 3y ago
There isn't enough text in the Tolkein works to generate a functional LLM. You need to start with a base model that contains enough English (or language of your choice) to become functional. Such a model (LLaMa is a good example) is not "ignorant," but rather a generalized model capable of a wide range of language tasks. This base model does not have specialized knowledge in any one area but has a broad understanding based on the diverse training data. If you were to "feed" Tolkien's specific books into this general LLM, the model wouldn't become a Middle Earth savant. It would still provide responses based on its broad training. It might generate text that reflects the style or themes of Tolkien's work if it has learned this from the broader training data, but its responses would be based on patterns learned from the entire dataset, not just those books.
- whartung 3y ago
  So, it wouldn't necessarily "know" much about Middle Earth, but might take a stab at writing like Tolkien?
  - flexagoon 3y ago
    If you give Tolkien books to a newborn child, they won't become a Tolkien expert. You need to first teach them English, and then give them the books. They will be able to answer questions about Middle Earth, but they'll also be able to form any other English sentence based on what they learned previously. It's basically the same with LLMs.
esafak 3y ago
You would fine tune a pretrained LLM because those books are written in English. And natural languages are in flux, and the corpuses that describe them are not neutral, so you can impose fairness after the fact. Sebastian Raschka has written some relevant popular articles, like: https://magazine.sebastianraschka.com/p/understanding-large-language-models https://magazine.sebastianraschka.com/p/understanding-large-... https://magazine.sebastianraschka.com/p/finetuning-large-language-models https://magazine.sebastianraschka.com/p/finetuning-large-lan...

LeoPanthera 3y ago

There isn't enough text in the Tolkein works to generate a functional LLM. You need to start with a base model that contains enough English (or language of your choice) to become functional. Such a model (LLaMa is a good example) is not "ignorant," but rather a generalized model capable of a wide range of language tasks. This base model does not have specialized knowledge in any one area but has a broad understanding based on the diverse training data. If you were to "feed" Tolkien's specific books into this general LLM, the model wouldn't become a Middle Earth savant. It would still provide responses based on its broad training. It might generate text that reflects the style or themes of Tolkien's work if it has learned this from the broader training data, but its responses would be based on patterns learned from the entire dataset, not just those books.

whartung 3y ago
So, it wouldn't necessarily "know" much about Middle Earth, but might take a stab at writing like Tolkien?
- flexagoon 3y ago
  If you give Tolkien books to a newborn child, they won't become a Tolkien expert. You need to first teach them English, and then give them the books. They will be able to answer questions about Middle Earth, but they'll also be able to form any other English sentence based on what they learned previously. It's basically the same with LLMs.

esafak 3y ago

You would fine tune a pretrained LLM because those books are written in English. And natural languages are in flux, and the corpuses that describe them are not neutral, so you can impose fairness after the fact. Sebastian Raschka has written some relevant popular articles, like: https://magazine.sebastianraschka.com/p/understanding-large-language-models https://magazine.sebastianraschka.com/p/understanding-large-... https://magazine.sebastianraschka.com/p/finetuning-large-language-models https://magazine.sebastianraschka.com/p/finetuning-large-lan...