3 ms·

It's a next token prediction library, not an LLM. The title says "Build LLMs from scratch" and that's exactly what you can do with it :) Keep in mind an LLM is

by _akhe 3y ago

It's a next token prediction library, not an LLM. The title says "Build LLMs from scratch" and that's exactly what you can do with it :) Keep in mind an LLM is not "ChatGPT" though ChatGPT uses an LLM.

georgehill 3y ago
> "Build LLMs from scratch" and that's exactly what you can do with it Can you pls explain what you just said as if I were a 5-year-old?
- _akhe 3y ago
  Instead of using someone else's language library pre-trained on a bunch of text, you can create your own language library trained on your own text. Use cases include: Auto-completion, auto-correct, spell checking, search/lookup, conversation simulation (chatbot), and more. "Trained" means that every word of every sentence of every book loaded in has an embedded score representing how likely it is to appear following a user-provided cursor. The more text you have trained on (see /training/embeddings/) the more scored words it has available, so the more robust and accurate it can be with language. What would make your implementation a "Large" Language Model is if it has many, many, many words and phrases modeled and ranked, not just a small amount. Chat Bots: For something like a realistic conversation simulator (AKA a chat bot) you usually need a Large Language Model (LLM) so that any user-provided cursor yields relevant responses. To filter out the noise of less relevant responses, a good LLM-based chatbot will also use parts-of-speech and sentiment analysis ("attention") to match to the most relevant one. Example cursor: "Paris is", you might get "in France" or "really nice" as equally ranked predictions, what would make a Chat-specific model choose one or the other would be analyzing the input - example if the question was "Where is Paris?" the Chat model would be biased toward answers that mention locations, and respond with "in France". User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" Hope this helps!

by _akhe 3y ago

georgehill 3y ago

> "Build LLMs from scratch" and that's exactly what you can do with it Can you pls explain what you just said as if I were a 5-year-old?

_akhe 3y ago
Instead of using someone else's language library pre-trained on a bunch of text, you can create your own language library trained on your own text. Use cases include: Auto-completion, auto-correct, spell checking, search/lookup, conversation simulation (chatbot), and more. "Trained" means that every word of every sentence of every book loaded in has an embedded score representing how likely it is to appear following a user-provided cursor. The more text you have trained on (see /training/embeddings/) the more scored words it has available, so the more robust and accurate it can be with language. What would make your implementation a "Large" Language Model is if it has many, many, many words and phrases modeled and ranked, not just a small amount. Chat Bots: For something like a realistic conversation simulator (AKA a chat bot) you usually need a Large Language Model (LLM) so that any user-provided cursor yields relevant responses. To filter out the noise of less relevant responses, a good LLM-based chatbot will also use parts-of-speech and sentiment analysis ("attention") to match to the most relevant one. Example cursor: "Paris is", you might get "in France" or "really nice" as equally ranked predictions, what would make a Chat-specific model choose one or the other would be analyzing the input - example if the question was "Where is Paris?" the Chat model would be biased toward answers that mention locations, and respond with "in France". User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" Hope this helps!