4 ms·
Please change the title; it seems very misleading. It has nothing to do with LLMs. https://github.com/bennyschmidt/next-token-prediction/blob/d982dbe2e86db434a
by georgehill 3y ago
Please change the title; it seems very misleading. It has nothing to do with LLMs.
https://github.com/bennyschmidt/next-token-prediction/blob/d982dbe2e86db434adb5c17eb95dd792f783188c/models/TextDecoder/index.js#L291C1-L296C6 https://github.com/bennyschmidt/next-token-prediction/blob/d...
- _akhe 3y agoIt's a next token prediction library, not an LLM. The title says "Build LLMs from scratch" and that's exactly what you can do with it :) Keep in mind an LLM is not "ChatGPT" though ChatGPT uses an LLM.
- georgehill 3y ago> "Build LLMs from scratch" and that's exactly what you can do with it Can you pls explain what you just said as if I were a 5-year-old?
- _akhe 3y agoInstead of using someone else's language library pre-trained on a bunch of text, you can create your own language library trained on your own text. Use cases include: Auto-completion, auto-correct, spell checking, search/lookup, conversation simulation (chatbot), and more. "Trained" means that every word of every sentence of every book loaded in has an embedded score representing how likely it is to appear following a user-provided cursor. The more text you have trained on (see /training/embeddings/) the more scored words it has available, so the more robust and accurate it can be with language. What would make your implementation a "Large" Language Model is if it has many, many, many words and phrases modeled and ranked, not just a small amount. Chat Bots: For something like a realistic conversation simulator (AKA a chat bot) you usually need a Large Language Model (LLM) so that any user-provided cursor yields relevant responses. To filter out the noise of less relevant responses, a good LLM-based chatbot will also use parts-of-speech and sentiment analysis ("attention") to match to the most relevant one. Example cursor: "Paris is", you might get "in France" or "really nice" as equally ranked predictions, what would make a Chat-specific model choose one or the other would be analyzing the input - example if the question was "Where is Paris?" the Chat model would be biased toward answers that mention locations, and respond with "in France". User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" Hope this helps!
- dang 3y agoOk, we've taken the reference to LLMs out of the title now. (Submitted title was "Show HN: Next-token prediction in JavaScript – build fast LLMs from scratch")
- georgehill 3y agoThank you!
- _akhe 3y agoOut of curiosity, why did you link the comment block of the `train` function? Can't figure out the significance of linking this or how it supports your claim.
- _akhe 3y agoHe ain't gonna answer is he? Kinda crazy how karma/reputation works on these sites, you just get what you want and have to explain nothing
- georgehill 3y agoYou've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - https://karpathy.ai/zero-to-hero.html - Highly recommend this one https://www.youtube.com/watch?v=PaCmpygFfXo - https://github.com/karpathy/llm.c - https://ig.ft.com/generative-ai 1: ex: https://news.ycombinator.com/reply?id=40007096&goto=item%3Fid%3D39995908%2340007096 https://news.ycombinator.com/reply?id=40007096&goto=item%3Fi...