5 ms·

This is what HN is all about. This probably isn't your next big LLM project but its a wonderfully simple example that shows what is happening underneath the hoo

This is what HN is all about. This probably isn't your next big LLM project but its a wonderfully simple example that shows what is happening underneath the hood. Do you plan on writing up something like a blog post to go with it?

_akhe 3y ago
Hell yeah :) At first I thought a simple Show HN titled "Next token prediction in 100 lines of JavaScript" would be cool, but the project grew and now it's a few different modules and things going on. A couple people have basically said "but what does it do?", "what value does it add?" So I am planning a series of examples with videos for use cases like: Auto-completion, auto-correct, spell checking, search/lookup, and if I can get around to making an ImageDecoder, could get into "next-pixel prediction". Beyond prediction, I want to make a YouTube video when I can show a working example of a really good chatbot than anyone can just npm install and use, would be funny to give some of the open models a run for their money - but you're right, it's just a hobby project for now. Working on that in a separate project: https://github.com/bennyschmidt/llimo https://github.com/bennyschmidt/llimo Thanks for the comment and motivation to write about it!
kybernetikos 3y ago
It is a fun and interesting project, but it doesn't really show what is happening under the hood of LLMs since they work on a completely different principle.
- _akhe 3y ago
  > LLMs work on a completely different principle News to me - can you tell me what principle you think an LLM works on?
  - kybernetikos 3y ago
    LLMs are artificial neural networks. What you seem to be describing is a trie. You dont use tries at all when building neural networks, you use matrices, and typically algorithms like backpropagation to train the weights of the network. Now, I haven't gone through your code, I'm relying mainly on your description, so what you've said may have confused me and be different from what you've done, but what you seem to be calling an 'embedding' is a trie storing a dataset of sentences. The actual embedding used in LLMs is a mapping from words (a word/token list is effectively a 1 dimensional space) into a much higher dimensional space that captures aspects of the meaning of the words being embedded (e.g. similar concepts may map to similar vectors in the space, or you might be able to do classic word arithmetic like king - man + woman = queen with the vectors that each word maps to). Then the neural network works, ultimately resulting in a vector in that space, which we map into the next token. This is a very different process to looking up a set of options in a trie, and I would expect it to outperform the trie method by an enormous amount on longer outputs, or where outputs that are not in the training data are needed.
    - _akhe 3y ago
      Sure, artificial neural network is a great analogy. But that leaves out what it literally is, which is a next-token prediction library that you can use like this: // Predict the next word agent.getTokenPrediction('what'); // Predict the next 5 words agent.getTokenSequencePrediction('what is', 5); // Complete the phrase agent.complete('hopefully'); // Get a top k sample of completion predictions agent.getCompletions('The sun'); And it has this ability to predict because it's based on a language model that turns mere sentences like: "I want to be at the beach", "I will do it later", "I want to know the answer" into a single model: { I: { want: { to: { be: { ... }, know: { ... } } }, will: { ... } }, ... } This model - call it a prefix tree (trie), call it a Markov model or "chain", or "just lookups", anything you like - but it's more specifically than all those a language model. Embeddings: Where my embeddings differ is that they're kept in a structure reflective of their source context. That's how I know what the meaning of the word is. OpenAI's embedding is like this: Text: "The dog" Embedding: { "object": "embedding", "index": 0, "embedding": [ -0.006929283495992422, -0.005336422007530928, -4.547132266452536e-05, -0.024047505110502243, ... ], } And those values, particularly those for "dog" will be - like you said - computable with others to find things like similarity, etc. And using that math, they can tell you what the next word will be - and probably a lot of other useful chat-specific things I haven't gotten into. My method is also commonly used in machine learning, and would be TF-IDF vectorization except that I don't fully vectorize the semantics, because it's just based on frequency and token structure: { "the": { "dog": { "was": 834, "is": 721, "will": 36 "has": 442 } } } This system predicts "was" as the next word because it usually is the next word after "dog" (in the source data). This library was built to ultimately provide completions, not have a conversation, so no doubt OpenAI's approach works better for chat. I am however already making a chat model. Here's my approach if anyone cares: The completer already gives great completions and fast, but some of them make no sense to what was asked. The chat model I'm working on here (https://github.com/bennyschmidt/llimo/pull/1 https://github.com/bennyschmidt/llimo/pull/1) can just get all completions and use parts-of-speech codes to match a completion to the cursor. I don't have this fully implemented yet, but you can get the idea in this PR. This is like an NLP layer specific to chat - has nothing to do with the next-token prediction in general, and there are no NLP libraries in `next-token-prediction` (the npm). The example I've been using to explain this is: User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" I hope it makes sense! Thanks for your comment. Edit: And yes, again, the library does not claim to "be an LLM" - it's not that at all, it's actually a next-token prediction library that has methods like `getTokenPrediction` etc. Though you are set up to build an LLM with this if you train enough data :) The library is not going to provide a `chat` function like GPT, not going to provide an img2img API like Stable Diffusion, that would be whatever chat-based or image-based LLM is being made with this library.

_akhe 3y ago

Hell yeah :) At first I thought a simple Show HN titled "Next token prediction in 100 lines of JavaScript" would be cool, but the project grew and now it's a few different modules and things going on. A couple people have basically said "but what does it do?", "what value does it add?" So I am planning a series of examples with videos for use cases like: Auto-completion, auto-correct, spell checking, search/lookup, and if I can get around to making an ImageDecoder, could get into "next-pixel prediction". Beyond prediction, I want to make a YouTube video when I can show a working example of a really good chatbot than anyone can just npm install and use, would be funny to give some of the open models a run for their money - but you're right, it's just a hobby project for now. Working on that in a separate project: https://github.com/bennyschmidt/llimo https://github.com/bennyschmidt/llimo Thanks for the comment and motivation to write about it!

kybernetikos 3y ago

It is a fun and interesting project, but it doesn't really show what is happening under the hood of LLMs since they work on a completely different principle.

_akhe 3y ago
> LLMs work on a completely different principle News to me - can you tell me what principle you think an LLM works on?
- kybernetikos 3y ago
  LLMs are artificial neural networks. What you seem to be describing is a trie. You dont use tries at all when building neural networks, you use matrices, and typically algorithms like backpropagation to train the weights of the network. Now, I haven't gone through your code, I'm relying mainly on your description, so what you've said may have confused me and be different from what you've done, but what you seem to be calling an 'embedding' is a trie storing a dataset of sentences. The actual embedding used in LLMs is a mapping from words (a word/token list is effectively a 1 dimensional space) into a much higher dimensional space that captures aspects of the meaning of the words being embedded (e.g. similar concepts may map to similar vectors in the space, or you might be able to do classic word arithmetic like king - man + woman = queen with the vectors that each word maps to). Then the neural network works, ultimately resulting in a vector in that space, which we map into the next token. This is a very different process to looking up a set of options in a trie, and I would expect it to outperform the trie method by an enormous amount on longer outputs, or where outputs that are not in the training data are needed.
  - _akhe 3y ago
    Sure, artificial neural network is a great analogy. But that leaves out what it literally is, which is a next-token prediction library that you can use like this: // Predict the next word agent.getTokenPrediction('what'); // Predict the next 5 words agent.getTokenSequencePrediction('what is', 5); // Complete the phrase agent.complete('hopefully'); // Get a top k sample of completion predictions agent.getCompletions('The sun'); And it has this ability to predict because it's based on a language model that turns mere sentences like: "I want to be at the beach", "I will do it later", "I want to know the answer" into a single model: { I: { want: { to: { be: { ... }, know: { ... } } }, will: { ... } }, ... } This model - call it a prefix tree (trie), call it a Markov model or "chain", or "just lookups", anything you like - but it's more specifically than all those a language model. Embeddings: Where my embeddings differ is that they're kept in a structure reflective of their source context. That's how I know what the meaning of the word is. OpenAI's embedding is like this: Text: "The dog" Embedding: { "object": "embedding", "index": 0, "embedding": [ -0.006929283495992422, -0.005336422007530928, -4.547132266452536e-05, -0.024047505110502243, ... ], } And those values, particularly those for "dog" will be - like you said - computable with others to find things like similarity, etc. And using that math, they can tell you what the next word will be - and probably a lot of other useful chat-specific things I haven't gotten into. My method is also commonly used in machine learning, and would be TF-IDF vectorization except that I don't fully vectorize the semantics, because it's just based on frequency and token structure: { "the": { "dog": { "was": 834, "is": 721, "will": 36 "has": 442 } } } This system predicts "was" as the next word because it usually is the next word after "dog" (in the source data). This library was built to ultimately provide completions, not have a conversation, so no doubt OpenAI's approach works better for chat. I am however already making a chat model. Here's my approach if anyone cares: The completer already gives great completions and fast, but some of them make no sense to what was asked. The chat model I'm working on here (https://github.com/bennyschmidt/llimo/pull/1 https://github.com/bennyschmidt/llimo/pull/1) can just get all completions and use parts-of-speech codes to match a completion to the cursor. I don't have this fully implemented yet, but you can get the idea in this PR. This is like an NLP layer specific to chat - has nothing to do with the next-token prediction in general, and there are no NLP libraries in `next-token-prediction` (the npm). The example I've been using to explain this is: User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" I hope it makes sense! Thanks for your comment. Edit: And yes, again, the library does not claim to "be an LLM" - it's not that at all, it's actually a next-token prediction library that has methods like `getTokenPrediction` etc. Though you are set up to build an LLM with this if you train enough data :) The library is not going to provide a `chat` function like GPT, not going to provide an img2img API like Stable Diffusion, that would be whatever chat-based or image-based LLM is being made with this library.