3 ms·
Sure, artificial neural network is a great analogy. But that leaves out what it literally is, which is a next-token prediction library that you can use like thi
by _akhe 3y ago
Sure, artificial neural network is a great analogy. But that leaves out what it literally is, which is a next-token prediction library that you can use like this:
// Predict the next word
agent.getTokenPrediction('what');
// Predict the next 5 words
agent.getTokenSequencePrediction('what is', 5);
// Complete the phrase
agent.complete('hopefully');
// Get a top k sample of completion predictions
agent.getCompletions('The sun');
And it has this ability to predict because it's based on a language model that turns mere sentences like: "I want to be at the beach", "I will do it later", "I want to know the answer" into a single model:
{
I: {
want: {
to: {
be: { ... },
know: { ... }
}
},
will: { ... }
},
...
}
This model - call it a prefix tree (trie), call it a Markov model or "chain", or "just lookups", anything you like - but it's more specifically than all those a language model.
Embeddings: Where my embeddings differ is that they're kept in a structure reflective of their source context. That's how I know what the meaning of the word is. OpenAI's embedding is like this:
Text: "The dog"
Embedding:
{
"object": "embedding",
"index": 0,
"embedding": [
-0.006929283495992422,
-0.005336422007530928,
-4.547132266452536e-05,
-0.024047505110502243,
...
],
}
And those values, particularly those for "dog" will be - like you said - computable with others to find things like similarity, etc. And using that math, they can tell you what the next word will be - and probably a lot of other useful chat-specific things I haven't gotten into.
My method is also commonly used in machine learning, and would be TF-IDF vectorization except that I don't fully vectorize the semantics, because it's just based on frequency and token structure:
{
"the": {
"dog": {
"was": 834,
"is": 721,
"will": 36
"has": 442
}
}
}
This system predicts "was" as the next word because it usually is the next word after "dog" (in the source data). This library was built to ultimately provide completions, not have a conversation, so no doubt OpenAI's approach works better for chat.
I am however already making a chat model. Here's my approach if anyone cares: The completer already gives great completions and fast, but some of them make no sense to what was asked. The chat model I'm working on here (https://github.com/bennyschmidt/llimo/pull/1 https://github.com/bennyschmidt/llimo/pull/1) can just get all completions and use parts-of-speech codes to match a completion to the cursor. I don't have this fully implemented yet, but you can get the idea in this PR. This is like an NLP layer specific to chat - has nothing to do with the next-token prediction in general, and there are no NLP libraries in `next-token-prediction` (the npm). The example I've been using to explain this is:
User: "Where is Paris?"
Convert to cursor: "Paris is"
Get completions: "in France", "really nice", "a first name", etc.
Analyze input: "Where is" refers to a location
Analyze completions: "in France" refers to a location
Answer: "Paris is in France"
I hope it makes sense! Thanks for your comment.
Edit: And yes, again, the library does not claim to "be an LLM" - it's not that at all, it's actually a next-token prediction library that has methods like `getTokenPrediction` etc. Though you are set up to build an LLM with this if you train enough data :) The library is not going to provide a `chat` function like GPT, not going to provide an img2img API like Stable Diffusion, that would be whatever chat-based or image-based LLM is being made with this library.
- kybernetikos 3y ago> Sure, artificial neural network is a great analogy. But that leaves out what it literally is It's not an analogy - an artificial neural network is literally what an LLM is. A trie is not an LLM, because it's not a neural network. If someone asked how many parameters a trie loaded with many sentences has, that question wouldn't even make sense, because it doesn't model weights between neurons, unlike an LLM. I think perhaps you're getting confused between the class of 'next token predicting algorithms' (which is a class with lots of different algorithms in it) and 'LLMs'. LLMs are a specific kind of next token predicting algorithm that work in a particular way. You can make a next token predictor out of a trie (as you have), but it is not an LLM.
- _akhe 3y agoAgain, the library does not claim to "be an LLM" - it's not that at all, it's actually a next-token prediction library that has methods like `getTokenPrediction` etc. Though you are set up to build an LLM with this if you train enough data :) The library is not going to provide a `chat` function like GPT, not going to provide an img2img API like Stable Diffusion, that would be whatever chat-based or image-based LLM is being made with this library. (reply was not visible before, so I appended as an Edit)
- kybernetikos 3y ago> Though you are set up to build an LLM with this if you train enough data But you aren't. No amount of data loaded into a trie will turn the trie into a neural net. > The library is not going to provide a `chat` function like GPT, Providing a 'chat' function is nothing to do with what a LLM is. Here's the first sentence from the second paragraph of wikipedia on Large Language Models: "LLMs are artificial neural networks." When someone points out that you wouldn't use this kind of library to create an LLM, you generally say something about how you think they're complaining that it doesn't chat with you. It's not about chat, it's about the fact that the functionality you've written about has nothing to do with the way that LLMs work. LLMs are artificial neural networks. What you've written is a trie, not an artificial neural network. Artificial neural networks need fast matrix multiplication, not tries.