6 ms·

Does anyone have any resources they recommend for just understanding the base terminology of models like this? I always see the terms "weights", "tokens", "mode

by jjice 4y ago

Does anyone have any resources they recommend for just understanding the base terminology of models like this? I always see the terms "weights", "tokens", "model", etc. I feel like I understand what these mean, but I have no idea what I need to care about them for in open models like this? If I were to download an open model to run on my machine, would I download the weights? I'm just ignorant in the ML space I guess but not sure where to start.

zoogeny 4y ago
Andrej Karpathy's Zero to Hero video series [1] is a good middle ground. It isn't super low-level but it also isn't super high-level. I think seeing how the pieces actually fit together in a working project is valuable to get a real understanding. After going through this series I can say I basically understand weights, tokens, back-propagation, layers, embeddings, etc. 1. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- data_maan 4y ago
  When was this published? Is this an older tutorial by Karpathy? Just curious, didn't see any date...
  - knutzui 4y ago
    The first class is 8 months old and the latest one is 3 months old. If you click on the links, they'll direct you to YouTube videos.
  - rini17 4y ago
    On youtube you can. First video 8 months ago.
- CamperBob2 4y ago
  I'm working my way through that series now. He really is a good teacher -- I keep waiting for the inevitable "Next, draw the rest of the fucking owl" moment, but so far he does seem to be sticking to his commitment to a from-scratch approach.
mabbo 4y ago
Weights are basically number/float variables. In neural networks, vectors of values are multiplied (or math'd in some way) by weights to get new vectors of values. A 500 billion weight model has 500 billion variables, all carefully chosen via training. A model is some architecture of how data will flow through these weight matrices, along with the values of each weight. Tokens are sort of "words" in a sentence, but the ML may be translating the word itself into a more abstract concept in 'word space': eg, a bunch of floating point values. At least some of what I just said is probably wrong, but now someone will correct me and we'll both me more right!
- mrtranscendence 4y ago
  At a first approximation this is pretty good. I wouldn't say this exactly: > A model is some architecture of how data will flow through these weight matrices, along with the values of each weight. Because data doesn't really flow through weight matrices, though perhaps this is true if you squint at very simple models. Deep learning architectures are generally more complicated than multiplying values by weights and pushing the results to the next layer, though which architecture to use depends heavily on context. > Tokens are sort of "words" in a sentence Tokens are funny. What a token is depends on the context of the model you're using, but generally a token is a portion of a word. (Why? Efficiency is one reason; handling unknown words is another.)
  - jstarfish 4y ago
    > What a token is depends on the context of the model you're using, but generally a token is a portion of a word. When doing quick estimates, I just assume every syllable is a token. It tends to overestimate, which is fine for my OOM mitigation purposes.
mhh__ 4y ago
A book about AI. (Norvig and Russell comes to mind)
heliophobicdude 4y ago
Probably not the answer you would like but I think your approach to download them and figure out how to run them on your machine is a good one. You don't need to understand everything to get something working. It can be overwhelming and unproductive to know everything before getting started. To learn more deeply though, get started with getting it to work and when you are curious or something doesn't work, try to understand why and recursively go back to fill in the foundational details. Example, download the code try to get it to work. Why is it not working? Oh it's trying to look for the model. Search for how to get the model and set it up. Then key step, recursively look up every single thing in the guide or set up. Don't try to set something up or fix some thing without truly understanding what it is you are doing (e.g. copy and paste). This gives you a structured why to fill in the foundations of what it is you are trying to get to work in a more focused and productive manner. At the end you might realize that their approach or yours is not optimal "oh it was telling me to download the 65k model when I can only run 7k on my machine bc ..."
visarga 4y ago
Psst ... why don't you spend 30 minutes of quality time with chatGPT and get to the bottom of this? Get those personalised explanations and enjoy its unlimited patience. I have felt the same in the past, related to a completely different topic. I know how it feels, it's like people are not saying things what they are, just using weird words. "weights" - synapses in the AI brain "tokens" - word fragments "model" - of course, the model is the AI brain "context" - the model can only handle a piece of text, can't put whole books in, so this limited window is the context "GPT" - predicts the next word, trained on everything; if you feed its last predicted word back in, it can write long texts "LoRA" - a lightweight plug-in model for tweaking the big model "loss" - a score telling how bad is the output "training" - change the model until it fits the data "quantisation" - making a low precision version of the model because it still works, but now is much faster and needs less compute "embedding" - just a vector, it stands for the meaning of a word token or a piece of image; these embeddings are learned
- agentdrtran 4y ago
  > why don't you spend 30 minutes of quality time with chatGPT and get to the bottom of this? when it can hallucinate content, why do that instead of reading a blog post from an expert?
  - visarga 4y ago
    Oh no, it will hallucinate an obscure fact, but not basics. It's pretty good at reciting theory, it would pass many ML engineering theoretical interviews. If you don't trust its memory, copy a piece of high quality text in the topic of interest inside the context, as reference.
    - agentdrtran 3y ago
      it's repeatedly made up entire quotes and research papers?
- tikkun 4y ago
  I think this is the right answer, ChatGPT is an excellent 1-1 tutor.
2devnull 4y ago
For a good general non-technical introduction I recommend the YouTube computerphile series related to language models, transformers and other general concepts. If you are interested in actually doing stuff there’s an over abundance of material out there already, if you try looking.
bobbyi 4y ago
I haven't watched it yet, but the Practical Deep Learning for Coders course that's available on YouTube is often recommended https://course.fast.ai/ https://course.fast.ai/

by jjice 4y ago

zoogeny 4y ago

Andrej Karpathy's Zero to Hero video series [1] is a good middle ground. It isn't super low-level but it also isn't super high-level. I think seeing how the pieces actually fit together in a working project is valuable to get a real understanding. After going through this series I can say I basically understand weights, tokens, back-propagation, layers, embeddings, etc. 1. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

data_maan 4y ago
When was this published? Is this an older tutorial by Karpathy? Just curious, didn't see any date...
- knutzui 4y ago
  The first class is 8 months old and the latest one is 3 months old. If you click on the links, they'll direct you to YouTube videos.
- rini17 4y ago
  On youtube you can. First video 8 months ago.
CamperBob2 4y ago
I'm working my way through that series now. He really is a good teacher -- I keep waiting for the inevitable "Next, draw the rest of the fucking owl" moment, but so far he does seem to be sticking to his commitment to a from-scratch approach.

mabbo 4y ago

Weights are basically number/float variables. In neural networks, vectors of values are multiplied (or math'd in some way) by weights to get new vectors of values. A 500 billion weight model has 500 billion variables, all carefully chosen via training. A model is some architecture of how data will flow through these weight matrices, along with the values of each weight. Tokens are sort of "words" in a sentence, but the ML may be translating the word itself into a more abstract concept in 'word space': eg, a bunch of floating point values. At least some of what I just said is probably wrong, but now someone will correct me and we'll both me more right!

mrtranscendence 4y ago
At a first approximation this is pretty good. I wouldn't say this exactly: > A model is some architecture of how data will flow through these weight matrices, along with the values of each weight. Because data doesn't really flow through weight matrices, though perhaps this is true if you squint at very simple models. Deep learning architectures are generally more complicated than multiplying values by weights and pushing the results to the next layer, though which architecture to use depends heavily on context. > Tokens are sort of "words" in a sentence Tokens are funny. What a token is depends on the context of the model you're using, but generally a token is a portion of a word. (Why? Efficiency is one reason; handling unknown words is another.)
- jstarfish 4y ago
  > What a token is depends on the context of the model you're using, but generally a token is a portion of a word. When doing quick estimates, I just assume every syllable is a token. It tends to overestimate, which is fine for my OOM mitigation purposes.

mhh__ 4y ago

A book about AI. (Norvig and Russell comes to mind)

heliophobicdude 4y ago

Probably not the answer you would like but I think your approach to download them and figure out how to run them on your machine is a good one. You don't need to understand everything to get something working. It can be overwhelming and unproductive to know everything before getting started. To learn more deeply though, get started with getting it to work and when you are curious or something doesn't work, try to understand why and recursively go back to fill in the foundational details. Example, download the code try to get it to work. Why is it not working? Oh it's trying to look for the model. Search for how to get the model and set it up. Then key step, recursively look up every single thing in the guide or set up. Don't try to set something up or fix some thing without truly understanding what it is you are doing (e.g. copy and paste). This gives you a structured why to fill in the foundations of what it is you are trying to get to work in a more focused and productive manner. At the end you might realize that their approach or yours is not optimal "oh it was telling me to download the 65k model when I can only run 7k on my machine bc ..."

visarga 4y ago

Psst ... why don't you spend 30 minutes of quality time with chatGPT and get to the bottom of this? Get those personalised explanations and enjoy its unlimited patience. I have felt the same in the past, related to a completely different topic. I know how it feels, it's like people are not saying things what they are, just using weird words. "weights" - synapses in the AI brain "tokens" - word fragments "model" - of course, the model is the AI brain "context" - the model can only handle a piece of text, can't put whole books in, so this limited window is the context "GPT" - predicts the next word, trained on everything; if you feed its last predicted word back in, it can write long texts "LoRA" - a lightweight plug-in model for tweaking the big model "loss" - a score telling how bad is the output "training" - change the model until it fits the data "quantisation" - making a low precision version of the model because it still works, but now is much faster and needs less compute "embedding" - just a vector, it stands for the meaning of a word token or a piece of image; these embeddings are learned

agentdrtran 4y ago
> why don't you spend 30 minutes of quality time with chatGPT and get to the bottom of this? when it can hallucinate content, why do that instead of reading a blog post from an expert?
- visarga 4y ago
  Oh no, it will hallucinate an obscure fact, but not basics. It's pretty good at reciting theory, it would pass many ML engineering theoretical interviews. If you don't trust its memory, copy a piece of high quality text in the topic of interest inside the context, as reference.
  - agentdrtran 3y ago
    it's repeatedly made up entire quotes and research papers?
tikkun 4y ago
I think this is the right answer, ChatGPT is an excellent 1-1 tutor.

2devnull 4y ago

For a good general non-technical introduction I recommend the YouTube computerphile series related to language models, transformers and other general concepts. If you are interested in actually doing stuff there’s an over abundance of material out there already, if you try looking.

bobbyi 4y ago

I haven't watched it yet, but the Practical Deep Learning for Coders course that's available on YouTube is often recommended https://course.fast.ai/ https://course.fast.ai/