6 ms·

Tangent: where can a mortal go learn what this title means to the point where they can have something in their computer that allows them to change settings (and

Tangent: where can a mortal go learn what this title means to the point where they can have something in their computer that allows them to change settings (and know which ones) and see what happens?

cypress66 3y ago
Just learn AI in general and the rest will be easy to process.
- taneq 3y ago
  “Learn to draw the rest of the owl first, then those first three ovals are super easy!”
tekno45 3y ago
good explanation on tokens and context https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoaWxs https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoa...
quickthrower2 3y ago
Good question. I have done the andrej karpathy course on youtube. It is not easy. And it is fast paced (about 12 or so hours that would be 50 if it were a university designed course plus same again for practice). Then even with all that debugging the model, and making architecture choices is a whole other thing he barely covers. Would love a good course on that. If you learn the nuts and bolts you can point to any part of the transformer model and describe what numbers and operations are happening in the forward and backward pass. That sort of is the reality. Reading other people’s fuzzy explanations is probably like understanding quantum mechanics by reading quanta (you get the layman example but don’t really understand… btw I know little about QM!) I needed to hop on other materials to exist. I van understand some of this but not all and running it and trying things out… probably not yet. Well never for 100k context size as I don’t want to sell my house to pay for compute :-).
wokwokwok 3y ago
There are lots of places that explain LLMs. …but you will be disappointed if you expect/hope to be able to recreate or modify things yourself. You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards. The barrier to entry isn’t knowledge; it’s money. (So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)
- sp332 3y ago
  The amount of resources needed depends on the size. If you're looking for understanding, you can get a toy model running with a lot less hardware and time.
  - Swizec 3y ago
    But unlike other areas of computing, this stuff really doesn’t scale intuitively. The toy model you run on your computer will barely work better than a naive Markov chain and you’ll have a hard time seeing the impact your choices because everything will feel like trash. Add a few orders of magnitude of data and suddenly the exact same thing works like magic.
    - Solvency 3y ago
      Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or approach here is gone, because...even if you have the inkling of a good idea, you'd literally never be able to realize it unless you were independently exorbitantly wealthy or had venture backing. Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc. The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
      - 3y ago
version_five 3y ago
Just look up llama.cpp and read the instructions plus look at the arguments you can pass to the main program, of which context size is one. Don't listen to people telling you to learn the ML theory, if you don't know it, learn the functionality of the program. The one I recommended is one you can run on a normal computer.

Tangent: where can a mortal go learn what this title means to the point where they can have something in their computer that allows them to change settings (and know which ones) and see what happens?

cypress66 3y ago

Just learn AI in general and the rest will be easy to process.

taneq 3y ago
“Learn to draw the rest of the owl first, then those first three ovals are super easy!”

tekno45 3y ago

good explanation on tokens and context https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoaWxs https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoa...

quickthrower2 3y ago

Good question. I have done the andrej karpathy course on youtube. It is not easy. And it is fast paced (about 12 or so hours that would be 50 if it were a university designed course plus same again for practice). Then even with all that debugging the model, and making architecture choices is a whole other thing he barely covers. Would love a good course on that. If you learn the nuts and bolts you can point to any part of the transformer model and describe what numbers and operations are happening in the forward and backward pass. That sort of is the reality. Reading other people’s fuzzy explanations is probably like understanding quantum mechanics by reading quanta (you get the layman example but don’t really understand… btw I know little about QM!) I needed to hop on other materials to exist. I van understand some of this but not all and running it and trying things out… probably not yet. Well never for 100k context size as I don’t want to sell my house to pay for compute :-).

wokwokwok 3y ago

There are lots of places that explain LLMs. …but you will be disappointed if you expect/hope to be able to recreate or modify things yourself. You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards. The barrier to entry isn’t knowledge; it’s money. (So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)

sp332 3y ago
The amount of resources needed depends on the size. If you're looking for understanding, you can get a toy model running with a lot less hardware and time.
- Swizec 3y ago
  But unlike other areas of computing, this stuff really doesn’t scale intuitively. The toy model you run on your computer will barely work better than a naive Markov chain and you’ll have a hard time seeing the impact your choices because everything will feel like trash. Add a few orders of magnitude of data and suddenly the exact same thing works like magic.
  - Solvency 3y ago
    Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or approach here is gone, because...even if you have the inkling of a good idea, you'd literally never be able to realize it unless you were independently exorbitantly wealthy or had venture backing. Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc. The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
    - 3y ago

version_five 3y ago

Just look up llama.cpp and read the instructions plus look at the arguments you can pass to the main program, of which context size is one. Don't listen to people telling you to learn the ML theory, if you don't know it, learn the functionality of the program. The one I recommended is one you can run on a normal computer.