3 ms·

There are lots of places that explain LLMs. …but you will be disappointed if you expect/hope to be able to recreate or modify things yourself. You need a mass

There are lots of places that explain LLMs. …but you will be disappointed if you expect/hope to be able to recreate or modify things yourself. You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards. The barrier to entry isn’t knowledge; it’s money. (So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)

sp332 3y ago
The amount of resources needed depends on the size. If you're looking for understanding, you can get a toy model running with a lot less hardware and time.
- Swizec 3y ago
  But unlike other areas of computing, this stuff really doesn’t scale intuitively. The toy model you run on your computer will barely work better than a naive Markov chain and you’ll have a hard time seeing the impact your choices because everything will feel like trash. Add a few orders of magnitude of data and suddenly the exact same thing works like magic.
  - Solvency 3y ago
    Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or approach here is gone, because...even if you have the inkling of a good idea, you'd literally never be able to realize it unless you were independently exorbitantly wealthy or had venture backing. Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc. The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
    - Swizec 3y ago
      I think there’s room, we’re just old and stuck in our ways. A bedroom hacker can get access to unimaginable technology for like $10 per month. The things that AWS, Azure, OpenAI, and friends make available for a smol card swipe would literally break my brain when I was a bedroom hacker and my parents sunk 2 or 3 monthly salaries into a 166Mhz Pentium 1. > The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future. Wasn’t Huffman backed by a university? And didn’t Carmack do his best work when id software was printing so much money they literally didn’t know what to do with it all? PS: many of the large datasets people use for these things are fairly standardized and keep showing up in paper after paper. I assume that means they’re available somewhere.
    - two_in_one 3y ago
      World doesn't end on LLMs. Even with LLMs we have available pre-trained which can be used for something else. I think next hot area will be applications in different domains. Here even less powerful model can be a game changer. Big LLMs as services are here to stay. They will become irreplaceable and incompatible with each other. As for "cracking", people are still trying to make "the best game ever". This will never end ;)
    - Me1000 3y ago
      Nah, the work and pace of progress in the open source community is mind blowing. There’s a lot of brain power going into getting models running on consumer hardware. Imho, this reminds me a lot of when people thought personal computers were just toys, and you needed a huge IBM machine if you wanted to do “real computing”.
- wokwokwok 3y ago
  I don’t see how any toy model you could trivially create could have the settings / tweaks for 100k context windows applied / played with. I mean, in general yes, but in this specific example? Hmm…

sp332 3y ago

The amount of resources needed depends on the size. If you're looking for understanding, you can get a toy model running with a lot less hardware and time.

Swizec 3y ago
But unlike other areas of computing, this stuff really doesn’t scale intuitively. The toy model you run on your computer will barely work better than a naive Markov chain and you’ll have a hard time seeing the impact your choices because everything will feel like trash. Add a few orders of magnitude of data and suddenly the exact same thing works like magic.
- Solvency 3y ago
  Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or approach here is gone, because...even if you have the inkling of a good idea, you'd literally never be able to realize it unless you were independently exorbitantly wealthy or had venture backing. Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc. The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
  - Swizec 3y ago
    I think there’s room, we’re just old and stuck in our ways. A bedroom hacker can get access to unimaginable technology for like $10 per month. The things that AWS, Azure, OpenAI, and friends make available for a smol card swipe would literally break my brain when I was a bedroom hacker and my parents sunk 2 or 3 monthly salaries into a 166Mhz Pentium 1. > The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future. Wasn’t Huffman backed by a university? And didn’t Carmack do his best work when id software was printing so much money they literally didn’t know what to do with it all? PS: many of the large datasets people use for these things are fairly standardized and keep showing up in paper after paper. I assume that means they’re available somewhere.
  - two_in_one 3y ago
    World doesn't end on LLMs. Even with LLMs we have available pre-trained which can be used for something else. I think next hot area will be applications in different domains. Here even less powerful model can be a game changer. Big LLMs as services are here to stay. They will become irreplaceable and incompatible with each other. As for "cracking", people are still trying to make "the best game ever". This will never end ;)
  - Me1000 3y ago
    Nah, the work and pace of progress in the open source community is mind blowing. There’s a lot of brain power going into getting models running on consumer hardware. Imho, this reminds me a lot of when people thought personal computers were just toys, and you needed a huge IBM machine if you wanted to do “real computing”.
wokwokwok 3y ago
I don’t see how any toy model you could trivially create could have the settings / tweaks for 100k context windows applied / played with. I mean, in general yes, but in this specific example? Hmm…