4 ms·

Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or a

Doesn't this seem almost sad for the future bedroom hacker/savants/prodigies? Like, the era of being able to theorize and program a game changing new model or approach here is gone, because...even if you have the inkling of a good idea, you'd literally never be able to realize it unless you were independently exorbitantly wealthy or had venture backing. Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc. The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.

Swizec 3y ago
I think there’s room, we’re just old and stuck in our ways. A bedroom hacker can get access to unimaginable technology for like $10 per month. The things that AWS, Azure, OpenAI, and friends make available for a smol card swipe would literally break my brain when I was a bedroom hacker and my parents sunk 2 or 3 monthly salaries into a 166Mhz Pentium 1. > The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future. Wasn’t Huffman backed by a university? And didn’t Carmack do his best work when id software was printing so much money they literally didn’t know what to do with it all? PS: many of the large datasets people use for these things are fairly standardized and keep showing up in paper after paper. I assume that means they’re available somewhere.
two_in_one 3y ago
World doesn't end on LLMs. Even with LLMs we have available pre-trained which can be used for something else. I think next hot area will be applications in different domains. Here even less powerful model can be a game changer. Big LLMs as services are here to stay. They will become irreplaceable and incompatible with each other. As for "cracking", people are still trying to make "the best game ever". This will never end ;)
Me1000 3y ago
Nah, the work and pace of progress in the open source community is mind blowing. There’s a lot of brain power going into getting models running on consumer hardware. Imho, this reminds me a lot of when people thought personal computers were just toys, and you needed a huge IBM machine if you wanted to do “real computing”.

Swizec 3y ago

I think there’s room, we’re just old and stuck in our ways. A bedroom hacker can get access to unimaginable technology for like $10 per month. The things that AWS, Azure, OpenAI, and friends make available for a smol card swipe would literally break my brain when I was a bedroom hacker and my parents sunk 2 or 3 monthly salaries into a 166Mhz Pentium 1. > The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future. Wasn’t Huffman backed by a university? And didn’t Carmack do his best work when id software was printing so much money they literally didn’t know what to do with it all? PS: many of the large datasets people use for these things are fairly standardized and keep showing up in paper after paper. I assume that means they’re available somewhere.

two_in_one 3y ago

World doesn't end on LLMs. Even with LLMs we have available pre-trained which can be used for something else. I think next hot area will be applications in different domains. Here even less powerful model can be a game changer. Big LLMs as services are here to stay. They will become irreplaceable and incompatible with each other. As for "cracking", people are still trying to make "the best game ever". This will never end ;)

Me1000 3y ago

Nah, the work and pace of progress in the open source community is mind blowing. There’s a lot of brain power going into getting models running on consumer hardware. Imho, this reminds me a lot of when people thought personal computers were just toys, and you needed a huge IBM machine if you wanted to do “real computing”.