3 ms·

Tell HN: We take LLMs and Diffusion models for granted

5 points by behnamoh 3y ago

Just 3 years ago, the state of the art image generation models were mostly academic papers that would work in specific scenarios, but nowhere near the capabilities of DALL-E 3, Midjourney, SD, etc. Just 3 years ago, the state of the art chatbots were so crappy that no one thought soon they'd easily pass the Turing test. Nowadays we talk to ChatGPT, download a variety of open-source LLMs, talk about some LLMs being better than the others. But just remember: 3 years ago we didn't even have something like Llama 7b!

deleted 3y ago
[deleted]
cpach 3y ago
Yes. You are so right. It’s fascinating and quite scary, how fast our minds can get used to things. The first time I saw output from DALL-E I thought: Is this actual output from ML/“AI” or is it a hoax? It wasn’t a hoax, so I was very impressed. And now it’s the new normal.
ActorNightly 3y ago
To the layman, sure, this stuff seems high tech. However, for anyone that has been exposed to neural nets in the past decade, the solution of "throwing more compute" at it has been obvious. The big driver of this has been the actual compute availability. Even in early 2000s, it was simply unfeasible to just make computers iterate on a bunch of things until they got it right. When deep learning took hold and showed promise, hardware start to catch up, and now a rack of GPUs can pretty much out-compute buildings of servers back from 2000s. Same thing is going to happen in the future. If economy works out in such a way that someone with median salary can buy a box that allows them to train a model like Llama in a week (with of course cheap electricity and updated power grid), we are going to get much cooler things. Because in that case, it wouldn't be a matter of figuring out what the best architecture is for LLM - it would be a matter of generalizing the compute graph, which means that the actual architecture would be a learnable parameter. I mean if you look at the human brain, and take the neural activations per second (which is essentially a floating point operation of add/multiply) times the number of neurons, you get some petaflop number. We have racks of hardware that can match that number right now. We just don't know the correct structure of all the compute graph, and to figure it out is a matter of iteration, and computers are better at iteration than humans are.

5 points by behnamoh 3y ago

deleted 3y ago

[deleted]

cpach 3y ago

Yes. You are so right. It’s fascinating and quite scary, how fast our minds can get used to things. The first time I saw output from DALL-E I thought: Is this actual output from ML/“AI” or is it a hoax? It wasn’t a hoax, so I was very impressed. And now it’s the new normal.

ActorNightly 3y ago

To the layman, sure, this stuff seems high tech. However, for anyone that has been exposed to neural nets in the past decade, the solution of "throwing more compute" at it has been obvious. The big driver of this has been the actual compute availability. Even in early 2000s, it was simply unfeasible to just make computers iterate on a bunch of things until they got it right. When deep learning took hold and showed promise, hardware start to catch up, and now a rack of GPUs can pretty much out-compute buildings of servers back from 2000s. Same thing is going to happen in the future. If economy works out in such a way that someone with median salary can buy a box that allows them to train a model like Llama in a week (with of course cheap electricity and updated power grid), we are going to get much cooler things. Because in that case, it wouldn't be a matter of figuring out what the best architecture is for LLM - it would be a matter of generalizing the compute graph, which means that the actual architecture would be a learnable parameter. I mean if you look at the human brain, and take the neural activations per second (which is essentially a floating point operation of add/multiply) times the number of neurons, you get some petaflop number. We have racks of hardware that can match that number right now. We just don't know the correct structure of all the compute graph, and to figure it out is a matter of iteration, and computers are better at iteration than humans are.