4 ms·

I find it incredibly hard to believe we stumbled upon an efficient architecture that requires nothing but more compute not 10 years after the AI winter thawed.

by uoaei 3y ago

I find it incredibly hard to believe we stumbled upon an efficient architecture that requires nothing but more compute not 10 years after the AI winter thawed. That's incredibly optimistic to the point of blind hope. What is your background and what makes you think we've somehow already figured everything out?

sigmoid10 3y ago
I have been working on architectures in this field for almost a decade now and I've seen firshand how things have changed. It might seem hard to believe if you have been to university ~10 years ago and only know the state of deep learning from the early revolutions back then, but we are in a totally different era now. With the transformer, we now have a true general-purpose, efficiently scalable, end-to-end differentiable algorithm. Meaning you can apply it to any task as long as you convert it to the right embedding space, you can train gigantic models that compress huge amounts of information using enormous datasets and you can still use good-ol' gradient descent to optimize it (which is kind of sad since we still haven't found a better way of training models, but hey it works).
- asadotzler 3y ago
  and yet SamA says it's actual trillions of dollars in entirely new compute capacity to reach the next level. hmmm. to believe him or you... so hard to decide.
  - sigmoid10 3y ago
    SamA is a business bro who tries to hoard investor capital. He has to say these things to collect more VCs. If you want to learn about the tech, listen to what the actual techies at openai have to say. This stuff is no secret.
- uoaei 3y ago
  > Meaning you can apply it to any task as long as you convert it to the right embedding space This glosses over a massive issue which is that not everything can be efficiently represented as a vector space via embeddings. So your claim of "general purpose" rings hollow. Not to mention that there is no feedback mechanism for the supposed "knowledge" advocates claim Transformer-based models have, so things like metacognition are literally impossible with this architecture. As it stands LLM outputs are isomorphic to psychotic stream-of-consciousness babble. You've managed to find a tall tree, but from your response it seems like you haven't yet gotten to considering rockets.
  - sigmoid10 3y ago
    There is no basis to any of your arguments. Embedding spaces are not defined by humans, but learned. A priori you have zero idea in what way or how efficient things will be encoded. It all depends on the model and it's internally learned world representation. And things like "metacognition" are meaningless terms used by quacks. We don't know how the brain works, we only know what it can do on the outside. And it is mathematically proven that neural networks can do literally everything in principle as well, thanks to universal approximation.

by uoaei 3y ago

sigmoid10 3y ago

I have been working on architectures in this field for almost a decade now and I've seen firshand how things have changed. It might seem hard to believe if you have been to university ~10 years ago and only know the state of deep learning from the early revolutions back then, but we are in a totally different era now. With the transformer, we now have a true general-purpose, efficiently scalable, end-to-end differentiable algorithm. Meaning you can apply it to any task as long as you convert it to the right embedding space, you can train gigantic models that compress huge amounts of information using enormous datasets and you can still use good-ol' gradient descent to optimize it (which is kind of sad since we still haven't found a better way of training models, but hey it works).

asadotzler 3y ago
and yet SamA says it's actual trillions of dollars in entirely new compute capacity to reach the next level. hmmm. to believe him or you... so hard to decide.
- sigmoid10 3y ago
  SamA is a business bro who tries to hoard investor capital. He has to say these things to collect more VCs. If you want to learn about the tech, listen to what the actual techies at openai have to say. This stuff is no secret.
uoaei 3y ago
> Meaning you can apply it to any task as long as you convert it to the right embedding space This glosses over a massive issue which is that not everything can be efficiently represented as a vector space via embeddings. So your claim of "general purpose" rings hollow. Not to mention that there is no feedback mechanism for the supposed "knowledge" advocates claim Transformer-based models have, so things like metacognition are literally impossible with this architecture. As it stands LLM outputs are isomorphic to psychotic stream-of-consciousness babble. You've managed to find a tall tree, but from your response it seems like you haven't yet gotten to considering rockets.
- sigmoid10 3y ago
  There is no basis to any of your arguments. Embedding spaces are not defined by humans, but learned. A priori you have zero idea in what way or how efficient things will be encoded. It all depends on the model and it's internally learned world representation. And things like "metacognition" are meaningless terms used by quacks. We don't know how the brain works, we only know what it can do on the outside. And it is mathematically proven that neural networks can do literally everything in principle as well, thanks to universal approximation.