4 ms·
Of course. But we must acknowledge that many have blinders on, assuming that scale is all you need to beat statistical errors.
by uoaei 3y ago
Of course. But we must acknowledge that many have blinders on, assuming that scale is all you need to beat statistical errors.
- sigmoid10 3y agoWell, these people are not wrong per se. Scale is what drove what we have today and as hardware improves, the models will too. It's just that in the very short term it turns out to be faster to just code around some of these issues on the backend of an API rather than increase the compute you spend on the model itself.
- uoaei 3y agoMonkey sees moon. Monkey climbs tree. "See? Monkey is closer to moon than before. To reach moon, monkey just needs taller tree." How long before monkey finds tall enough tree to reach moon?
- sigmoid10 3y agoWe're rapidly approaching the compute capacity of the human brain in individual server racks. This "moon" is neither unreachable nor is there any doubt that we will cross the threshold soon.
- uoaei 3y agoI find it incredibly hard to believe we stumbled upon an efficient architecture that requires nothing but more compute not 10 years after the AI winter thawed. That's incredibly optimistic to the point of blind hope. What is your background and what makes you think we've somehow already figured everything out?
- sigmoid10 3y agoI have been working on architectures in this field for almost a decade now and I've seen firshand how things have changed. It might seem hard to believe if you have been to university ~10 years ago and only know the state of deep learning from the early revolutions back then, but we are in a totally different era now. With the transformer, we now have a true general-purpose, efficiently scalable, end-to-end differentiable algorithm. Meaning you can apply it to any task as long as you convert it to the right embedding space, you can train gigantic models that compress huge amounts of information using enormous datasets and you can still use good-ol' gradient descent to optimize it (which is kind of sad since we still haven't found a better way of training models, but hey it works).
- asadotzler 3y agoand yet SamA says it's actual trillions of dollars in entirely new compute capacity to reach the next level. hmmm. to believe him or you... so hard to decide.
- sigmoid10 3y agoSamA is a business bro who tries to hoard investor capital. He has to say these things to collect more VCs. If you want to learn about the tech, listen to what the actual techies at openai have to say. This stuff is no secret.
- uoaei 3y ago> Meaning you can apply it to any task as long as you convert it to the right embedding space This glosses over a massive issue which is that not everything can be efficiently represented as a vector space via embeddings. So your claim of "general purpose" rings hollow. Not to mention that there is no feedback mechanism for the supposed "knowledge" advocates claim Transformer-based models have, so things like metacognition are literally impossible with this architecture. As it stands LLM outputs are isomorphic to psychotic stream-of-consciousness babble. You've managed to find a tall tree, but from your response it seems like you haven't yet gotten to considering rockets.
- sigmoid10 3y agoThere is no basis to any of your arguments. Embedding spaces are not defined by humans, but learned. A priori you have zero idea in what way or how efficient things will be encoded. It all depends on the model and it's internally learned world representation. And things like "metacognition" are meaningless terms used by quacks. We don't know how the brain works, we only know what it can do on the outside. And it is mathematically proven that neural networks can do literally everything in principle as well, thanks to universal approximation.