3 ms·
Your brain has roughly a quadrillion synapses. These are (in a very rough manner) the closest thing there are to parameters in the ML sense. I think you can pro
by Tuna-Fish 4y ago
Your brain has roughly a quadrillion synapses. These are (in a very rough manner) the closest thing there are to parameters in the ML sense. I think you can probably do better with smarter software, but if you want to match humans on general tasks, I doubt you can do 100x better.
So there is clearly still ways to go.
- monkmartinez 4y agoI agree there is a way to go... I am just curious if the answer lies in a super computer. I don't think it does, but I am also openly admitting my naivety. Since you brought up the brain, I will roll with some thoughts. Instead of thinking about a hunk of wet meat that contains a quadrillion synapses... wouldn't it make sense to break those into discrete, specialized and smaller hunks of wet meat of say 10 billion? Then we link those discrete units under a unified command which directs information to the correct specialized core and then the next until some arbitrary stopping point was reached. AKA: An ensemble of models. My point; It seems that throwing more compute at individual problems has probably reached a kind of apogee in time and cost. PaLM, Bloom, GPT-NeoX and GPT-3 all live in silos. Cool story, but you can't use GPT-3 and PaLM at the same time without an absolute ton of overhead right now. At some point, I think there will be a unifying middleware that can utilize these models in there current form. That is, you will not need to train yet another model to combine the networks of one or more discrete models. Just a hunch.... could be wrong.