3 ms·

Full disclosure; I am not a pro ML'er. I dabble as a hobby and I am trying to learn as much as possible. Question; Does the innovation need to happen on the ha

Full disclosure; I am not a pro ML'er. I dabble as a hobby and I am trying to learn as much as possible. Question; Does the innovation need to happen on the hardware side? I ask because there is a concerted effort, at least for inference, to reduce memory requirements and provide more access to giant LLMs.[0] It seems like everyone for the past few years has just thrown more and more compute at the wall to see what sticks. Where does it stop? 3 or 4 trillion parameter models that cost a $Billion to train? Is there a law of diminishing returns? Have we reached the apogee or will it be the 15 trillion parameter that ate $100Billion? Accessibility to all manner of models, for knuckle heads like myself, is relatively new. Maybe I should rephrase, accessibility to interesting and FUN models that mere mortals can use and hack on is relatively new. Stable Diffusion is nothing short of spectacular for just having some fun with ML/DL. My naïve thought is that, similar to most natural systems, smaller and more specialized models that can be "glued" together as a way forward. In my mind this makes more sense than these ridiculously large language models. That is to say, purpose built or fine--tuned models that can communicate seems like a good idea. GPT-NeoX "communicates" with to Bloom and makes a gaggle of small models that can be fine-tuned further. If regular people with a consumer GPU can train/fine-tune a model at home, this happens a lot faster. [0] https://arxiv.org/abs/2208.07339 https://arxiv.org/abs/2208.07339

Tuna-Fish 4y ago
Your brain has roughly a quadrillion synapses. These are (in a very rough manner) the closest thing there are to parameters in the ML sense. I think you can probably do better with smarter software, but if you want to match humans on general tasks, I doubt you can do 100x better. So there is clearly still ways to go.
- monkmartinez 4y ago
  I agree there is a way to go... I am just curious if the answer lies in a super computer. I don't think it does, but I am also openly admitting my naivety. Since you brought up the brain, I will roll with some thoughts. Instead of thinking about a hunk of wet meat that contains a quadrillion synapses... wouldn't it make sense to break those into discrete, specialized and smaller hunks of wet meat of say 10 billion? Then we link those discrete units under a unified command which directs information to the correct specialized core and then the next until some arbitrary stopping point was reached. AKA: An ensemble of models. My point; It seems that throwing more compute at individual problems has probably reached a kind of apogee in time and cost. PaLM, Bloom, GPT-NeoX and GPT-3 all live in silos. Cool story, but you can't use GPT-3 and PaLM at the same time without an absolute ton of overhead right now. At some point, I think there will be a unifying middleware that can utilize these models in there current form. That is, you will not need to train yet another model to combine the networks of one or more discrete models. Just a hunch.... could be wrong.
atty 4y ago
Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is NLP or CV. For instance, in physical simulations, the representation of the system you’re simulating could be on the order of 10s of gigabytes (a very large mesh, a fine 3D grid, etc.) These are the kinds of problems I work on, and we massively benefit every time a new chip comes out with more memory or increased memory bandwidth, because a single training example, plus it’s activations, the model, and optimizer state, can take about a half a TB of memory or more depending on model size. I’d also point out that as hardware devices get better, it’s good for everyone. It allows the large corporations to train even larger, more capable models, and it allows smaller players to train models they couldn’t have afforded before on smaller budgets. A rising tide lifts all boats, so to speak. Algorithmic and efficiency improvements are also important, but they’re additive, not a replacement. And while combining models is not novel and is certainly an avenue that should and is being explored, the models that have the most generalizability for recombination are these extremely large models! They have the capacity to learn very generalized patterns that are then useful for far more transfer/combination tasks.

Tuna-Fish 4y ago

Your brain has roughly a quadrillion synapses. These are (in a very rough manner) the closest thing there are to parameters in the ML sense. I think you can probably do better with smarter software, but if you want to match humans on general tasks, I doubt you can do 100x better. So there is clearly still ways to go.

monkmartinez 4y ago
I agree there is a way to go... I am just curious if the answer lies in a super computer. I don't think it does, but I am also openly admitting my naivety. Since you brought up the brain, I will roll with some thoughts. Instead of thinking about a hunk of wet meat that contains a quadrillion synapses... wouldn't it make sense to break those into discrete, specialized and smaller hunks of wet meat of say 10 billion? Then we link those discrete units under a unified command which directs information to the correct specialized core and then the next until some arbitrary stopping point was reached. AKA: An ensemble of models. My point; It seems that throwing more compute at individual problems has probably reached a kind of apogee in time and cost. PaLM, Bloom, GPT-NeoX and GPT-3 all live in silos. Cool story, but you can't use GPT-3 and PaLM at the same time without an absolute ton of overhead right now. At some point, I think there will be a unifying middleware that can utilize these models in there current form. That is, you will not need to train yet another model to combine the networks of one or more discrete models. Just a hunch.... could be wrong.

atty 4y ago

Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is NLP or CV. For instance, in physical simulations, the representation of the system you’re simulating could be on the order of 10s of gigabytes (a very large mesh, a fine 3D grid, etc.) These are the kinds of problems I work on, and we massively benefit every time a new chip comes out with more memory or increased memory bandwidth, because a single training example, plus it’s activations, the model, and optimizer state, can take about a half a TB of memory or more depending on model size. I’d also point out that as hardware devices get better, it’s good for everyone. It allows the large corporations to train even larger, more capable models, and it allows smaller players to train models they couldn’t have afforded before on smaller budgets. A rising tide lifts all boats, so to speak. Algorithmic and efficiency improvements are also important, but they’re additive, not a replacement. And while combining models is not novel and is certainly an avenue that should and is being explored, the models that have the most generalizability for recombination are these extremely large models! They have the capacity to learn very generalized patterns that are then useful for far more transfer/combination tasks.