7 ms·
For training how do you get any kind of meaningful derivative with it?
by cma 3y ago
For training how do you get any kind of meaningful derivative with it?
- twelfthnight 3y agoMaybe evolutionary algorithms instead? Hasn't proven super useful historically, but maybe at the scale of enormous LLMs it will be?
- bionhoward 3y agoEvolutionary algorithms made you, didn’t they?
- TheDudeMan 3y agoThat does not prove that they can beat gradient decent.
- bick_nyers 3y agoIt took a lot of human brain flops to get to this point in time though, I wonder how many orders of magnitude more than it took to train ChatGPT...
- VHRanger 3y agoNope, they're orders of magnitude more inefficient because they don't leverage gradient descent. Rule of thumb in optimization: real numbers are easy, integers are hard
- markisus 3y agoThis may be the status quo because of the so called "hardware lottery" which has historically been optimized for floating point. I'm speculating, but if hardware designers were instead only concerned about raw xnor density and throughput, we might end up with chips powerful enough that giant 1-bit nets could be trained purely through evolution.
- imtringued 3y agoHow do you optimize memory for floating point?
- VHRanger 3y agoBF8 and other similar formats?
- VHRanger 3y agoNo, it's a fact at the mathematical level that you can enshrine in big O terms if you want to
- bick_nyers 3y agoGradient-directed evolutionary algorithm sounds kinda interesting.
- chalst 3y agoThe OP explicitly excludes training.
- cma 3y agoThe one I replied to said 1-bit for both training and inference.
- scotty79 3y agoMaybe something probabilistic?
- concurrentsquar 3y agoYou don't (you have to use real-valued inertial 'latent weights' during training): https://arxiv.org/abs/1906.02107 https://arxiv.org/abs/1906.02107 (there is still a reduction in memory usage though (just not 24x): > "Furthermore, Bop reduces the memory requirements during training: it requires only one real-valued variable per weight, while the latent-variable approach with Momentum and Adam require two and three respectively.")