3 ms·
It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future n
by ampdepolymerase 6y ago
It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.
- bitL 6y ago"Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
- heyitsguay 6y agoI don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
- ampdepolymerase 6y agoFor aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.