4 ms·
If a paper provided general tools that let me sit down with pen and paper and derive that SWIGLU is better than RELU, and batch normalization trains 6 pi /root
by QuadmasterXLII 3y ago
If a paper provided general tools that let me sit down with pen and paper and derive that SWIGLU is better than RELU, and batch normalization trains 6 pi /root 5 times faster than no normalization, etc, and I could get the derivation right before running the experiment, then I wpuld believe that I understood how neural networks train
- SiempreViernes 3y agoPretty much this, so long as you can't make any prediction ahead of time of the approximate performance of a architecture on a dataset one shouldn't claim they have a good understanding of how a network "learns".
- chermi 3y agoThis doesn't make sense as a requirement - you make no mention of the data. "How it works" will always depend, in part, on the data, you can't fully separate the algorithm from the data if you want to answer questions about why learning works the way it does. IMO.
- ogogmad 3y agoIt does make sense. An explanation of NNs must help people choose hyperparameter values ***based on*** the data.