4 ms·
The sigmoid approximation is nice, but not 100% necessary (I don't know the cost of doing sigmoid in GPU but I suspect the modern ones by nvidia probably have s
by dnautics 9y ago
The sigmoid approximation is nice, but not 100% necessary (I don't know the cost of doing sigmoid in GPU but I suspect the modern ones by nvidia probably have some way of doing it relatively fast) - note that sigmoid and tanh are still used in GRUs and LSTMs. And sigmoid, tanh, and their derivatives all have approximative shortcuts in posits.
There is a single critical step in training which I find gets stuck in early phases of learning due to accumulation issues (I solved it by expanding to 16 bits during this phase, which should be "on-chip"). It's nice to see another method to do the same! I suspect that batch normalization (which seems similar to what HALP is) after each step will also help.
- vanderZwan 9y agoYes, while I haven't done anything with ML myself, I heard via the 3Blue1Brown video on Deep Learning[0] that the sigmoid function isn't really that used anymore. But I figured allowing that being able to tweak the dynamic range could make it a good fit for the rescaling and recentering approach. You know what, I'll just go ahead and post a link to this article on the Unum google group, perhaps someone there can add some thoughts[1]. [0] https://www.youtube.com/watch?v=aircAruvnKk&t=17m https://www.youtube.com/watch?v=aircAruvnKk&t=17m [1]https://groups.google.com/forum/#!topic/unum-computing/RHrQUoyDl5c https://groups.google.com/forum/#!topic/unum-computing/RHrQU...
- dnautics 9y agoSigmoid is indeed hardly used as layer-to-layer transfer functions. But I gurantee you > 20% of the world is activating sigmoid functions in ML apps every day.
- vanderZwan 9y agoAh, thank you for that correction. Guess the video was accidentally misleading (they technically didn't say anything about other ML approaches, but overgeneralising like I did isn't that big of a leap)