4 ms·
This is just a practical approximation to the actual mathematical definition of GELU, which is `GELU(x) := x * Φ(x)` where Φ(x) is the CDF of the Gaussian distr
by jey 2y ago
This is just a practical approximation to the actual mathematical definition of GELU, which is `GELU(x) := x * Φ(x)` where Φ(x) is the CDF of the Gaussian distribution.
- fluoridation 2y agoIsn't that just erf()?
- kgwgk 2y agoThey are related but the error function approaches -1 for large negative numbers. Φ(x) approaches 0 and so does x * Φ(x).
- deleted 2y ago[deleted]