3 ms·
Note llama's feed forward is a bit different too: self.w2(F.silu(self.w1(x)) * self.w3(x)) I.e. the nonlinearity is a gate. https://github.com/meta-llama/l
by jorlow 2y ago
Note llama's feed forward is a bit different too:
self.w2(F.silu(self.w1(x)) * self.w3(x))
I.e. the nonlinearity is a gate.
https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f/llama/model.py#L219 https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
- soraki_soladead 2y agoFwiw, that's SwiGLU in #3 above. Swi = Swish = silu. GLU is gated linear unit; the gate construction you describe.