4 ms·
HF says activation function is "SiTU-GLU", but I can't find any info on that? Anyone know / is this a typo? Even the python code for inference seems to use no
by minimaltom 3mo ago
HF says activation function is "SiTU-GLU", but I can't find any info on that?
Anyone know / is this a typo?
Even the python code for inference seems to use normal activations.
- lightningdev1 3mo agoIt stands for Sigmoid Tanh Unit Gated Linear Unit. Check the tech report/paper (page 7): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...
- minimaltom 3mo agoThanks! I had a clanker graph it with sliders, its a very interesting function to play with! seems like it has the negative 'hump' typical of a GELU coupled with the saturation of a sigmoid. Its interesting that saturation is ideal at this scale, i thought we were all in on self-activating functions (i.e. swish) but in fairness all those papers lacked ablations at scale nor discussion of stability.