4 ms·
> Add a dedicated "parking spot" like GPT-OSS does and eat the gradient flow tax on that Not familiar with this topic, but intrigued-anywhere I can read more a
by skissane 1y ago
> Add a dedicated "parking spot" like GPT-OSS does and eat the gradient flow tax on that
Not familiar with this topic, but intrigued-anywhere I can read more about it?
- ACCount37 1y agoLooked for it briefly, think the best I got is this older discussion: https://news.ycombinator.com/item?id=44834918 https://news.ycombinator.com/item?id=44834918
- qcnguy 1y agoOpenAI have talked about it. The neural architecture needs to let the model handle the case where there's nothing worth attending to, as softmax requires attention to be allocated to all tokens but sometimes there's nothing worth it.