4 ms·
Full distributions are a fucking pain to save - at this point just save the hidden states. But there are lossy compression tricks there.
by ACCount37 5mo ago
Full distributions are a fucking pain to save - at this point just save the hidden states. But there are lossy compression tricks there.
- rao-v 5mo agoTo the previous poster's point, soft distributions are useful, even saving the top 10 logits is significantly more training signal than just the final token.