4 ms·
Yes, that much I do understand. I'm just confused why that is important when predicting the next token? What about the calculation requires that you keep track
by xrd 3y ago
Yes, that much I do understand. I'm just confused why that is important when predicting the next token? What about the calculation requires that you keep track of really big numbers (and don't overflow) and requires that you have very precise numbers (and don't truncate the precision)?
- xeonmc 3y agoBecause AI is an alchemical art involving heuristically selecting arcane “activation functions” without critically thinking about precision distribution or symmetry, so they rely on band-aiding over it via altering number formats rather than properly matching the function to the appropriate numeric domain.
- deleted 3y ago[deleted]