3 ms·
bfloat16 is stable enough for ML training, which is what TPUs exist for
by why_only_15 4y ago
bfloat16 is stable enough for ML training, which is what TPUs exist for
- rubatuga 4y agoNot always though!
- pfortuny 4y agoBe careful: 16 bits was quite a lot of colors around 20 years ago. Now we would laugh at it. Ditto for 640Kb of RAM (who would need more?) etc. Not trying to be dismissive just saying that... computational limits are limits on what can be done, in the end.
- mjan22640 4y ago30 years ago. The time flies...
- pfortuny 4y agowell, you are right and I grow old, I grow old, I shall wear the bottoms of my trousers rolled… I stand corrected.
- omegalulw 4y agoWrong way to think about the problem. Lowering bits does not mean lowering model capacity. Its the opposite in fact - it allows you to you to fit more parameters.
- pfortuny 4y agoWell, yes within your constraints. In the end, you are choosing between two aspects. Same as with screens: you could have resolution (1024x768!!) or color (16bits!). Edit: the term I could not remember is tradeoff.
- adgjlsfhk1 4y agokind of. if we get computers that are 1000x faster, it just becomes a tradeoff between higher precision or 1000x more parameters. the reason resolution has stopped being pushed is that our eyes have severe diminishing returns. it's not yet known whether brains do.