3 ms·
And this is an asic that is still operating digitally. Imagine a chip with baked it weights that does its math analogue with 20x reduction in number of circuit
by LarsDu88 5mo ago
And this is an asic that is still operating digitally. Imagine a chip with baked it weights that does its math analogue with 20x reduction in number of circuit elements needed to do a multiplication op.
If there's a breakthrough in memristors, you could end up with another 20x reduction in circuit elements (get rid of memory bottlnecks, start doing multiplication ops as log transform voltage addition)
The ceiling is ultra high for how far AI can go.
- mordae 5mo agoExcept weights will be unstable - temperature and frequency dependent - and we still have issues delivering analog circuits reliably to the spec. So it would take multiple attempts. But yeah, as soon as the digital models start to plateau, ASICs and then this will happen.
- ACCount37 5mo agoYou could SFT for temperature and frequency stability. I'm not even kidding. Modern ML systems already eat errors - what's one more error type for them to eat?
- imtringued 5mo agofp8 and fp4 are already the standard data types, where are you going to find a 20x reduction? At 1 sign bit, 1 exponent and 2 mantissa bits, there is barely any work done during multiplication into a bf16 accumulator. You're performing two shifted adds at this point.