3 ms·
The reason they need terabytes per second is they're constantly loading weights and data into and then back out of multiply accumulator chips. If you program h
by mikewarot 1y ago
The reason they need terabytes per second is they're constantly loading weights and data into and then back out of multiply accumulator chips.
If you program hardware to just do the multiply/accumulate with the weights built in, you can then reduce the bandwidth required to just putting data into a "layer" and getting the results out, a much, MUCH lower amount of data in most cases.
- thrtythreeforty 1y agoDoesn't this presume that you can fit the model's weights into the SRAMs of a single chip, or of multiple chips that you plan to connect together? A big reason HBM is a thing is because it's much, much denser than SRAM.