2 ms·
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient archi
by danmaz74 1mo ago
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
- dotancohen 1mo agoExactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference. If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.
- PessimalDecimal 1mo agoTurning RAM into an active coprocessor does feel like an awkward model for that though. Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit. But Samsung doesn't have that option so they're playing the card they have.
- londons_explore 1mo agoI think useful flops per dollar will outweigh any "awkwardness" factor. Elegant designs don't win anymore.
- IsTom 1mo agoWith the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.