5 ms·
It addresses b too since decompositions are always smaller than the original tensor. It's usually the case that memory access is also slower than matrix multipl
by llm_trw 2y ago
It addresses b too since decompositions are always smaller than the original tensor. It's usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement.
- menaerus 2y ago> It's usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement. I haven't read this paper (yet) but isn't this the case that mostly applies to training and not so much to inference? A good example would be flash-attention, it trades the higher flops for better memory utilization but it's mostly irrelevant in inference workloads.
- verdverm 2y agoThey claim an inference time savings to the kv cache
- menaerus 2y agoI skimmed through the paper real quickly. There's no performance data on inference speedups in the paper. Only the benchmarks relevant for training. They also, interestingly, don't compare against the flash-attention. Flash-attention outperforms all of the other attention mechanisms mentioned in the paper: MHA, MQA, GQA, and MLA.
- verdverm 2y agoThey aren't claiming speedups, they are claiming up to an order of magnitude less space needed for the kv cache at runtime. This translates to a smaller GPU or longer sequences in the same GPU
- menaerus 2y agoUnder what circumstances can you cut down your LOADS and STORE from and to main memory by an order of magnitude without observing major improvements in algorithm runtime that is memory-bound?
- verdverm 2y agoAI models are compute bound, it's why we use GPUs
- menaerus 2y agoIncorrect. Self-attention is a highly parallel algorithm that makes it a great candidate for being a memory-bound workload once you have enough compute. Both datacenter grade CPUs and GPUs have enough compute to carry out the self-attention computation but it is only the latter that has enough hi-bandwidth memory to make the algorithm really perform. If this hadn't been the case, the theory behind flash-attention wouldn't materialize, and it does, and reason being that (main) memory is slow. Deep FFWD networks OTOH are compute-bound.
- zaptrem 2y agoTransformers are deep feedforward networks that happen to also have attention. Causal LMs are super memory bound during inference due to kv caching as all of those linear layers need to be loaded onto the core to transform only a single token per step.
- menaerus 2y agoAnd I said something else?
- lostmsu 2y agoMemory bound only applies to low batch size scenarios AFAIK
- apophis-ren 2y agoFlash attention is an implementation trick; you can implement MHA/GQA, for example, with flash attention.