keivanalizadeh

3 ms·

1. 1.▲
   by keivanalizadeh 3y ago

   By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 and dejavu
2. 2.▲
   by keivanalizadeh 3y ago

   Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash. p.s: The window size in the paper is showing how many token's feed forw
3. 3.▲
   by keivanalizadeh 3y ago

   Yes, the goal is at the end to run larger models on the phone as phones have very limited DRAM.
4. 4.▲
   by keivanalizadeh 3y ago

   And how does mmap gonna help with the performance when the bottleneck is flash to DRAM bandwidth?
5. 5.▲
   by keivanalizadeh 3y ago

   One of the initial ideas was of course doing computation inside flash, but we didn't try to go that path for two reasons: 1. It's not as easy to change the controller, even if you do it was not obvious for me if we need for softwa