3 ms·
Does it support paged attention like vLLM though? Without that they will run into memory fragmentation quickly.
by storus 7mo ago
Does it support paged attention like vLLM though? Without that they will run into memory fragmentation quickly.
- lukebechtel 7mo agoYes, great question! The system started without paged attention, and recreated its own paged attention implementation automatically once it realized it was a bottleneck. Pretty cool!
- deleted 7mo ago[deleted]