3 ms·
What part of the optimization process gave you the biggest speed gain?
by huangsemao 2mo ago
What part of the optimization process gave you the biggest speed gain?
- deleted 2mo ago[deleted]
- gitpusher42 2mo agoSwitching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7