3 ms·
Well, yeah. Short version: the claim that llama could use much less memory by memory-mapping the model data file was wrong. It's just that the Linux utilities
by Animats 4y ago
Well, yeah.
Short version: the claim that llama could use much less memory by memory-mapping the model data file was wrong. It's just that the Linux utilities for memory use don't count memory-mapped file area.
- pavon 4y agoI'm not convinced that there isn't more to the story than that. People (including Justine who implemented MMAPing the model and knows how to monitor memory use properly) are seeing that much of the file isn't being paged into memory, and are not quite sure why[1]. Is it a bug? Is the distribution of weights used highly dependent on the prompts? Are some weights somehow unused altogether? This twitter thread rules out some of those, but I don't think it closes the door. [1] https://news.ycombinator.com/item?id=35393615 https://news.ycombinator.com/item?id=35393615 edit: softened some of my claims after reading more updates from over the weekend.
- Animats 4y agoNow that's interesting. Entire memory pages of the model aren't being referenced?
- freilanzer 4y agoIf correct could this hint at sub models for specific tasks?
- sitkack 4y agoeBPF echo 3 > /proc/sys/vm/drop_caches
- wmf 4y agoOther people are saying that the whole file does get paged in and transformers access all their weights by design.
- peterfirefly 4y agoThe kernel will read more than just the page that faulted. Jart counts just the page faults and multiplies by the page size.
- astrange 4y agoFile backed memory is "less" than heap memory, because it can be thrown away when needed instead of being swapped out to disk. And because reading the file probably allocated file backed pages for it anyway unless you used O_DIRECT (and profiled it first.) There's also wired/mlock() memory which is "worse" than regular memory because it can't be swapped/evicted at all.