3 ms·

30B model now needs only 5.8GB of RAM? How?

31 points by olalonde 4y ago

LeoPanthera 4y ago
The answer: "it did actually not, you are not seeing the actual ram usage, because the os now counts it as filesystem cache. Since #613 the model is a memory mapped file. It does not do the impossible, it's just that gnome-system-monitor does not show cached files."
- jasonmp85 4y ago
  This is incredibly mundane; why is this on the front page?
  - dagmx 4y ago
    Because it’s a response to a post that had a lot of hype on the front page yesterday
- deleted 4y ago
  [deleted]
kristianp 4y ago
Looks like the mmap functionality was done by the same person as https://justine.lol https://justine.lol what a great hacker! Edit: Dup of https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284
- olalonde 4y ago
  Oops, missed that.
tedunangst 4y ago
https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284
19h 4y ago
If you have enough memory, I the entire file eventually is in memory because it doesn't need to be paged out. If it takes up less memory (because your memory is limited), it'll still need to page the file off disk when it hits the cache bounds.
mayli 4y ago
80% percent of code writers probably never heard mmap, and might less than 1% ever use that.
- wyldfire 4y ago
  It's truly magical. And flags like MAP_LOCKED, MAP_ANONYMOUS, MAP_POPULATE - you can tune it in closer to the behavior you want. That said, if you are concerned about memory consumption, you are ceding control to the kernel. It gets to decide when and whether to evict pages (though you can give hints).
  - gaudat 4y ago
    qBittorrent builds using libtorrent 2.0 turned into an absolute memory hog because it used mmap. I assume it is the page table backing the hundred of GBs of memory mapped files. I remember seeing a joke issue asking the author to update the system requirements to be hundreds of GBs of RAM.
chrisMyzel 4y ago
According to the PR on github this is likely just a bug with the implementation that makes htop (etc) display wrong RAM usage numbers. It seems as of now as there is "no magic happening". People reported much higher actual RAM usage. The author acknowledged that it might be a bug when he posted the PR.

31 points by olalonde 4y ago

The answer: "it did actually not, you are not seeing the actual ram usage, because the os now counts it as filesystem cache. Since #613 the model is a memory mapped file. It does not do the impossible, it's just that gnome-system-monitor does not show cached files."

jasonmp85 4y ago
This is incredibly mundane; why is this on the front page?
- dagmx 4y ago
  Because it’s a response to a post that had a lot of hype on the front page yesterday
deleted 4y ago
[deleted]

kristianp 4y ago

Looks like the mmap functionality was done by the same person as https://justine.lol https://justine.lol what a great hacker! Edit: Dup of https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284

olalonde 4y ago
Oops, missed that.

tedunangst 4y ago

https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284

19h 4y ago

If you have enough memory, I the entire file eventually is in memory because it doesn't need to be paged out. If it takes up less memory (because your memory is limited), it'll still need to page the file off disk when it hits the cache bounds.

mayli 4y ago

80% percent of code writers probably never heard mmap, and might less than 1% ever use that.

wyldfire 4y ago
It's truly magical. And flags like MAP_LOCKED, MAP_ANONYMOUS, MAP_POPULATE - you can tune it in closer to the behavior you want. That said, if you are concerned about memory consumption, you are ceding control to the kernel. It gets to decide when and whether to evict pages (though you can give hints).
- gaudat 4y ago
  qBittorrent builds using libtorrent 2.0 turned into an absolute memory hog because it used mmap. I assume it is the page table backing the hundred of GBs of memory mapped files. I remember seeing a joke issue asking the author to update the system requirements to be hundreds of GBs of RAM.

chrisMyzel 4y ago

According to the PR on github this is likely just a bug with the implementation that makes htop (etc) display wrong RAM usage numbers. It seems as of now as there is "no magic happening". People reported much higher actual RAM usage. The author acknowledged that it might be a bug when he posted the PR.