5 ms·

Vmtouch – Portable file system cache diagnostics and control

82 points by old-gregg 10y ago

currriuosly 10y ago
Where can I read more about Linux internals? Any good resource with design decisions and algorithms?
- thedjinn 10y ago
  I read Understanding The Linux Kernel 3rd edition a couple of years ago and that book covers quite a lot of it.
- alpb 10y ago
  https://www.gitbook.com/book/0xax/linux-insides https://www.gitbook.com/book/0xax/linux-insides
- hoytech 10y ago
  In terms of virtual memory, there's a somewhat dated book that explains the linux implementation: https://www.kernel.org/doc/gorman/pdf/understand.pdf https://www.kernel.org/doc/gorman/pdf/understand.pdf And there are some maintained text files in the kernel source tree itself: https://www.kernel.org/doc/Documentation/vm/ https://www.kernel.org/doc/Documentation/vm/ Of particular interest is the pagemap.txt file. There's also a really neat utility in the kernel source called page-types and you can learn a lot by compiling it and playing around, and of course reading the source (there are lots of tricks in there): http://lxr.free-electrons.com/source/tools/vm/page-types.c http://lxr.free-electrons.com/source/tools/vm/page-types.c Finally, you could also read the source to vmtouch or my vmprobe utilities, or ask questions in the vmtouch github repo -- we can probably point you in the right direction.
hoytech 10y ago
Hi! I've also been working on another program called vmprobe which does some neat things vmtouch doesn't: https://vmprobe.com/ https://vmprobe.com/
deleted 10y ago
[deleted]
martythemaniak 10y ago
vmtouch and vmprobe are both pretty cool utilities. There's an interesting presentation where you can see how it's used: https://www.youtube.com/watch?v=yoBIr9sFCS8 https://www.youtube.com/watch?v=yoBIr9sFCS8
boomskats 10y ago
I'm so happy to have come across this. Had no idea it existed.
mark212 10y ago
Now _this_ is how you write a readme. The first sentence has a clear, concise explanation of what the code is for and why you might care, without a garble of acronyms and abbreviations.
ausjke 10y ago
Great tools that are very useful for my purpose, installing it now and Thanks!
TheAceOfHearts 10y ago
I think linking to the homepage would've been better. It includes a "What is it good for?" section, as well as examples. The articles on the vmprobe[1] homepage are also very well written. They present a simple scenario, and talk through it without lots of buzzwords. A bit sad that there's no macOS support. I just tried using vmtouch on the root of a web project with lots of dependencies. I tried evicting and touching a few times, and it seems to consistently take ~1 second off its startup time. Unfortunately, touching seems to take ~3 seconds. Does anyone regularly use this in their daily workflow? [0] https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/ [1] https://vmprobe.com/ https://vmprobe.com/
- hoytech 10y ago
  Thanks! Ya I'd like to add macOS support at some point to vmprobe, but I'm using a bunch of linux-specific stuff at the moment: mostly pagemap/kpageflags, although also some mmap/madvise flags that are linux extensions. Still, the basic use cases don't need this. Regarding your web-app, it's possible that you are touching a bunch of files that you don't need to. Rarely-accessed static files, the .git directory, and so on. vmprobe snapshots might help here, or you could figure out which files are used for start-up and just touch those. There's also the -I switch. You might try something like: vmtouch -t -I '*.html' -I '*.js' -I '*.css' /var/htdocs/ Also, sometimes it is possible to page in the necessary files before your web-app is started, or concurrently while it's starting (maybe at boot-time, container start, etc). Also, sometimes assets are paged in before the first request is received. I know of quite a few people and companies that use vmtouch in their daily workflows. vmprobe is still largely experimental at this point though.
- microcolonel 10y ago
  For what it's worth, the homepage is basically the same document as the README.md.

82 points by old-gregg 10y ago

currriuosly 10y ago

Where can I read more about Linux internals? Any good resource with design decisions and algorithms?

thedjinn 10y ago
I read Understanding The Linux Kernel 3rd edition a couple of years ago and that book covers quite a lot of it.
alpb 10y ago
https://www.gitbook.com/book/0xax/linux-insides https://www.gitbook.com/book/0xax/linux-insides
hoytech 10y ago
In terms of virtual memory, there's a somewhat dated book that explains the linux implementation: https://www.kernel.org/doc/gorman/pdf/understand.pdf https://www.kernel.org/doc/gorman/pdf/understand.pdf And there are some maintained text files in the kernel source tree itself: https://www.kernel.org/doc/Documentation/vm/ https://www.kernel.org/doc/Documentation/vm/ Of particular interest is the pagemap.txt file. There's also a really neat utility in the kernel source called page-types and you can learn a lot by compiling it and playing around, and of course reading the source (there are lots of tricks in there): http://lxr.free-electrons.com/source/tools/vm/page-types.c http://lxr.free-electrons.com/source/tools/vm/page-types.c Finally, you could also read the source to vmtouch or my vmprobe utilities, or ask questions in the vmtouch github repo -- we can probably point you in the right direction.

hoytech 10y ago

Hi! I've also been working on another program called vmprobe which does some neat things vmtouch doesn't: https://vmprobe.com/ https://vmprobe.com/

deleted 10y ago

[deleted]

martythemaniak 10y ago

vmtouch and vmprobe are both pretty cool utilities. There's an interesting presentation where you can see how it's used: https://www.youtube.com/watch?v=yoBIr9sFCS8 https://www.youtube.com/watch?v=yoBIr9sFCS8

boomskats 10y ago

I'm so happy to have come across this. Had no idea it existed.

mark212 10y ago

Now _this_ is how you write a readme. The first sentence has a clear, concise explanation of what the code is for and why you might care, without a garble of acronyms and abbreviations.

ausjke 10y ago

Great tools that are very useful for my purpose, installing it now and Thanks!

TheAceOfHearts 10y ago

I think linking to the homepage would've been better. It includes a "What is it good for?" section, as well as examples. The articles on the vmprobe[1] homepage are also very well written. They present a simple scenario, and talk through it without lots of buzzwords. A bit sad that there's no macOS support. I just tried using vmtouch on the root of a web project with lots of dependencies. I tried evicting and touching a few times, and it seems to consistently take ~1 second off its startup time. Unfortunately, touching seems to take ~3 seconds. Does anyone regularly use this in their daily workflow? [0] https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/ [1] https://vmprobe.com/ https://vmprobe.com/

hoytech 10y ago
Thanks! Ya I'd like to add macOS support at some point to vmprobe, but I'm using a bunch of linux-specific stuff at the moment: mostly pagemap/kpageflags, although also some mmap/madvise flags that are linux extensions. Still, the basic use cases don't need this. Regarding your web-app, it's possible that you are touching a bunch of files that you don't need to. Rarely-accessed static files, the .git directory, and so on. vmprobe snapshots might help here, or you could figure out which files are used for start-up and just touch those. There's also the -I switch. You might try something like: vmtouch -t -I '*.html' -I '*.js' -I '*.css' /var/htdocs/ Also, sometimes it is possible to page in the necessary files before your web-app is started, or concurrently while it's starting (maybe at boot-time, container start, etc). Also, sometimes assets are paged in before the first request is received. I know of quite a few people and companies that use vmtouch in their daily workflows. vmprobe is still largely experimental at this point though.
microcolonel 10y ago
For what it's worth, the homepage is basically the same document as the README.md.