5 ms·
Shouldn't you also compare to mmap with huge page option? My understanding is its presicely meant for this circumstance. I don't think its a fair comparison wit
by bawolff 1y ago
Shouldn't you also compare to mmap with huge page option? My understanding is its presicely meant for this circumstance. I don't think its a fair comparison without it.
Respectfully, the title feels a little clickbaity to me. Both methods are still ultimately reading out of memory, they are just using different i/o methods.
- mrlongroots 1y agoYou don't need hugepages for basic 5GB/s sequential scans. I don't know the exact circumstances that would cause TLB pressure, but this is not it. You can maybe reduce the number of page faults, but you can do that by walking the mapped address space once before the actual benchmark too.
- jared_hulbert 1y agoThe original blog post title is intentionally clickbaity. You know, to bait people into clicking. Also I do want to challenge people to really think here. Seeing if the cached file data can be accessed quickly is the point of the experiment. I can't get mmap() to open a file with huge pages. void* buffer = mmap(NULL, size_bytes, PROT_READ, (MAP_HUGETLB | MAP_HUGE_1GB), fd, 0); doesn't work. You can can see my code here https://github.com/bitflux-ai/blog_notes https://github.com/bitflux-ai/blog_notes. Any ideas?
- mastax 1y agoMAP_HUGETLB can't be used for mmaping files on disk, it can only be used with MAP_ANONYMOUS, with a memfd, or with a file on a hugetlbfs pseudo-filesystem (which is also in memory).
- inetknght 1y ago> MAP_HUGETLB can't be used for mmaping files on disk False. I've successfully used it to memory-map networked files.
- inetknght 1y agoMy bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb https://github.com/inetknght/mmap-hugetlb
- jared_hulbert 1y agoAdding MAP_HUGE_1GB and not MAP_HUGETLB does compile and run for me. Not convinced that its' actually doing anything. Performance is the same.
- inetknght 1y agoWell now that it works, feel free to start poking around at it for a follow-up blog post :)
- bawolff 1y agoThe mmap man page kind of implies that would be a no-op, but i haven't tested myself.
- loloquwowndueo 1y agoShare your code?
- inetknght 1y agoMy bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb https://github.com/inetknght/mmap-hugetlb
- inetknght 1y agoI don't work there any more (it was a decade ago) and I'm pretty busy right now with a new job coming up (offered today). Do you have kernel documentation that says that hugetlb doesn't work for files? I don't see that stated anywhere.
- mananaysiempre 1y agoIt looks like there is in theory support for that[1]? But the patches for ext4[2] did not go through. [1] https://lwn.net/Articles/686690/ https://lwn.net/Articles/686690/ [2] https://lwn.net/Articles/718102/ https://lwn.net/Articles/718102/
- jandrewrogers 1y agoRead the man pages, there are restrictions on using the huge page option with mmap() that mean it won’t do what you might intuit it will in many cases. Getting reliable huge page mappings is a bit fussy on Linux. It is easier to control in a direct I/O context.