3 ms·
If your application really wants to random access over 2 GB of files you might have a point. Most applications do not, except database servers. Cache works with
by AstralStorm 8y ago
If your application really wants to random access over 2 GB of files you might have a point. Most applications do not, except database servers. Cache works with all your memory and is handled with 64 bit offsets in kernel, the limitation is in mmap, but you can mmap parts of a big file no problem with large file support in libc.
- pedrocr 8y agoThe use case I'm thinking of is not random access over large files. It's using lots of very small files mmap()'ed so that you get the Linux VM helping you run an efficient cache. I've run across this with thumbnail caches in image processing software. It's easy for these kinds of things to be present in generally used libraries that your standard app will be using just because it has a GTK file selector for example. > you can mmap parts of a big file no problem with large file support in libc. For the speedups discussed here I definitely don't want to spend any time writing special code just to be able to read a 3GB file. That ship has long sailed and good riddance to all those subtle issues where things seem to work fine until they don't and it's not clear why.
- CJefferson 8y agoActually, you can get a lot less than that once libraries are loaded and such like. I've found on some 32 bit linuxes you can't mmap a single file bigger than around 500mb, which means I couldn't check out a git repository containing 650mb cd images on a 32 bit Linux.
- geocar 8y agoYou might check resource limits and/or disable address randomisation (or other "security" features) since they can really eat up your (limited) address space. Don't forget to not put the bugs in, though!