5 ms·
4+ GB inputs are often fed to hashing functions. Verifying the integrity of file downloads before running them for example.
by mike_d 4y ago
4+ GB inputs are often fed to hashing functions. Verifying the integrity of file downloads before running them for example.
- hot_gril 4y agoBut that'll probably run in chunks. Not gonna load 4GB into memory. Edit: Turns out there was another comment like this, and a response was that the file could be mmap'd.
- Cthulhu_ 4y agoYou're optimistic, which is commendable, but unfortunately a naive implementation would probably just read it all. I probably would, because the happy path doesn't deal with 4 GB inputs.
- hot_gril 4y agoI guess if your test machine always has a lot of contiguous* free memory or you never test it with a large enough file. Otherwise, it's impossible to ignore. One large download, and it wouldn't just be slow, it'd not work at all. * Which is usually much harder than just free mem. I recall macOS always being able to give a huge chunk of memory, I guess by rearranging on the fly, but it's very slow if you ask for a lot. And Linux just says no.
- kevingadd 4y agoIt doesn't need to be contiguous on any modern operating system I'm aware of. You just need 4gb of free contiguous address space, which virtually every 64-bit application will have.
- hot_gril 4y agoOh, then I ought to try my experiment again sometime. Last time was 8 years ago on Ubuntu Server, and maybe I'm missing a detail, like unknowingly running a 32-bit OS. But yeah the behavior you describe is more what I'd expect.
- someweirdperson 4y ago> And Linux just says no. Linux might say yes, even if the memory isn't available. See notes of malloc: "This means that when malloc() returns non-NULL there is no guarantee that the memory really is available." [0] [0] https://man7.org/linux/man-pages/man3/malloc.3.html https://man7.org/linux/man-pages/man3/malloc.3.html
- YetAnotherNick 4y agoThis won't trigger overflow. It will only be triggered if file is broken into chunks.
- YetAnotherNick 4y agoThe vulnerability only triggers when you break the file into chunks and make one chunk larger than 4GB.
- magicalhippo 4y agoCould easily happen if you compute the combined hash of a set of files, with one chunk per file (mmap'ed fex).