3 ms·

I feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz

I feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally). Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file. I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways.

samcday 8y ago
But now you have two completely independent code paths. Both of which will need to go through the same maturation phase that the ST folks evidently went through with mmap. And if the code needs to evolve for other reasons, potentially both of these paths will need some love too. Seems like the worst choice in a situation like this!
- CoolGuySteve 8y ago
  The read() API is so basic, mainly because it is synchronous and has clear return values, that I have a hard time believing it would present nearly as many issues.
- theoh 8y ago
  Butler Lampson talks about the general concept of an overlay in operating system design. It's a slightly different case, on the surface, but the possible application in this case is that if mmap fails, you probably should have a fallback code path that tries to access the file normally. That's obviously not going to work for all applications. But it's something you consciously have to do anyway if mmap may not be available (e.g. when working across the network). The OS design should make it easy to move up and down in this hierarchy of access methods, but the application programmer still needs to know which "level" of access they have to the file. There's work to be done here in improving the design of the OS interface. That would lift part of the the burden of maintaining multiple paths in the application code. https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s slides: https://bwlampson.site/Slides/Hints%20and%20principles%20(HLF%202015).pdf https://bwlampson.site/Slides/Hints%20and%20principles%20(HL... see also: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-826-principles-of-computer-systems-spring-2002/lecture-notes/30.pdf https://ocw.mit.edu/courses/electrical-engineering-and-compu...
ben-schaaf 8y ago
> You want to abstract two different kinds of file reader: an mmap reader and a regular reader. We already do this. Small files aren't mmap'd in Sublime Merge and instead copied into memory. > I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways. Speed in this case was the absolute least of our concerns. mmap was used to deal with large files without refactoring large parts of the codebase. Not using it for networked files doesn't fix the memory usage issue.

samcday 8y ago

But now you have two completely independent code paths. Both of which will need to go through the same maturation phase that the ST folks evidently went through with mmap. And if the code needs to evolve for other reasons, potentially both of these paths will need some love too. Seems like the worst choice in a situation like this!

CoolGuySteve 8y ago
The read() API is so basic, mainly because it is synchronous and has clear return values, that I have a hard time believing it would present nearly as many issues.
theoh 8y ago
Butler Lampson talks about the general concept of an overlay in operating system design. It's a slightly different case, on the surface, but the possible application in this case is that if mmap fails, you probably should have a fallback code path that tries to access the file normally. That's obviously not going to work for all applications. But it's something you consciously have to do anyway if mmap may not be available (e.g. when working across the network). The OS design should make it easy to move up and down in this hierarchy of access methods, but the application programmer still needs to know which "level" of access they have to the file. There's work to be done here in improving the design of the OS interface. That would lift part of the the burden of maintaining multiple paths in the application code. https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s slides: https://bwlampson.site/Slides/Hints%20and%20principles%20(HLF%202015).pdf https://bwlampson.site/Slides/Hints%20and%20principles%20(HL... see also: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-826-principles-of-computer-systems-spring-2002/lecture-notes/30.pdf https://ocw.mit.edu/courses/electrical-engineering-and-compu...

ben-schaaf 8y ago

> You want to abstract two different kinds of file reader: an mmap reader and a regular reader. We already do this. Small files aren't mmap'd in Sublime Merge and instead copied into memory. > I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways. Speed in this case was the absolute least of our concerns. mmap was used to deal with large files without refactoring large parts of the codebase. Not using it for networked files doesn't fix the memory usage issue.