5 ms·
Interesting they would discuss system call overhead of opening a file, reading from it and closing it. Seems like in almost all cases the open and close calls w
by ec109685 3y ago
Interesting they would discuss system call overhead of opening a file, reading from it and closing it. Seems like in almost all cases the open and close calls would be overwhelmed by the other operations.
- eru 3y agoFor lots of small files, that might not be the case. (I worked on a FUSE filesystem that had these issues.)
- loeg 3y agoIt seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. I think the main benefit here is not having to do memory copies through the CPU, which frees up memory bandwidth for other things.
- maxcoder4 3y agoThere are plenty of cases where you can't just change the file layout. And the GPU filesystem is being implemented by someone else, so the choice is: migrate your data to another filesystem OR fix the data-in-files layout, even though the files may come from completely different source than your application, the layout may be a standard or other applications may depend on it, or you can't easily change it for another reason.
- loeg 3y agoIf you can get the data into the GPU-native filesystem, you can change the data layout at least as easily. The point is there is some sort of data ingestion pipeline involved.
- eru 3y ago> It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. You can improve file-open overhead in conventional filesystems, too. Including the FUSE one I was working on.
- loeg 3y agoSure!
- deleted 3y ago[deleted]