3 ms·
BTW, I've always thought Python uses way too many syscalls when working with files. Simple code like this uses something like 9 syscalls (shown in the article):
by collinmanderson 3y ago
BTW, I've always thought Python uses way too many syscalls when working with files. Simple code like this uses something like 9 syscalls (shown in the article):
with open('myfile') as f:
data = f.read()
I'm not much of a C programmer myself. but I at least reported part of the issue to Python: https://bugs.python.org/issue45944 https://bugs.python.org/issue45944
This is the fastest way to read a file on python that I've found, using only 3-4 syscalls (though os.fstat() doesn't work for some special files kernel files like those in /proc/ and /dev/):
def read_file(path: str, size=-1) -> bytes:
fd = os.open(path, os.O_RDONLY)
try:
if size == -1:
size = os.fstat(fd).st_size
return os.read(fd, size)
finally:
os.close(fd)
- the8472 3y agoAs you say, the reported size is not necessarily correct so it should only be treated as a hint. And if os.read directly translates to a read syscall then you're also not handling short reads.
- collinmanderson 3y agoAhh ok so to be correct you have to keep reading until you get an empty read? Maybe I don’t need to query the file size at all?
- the8472 3y agoquerying the file size can be useful to choose the allocation size for a buffer. but yes, you have to keep reading until you get a zero-length read. https://man7.org/linux/man-pages/man2/read.2.html https://man7.org/linux/man-pages/man2/read.2.html > On success, the number of bytes read is returned (zero indicates end of file), [...] It is not an error if this number is smaller than the number of bytes requested