5 ms·
Go can only read 1GiB per Read call
- hmottestad 3y agoA good comparison. I couldn’t see that the author has created an issue on the golang GitHub. I assume that this would be considered a bug and that the golang community would be open to a fix.
- rini17 3y agoOr not, because it simplifies code, no differences between platforms. And if syscall overhead once per each GB is really a problem you should probably try mmap or some other IO style.
- oflebbe 3y agoOnly that there is no official mmap support in go, due to gc concerns, afaik
- barsonme 3y agoHm? https://pkg.go.dev/golang.org/x/sys/unix#Mmap https://pkg.go.dev/golang.org/x/sys/unix#Mmap
- hmottestad 3y agoThe author saw performance issues with this restriction, which is why they wrote the post.
- xyzzy123 3y agoI would be curious to see the numbers on that.
- hmottestad 3y agoYou should probably ask the author then. There are a bunch of links to various socials on the bottom of the post.
- technicolor 3y agoHello, author here. Just to clarify, I wasn't saying that this has any perf impact per se. I was trying to see why syscall showed up quite a bit in the profile, when reading a large file, no matter what buffer size I pass in. Rather than spend time finishing the actual work, I spent way too much time trying to understand the reason for the limit.
- hmottestad 3y agoMmap seems specific to POSIX according to this post[1], so it's not a portable way of using memory mapped files. Since there are already differences between platform capabilities in golang there really shouldn't need to be an artificial limit to how large of a chunk you can read in one operation. In 20 years I'm sure 1 GiB will be considered small. 1: https://stackoverflow.com/questions/52589949/undefined-syscall-mmap-on-windows-10 https://stackoverflow.com/questions/52589949/undefined-sysca...
- xyzzy123 3y agoInteresting factoid, but I can't see how this would lead to any real world performance impacts. I am also baffled as to why you would want 2gb reads for wc. System calls are not that expensive and at some point touching so many pages has gotta hurt.
- aydio 3y agoFYI, factoid has a different meaning: https://en.wikipedia.org/wiki/Factoid https://en.wikipedia.org/wiki/Factoid
- DemocracyFTW2 3y agoÄckschully no: A factoid is either an invented or assumed statement presented as a fact, or a true but brief or trivial item of news or information. which is exacktly what this is. Innit.
- deaddodo 3y ago> Since the term's invention in 1973, it has become used to describe a brief or trivial item of news or information. In the very first paragraph, in fact.
- technicolor 3y agoAuthor here. I wasn't suggesting it would have any perf impact. Just that it was an interesting change set.
- xyzzy123 3y agoMakes sense, I guess I misunderstood the premise of the article. It's a good dive.
- jsnell 3y agoI feel I'm missing something here, but it feels like the intro is not only suggesting there was a performance impact, but very explicitly stating there was: > profile of a sample word count program I was writing, which showed the program was spending way too much time in the syscall module. That in this context can only mean one thing: way too many read syscalls were getting called. I find it hard to believe that the profile would look any different with 1 vs 2 syscalls per 2GB chunk. The syscall overhead is going to be insignificant compared to actually copying the data. The program is going to be spending a lot of time doing syscalls no matter how many there are, because the individual syscalls will just start to take more time as you increase the size of the buffer. Edit: Compare: strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 1*1024*1024*1024) {} ' read(3, ..., 1073741824) = 1073741824 <0.455901> read(3, ..., 1073741824) = 1073741824 <0.219711> read(3, ..., 1073741824) = 1073741824 <0.213923> read(3, ..., 1073741824) = 1073741824 <0.211783> Vs. strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 2*1024*1024*1024) {} ' read(3, ..., 2147483648) = 2147479552 <0.921789> read(3, ..., 2147483648) = 2147479552 <0.487007> read(3, ..., 2147483648) = 8192 <0.000031>
- cm2187 3y agoIsn't GiB = 10^9, GB = 1024^3? Some people seem to use them interchangeably.
- progbits 3y agoThe other way around, author uses them correctly. k/M/G/... come from SI which is based on 10^3. Ki/Mi/Gi/... are for 1024.
- pkage 3y agoOther way around, 10^9 bytes is a gigabyte (GB) and 1024^3 is a gibibyte (GiB).
- none_to_remain 3y agoI have yet to hear anyone say "kibi" or "gibi" that wasn't mocking the term. I believe the virtuous answer is that bytes are measured in powers of two, & hard drive manufacturers are dishonest. (Bits are measured in powers of ten)
- DaSHacka 3y ago> hard drive manufacturers are dishonest. (Bits are measured in powers of ten) You'd have to add ISPs to that too, as network links are typically measured in decimal increments.
- syene 3y agoI say “mibi” and “gibi” all the time, because my system denotes them as such and it sounds much cuter
- DemocracyFTW2 3y agoYeah sure and it's completely irrelevant in this context and also the other way around
- PaulKeeble 3y agoIn the past when I have tested buffer sizes with reads I can't say I have noticed much performance difference past really quite small buffers like 16k. I am sure there is some CPU usage cost associated with more calls but its not much and its just not worth using so much memory for streaming purposes.
- silisili 3y agoThis seems like an article that could have just been summed up and rewritten as the title. It's a conscious choice and consistent. I have no idea what use case would need to use 2GB per read call, if anything the limit seems excessive.
- neonsunset 3y agoDoes not reproduce :) // Assuming 2GiB length using var file = File.OpenRead("verylargefile.txt"); var read = 0; var iterations = 0; var buffer = new byte[Array.MaxLength].AsMemory(); while ((read = await file.ReadAsync(buffer)) > 0) { buffer = buffer[read..]; iterations++; } Console.WriteLine($"Read {Array.MaxLength} in {iterations} call(s).");