10 ms·

This reminds me of a story from 15 years ago, where I was developing a technology to download games on demand by hooking into the OS calls. There was a particu

This reminds me of a story from 15 years ago, where I was developing a technology to download games on demand by hooking into the OS calls. There was a particular game that was superslow when this tech was applied. Original game loading took around 15-20 seconds, whereas once the tech was applied it took easily 3-5 min, even with all data already downloaded. When I started digging into it, I realized the reason was the game was using something like fread(data, 1, 65536, fptr); instead of fread(data, 65536, 1, fptr); Which basically expanded back in the day to 65k reads of 1 byte for several MB file. Each fread translated to 65k reads of ReadFile Windows API. Since my code was hooking on ReadFile system call, and my call was heavier than ReadFile, the game loading felt really slow. Unusable. It would have not been fun for players. The easy fix was to swap arguments for certain calls. The long fix required to use an internal cache to account for these cases so that the hooked ReadFile was faster when data was already in disk. Funny thing is that as we started rolling out the tech and applying it to more and more games we realized lots of games did this. We went for the cache fix and games ended up loading faster than before. Honestly, games could have load all the data in a couple of seconds by just swapping the args. I'm guessing developers did this on purpose so that games seemed like they were loading a lot of stuff, although you never know.

Taniwha 4mo ago
I used to be a graphics card/chip architect for macs in the early/mid 90s - our chips were the fastest, but some programs were resistant because they did stupid stuff: pagemaker invalidated the font cache every time it went thru its main loop, quark with ATM did an n*2 thing every time it wrote text etc etc. We had special hardware to accelerate text drawing and it did nothing because the software pissed it away. We considered creating a plugin that fixed all these things, it would have been hard to maintain, in the end we travelled around to the people who made these apps and talked them through their problems To be fair excel would erase places white that it wanted to write up to 9 times before it drew any black pixels, we made that very fast! we didn't tell them :-) At the time 24-bit framebuffers were so slow that before we built graphics acceleration hardware people would switch back to 8-bit to get stuff done, making 24-bit/true colour your daily driver was a big step forward.
- urbandw311er 4mo ago
  This is a horrible and yet not unexpected insight into the internals of Excel
  - Taniwha 4mo ago
    To be fair this was Excell 25 years ago, may no longer be true. One of the other bugs (the Quark/ATM one) was also because of the programmers were worried about writing over stuff that hadn't been completely erased, the Quark guys wrote a string with 2 spaces at the end through a box that masked the end of the string, the ATM font renderer saw it couldn't fit the text so it split it in half and tried again so it drew N/2 N/4 N/8 ... strings. It spent all it's time in the 68k's multiply instructions figuring out how wide the strings (and substrings) were, our fancy 24-bit character rendering hardware was an afterthought
    - sidewndr46 4mo ago
      Nowadays Excel is a webapp that reflows the entire DOM 400 times per keystroke!
  - bathtub365 4mo ago
    In all of the software you’ve written, are you aware of how many on-screen pixels you’ve overdrawn?
  - trelbutate 4mo ago
    > To be fair excel would erase places white that it wanted to write up to 9 times before it drew any black pixels I feel like I'm having a stroke trying to read this, what does it mean??
    - sixeyes 4mo ago
      before writing to some area, it would erase it (clearing with white) up to 9 times
    - NSUserDefaults 4mo ago
somenameforme 4mo ago
Doesn't that break anything relying on the return value? fread gives you the number of objects read as a return. So I think a pretty typical thing would be to fread and then parse that number of characters, and that'd just break?
- jcul 4mo ago
  I've seen a lot of code that just assumes fread / fwrite succeeded without bothering to check the return value... But in this case if the code was calling fread 65536 times in a loop and getting 64KiB each time it wouldn't be good either! Sounds like the parent comment had to fix this with the internal cache thing to speed up the small freads. I think they meant the easy fix would have been swapping the args in the original / caller code.
  - account42 4mo ago
    There are no small freads in the story, whatever implements those freads supposedly split them up into many calls. But that sound more like a problem of that implementation than the fread callers as size == 1 is correct when you are reading a bag of bytes.
    - jcul 4mo ago
      Ah you're right, I misread it.
- DonHopkins 4mo ago
  The type of programmer who swaps the args to fread tends to be the type of programmer who doesn't bother to check the return value, fortunately. Edit: mort96: So did you check the return value or not?
  - account42 4mo ago
    But the args aren't necessarily swapped just because they end up in a slow case in some implementation.
lukan 4mo ago
"I'm guessing developers did this on purpose so that games seemed like they were loading a lot of stuff" I really hope that was not the case and rather think incompetence or to deal with obscure legacy problems, but the gamer in me gets enraged at the thought someone would artificially increase loading times.
Someone 4mo ago
> Which basically expanded back in the day to 65k reads of 1 byte for several MB file. Each fread translated to 65k reads of ReadFile Windows API What software did that that badly? If the code asks for (up to) 65,536 single byte items, why would you split that into 65,536 calls? Also, that change changes behavior. The old call could read anything from zero to 65,536 bytes, the new one only can read zero or 65,536 bytes. (Reading the source of a few implementations, I think most implementations will fill the output buffer with partial objects if the input doesn’t supply an integral number of them, but the return value of fread cannot signal that to the caller)
- micampe 4mo ago
  A long time ago I worked with someone who read 1 byte at a time from a socket because they insisted data was cached so the kernel was going to batch it magically somehow. It took me days to convince them to measure it.
  - quietbritishjim 4mo ago
    That's different: you're talking about the application code, like OP. But I think the parent comment's point is that the issue is in the implementation of fread itself in the standard library. It's perfectly reasonable for an application to pass it 1, 65536 (i.e. one byte, up to 65536 times) and expect it not to issue 65536 separate OS calls.
    - b112 4mo ago
      Is it? I get what you're saying, but asking for 1 byte 65536 times, is indeed different than asking for 65536 bytes, 1 time. There may be reasons, such as when you pull off the end of a buffer, it shifts. And the buffer size is 1 byte. Or 10. Or whatever. No, I'm not saying that's why. I'm simply saying there is a difference between asking for 1 byte or 65k bytes of something. Even dd runs the same under Linux. dd bs=10k count=1 is faster than bs=1 count=10k I remember trying to recover some data from a spinning disk, and trying to slowly creep up on the data. So I wanted 1 byte per, I wanted it to nibble, until it hit whatever the errored part was. If I just grabbed the lot, it'd error out from the whole read.
mort96 4mo ago
Wait, is that wrong? I always call fread as: fread(data, 1, sizeof(buffer), f); with the rationale that I'm interested in reading sizeof(buffer) individual bytes. The buffer size is incidental, not the size of the items I'm trying to read from the file; "read one item whose size is sizeof(buffer)" seems semantically wrong. Is this just the case of Windows having a bad stdlib fread implementation 15 years ago or is my thinking here actually wrong?
- chadgpt3 4mo ago
  It's not wrong. Guy just wrote a bad implementation of fread and blamed everyone else.
  - DarkUranium 4mo ago
    He didn't write it. The C runtime authors did (presumably Microsoft, if it's MSVCRT). He's hooking into ReadFile, a layer below the stdlib. By the time it reaches the hook, it's already split.
- projektfu 4mo ago
  fread should be buffered, but different values may cause buffering at different rates. Perhaps it didn't generate 65535 calls to ReadFile but it generated 16 or 64.
chadgpt3 4mo ago
Why does your fread to anything other than multiplying the two arguments?
- Sesse__ 4mo ago
  The idea of having two arguments to fread() is presumably to be able to do something else than all-or-nothing when there's a short read.
  - chadgpt3 4mo ago
    Yes, it divides the bytes read by the element size to get the return value. Which is the obvious reason you'd pass an element size of 1: you want to know how many bytes were read.
fsfod 4mo ago
Part of Windows Explorer actually does tons of tiny 4 byte ReadFile calls in to its tracking database like file when you delete a file. If you deleting lots of files this quickly adds up.
- pbhjpbhj 4mo ago
  Is this why Windows takes so long to delete things?? Presumably those reads aren't done when using del from a console as that always seems a bit faster.
  - Asmod4n 4mo ago
    Windows Explorers zip implementation also seams to do 1 byte reads by the speed is has compared to every other zip implementation.
    - jonathanlydall 4mo ago
      It is frustrating how slow .zip (and more recently .7z) support built into Windows Explorer is. This is a great article on why it's so unreasonably slow to modify these archives: https://textslashplain.com/2021/06/02/leaky-abstractions/ https://textslashplain.com/2021/06/02/leaky-abstractions/ But it doesn't seem to explain why it's so much slower at regular extraction.
  - jonathanlydall 4mo ago
    Its slowness is also a function of security software or any other file system "filters" (I believe they're called) are installed. For example, I run TortoiseGit which has a caching feature which is supposed to make it faster at showing what to commit. Disabling it increases the number of items I can delete per second in my Windows Explorer from about 1000 to about 3000 while making not making TortoiseGit operations meaningfully slower (that I can tell). This is a Dev Drive [0] on my machine, it would probably be slower on my C: drive which has full Windows Defender real time file scanning. [0]: https://learn.microsoft.com/windows/dev-drive/ https://learn.microsoft.com/windows/dev-drive/
dfox 4mo ago
The most important fix in SP1 for Office 2007 was fixing exactly that in Excel. Doing ridiculous amount of 4 byte reads made it basically unusable on network filesystems.
Xirdus 4mo ago
Reminds me of the "community patch" to GTA Online from a few years ago. The game was plagued by 10+ minute loading times. The situation remained for years and only got worse with time. Some hacker figured out that the game spent 80% of loading time reading the in-game store listing file. The file was tens of megabytes IIRC, and it literally used the Schlemiel the Painter's Algorithm - for each entry, start reading from the beginning byte after byte. The hacker made a tiny patch that made it remember where it found the last entry. This cut the total loading time by 80%, from over 10 minutes to less than 3. Edit: removed incorrect information.
- exrook 4mo ago
  This is not quite an accurate telling of rockstar's reaction, there were actually receptive to it and paid out $10k for the discovery. Though it's an understandable mistake given rockstar's hostile history with the gta modding scene. See the original post and discussion for the whole story: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times... https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
  - Xirdus 4mo ago
    That's not how I remember these events when they were playing out. I distincly remember social media posts warning about the dangers of modifying game files, plus refusal to acknowledge the issue. Note there were 2 full weeks between the blog post and the update mentioning the bounty. I'm pretty sure the massive community outrage in between has played a role in it. But I don't have any sources and I was wrong about at least one thing (lack of attribution), so I'm okay assuming I'm wrong about everything else too.
    - jayd16 4mo ago
      Wowee two full weeks? You mean like a single sprint to discover, verify, and post PR about a perf patch that was good among the sea of rumors and reports a billion dollar game usually gets?
gwbas1c 4mo ago
> The long fix required to use an internal cache to account for these cases That's because the OS does the same thing too. It's the right fix, when I implemented something similar, we implemented caching right away.
Dwedit 4mo ago
Is this actually real? I thought fread just multiplied the two numbers together to compute a total size. Meanwhile, the Win32 API call ReadFile actually does do a separate system call if you call it multiple times.

Taniwha 4mo ago

I used to be a graphics card/chip architect for macs in the early/mid 90s - our chips were the fastest, but some programs were resistant because they did stupid stuff: pagemaker invalidated the font cache every time it went thru its main loop, quark with ATM did an n*2 thing every time it wrote text etc etc. We had special hardware to accelerate text drawing and it did nothing because the software pissed it away. We considered creating a plugin that fixed all these things, it would have been hard to maintain, in the end we travelled around to the people who made these apps and talked them through their problems To be fair excel would erase places white that it wanted to write up to 9 times before it drew any black pixels, we made that very fast! we didn't tell them :-) At the time 24-bit framebuffers were so slow that before we built graphics acceleration hardware people would switch back to 8-bit to get stuff done, making 24-bit/true colour your daily driver was a big step forward.

urbandw311er 4mo ago
This is a horrible and yet not unexpected insight into the internals of Excel
- Taniwha 4mo ago
  To be fair this was Excell 25 years ago, may no longer be true. One of the other bugs (the Quark/ATM one) was also because of the programmers were worried about writing over stuff that hadn't been completely erased, the Quark guys wrote a string with 2 spaces at the end through a box that masked the end of the string, the ATM font renderer saw it couldn't fit the text so it split it in half and tried again so it drew N/2 N/4 N/8 ... strings. It spent all it's time in the 68k's multiply instructions figuring out how wide the strings (and substrings) were, our fancy 24-bit character rendering hardware was an afterthought
  - sidewndr46 4mo ago
    Nowadays Excel is a webapp that reflows the entire DOM 400 times per keystroke!
- bathtub365 4mo ago
  In all of the software you’ve written, are you aware of how many on-screen pixels you’ve overdrawn?
- trelbutate 4mo ago
  > To be fair excel would erase places white that it wanted to write up to 9 times before it drew any black pixels I feel like I'm having a stroke trying to read this, what does it mean??
  - sixeyes 4mo ago
    before writing to some area, it would erase it (clearing with white) up to 9 times
  - NSUserDefaults 4mo ago

somenameforme 4mo ago

Doesn't that break anything relying on the return value? fread gives you the number of objects read as a return. So I think a pretty typical thing would be to fread and then parse that number of characters, and that'd just break?

jcul 4mo ago
I've seen a lot of code that just assumes fread / fwrite succeeded without bothering to check the return value... But in this case if the code was calling fread 65536 times in a loop and getting 64KiB each time it wouldn't be good either! Sounds like the parent comment had to fix this with the internal cache thing to speed up the small freads. I think they meant the easy fix would have been swapping the args in the original / caller code.
- account42 4mo ago
  There are no small freads in the story, whatever implements those freads supposedly split them up into many calls. But that sound more like a problem of that implementation than the fread callers as size == 1 is correct when you are reading a bag of bytes.
  - jcul 4mo ago
    Ah you're right, I misread it.
DonHopkins 4mo ago
The type of programmer who swaps the args to fread tends to be the type of programmer who doesn't bother to check the return value, fortunately. Edit: mort96: So did you check the return value or not?
- account42 4mo ago
  But the args aren't necessarily swapped just because they end up in a slow case in some implementation.

lukan 4mo ago

"I'm guessing developers did this on purpose so that games seemed like they were loading a lot of stuff" I really hope that was not the case and rather think incompetence or to deal with obscure legacy problems, but the gamer in me gets enraged at the thought someone would artificially increase loading times.

Someone 4mo ago

> Which basically expanded back in the day to 65k reads of 1 byte for several MB file. Each fread translated to 65k reads of ReadFile Windows API What software did that that badly? If the code asks for (up to) 65,536 single byte items, why would you split that into 65,536 calls? Also, that change changes behavior. The old call could read anything from zero to 65,536 bytes, the new one only can read zero or 65,536 bytes. (Reading the source of a few implementations, I think most implementations will fill the output buffer with partial objects if the input doesn’t supply an integral number of them, but the return value of fread cannot signal that to the caller)

micampe 4mo ago
A long time ago I worked with someone who read 1 byte at a time from a socket because they insisted data was cached so the kernel was going to batch it magically somehow. It took me days to convince them to measure it.
- quietbritishjim 4mo ago
  That's different: you're talking about the application code, like OP. But I think the parent comment's point is that the issue is in the implementation of fread itself in the standard library. It's perfectly reasonable for an application to pass it 1, 65536 (i.e. one byte, up to 65536 times) and expect it not to issue 65536 separate OS calls.
  - b112 4mo ago
    Is it? I get what you're saying, but asking for 1 byte 65536 times, is indeed different than asking for 65536 bytes, 1 time. There may be reasons, such as when you pull off the end of a buffer, it shifts. And the buffer size is 1 byte. Or 10. Or whatever. No, I'm not saying that's why. I'm simply saying there is a difference between asking for 1 byte or 65k bytes of something. Even dd runs the same under Linux. dd bs=10k count=1 is faster than bs=1 count=10k I remember trying to recover some data from a spinning disk, and trying to slowly creep up on the data. So I wanted 1 byte per, I wanted it to nibble, until it hit whatever the errored part was. If I just grabbed the lot, it'd error out from the whole read.

mort96 4mo ago

Wait, is that wrong? I always call fread as: fread(data, 1, sizeof(buffer), f); with the rationale that I'm interested in reading sizeof(buffer) individual bytes. The buffer size is incidental, not the size of the items I'm trying to read from the file; "read one item whose size is sizeof(buffer)" seems semantically wrong. Is this just the case of Windows having a bad stdlib fread implementation 15 years ago or is my thinking here actually wrong?

chadgpt3 4mo ago
It's not wrong. Guy just wrote a bad implementation of fread and blamed everyone else.
- DarkUranium 4mo ago
  He didn't write it. The C runtime authors did (presumably Microsoft, if it's MSVCRT). He's hooking into ReadFile, a layer below the stdlib. By the time it reaches the hook, it's already split.
projektfu 4mo ago
fread should be buffered, but different values may cause buffering at different rates. Perhaps it didn't generate 65535 calls to ReadFile but it generated 16 or 64.

chadgpt3 4mo ago

Why does your fread to anything other than multiplying the two arguments?

Sesse__ 4mo ago
The idea of having two arguments to fread() is presumably to be able to do something else than all-or-nothing when there's a short read.
- chadgpt3 4mo ago
  Yes, it divides the bytes read by the element size to get the return value. Which is the obvious reason you'd pass an element size of 1: you want to know how many bytes were read.

fsfod 4mo ago

Part of Windows Explorer actually does tons of tiny 4 byte ReadFile calls in to its tracking database like file when you delete a file. If you deleting lots of files this quickly adds up.

pbhjpbhj 4mo ago
Is this why Windows takes so long to delete things?? Presumably those reads aren't done when using del from a console as that always seems a bit faster.
- Asmod4n 4mo ago
  Windows Explorers zip implementation also seams to do 1 byte reads by the speed is has compared to every other zip implementation.
  - jonathanlydall 4mo ago
    It is frustrating how slow .zip (and more recently .7z) support built into Windows Explorer is. This is a great article on why it's so unreasonably slow to modify these archives: https://textslashplain.com/2021/06/02/leaky-abstractions/ https://textslashplain.com/2021/06/02/leaky-abstractions/ But it doesn't seem to explain why it's so much slower at regular extraction.
- jonathanlydall 4mo ago
  Its slowness is also a function of security software or any other file system "filters" (I believe they're called) are installed. For example, I run TortoiseGit which has a caching feature which is supposed to make it faster at showing what to commit. Disabling it increases the number of items I can delete per second in my Windows Explorer from about 1000 to about 3000 while making not making TortoiseGit operations meaningfully slower (that I can tell). This is a Dev Drive [0] on my machine, it would probably be slower on my C: drive which has full Windows Defender real time file scanning. [0]: https://learn.microsoft.com/windows/dev-drive/ https://learn.microsoft.com/windows/dev-drive/

dfox 4mo ago

The most important fix in SP1 for Office 2007 was fixing exactly that in Excel. Doing ridiculous amount of 4 byte reads made it basically unusable on network filesystems.

Xirdus 4mo ago

Reminds me of the "community patch" to GTA Online from a few years ago. The game was plagued by 10+ minute loading times. The situation remained for years and only got worse with time. Some hacker figured out that the game spent 80% of loading time reading the in-game store listing file. The file was tens of megabytes IIRC, and it literally used the Schlemiel the Painter's Algorithm - for each entry, start reading from the beginning byte after byte. The hacker made a tiny patch that made it remember where it found the last entry. This cut the total loading time by 80%, from over 10 minutes to less than 3. Edit: removed incorrect information.

exrook 4mo ago
This is not quite an accurate telling of rockstar's reaction, there were actually receptive to it and paid out $10k for the discovery. Though it's an understandable mistake given rockstar's hostile history with the gta modding scene. See the original post and discussion for the whole story: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times... https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
- Xirdus 4mo ago
  That's not how I remember these events when they were playing out. I distincly remember social media posts warning about the dangers of modifying game files, plus refusal to acknowledge the issue. Note there were 2 full weeks between the blog post and the update mentioning the bounty. I'm pretty sure the massive community outrage in between has played a role in it. But I don't have any sources and I was wrong about at least one thing (lack of attribution), so I'm okay assuming I'm wrong about everything else too.
  - jayd16 4mo ago
    Wowee two full weeks? You mean like a single sprint to discover, verify, and post PR about a perf patch that was good among the sea of rumors and reports a billion dollar game usually gets?

gwbas1c 4mo ago

> The long fix required to use an internal cache to account for these cases That's because the OS does the same thing too. It's the right fix, when I implemented something similar, we implemented caching right away.

Dwedit 4mo ago

Is this actually real? I thought fread just multiplied the two numbers together to compute a total size. Meanwhile, the Win32 API call ReadFile actually does do a separate system call if you call it multiple times.