5 ms·

Holy cow, I'm a very casual gamer, I was excited about the game but when it came out I decided I don't want to wait that long and I'll wait until they sort it o

by comboy 6y ago

Holy cow, I'm a very casual gamer, I was excited about the game but when it came out I decided I don't want to wait that long and I'll wait until they sort it out. 2 years later it still sucked. So I abandoned it. But.. this... ?! This is unbelievable. I'm certain that many people left this game because of the waiting time. Then there are man-years wasted (in a way different than desired). Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020. Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.

dan-robertson 6y ago
I don’t think the lesson here is “be careful when parsing json” so much as it’s “stop writing quadratic code.” The json quadratic algorithm was subtle. I think most people’s mental model of sscanf is that it would be linear in the number of bytes it scans, not that it would be linear in the length of the input. With smaller test data this may have been harder to catch. The linear search was also an example of bad quadratic code that works fine for small inputs. Some useful lessons might be: - try to make test more like prod. - actually measure performance and try to improve it - it’s very easy to write accidentally quadratic code and the canonical example is this sort of triangular computation where you do some linear amount of work processing all the finished or remaining items on each item you process. As I read the article, my guess was that it was some terrible synchronisation bug (eg download a bit of data -> hand off to two sub tasks in parallel -> each tries to take out the same lock on something (eg some shared data or worse, a hash bucket but your hash function is really bad so collisions are frequent) -> one process takes a while doing something, the other doesn’t take long but more data can’t be downloaded until it’s done -> the slow process consistently wins the race on some machines -> downloads get blocked and only 1 cpu is being used)
- petters 6y ago
  Is there some reason sscanf can not be implemented without calling strlen?
  - ddulaney 6y ago
    It could be, and the article acknowledges that possibility. For example, a cursory check of the musl sscanf [0] suggests that it does not (though I may have missed something). However, whichever implementation Rockstar used apparently does. [0]: https://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c https://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c
    - JdeBP 6y ago
      Slightly less cursory: https://news.ycombinator.com/item?id=26298300 https://news.ycombinator.com/item?id=26298300
    - pja 6y ago
      A lot of libc implementations seem to implement sscanf() this way: As well as the windows libc ones mentioned above, I checked the OpenBSD & glibc implemtations & they worked the same way.
    - SolarNet 6y ago
      Part of this is that game companies are notorious for re-implementing standard libraries for "performance". I suspect both shitty implementations of sscanf and the not-a-hashmap stem from this.
- Nitramp 6y ago
  - do not implement your own JSON parser (I mean, really?). - if you do write a parser, do not use scanf (which is complex and subtle) for parsing, write a plain loop that dispatches on characters in a switch. But really, don't.
  - dan-robertson 6y ago
mdoms 6y ago
Me and my twice-a-week gaming group enjoyed GTA V but abandoned it years ago simply because of the load times. We have 2x short slots (90-120 minutes) each week to play and don't want to waste them in loading screens. We all would have picked this game back up in a second if the load times were reduced. Although I must say even with the same results as this author, 2 minutes is still too long. But I'll bet that, given the source code, there are other opportunities to improve.
- thw0rted 6y ago
  I wonder if a paid subscription would have fixed this? If you left a paid MMO, they'd probably ask you to fill out an exit survey, and you could say "I'm canceling because load times are terrible", which would (hopefully) raise the priority of reducing load times. But since GTA online is "free", there's not a single exit point where they can ask "why did you stop playing".
  - thatguy0900 6y ago
    Gta has made billions off of its shark card micro transaction system. So the incentives are probably pretty similar for player retention. Granted, the players leaving over load times are probably not the players who are invested enough to spend thousands in micro transactions.
    - thw0rted 6y ago
      That's my point though, you don't get to survey people not buying the microtransaction because they quit due to terrible load times, whereas you could survey people who cancel a subscription. I guess they could still gather data by reading reviews, looking through forums / Reddit / whatever, and tallying up complaints, though.
XCSme 6y ago
> But how could R* not do this? GTAV is so full of great engineering I assume there were different people working on the core game engine and mechanics VS loading. It could as well be some modular system, where someone just implemented the task "load items during online mode loading screen screen".
jiggawatts 6y ago
> Parsing JSON?! Many developers I have spoken to out there in the wild in my role as a consultant have wildly distorted mental models of performance, often many orders of magnitude incorrect. They hear somewhere that "JSON is slow", which it is, but you and I know that it's not this slow. But "slow" can encompass something like 10 orders of magnitude, depending on context. Is it slow relative to a non-validating linear binary format? Yes. Is it minutes slow for a trivial amount of data? No. But in their mind... it is, and there's "nothing" that can be done about it. Speaking of which: A HTTPS REST API call using JSON encoding between two PaaS web servers in Azure is about 3-10ms. A local function call is 3-10ns. In other words, a lightweight REST call is one million times slower than a local function call, yet many people assume that a distributed mesh microservices architecture has only "small overheads"! Nothing could be further from the truth. Similarly, a disk read on a mechanical drive is hundreds of thousands of times slower than local memory, which is a thousand times slower than L1 cache. With ratios like that being involved on a regular basis, it's no wonder that programmers make mistakes like this...
- salawat 6y ago
  Tbe funny thing is, as a long time SDET, I had to give up trying to get people to write or architect in a more "local first" manner. Everyone thinks the network is free... Until it isn't. Every bit move in a computer has a time cost, and yes, it's small... But... When you have processors as fast as what exist today, it seems a sin that we delegate so much functionality out to some other machine across a network boundary when the same work could be done locally. The reason why though? Monetizability and trust. All trivial computation must be done on my services so they can be metered and charged for. We're hamstringing the programs we run for the sole reason that we don't want to make tools. We want to make invoices.
  - gridspy 6y ago
    And like so many things, we're blind to how our economic systems are throwing sand in the gears of our technical ones. I love your point that shipping a library (of code to locally execute) with a good API would outperform an online HTTPS API for almost all tasks.
lumberingjack 6y ago
It gets worse they're brand new game Red Dead online does the same thing as soon as it did it the first time I logged out and charged back

by comboy 6y ago

dan-robertson 6y ago

I don’t think the lesson here is “be careful when parsing json” so much as it’s “stop writing quadratic code.” The json quadratic algorithm was subtle. I think most people’s mental model of sscanf is that it would be linear in the number of bytes it scans, not that it would be linear in the length of the input. With smaller test data this may have been harder to catch. The linear search was also an example of bad quadratic code that works fine for small inputs. Some useful lessons might be: - try to make test more like prod. - actually measure performance and try to improve it - it’s very easy to write accidentally quadratic code and the canonical example is this sort of triangular computation where you do some linear amount of work processing all the finished or remaining items on each item you process. As I read the article, my guess was that it was some terrible synchronisation bug (eg download a bit of data -> hand off to two sub tasks in parallel -> each tries to take out the same lock on something (eg some shared data or worse, a hash bucket but your hash function is really bad so collisions are frequent) -> one process takes a while doing something, the other doesn’t take long but more data can’t be downloaded until it’s done -> the slow process consistently wins the race on some machines -> downloads get blocked and only 1 cpu is being used)

petters 6y ago
Is there some reason sscanf can not be implemented without calling strlen?
- ddulaney 6y ago
  It could be, and the article acknowledges that possibility. For example, a cursory check of the musl sscanf [0] suggests that it does not (though I may have missed something). However, whichever implementation Rockstar used apparently does. [0]: https://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c https://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c
  - JdeBP 6y ago
    Slightly less cursory: https://news.ycombinator.com/item?id=26298300 https://news.ycombinator.com/item?id=26298300
  - pja 6y ago
    A lot of libc implementations seem to implement sscanf() this way: As well as the windows libc ones mentioned above, I checked the OpenBSD & glibc implemtations & they worked the same way.
  - SolarNet 6y ago
    Part of this is that game companies are notorious for re-implementing standard libraries for "performance". I suspect both shitty implementations of sscanf and the not-a-hashmap stem from this.
Nitramp 6y ago
- do not implement your own JSON parser (I mean, really?). - if you do write a parser, do not use scanf (which is complex and subtle) for parsing, write a plain loop that dispatches on characters in a switch. But really, don't.
- dan-robertson 6y ago

mdoms 6y ago

Me and my twice-a-week gaming group enjoyed GTA V but abandoned it years ago simply because of the load times. We have 2x short slots (90-120 minutes) each week to play and don't want to waste them in loading screens. We all would have picked this game back up in a second if the load times were reduced. Although I must say even with the same results as this author, 2 minutes is still too long. But I'll bet that, given the source code, there are other opportunities to improve.

thw0rted 6y ago
I wonder if a paid subscription would have fixed this? If you left a paid MMO, they'd probably ask you to fill out an exit survey, and you could say "I'm canceling because load times are terrible", which would (hopefully) raise the priority of reducing load times. But since GTA online is "free", there's not a single exit point where they can ask "why did you stop playing".
- thatguy0900 6y ago
  Gta has made billions off of its shark card micro transaction system. So the incentives are probably pretty similar for player retention. Granted, the players leaving over load times are probably not the players who are invested enough to spend thousands in micro transactions.
  - thw0rted 6y ago
    That's my point though, you don't get to survey people not buying the microtransaction because they quit due to terrible load times, whereas you could survey people who cancel a subscription. I guess they could still gather data by reading reviews, looking through forums / Reddit / whatever, and tallying up complaints, though.

XCSme 6y ago

> But how could R* not do this? GTAV is so full of great engineering I assume there were different people working on the core game engine and mechanics VS loading. It could as well be some modular system, where someone just implemented the task "load items during online mode loading screen screen".

jiggawatts 6y ago

> Parsing JSON?! Many developers I have spoken to out there in the wild in my role as a consultant have wildly distorted mental models of performance, often many orders of magnitude incorrect. They hear somewhere that "JSON is slow", which it is, but you and I know that it's not this slow. But "slow" can encompass something like 10 orders of magnitude, depending on context. Is it slow relative to a non-validating linear binary format? Yes. Is it minutes slow for a trivial amount of data? No. But in their mind... it is, and there's "nothing" that can be done about it. Speaking of which: A HTTPS REST API call using JSON encoding between two PaaS web servers in Azure is about 3-10ms. A local function call is 3-10ns. In other words, a lightweight REST call is one million times slower than a local function call, yet many people assume that a distributed mesh microservices architecture has only "small overheads"! Nothing could be further from the truth. Similarly, a disk read on a mechanical drive is hundreds of thousands of times slower than local memory, which is a thousand times slower than L1 cache. With ratios like that being involved on a regular basis, it's no wonder that programmers make mistakes like this...

salawat 6y ago
Tbe funny thing is, as a long time SDET, I had to give up trying to get people to write or architect in a more "local first" manner. Everyone thinks the network is free... Until it isn't. Every bit move in a computer has a time cost, and yes, it's small... But... When you have processors as fast as what exist today, it seems a sin that we delegate so much functionality out to some other machine across a network boundary when the same work could be done locally. The reason why though? Monetizability and trust. All trivial computation must be done on my services so they can be metered and charged for. We're hamstringing the programs we run for the sole reason that we don't want to make tools. We want to make invoices.
- gridspy 6y ago
  And like so many things, we're blind to how our economic systems are throwing sand in the gears of our technical ones. I love your point that shipping a library (of code to locally execute) with a good API would outperform an online HTTPS API for almost all tasks.

lumberingjack 6y ago

It gets worse they're brand new game Red Dead online does the same thing as soon as it did it the first time I logged out and charged back