5 ms·

Great article, I am baffled nobody at R* cared to investigate a 6min load time that was there for years. Doesn't sound very AAA-gamedev to me.

prox 3y ago
If no one carries the responsibility, no one will care. Maybe the team who worked on that moved on and what’s done is done. Not sure if R* has a team that does optimization only?
Hamuko 3y ago
I imagine the focus is on creating new things instead of fixing old things, since that will have more obvious and marketable impacts. The whales might tolerate longer load times over stale content.
spuz 3y ago
I also cannot explain it. If I was a dev on a system that had a 6 minute loading time, I'd be stopping what I was supposed to be doing and debugging it after a couple of days of suffering through it.
- DANmode 3y ago
  That's the fun part: their dev systems almost certainly are not console dev kits, or consumer PCs. They're absolutely jacked workstations that probably loaded it in a blazing 2.7 minutes!
  - charles_f 3y ago
    Or even simpler, they have flags to deactivate loading that catalog, so the issue doesn't exist for them
    - spuz 3y ago
      But if that's true, then there's even less excuse for not fixing the bug because it would imply at least one person knew the slow loads were caused by loading a small 10MB json file.
      - Verdex 3y ago
        Maybe, but not necessarily. The devs may even have been begging to fix it, but project management kept prioritizing other items in the backlog. If there's a million fires and half of those crash the game, fixing something like a load time might easily stay under the radar. And that's even before we consider internal political issues. The code could have been owned by a group that was refusing any PRs that would have fixed the issue. Or maybe the code was written by the big boss' nephew and fixing it would have removed his contribution.
        charles_f 3y ago
        Yeah there's definitely a scenario where the guideline was "let's hit the deadline, we'll fix it after"
- charles_f 3y ago
  I started working on a codebase where the git repo is 56gb. Turns out that it's a bunch of binary files that were committed and updated a buncha times more than 2y ago. They're not even in the main branch anymore. Nobody in the main team working on that seem to have cared enough to fix it before. Don't underestimate the power of complacency
  - DistractionRect 3y ago
    It's kinda careless to rewrite history just because a repo is large. Any reason you aren't doing a partial/shallow clone?
    - charles_f 3y ago
      Why though? I understand the point of keeping some history, but what's the point of keeping individual commits from years ago? It's not as if you'd cherry pick anything from back then. And then, you can still keep a full version of the history in another repo if you really have auditing requirements that mandate keeping history forever.
      - DistractionRect 3y ago
        > what's the point of keeping individual commits from years ago? It's not as if you'd cherry pick anything from back then. If you extend the logic, what's the point of keeping any commits from years ago? What's a good cut off date to just start discarding history? Rewriting it? But in the same breath, what's the point of cloning commits from years ago? It's not as if you'd cherry pick anything from back then. And that's my point, the former requires an org decision and active action, and the other is a blobless clone. The nice thing about the blobless clone is you can always change your mind, you can get old blobs if you find you need it. But if you threw out old commits, and you need them later... that's the brakes.
flohofwoe 3y ago
I can exactly imagine how it would happen in a large company: 1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added. 2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game. 3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that". 4. Any specific feedback from the outside probably gets lost in the noise of other requests. 5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)
- johannes1234321 3y ago
  Plus: Most developers probably don't use the full store bundle and skip it, thus don't notice it at all.
- rozab 3y ago
  This isn't some obscure b2b product, this is GTA5. Surely somebody at rockstar had, yknow, played the game? And noticed they were losing hours of their life on a loading screen? Hell, it's so long you could file a complete bug report in that time. My theory is the fix existed internally but was never merged for weird political reasons. Some have guessed it was due to advertising being done on the loading screen. Or it could have been pure dysfunction, like how fixes for all the notepad.exe problems apparently existed at Microsoft for decades, but it took a full rewrite to have working undo.
  - flohofwoe 3y ago
    > This isn't some obscure b2b product, this is GTA5. The 'large-organization-dynamics' are all the same though across all industries. I bet enough people down in the trenches noticed, but nobody pushed enough to get the problem investigated and fixed. There's probably several dozen bug duplicates about that problem buried somewhere in GTA5's JIRA (and indeed probably also with exact instructions how to fix, and maybe even somebody did that work but then didn't push enough to get it integrated).
sonicanatidae 3y ago
Why? Clowns STILL buy GTA by the truckload.
- wruza 3y ago
  Why clowns?
  - sonicanatidae 3y ago
    Because they keep supporting garbage like 6 minute load times with their money. 6 Minutes?!?! Are we using Commodore 1541 FDDs again? Per the article, most of that wait is garbage coding and that's apparently just fine, because....clowns will continue to buy it.
    - mardifoufs 3y ago
      They buy the game for the... game. The game is fun. The loading times are bad but don't affect gameplay once the game starts (for the most part, but they can in heists I guess). Still, they buy the game for the game content not for the loading times
      - sonicanatidae 3y ago
        It's about incentives. Why would a company be incentivized to write better code, when people are willing to pay full price and more for, what at times, at release, is garbage. They won't be. They will push out worse and worse code and it's fine, because it's still being purchased. Have you heard of a game called Fallout 76? or maybe Red Dead Redemption 2? Both released as hot garbage and apparently, that was fine.

prox 3y ago

If no one carries the responsibility, no one will care. Maybe the team who worked on that moved on and what’s done is done. Not sure if R* has a team that does optimization only?

Hamuko 3y ago

I imagine the focus is on creating new things instead of fixing old things, since that will have more obvious and marketable impacts. The whales might tolerate longer load times over stale content.

spuz 3y ago

I also cannot explain it. If I was a dev on a system that had a 6 minute loading time, I'd be stopping what I was supposed to be doing and debugging it after a couple of days of suffering through it.

DANmode 3y ago
That's the fun part: their dev systems almost certainly are not console dev kits, or consumer PCs. They're absolutely jacked workstations that probably loaded it in a blazing 2.7 minutes!
- charles_f 3y ago
  Or even simpler, they have flags to deactivate loading that catalog, so the issue doesn't exist for them
  - spuz 3y ago
    But if that's true, then there's even less excuse for not fixing the bug because it would imply at least one person knew the slow loads were caused by loading a small 10MB json file.
    - Verdex 3y ago
      Maybe, but not necessarily. The devs may even have been begging to fix it, but project management kept prioritizing other items in the backlog. If there's a million fires and half of those crash the game, fixing something like a load time might easily stay under the radar. And that's even before we consider internal political issues. The code could have been owned by a group that was refusing any PRs that would have fixed the issue. Or maybe the code was written by the big boss' nephew and fixing it would have removed his contribution.
      - charles_f 3y ago
        Yeah there's definitely a scenario where the guideline was "let's hit the deadline, we'll fix it after"
charles_f 3y ago
I started working on a codebase where the git repo is 56gb. Turns out that it's a bunch of binary files that were committed and updated a buncha times more than 2y ago. They're not even in the main branch anymore. Nobody in the main team working on that seem to have cared enough to fix it before. Don't underestimate the power of complacency
- DistractionRect 3y ago
  It's kinda careless to rewrite history just because a repo is large. Any reason you aren't doing a partial/shallow clone?
  - charles_f 3y ago
    Why though? I understand the point of keeping some history, but what's the point of keeping individual commits from years ago? It's not as if you'd cherry pick anything from back then. And then, you can still keep a full version of the history in another repo if you really have auditing requirements that mandate keeping history forever.
    - DistractionRect 3y ago
      > what's the point of keeping individual commits from years ago? It's not as if you'd cherry pick anything from back then. If you extend the logic, what's the point of keeping any commits from years ago? What's a good cut off date to just start discarding history? Rewriting it? But in the same breath, what's the point of cloning commits from years ago? It's not as if you'd cherry pick anything from back then. And that's my point, the former requires an org decision and active action, and the other is a blobless clone. The nice thing about the blobless clone is you can always change your mind, you can get old blobs if you find you need it. But if you threw out old commits, and you need them later... that's the brakes.

flohofwoe 3y ago

I can exactly imagine how it would happen in a large company: 1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added. 2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game. 3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that". 4. Any specific feedback from the outside probably gets lost in the noise of other requests. 5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)

johannes1234321 3y ago
Plus: Most developers probably don't use the full store bundle and skip it, thus don't notice it at all.
rozab 3y ago
This isn't some obscure b2b product, this is GTA5. Surely somebody at rockstar had, yknow, played the game? And noticed they were losing hours of their life on a loading screen? Hell, it's so long you could file a complete bug report in that time. My theory is the fix existed internally but was never merged for weird political reasons. Some have guessed it was due to advertising being done on the loading screen. Or it could have been pure dysfunction, like how fixes for all the notepad.exe problems apparently existed at Microsoft for decades, but it took a full rewrite to have working undo.
- flohofwoe 3y ago
  > This isn't some obscure b2b product, this is GTA5. The 'large-organization-dynamics' are all the same though across all industries. I bet enough people down in the trenches noticed, but nobody pushed enough to get the problem investigated and fixed. There's probably several dozen bug duplicates about that problem buried somewhere in GTA5's JIRA (and indeed probably also with exact instructions how to fix, and maybe even somebody did that work but then didn't push enough to get it integrated).

sonicanatidae 3y ago

Why? Clowns STILL buy GTA by the truckload.

wruza 3y ago
Why clowns?
- sonicanatidae 3y ago
  Because they keep supporting garbage like 6 minute load times with their money. 6 Minutes?!?! Are we using Commodore 1541 FDDs again? Per the article, most of that wait is garbage coding and that's apparently just fine, because....clowns will continue to buy it.
  - mardifoufs 3y ago
    They buy the game for the... game. The game is fun. The loading times are bad but don't affect gameplay once the game starts (for the most part, but they can in heists I guess). Still, they buy the game for the game content not for the loading times
    - sonicanatidae 3y ago
      It's about incentives. Why would a company be incentivized to write better code, when people are willing to pay full price and more for, what at times, at release, is garbage. They won't be. They will push out worse and worse code and it's fine, because it's still being purchased. Have you heard of a game called Fallout 76? or maybe Red Dead Redemption 2? Both released as hot garbage and apparently, that was fine.