22 ms·
I cut GTA Online loading times (2021)
- NickRandom 4y agoPrevious submission (as linked in tfa) https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339 (Feb 2021, 699 comments, 3883 points)
- flohofwoe 4y agoClickbait title ;) The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling. IIRC the actual problem was an exceptionally dumb JSON parser combined with parsing a pretty big JSON file (which probably also grew much bigger than the original developer working on that feature anticipated). Edit: the title has been edited in the meantime, originally it was something about strlen() being responsible for the slow loading time.
- ConceptJunkie 4y agoIt's funny though. Several years ago a coworker did some profiling on one of our server software components and found that it was spending up to 10% of the CPU time in strlen()... for very similar reasons.
- sfteus 4y agoHad something similar happen with low-level components in a PHP application I inherited repeatedly calling trim() and/or str_pad() on values that had already been processed. Refactoring those out led to a 10-15% performance boost in our test suites.
- Sesse__ 4y agoI've seen PHP applications use 30% of their CPU in the “realpath cache”, which at the time was a super-slow (O(n²) in this case) home-grown hash table for caching the result of realpath(), which was called all the time because someone decided include_once() needed to run its paths through realpath() in case someone included a file multiple time through symlinks. And I guess they couldn't trust inode numbers or Windows didn't have them or something… I hope it's gone. I really do hope it's gone :-)
- ddtaylor 4y agostrlen() was a major component of the slowdown. Yes, the JSON file was large and strlen() was indirectly caused by sscanf() being used. But it's not accurate to say it's a clickbait title when a bulk of the slowdown was caused by constantly recalculating the length of the C string. The article shows in detail how he created a monkey-patched strlen() that cached the length to avoid recalculating it over and over which was attributed to the speedup. The speedup gains were not caused by reducing the size of the JSON file or doing any exceptionally better JSON parsing. The speedups were a direct result of simply not calling strlen() over and over in an O(N^2) fashion.
- flohofwoe 4y agoAll true, but replacing the strlen() with a caching version is just fixing the symptoms (of calling strlen too often... but the only realistic solution without the ability to recompile the game - so kudos to the author to come up with this hack). I bet the "official" patch by Rockstar simply dropped in a less embarassing JSON parser library :)
- ddtaylor 4y agoThis article was written before Rockstar released a patch and the 70% speedup is from his strlen() caching.
- flohofwoe 4y agoYes I'm aware, that article basically triggered the official patch. Would be interesting to know if the Rockstar patch provided a further speedup though.
- aardvarkr 4y agoThe author provided an update at the bottom. His hack cut the tone from 6m to 1m 30s. The official rockstar patch took it down to 1m 53s. It’s MUCH better but still off the mark by a bit.
- gbraad 4y agoThe title is still not right as the poster is not the 'I' as in the title.
- Joker_vD 4y agoThe JSON parser was not "exceptionally dumb": it just used sscanf(s, "%d", ...) for parsing numbers. But most implementations of C's standard library (yes, including BSD and Linux's) implement sscanf() by converting the source string into a dummy FILE* object (which involves calling strlen() to set the stream's size) and calling fscanf() on it. Apparently most people who use sscanf() are unaware of this "lovely" implementation quirk and I can't really blame them.
- AlexanderTheGr8 4y agoIs C's std impl slow because of calling strlen() or due to converting into a dummy FILE* object or due to calling fscanf?
- unwind 4y agoAll of the above, I guess. It wants to reuse code in fscanf(), but for that it needs a FILE * interface. But to create that, it needs to implement EOF-handling, and to do that it needs to provide a size, so it has to call strlen(). I had no idea. :|
- malka 4y agoDid rockstar ever release a patch for this ?
- MontagFTB 4y agoA user released one first, then rockstar did subsequently: https://www.pcgamer.com/rockstar-thanks-gta-online-player-who-fixed-poor-load-times-official-update-coming/ https://www.pcgamer.com/rockstar-thanks-gta-online-player-wh...
- legobmw99 4y agoYes - https://www.gamesindustry.biz/articles/2021-03-16-rockstar-pays-usd10-000-to-modder-who-fixed-grand-theft-auto-online-loading-times https://www.gamesindustry.biz/articles/2021-03-16-rockstar-p...
- NickRandom 4y agoI'm not sure if you made it to the end of TFA, but yes. And the author got a 10k bug bounty
- Hamuko 4y agoThey took his fix, applied it to the game in an update and paid him a bug bounty of $10,000. "Just got awarded $10k through their H1 in-game bounty as an exception" https://support.rockstargames.com/articles/360061161574/GTAV-Title-Update-1-53-Notes-PS4-Xbox-One-PC https://support.rockstargames.com/articles/360061161574/GTAV...
- yeahSo 4y agoAnecdotally; I still wait 5-6 minutes for online to load, and 3-4 for single player.
- Hamuko 4y agoDigital Foundry measured a 20.2-second boot load time for GTA V single player on the PS5 and 28.9-second boot load time on a PC. Not even the PS4 takes 3-4 minutes. https://www.youtube.com/watch?v=m5QHInJShQw https://www.youtube.com/watch?v=m5QHInJShQw
- blibble 4y agoI tried playing GTA online once and actually gave up due to the loading times (on a top end gaming rig) I can't believe I'm alone, I wonder how much this cost rockstar
- dieulot 4y agoSame. Asking around why players tolerate it, the anecdotal conclusion I’ve got is that what makes these frequent and extremely slow loads tolerable is playing with friends on audio chat.
- zionic 4y agoThe same thing happened to me. GTAO earns billions/year, crazy that "fix the insane loading times" didn't bubble up to their top-priority.
- SketchySeaBeast 4y agoMy exact same experience. As to what it cost, in the old world paradigm I'd say "nothing, we already bought the game", but I guess these days there's probably a decent argument for a cohort of people who are too impatient to wait to get into the game and who pay extravagant sums of real-world money to get in-game items immediately.
- blibble 4y agoyes, GTA online's business model is making you buy in-game currency and paid DLC which is kinda hard if the game takes 30 minutes to load
- hermitdev 4y agoThere's other costs: how many times per day to R* devs & testers have to go through the same loading? How many man-hours wasted? How many compute resources wasted? How many upgrades were justified by "I need better hardware to get my job done in a timely manner"?
- bastardoperator 4y agoNot much, the GTA5 online scene is alive and well. Considering they've sold this game over the course of three generations makes me believe they've made a mega flipton of money off this game.
- roastedpeacock 4y agoAlthough a bit different rather sad seeing Take Two/Rockstar attempt to censor other fan projects through DMCA or lawsuits.
- kevinventullo 4y agoThere’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.
- pinche_gazpacho 4y agonor the incentive
- deleted 4y ago[deleted]
- azemetre 4y agoI’m starting to slowly realize how terrible it is working in environments where people simply aren’t curious. It’s so hard for me as an individual contributor to decouple this way of thinking. Curiosity is how we find better ways of doing things or finding out hidden flows that can be better. Incurious people are a contagion, but I want to believe you can make people curious. It can’t be innate. Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why. It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening). The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.
- TheAceOfHearts 4y agoDon't blame the people, blame the environment. People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them. If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
- 4y ago
- aliswe 4y agoThis is why you need a language (on a framework) with high introspection, with smart insights. Excuse my opinionatedism, but this is what .NET Core gives you. It even streams JSON serialization/deserialization without much memory requirements, iirc.
- 0xFACEFEED 4y ago> This is why you need a language (on a framework) with high introspection, with smart insights. That would be nice if it were true. What happens in reality is you end up paying the performance cost in other areas specifically due to the framework. Frameworks tend to cater to very broad use cases and will generally be less performant than custom code written for one specific use case. Frameworks also tend to be extremely interdependent meaning that to rip out one part (for perf) you need to rip out another huge part. That is why, to this day, if one's business NEEDS something to be performant (no do-overs! no second chances!) then industry veterans will opt for custom systems. Yes they'll have more edge case bugs. Yes they won't be performant in areas you don't care about. But you will still sell 165 million copies of your game because it performs well on a bunch of different platforms with varying specs. This particular case is more of an organizational problem. No one was responsible for load times, it wasn't a priority to improve, and users weren't complaining enough about it to matter.
- outworlder 4y agoNah. We just need to stop this idea that JSON(or similar formats) is an acceptable data storage mechanism. It's fine when interacting with low bandwidth APIs across different systems, where the need for debugging is greater than performance. But on a shipped game? Just bake this JSON into a better, binary serialization format. No need to parse data that seldom changes over and over again. It doesn't matter if you can stream json, if you don't need to call strlen, what have you. Just don't parse unnecessary stuff.
- tialaramex 4y agoThis is GTA Online, so while it's a "shipped game" that JSON changes, Rockstar decides to offer the "Pfister Astron Custom" for $2.5M and they just change the JSON. I don't know whether it's fetched from their servers on startup, or whether it's on disk and just updated periodically, but either way choosing JSON is not as crazy as you seem to think.
- kingcharles 4y agoI used to be a video game developer. I guess it's down to priorities and passing the buck. Our priorities were always on profiling the running game to make sure we got every ounce of juice out of the hardware and the game ran smoothly on the lowest systems. I can't think of one time we would have had the spare time to look at loading times. In those days we were being pushed into 80+ hour weeks just to make the game work at all, never mind finding time for "luxuries" such as this. And of course, on a big team, a problem like this would always be someone else's problem.
- blobbers 4y agoYou are saying a game had a 6 minute load screen and everybody would throw their hands up and say “but look at the dynamic lighting once it loads… it’ll all be worthwhile!” If that’s not group think gone bad, I don’t know what is.
- barrysteve 4y agoThe customer has put up with 6 minute load times, but a low framerate kills sales quickly (unless you're Star Citizen).
- kingcharles 4y agoAnd kills reviews. Most reviewers won't freak about the load time, but the framerate they will.
- blobbers 4y agoAlright, this comment thread has some potential R* devs. (Jokes jokes)
- kingcharles 4y agoI'm not saying there is a R* cabal, because of course there is no cabal, but if I told you what the *back* of this sign meant, I would have to kill you: https://www.reddit.com/r/gaming/comments/3ylmm4/a_staggering_127mb_san_andreas_bridge_facts/cyethcv/ https://www.reddit.com/r/gaming/comments/3ylmm4/a_staggering...
- thatswrong0 4y agoI cannot fathom how dysfunctional Rockstar must be that this slipped through the cracks for so long given the fact that GTA Online is Rockstar’s cash cow. 6 minutes is insanity and must have turned off so many people from playing.. this person deserved far more than $10k
- swarnie 4y agoBecause the user experience isn't important. Enough still play it and buy a few micro tractions per month regardless of the load times Dev time is better spent reskinning a hat or something.
- rvba 4y agoYou are probably being sarcastic, or cynical about current corporate business world where nobody cares. But user experience is important. With faster lpad timea they probably could get more players. More players probably means more money, since there are more chances to convert someone from a player to a paying customer. Perhaps if someone can endure a 6 minute load time they are more likely to buy something in the game (a fan of the series?), but we could say that those who like fast load times also use their wallets fast. But it is pure speculation - we dont have the data. Also obviously Rockstar doesnt have this data - they couldnt A/B test if load time converts to more sales - their game only had bad load times.
- ssharp 4y agoCertainly there has to be some portion of the potential audience who would have become microtransaction purchasers who didn't because they were turned off by the long load times.
- blobbers 4y agoA heart warming story of a hacker turned game dev! Nice job. They offer you a job too?
- Tarragon 4y agoI assume there is an upcoming: Show HN: Passive karma farming to 10k Two submissions an hour, 24 hours a day...
- mdrzn 4y agoHe's just reposting stuff from older years, mostly on the same exact day/month. Should this be allowed? It's not breaking any "rules" but then 10k people could setup a bot to do the same thing and repost stuff twice an hour..
- lostgame 4y agoWow. I wanted to call you out for just being an asshole, but seriously - is this account just a bot? And who cares about karma on HN?
- Tarragon 4y agoBoth things can be true. There's clearly a bot here but ddtaylor has commented on this thread, so there's also a person in there somewhere.
- deleted 4y ago[deleted]
- maccard 4y agoSeconding the other comments here - I'm a developer and I'd have to agree it's about priorities. Every game I've worked on has had load time targets but those targets were set based on what the loading time of the game was at the time the target was introduced, so the goal would be to not make them regress. The other thing to note is that with this particular problem, the issue is parsing a json blob on a live environment. I can count on one hand the number of times I connected a development client to a live environment _ever_, it's just not the done thing. Developers likely have local environments that have a much smaller json blob to parse (and that's likely where the load time benchmarks were run, if they exist).
- debdut 4y agoPlease add RSS to your blog!
- kuroguro 4y agoAdded a feed plugin since people keep asking. Never used RSS so hope it's working correctly.
- holtkam2 4y agoPerhaps my favorite thing I've ever read on HN :) I love to see it crop up again
- dusted 4y agoFirst of all, I find it an impressive skill to be able to do this on a binary blob. Secondly (optimistic version), it really shows that they (management) didn't care, that nobody ever made a ticket to fix loading times, since this would be trivial do find with a normal profiler on a debug build. Secondly (pessimistic version), nobody at rockstar had enough skill and/or professional pride/sense of ownership to find this out themselves.
- secant 4y agoI feel like these problems are all around and often present in games. There's one that particularly infuriates me in Hearthstone, a Blizzard developed TCG, where when a game ends it causes the processor to furiously spin up and causes all kinds of anomalous behaviour like webcam cutting out, mic distortion and other generally laggy behaviour. I'm aware that there's probably a lot going on at the game's end but I'm still utterly convinced there is a coding bug causing this amount of processor spiking but I lack the skills of debugging Unity, which is the game's engine, to determine what actually is going on. I guess that's the issue, the ability of conducting the research the kind in the OP article is extraordinarily rare even among game devs so these problems persist (Hearthstone is just turning 8 years old now) without anyone with the deep knowledge to rectify it.
- victorclf 4y agoI wonder if this code with poor time complexity was written by the same people who are always complaining about leetcode being useless.
- s-xyz 4y agoWow, wow, wow! This was such a nice read. Definitely buying the guy a beer (there is a button at the bottom).
- thewebcount 4y agoI've noticed similar stupid problems playing Portal 2 with a friend. Downloading the maps takes a really long time, but I've got a decent internet connection and so does my friend. It appears that when his machine is loading, my machine has no network activity whatsoever, and then once he's loaded the maps, then my machine will start loading them and his machine has no network activity. The game has a number of other weird issues like this. After you've played a map, it asks you to rate a thumbs up or thumbs down, but the hit areas don't match where the button is drawn, for example. Sometimes the mouse doesn't cause my character to turn. I can quit and restart the game and it's the same. But if I open a different game, quit it and then restart Portal, it works fine. WTF? I realize this is an old game so they aren't updating it (I have to boot into a 32-bit OS to even play it!) but it's a little depressing that such egregious issues went unnoticed for so long that we're now probably past the point of fixing them.
- Laforet 4y agoI had to do a double take when reading your comment before realizing that you were probably referring to the Mac version of Portal 2. If that is the case then it's not unexpected, for OSX routinely break compatibility between major versions unless the developer is able to keep up with the changes. This is not to say things are that things are that much better on the windows side. While most 32bit binary run fine in WoW64, driver issues become more common as the games get older. Another major issue is 3rd party services bundled with the games that are no longer available. I can name more than a dozen of games published around 2010 that are no longer available for purchase because the original build depends on the deprecated Games for Windows Live service and would refuse to launch without it installed. Microsoft abandoned GFWL around 2018 and turned off the online installer at a later date so you would have to download an offline installer from some dodgy non-official website. Thankfully it still works with current versions of windows as long as you still have the physical copy of the game, and the backend of GFWL is partially working with achievements still being synchronized with Xbox Live accounts. But who knows how long that's going to last?
- firefoxd 4y agoAnother take: Dev 1: Hey, I think we can improve the load time by fixing X, Y, and Z. Dev 2: that's neat, create a ticket so we can keep track of that. Product 1: hey, what's this ticket about? Dev 1: oh, that's to speed up the load time. It's a quick fix. Product 1: was that in the requirements? Dev 1: No, but it will be a huge improvement. Product 1: ok, I'll mark it as Tech debt, and put it in the back log. ~~ Many Month Later ~~ Dev 3: hey, did you know we can improve load time by updating X, Y, and Z? Dev 2: Oh yeah, we even have a ticket for that. Let's let Product know it's still an issue and maybe they'll prioritize it. Product 2: Was this part of the requirement? ~~ years later ~~ Product 3: hey, there's this old ticket here. Do you think it's still relevant? Dev 35: I'm not familiar with that, but it's too old, most likely not relevant any more. We can delete it. .... Dev N: Hey, I think we can improve the load time by fixing X, Y, and Z....
- tomschlick 4y agoThat's why you stop asking permission to make quick fixes and just submit the PR. Obviously there is a balance here but if you can't "steal" a few hours in a week from the product driven roadmap to do so then there is something wrong in the engineering culture.
- rrrrrrrrrrrryan 4y agoDo any orgs explicitly ask the engineers themselves decide how to allocate a certain percentage of story points in a given sprint? Product people are great for selecting which features to build out and which bugs to address, but are poorly equipped to decide how to prioritize things like: performance optimizations, bugfixes that haven't (yet) been reported by the end users, code refactoring, paying down tech debt, modernizing tooling, misc quality of life improvements for the engineering team, etc.
- Drew_ 4y agoMy last org did exactly that. The team lead that runs sprint planning was always an engineer. Product management is still there for direction and context, but the decision maker was always an engineer. I don't think this is particularly special way to do it, but I might be wrong.
- Panzer04 4y agoHow can not a single person have hooked up a debugger and just paused it during the 5 minute load time? Especially given the difference between story and online mode? After eight years?
- xyst 4y agoproprietary software combined with a small circle of people that play GTA5 and have the technical ability to disassemble code also: corporate life, original devs rotated out. new team that has been maintaining is just adding features, new micro transactions, or building new heists
- makecheck 4y agoA really cool example of good debugging especially without the source. Also a good reminder that the slow thing may not be what you think and if you aren’t profiling you don’t really know what’s going on. And an even better reminder that typical “metrics” trying to assess developer value can be pretty meaningless. This is the kind of code change that would drastically improve everything but show up as a blip on some of the pathetic measurements out there, e.g. hardly any lines of code or whatever.
- ChoGGi 4y agoCheaters and long load times are the two reasons I stopped playing. I couldn't be bothered to start again after this update came out. Ah well
- aperrien 4y agoIs the source for Luke Stackwalker available anywhere? I found an old Github project, but it's last update was 7 years ago.