7 ms·
I cut GTA Online loading times by 70% (2021)
- mcint 3y agoOther nice thoughts on it I cut GTA Online loading times (2021) (June 9, 2022 — 476 points, 213 comments) https://news.ycombinator.com/item?id=31681515 https://news.ycombinator.com/item?id=31681515 How I cut GTA Online loading times by 70% (February 28, 2021 — 3883 points, 699 comments) https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
- joshsyn 3y ago[dead]
- el_duderino 3y agoDiscussion from a previous submission: https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
- xxs 3y agoThe link to HN was added to the blog post, itself: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/#Small-update https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
- two_handfuls 3y agoAmazing story!
- m-a-r-c-e-l 3y agoRockstar fixed that bug because of this guy... Awesome!
- rightbyte 3y agoOne could argue the bug is in scanf on Windows?
- charcircuit 3y agoNo, the bug is not in Windows. scanf can not cache string length as it can't guarantee x, and x + offset are the same thing, nor can it guarantee the string was unmodified since the last call. Windows provides _snscanf_s if you want to keep track of the string length yourself instead of having it recompute it each time.
- Arnavion 3y agoThe fix would have nothing to do with caching the string length across multiple calls to sscanf. The fix would be to have sscanf not call strlen on the input string in the first place, and instead only process the input string up to the point where it satisfies the format string or the input string terminates. After all, regular scanf works fine without the length of stdin. As TFA also says: >To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this. I would assume it just scanned byte by byte and could stop on a NULL. The author's replacement strlen does the "cache the length across calls" thing only because bolting that on top of the default strlen was easier than doing the lazy parsing thing, since the latter would've required making an actual sscanf implementation to do that from scratch.
- pjc50 3y agoThe C standard library makes no performance guarantees. Besides, scanf is awful and should be avoided wherever possible.
- sumtechguy 3y agoscanf a great way to cause a memory over/underun. Dont think so? Go read the docs for the different CRT's out there. None of them really match on what all the % items mean.
- flohofwoe 3y agoThe problem was that the JSON parsing code was doing exceptionally stupid things in the first place (like calling scanf), which doesn't matter on small JSON files used in testing but exploded in production once that JSON file was growing to several MBytes. Just dropping in a different JSON parser library was probably enough to fix the problem on R*'s side.
- ErneX 3y agoAnd paid him 10k as bounty.
- Panzer04 3y agoOne wonders how many millions (or tens of millions) they lost from people giving up over a 10 minute loading screen...
- ngetchell 3y agoWikipedia mentions revenues for the game being $8.5 billion. It obviously didn't hurt too much!
- jerf 3y agoOn the other hand, if you consider the losses as a percentage based on people who gave up on the margin, you can easily derive losses relative to what they would have gotten without the bug in the tens of millions of dollars, and hundreds of millions is on the table. I can't "prove" that, because the data isn't there, but it is a completely reasonable guess. At an 8 minute load time it is reasonable to guess there were a lot of people who stopped playing and thus stopped spending because they thought their install was crashed or corrupted, and just silently wandered away. I certainly wouldn't. I've got some games that load more slowly than others but I don't think I've waited multiple minutes since the Commodore 64 era.
- 392 3y agoChiming in here as someone who used to play in competitive GTA leagues. I quit playing because of too much time spent in load screens. It was worse than just this slow loading bug: random disconnects that triggered loading again got more and more common over time, too.
- japanman185 3y ago[dead]
- thiago_fm 3y agoHow do I become that good in debugging stuff like this? I'm guessing he works with gamedev and does this sort of debugging and reverse engineering regularly?
- nolok 3y agoLearn to use a performance profiler and then a debugger/disassembler/whatever fit you needs. The profiler tells you where the slowdown happens, the rest then let you figure out what actually happens. By learn to use I mean really do, it's like refactoring tools, it's sort of boring and tiring to learn those, you end skipping large parts because "yeah I already know / it's obvious", then you need one day for an emergency / something right now and feel like you're fighting against it. For those three (and some others), the key to being very good at it is learning all the small tricks and details, when you don't need it, so the day you do it lets you skip 90% of the work. As an overall point, I'm sure many people have various different experiences, but for myself while being in high school I needed to parse "replay" files for some games to extract and process information and allow players to post their replays to get stats on web forums, and it helped me to learn a great deal about how to properly reverse engineer and figure out stuff, things that I have no idea how I would have learned otherwise.
- beezlewax 3y agoExcuse my ignorance... but what refactoring tools do you speak of? I'm accustomed to the basic ones in VS code but I have a feeling I'm missing something greater judging by your comment
- jonathanlydall 3y agoThere are probably multiple refactoring tools which would fit the description by the parent. Personally I have experience with JetBrains ReSharper for (full) Visual Studio, it has loads of functionality which makes it very fast for me to navigate and refactor even very large C# projects. JetBrains has different products for many different languages and I expect they offer similar capabilities in each. Learning all the features at once is impractical, but if you learn a new one every now and again, after a few months or years you become incredibly proficient. I had a similar experience with conversation macros back in the day when I was a GM for WoW. You don’t make too many at once as you’ll never remember them all, instead you make them kind of as you need them, which tends to be situations like “this is the third time today I’m writing this long explanation”.
- deleted 3y ago[deleted]
- aronhegedus 3y agoThis is my favorite reverse engineering article! Feel like I've seen it on ycombinator a few times now
- mywacaday 3y agoRemember reading this way back, glad to see he/she got a 10k reward from R*, great piece of debug work and a bit of inspiration to improve my own skills.
- bezier-curve 3y ago10k seems low considering millions of people wasted millions of hours of their time looking at a loading screen.
- lallysingh 3y agoI agree with you, but let's play devil's advocate. People waiting already paid for the game, so the business value here is hard to justify. And $10k is a decent amount of $ for the labor involved.
- danillonunes 3y agoGTA online also profits from existing consumers by selling in-game money.
- mardifoufs 3y agoGTA online made I think a few billion dollars in microtransactions. It's one of the biggest money maker in gaming history and most of it is in post game purchase revenue.
- bluedino 3y agoLike the story about Steve Jobs wanting the Mac to boot faster "You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day." "Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?" https://www.folklore.org/Saving_Lives.html https://www.folklore.org/Saving_Lives.html
- asadalt 3y agoare loading times any better now? i haven’t played in a while
- kramerger 3y agoGreat article, I am baffled nobody at R* cared to investigate a 6min load time that was there for years. Doesn't sound very AAA-gamedev to me.
- prox 3y agoIf no one carries the responsibility, no one will care. Maybe the team who worked on that moved on and what’s done is done. Not sure if R* has a team that does optimization only?
- Hamuko 3y agoI imagine the focus is on creating new things instead of fixing old things, since that will have more obvious and marketable impacts. The whales might tolerate longer load times over stale content.
- spuz 3y agoI also cannot explain it. If I was a dev on a system that had a 6 minute loading time, I'd be stopping what I was supposed to be doing and debugging it after a couple of days of suffering through it.
- DANmode 3y agoThat's the fun part: their dev systems almost certainly are not console dev kits, or consumer PCs. They're absolutely jacked workstations that probably loaded it in a blazing 2.7 minutes!
- charles_f 3y agoOr even simpler, they have flags to deactivate loading that catalog, so the issue doesn't exist for them
- spuz 3y agoBut if that's true, then there's even less excuse for not fixing the bug because it would imply at least one person knew the slow loads were caused by loading a small 10MB json file.
- kowlo 3y agoIt's a shame that's their only post (https://nee.lv/ https://nee.lv/)
- cedws 3y agoAgreed. I was rather hoping to see them post something about the GTA V source code leak.
- stevage 3y agoThat's insane to achieve that without access to source code. It's also completely baffling that a company would invest such resources building such a complex game and be fine with 6 minute load times, caused by such ... dumbness. Like selling a sportscar with a 5 litre petrol tank.
- DanielHB 3y agothe people maintaining GTA online systems are probably not the same ones who developed the main game (or even the initial GTA online systems) They are probably just adding more content, but the original dev's algorithm for loading that content didn't expect a bazillion new cosmetics
- wwalexander 3y agoYep. The entire game is like this, massive amounts of content clumsily tacked onto something that they clearly didn’t realize would become so popular. This also shows in the interface, where basically every piece of gameplay is triggered by plumbing through the massively overcrowded Quick Menu.
- jvmboi 3y agoMy sneaking suspicion is that the real reason that everything is so slow is that noticeable loading or processing times give the software a feeling of heft. It strokes our ego as programmers because if something takes a while to complete then it must be very complicated and important.
- rowanG077 3y agoInteresting. I have never thought about it like this. To me it always feels like bliss if something finishes instantly,
- deleted 3y ago[deleted]
- ionwake 3y ago
- jpeter 3y agoLoading times for GTA Online are still insanely long on PC
- haezee 3y agoYou must've never played before this was fixed.
- oscarteg 3y agoAre you using an HDD maybe?
- slifin 3y agoYeah I remember this blog post the first time it made the rounds Played recently and thought Rockstar must not have actioned the advice in the blog But no it's just it is still slow but they did fix these issues kudos to them but performance still needs attention People who play this regularly have their phones out/youtube videos on waiting for these load times
- whatyesaid 3y agoI mean GTA Online is a microtranscation/grindfest experience, mostly to milk kids of their money. It's not really worth the time due to how grindy it is. Most PC gamers play multiplayer mods like RageMP that allow connecting to servers with well-made scripted gamemodes like RP, but also stuff like death match where hackers are actually banned I'd guess.
- skywal_l 3y agoFYI, list of tools used: - Luke Stackwalker: For stack sampling (perf analysis) - ???: industry-standard disassembler - Process Dump: Dump process memory to file (for obfuscated code analysis) - x64dbg: For debug stepping. - MinHook: For patching/modding a binary executable.
- whackx 3y ago???: industry-standard disassembler guess that's IDA Pro
- jasondoty 3y agoDefinitely IDA Pro. It is an interesting case in piracy, as being the go-to software for reverse engineers, it also has the biggest target on its back for software crackers. Despite this, newer versions are not always cracked and released publicly so quickly. It seems each license owner gets their own custom watermarked build of the software. The crack of version 7.7 was leaked from Think-Cell, the excel add-ons company. 8.3 was recently seen released, with an anonymous supplier and Vietnamese cracking group behind it.
- kuroguro 3y agoThe funny thing is IDA added an x64 decompiler for the free version right after the post. The piracy could have been avoided if it wasn't for the timing.
- jcmontx 3y agoI remember reading this article when it first came out. It still gives me the same feeling of inadequacy. There are some people walking around us who just belong to a different league.
- akokanka 3y ago[flagged]
- NikolaNovak 3y agoAlways a fascinating read when it pops up on HN :) One thing to note - assumption that one might make is that "Rockstar shipped obviously ridiculously suboptimal code; how did this ever get past testing??" However, based on the author's discussion of the code, and the issue being in inefficient parsing of an online store content, it is likely that this not only passed testing, but was sufficiently efficient in production too... upon go-live, when the store content database was small. This likely grew slowly, and was especially bad for a specific subset of users (CPUs with poor single core performance). They may even have had trouble replicating it in non-production environments depending on their refresh policy and whatnot. This is not to excuse it, of course; but I've seen a repeated pattern where a not-perfectly-optimal algorithm works well enough in development, testing, and initial production... then blows up in production with growth a couple of years later. (In particular, I deal with a lot of complex SQL in my day job, and they have a nasty habit of going exponential with data growth. If you're implementing a greenfield application with no objective knowledge of its future size and growth patterns, it may be littered with trapmines despite reasonably good efforts to not make it so. Especially dangerous with "Big Bang Go-Lives" - though a game may have internal Agile sprints, I'd still consider it overall a bing-bang go-live in that majority of its content will go live at the same time, and then you get to see how your 100s/1000s/10,000s of SQL/code/libraries/etc behave:)
- deleted 3y ago[deleted]
- whatyesaid 3y agoThe loading times were atrocious even at launch though for Xbox 360. It wasn't a case of content piling up, they added a lot but probably a lot of their item catalogue was there from launch. There was also server instability near launch which it may have been thought to be linked, as in we thought we were waiting 10 minutes for a server. But any actual server issue was probably more demand/load scaling based.
- twelvedogs 3y agoyeah it was definitely a case of not wanting to spend a comparatively small amount of money on dev time to even look into it it was obviously not network based as connection speed didn't seem to change the time required
- asylteltine 3y agoI love reading this every time it gets posted. This guy is a genius and rockstar should feel bad.
- vicktorium 3y agoi read this article when it happened some time ago this really makes me think: if this obvious flaw exists ... imagine what other, more complicated and opaque mistakes exists and we're not aware of it ... just food for thought
- 2OEH8eoCRo0 3y agoIs there a safe way to do this and not get your online accounts banned? I'm interested in tinkering in this way but I'm not a cheater and I don't want to be confused for a cheater and banned.
- robblbobbl 3y agoGood job :)