27 ms·

It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something

It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple. I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context. This online gamemode alone made $1 billion in 2017 alone. Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler. Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures. The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top. (And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3 https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)

anoncake 6y ago
I find it absolutely believable that a for-profit company does not prioritize fixing a game that is already a cash cow anyway.
xyst 6y ago
is it really unbelievable? companies this big tend to prioritize hiring a shit ton of middlemen (VPs, project managers, developer managers, offshore managers) in order to avoid paying out for talent to build and constantly maintain the project. I guess paying a shit ton of money to 1 person to manage 10+ poorly paid contractors works out for them, accounting wise. If one really examined the accounting for GTAO, I would bet that most of the billions of dollars that were earned in micro transactions went to marketing, product research, and to middle management in the form of bonuses.
- spuz 6y ago
  Even if you view this as a business decision rather than a technical one, any smart project manager would realise a 6 minute loading time literally costs the company millions per year in lost revenue. (How many times have you felt like firing up GTA Online only to reconsider due to the agonising load time). I would guess this was simply a case of business folk failing to understand that such a technical issue could be so easily solved plus developers never being allowed the opportunity to understand and fix the issue in their spare time.
  - alien_ 6y ago
    The people who observe the slow loading time already paid for the game, so I guess R* won't lose much revenue because of this nasty bug.
    - bcrosby95 6y ago
      The game has microtransactions. Coincidentally, also a large reason why load times were so slow.
      - deleted 6y ago
        [deleted]
    - delecti 6y ago
      The online mode has microtransactions. People not playing anymore aren't paying for microtransactions either.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
    - AuryGlenz 6y ago
      I didn’t buy the game to play with my friends because I heard of how terrible the loading situation was.
- IshKebab 6y ago
  It's kind of hard to believe. GTA5's online mode is their cash cow, and 6 minute load times are common?! It's kind of amazing people even play it with those load times. It's such a simple problem that one dev could have found and fixed it within a day.
  - toast0 6y ago
    It's not at all hard to believe if you've been playing video games for a while. Everything is getting slower and slower, and nobody cares. When I played the Atari 2600, I had to wait for the TV to warm up, but otherwise there were no games with anything approaching load times (with 128 bytes of RAM in the console, who would know). The NES didn't have much in the way of load times either, but you did have to fiddle with the cartridge slot. SNES and Genesis didn't usually load (Air Buster being a noticeable exception). CD based systems sure did like to load, but that's somewhat understandable. In the mean time, more and longer boot screens. The Switch uses a cartridge (or system flash/SD cards), but it likes to load forever too. PC Gaming has had loading for longer, but it's been getting longer and longer. Some arcade games have lengthy loading sequences, but only when you turn them on while they read their entire storage into RAM so they can be fast for the rest of the time they're on (which in arcades is usually all day).
    - ubercow13 6y ago
      Shorter loading times were one of the main selling points of this console generation.
    - yoz-y 6y ago
      > Everything is getting slower and slower, and nobody cares. It really depends. The latest crop of games I’ve played (Doom Eternal, Cyberpunk) loads way faster than games from a few years back (aforementioned GTA-V, Shadow Warrior 2…). This is also on the same machine, so it’s not the hardware that makes it faster.
      - hnick 6y ago
        Doom Eternal's load times were so good I didn't even bother moving it to my SSD (I junction to a larger HDD by default).
- xyzelement 6y ago
  I am always amused by comments like this. You have no idea what development practices they follow (neither do I) but it's hilarious to read your tone. GTA has achieved tremendous success both as an entertaining game and as a business. It's enjoyed by millions of people and generates billions in revenue. As per this article, it has startup problems (which don't seem to actually really hurt the overall product but I agree sound annoying) but the bigger picture is: it's a huge success. So - Rockstar has nailed it. What exactly is your platform for analyzing/criticizing their processes or even having a shot of understanding what they are? What have you build that anyone uses? (not saying you haven't, but.. have you been involved with anything remotely close to that scale?) And if not, whence he high horse?
  - brokencode 6y ago
    Successful people and businesses can be wrong. You are not making a case for why those development practices are okay, but are simply appealing to authority. I and most other customers would argue that 6 minute loading times are atrocious, and if there is an easy fix like this, it makes me lose a lot of respect for the developer who doesn’t fix it. It maybe would even make me avoid them in the future. A reputation is built over years, but can be lost pretty much instantly. Companies have to continue serving their customers to enjoy ongoing success.
    - hombre_fatal 6y ago
      They didn’t make an appeal to authority but an appeal to commercial success, and they’re right on. The fact that GTAO is so popular should make most HNers rethink what they know about the commercial necessity of optimization vs building a compelling product.
      - incrudible 6y ago
        The loading times were not initially that long and then the slow CPU makes a big difference. This really only goes to show how much you can get away with if you have an outstandingly popular product that has no direct competition. Chances are that your product is not that compelling, if it performs poorly, that will hurt adoption. It will never become outstandingly popular in the first place.
masklinn 6y ago
> The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top. For what it's worth, 10MB of JSON is not much. Duplicating the example entry from the article 63000 times (replacing `key` by a uuid4 for unicity) yields 11.5MB JSON. Deserialising that JSON then inserting each entry in a dict (indexed by key) takes 450ms in Python. But as Bruce Dawson oft notes, quadratic behaviour is the sweet spot because it's "fast enough to go into production, and slow enough to fall over once it gets there". Here odds are there were only dozens or hundreds of items during dev so nobody noticed it would become slow as balls beyond a few thousand items. Plus load times are usually the one thing you start ignoring early on, just start the session, go take a coffee or a piss, and by the time you're back it's loaded. Especially after QA has notified of slow load times half a dozen times, the devs (with fast machines and possibly smaller development dataset) go "works fine", and QA just gives up.
- hanniabu 6y ago
  That QA bit is too true, "it works for me!" shrug.
- ldng 6y ago
  But is quadratic the real issue ? Isn't that a developer answer ? The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases. And what is small? Medium? Large? The truth is that there is no one size fits all and assumptions need to be reviewed periodically and adapted accordingly. And they never are... Ask a DBA.
  - gridspy 6y ago
    quadratic is a fancy way of saying "this code is super fast with no data, super slow once you have a decent amount" The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list. Why quadruple? Because you end up scanning a list which is twice as long. You have to scan that list twice as many times. 2x2 = 4. The larger list no longer fits in the fast (cache) memory, among other issues. The cache issue alone can add another 10x (or more!) penalty.
0xy 6y ago
Something I've noticed in highly successful companies is that problems never get fixed because the sound of the money printer from the core business is deafening. Our customer portal loads 2 versions of React, Angular, Knockout and jQuery on the same page? Doesn't matter, it's printing billions of dollars. Rockstar's money printer is so loud that they don't care about problems. Same thing for Valve, their money printer is so loud that they barely bother to make games anymore and let the Steam client languish for years (how did they let Discord/Twitch happen?).
- LukvonStrom 6y ago
  Look at sharepoint. It's a total nightmare of a platform to develop on, but the people just adapted and built their businesses on it
  - ip26 6y ago
    I suspect sharepoint is the platform version of excel & VBA. You & I might hate it, but it gets powerful capabilities into the hands of regular people. sharepoint is probably many non-engineer's very first exposure to actual version control with checkouts, checkins, version history, and merge.
    - ant6n 6y ago
      Sharepoint has version control? I thought it´s mostly a poorly functioning Dropbox copy.
      - easton 6y ago
        This comment is absolutely hilarious to me, because the main selling point for SharePoint (for the longest time) was that you could version control Microsoft Office documents. Then intranets happened, someone wanted to build a Turing machine with it, and now it's the monstrosity we know now.
  - passivate 6y ago
    What makes it a nightmare? Can you share a few issues you've run into?
tamrix 6y ago
Well this sprint we didn't release any new features but we reduced the load.... Dammit hackernews!
kuroguro 6y ago
Small nitpick: I believe these are items/prices for the in-game currency, not micro-transactions. You can buy in-game currency for real world money tho: https://gta.fandom.com/wiki/Cash_Cards https://gta.fandom.com/wiki/Cash_Cards Not 100% sure, never bought anything.
- account42 6y ago
  So what you are saying is they are in fact microtransactions.
Thaxll 6y ago
It was probably fast 10years ago when the store had couple of items, the dev back then never thought that it would grow to 60k items. Classic programming right there. As for profiling, Windows Performance Toolkit is the best available no?
- trulyme 6y ago
  Meh. It's ok to assume low number of items and code accordingly. What is not ok is for the company to ignore such a problem for years, instead if detecting and fixing it.
gameswithgo 6y ago
What is complicated about it is that an online modern 3d game is huge, and there are 95,000 places where a dumb mistake could hurt performance a lot for some customers. You catch 94,999 of them and then "unforgiveable"
- Plutoberth 6y ago
  If it was that way for a few months and then fixed... still pretty shoddy but sure. However, it has been that way for YEARS, and is one of the most common complaints among players. I wonder how much of the remaining loading time could actually be shaved off if someone with the source code took a crack at it.
moonchild 6y ago
> salty because their parent company unjustly DMCA'd re3 Unjustly, but legally. The people you should be salty at are the lawmakers.
- breakingcups 6y ago
  That still remains to be seen. A DMCA is not a court order. Anyone can file one and take a repository offline for two weeks.
  - moonchild 6y ago
    Yes but the code was clearly derived directly from a decompiled binary; not ‘clean room’ reverse engineering. Hence, illegal, regardless of whether a dmca takedown notice is filed.
    - xyzzy_plugh 6y ago
      That doesn't make it illegal, there are plenty of jurisdictions where non-clean-room reverse engineering is perfectly legal.
      - charcircuit 6y ago
        But distributing it without a license from the copyright holder is.
        jagger27 6y ago
        Distributing what, exactly? It required you already owned a copy of the game to install/build it.
        charcircuit 6y ago
        Distributing whatever was in the repo. Requiring a copy of the game doesn't magically make it legal. A modded game is a derivative work.
formerly_proven 6y ago
Also: Rockstar being too cheap to implement anti-cheat on the by far most successful online shooter on the planet. Also > I don’t think there’s any easier way out. lmfao
- buzzerbetrayed 6y ago
  This may be obvious, but is GTAV the most successful online shooter on the planet? (Never played it)
  - sickofparadox 6y ago
    Its the second best selling game of all time[1], and because Minecraft doesn't have guns, I suppose that would qualify it as the most successful online shooter (even if I think another genre would be more applicable). [1]https://en.wikipedia.org/wiki/List_of_best-selling_video_games https://en.wikipedia.org/wiki/List_of_best-selling_video_gam...
    - gambiting 6y ago
      Well but that's a bit like saying that since there are technically races in GTA it's also the most successful racing game. It will also be the most successful flying game and the most successful sailing simulator....etc etc. I don't know, even though there is shooting in GTA I don't think I'd call it a shooter.
      - sickofparadox 6y ago
        I agree that there is probably some other genre that would describe it better, but I think it wouldn't be unfair by any means to describe GTA as a third-person shooter.
    - charcircuit 6y ago
      That list is incomplete. CSGO is the most played game on Steam and is now free to play. I would not be surprised in the slightest if more people have CSGO.
AnonsLadder 6y ago
Does anyone have a link to a copy of re3? Iirc, there was a gitrepo that kept a copy of all DMCA'd repos
- djmips 6y ago
  try the hacker news search (bottom of the page) and you'll find stories on the takedown where there are links to backups posted in the comments.
nikanj 6y ago
The old maxim of "Premature optimization is the root of all evil" has over time evolved to "If you care one iota about performance, you are not a good programmer".
- deleted 6y ago
  [deleted]
- vendiddy 6y ago
  And fwiw, the full quote is: We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
  - blowski 6y ago
    Also, it’s not “never optimise”. It’s “only optimise once you’ve identified a bottleneck”. I guess in a profit-making business you only care about bottlenecks that are costing money. Perhaps this one isn’t costing money.
    - skzo 6y ago
      ding ding ding ding
    - saagarjha 6y ago
      This one isn’t measured to be costing money. Trying to figure out how much money you’re losing as a result of performance problems is extremely difficult to do.
    - CogitoCogito 6y ago
      Perhaps this one is costing them a lot of money.
    - nmfisher 6y ago
      Precisely. Hasn't GTA Online done over a billion in revenue? Given how incredibly successful it's been, it's conceivable the suits decided the opportunity cost of investing man-hours to fix the issue was too high, and that effort would be better spent elsewhere.
imglorp 6y ago
> This online gamemode alone made $1 billion in 2017 alone. There's the answer right there. They figure it's making $1B/yr, leave it alone. Maintenance? That cuts into the billion. Everyone moved onto the next project.
- tyingq 6y ago
  Or they fix it, see that their "in game time" average drops, and then back it out...
  - themeiguoren 6y ago
    I would not at all be surprised if the long load time made for a sunk cost that kept people playing for longer sessions rather than picking it up for less than half an hour at a time.
    - iknowstuff 6y ago
      I don't know. In GTA Online you encounter loading every 15min.
    - simias 6y ago
      I enjoyed GTA online but haven't touched it in well over a year, and the insane loading times are definitely a big reason why. For those who haven't played the game it's important to emphasize that it's usually in the 5minute range, and even then you'll regularly end up with connectivity issues or other problems that will kick you out of the lobby for yet an other ~5 minute load. When I played it wasn't uncommon to spend 30 minutes mostly looking at the loading screen while you were trying to set up a play session with a couple of friends. If you're an adult with limited playtime it's just a complete dealbreaker. You can't just decide to have a quick 20minute play session if you know that you'll have to spend at least half of it looking at loading screens.
    - antihero 6y ago
      This is actually very possible - if they allowed people to dip in and dip out, they will. I remember when CP2077 came out on GeForce NOW and the wait times were 30+mins. I'd play a 6 hour session until I was booted off simply to make it worth the wait.
neatze 6y ago
I am not saying it is the case, nor understand details of solution in depth to comment on it, but in analogy, this reads to me, like an yelling at person who figure out how to solve rubix cube puzzle steps, because once steps are known solution is simple.
- burlesona 6y ago
  No, other people have pointed this out, this should have been very easy to recognize as inefficient in the source code. More likely the code was written hastily and only tested against very small inputs, and then nobody ever actually tried to improve the famously long load times.
  - wnoise 6y ago
    The sscanf issue was not obvious: it looks linear. And should be, on a better sscanf implementation. The duplicate checking on the other hand is a classic "accidentally quadratic" case that is obvious.
oh_sigh 6y ago
Maybe long load times are advantageous? Because it creates longer user sessions on average? If you devote 10 minutes to loading the game you will probably want to play for at least 30 minutes.
- erik_seaberg 6y ago
  Wouldn't the most impatient customers be more likely to pay for items, rather than earn them in game?
zionic 6y ago
I stopped playing GTAV online a few years back because of the crazy load times, not only that but you have to go through 6+ minute load screens multiple times in many sessions. This oversight has cost them millions of dollars easy.
- wyaeld 6y ago
  If it made over $1b in a year previously, and had such insane load times, its very plausible this bad coding has cost them north of another $1b. Probably ranks pretty highly up there in terms of damage to company financials, due to a lack of care.
umanwizard 6y ago
“worth their salt” is doing a lot of work here. No true Scotsman fallacy? I think you might be surprised by how few programmers even know what a profiler is, let alone how to run one.
- thrwyoilarticle 6y ago
  That seems like a misapplication of the fallacy. If we assume 'worth their salt' is a synonym for 'good', then saying any good developer can operate a profiler is entirely reasonable.
thrwyoilarticle 6y ago
The popular view is that companies who write software know how to prioritise, so if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile. I disagree. If there are no internal incentives for the people who know how to fix this to fix it, or if there's no path from them thinking fixing it could improve revenues to being assigned the ticket, things like this won't get fixed. I can fully believe the load times will result in fewer users and lower expenditure. I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.
- saghm 6y ago
  > I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. The messenger website has been atrocious for me lately. On my high-powered desktop, it often lags a bit, and on my fairly high-end laptop, it's virtually unusable. I thought it must be something I changed in my setup, but it's oddly comforting to hear that I'm not the only one with such issues.
  - baby 6y ago
    For me it’s google maps, it has gotten so freaking slow both on mobile and on desktop. Actually google docs and sheet are the same.
    - amluto 6y ago
      Gmail. Typing in gmail is painfully laggy.
  - ianlevesque 6y ago
    Try https://www.messenger.com/desktop https://www.messenger.com/desktop
    - lethologica 6y ago
Scaevolus 6y ago
I suspect that the core engine programmers moved onto other projects long ago, leaving GTA:O running with mostly artists and scenario designers to produce more DLC. This bug wouldn't present in the first couple years with the limited amount of DLC, so by the time it got ridiculous there wasn't anyone left with the confidence to profile the game and optimize it. A junior dev could fix this, but would probably assume that slow loads are a deep complex engine problem that they won't be able to fix. Alternatively, management would declare that there's too much risk doing more technical engine work, and not sign off on any proposed "minor" optimizations because it's too risky.
- jorams 6y ago
  > This bug wouldn't present in the first couple years with the limited amount of DLC GTA Online loading times have been infamous for a very long time. They were already unjustifiably bad when they released the game for PC, and at that point engine programmers would surely be involved.
  - Aeolun 6y ago
    The 1.5 minute default load time is also ridiculous.
- anotherfish 6y ago
  This is very much the likely scenario. The money is in further DLC. The existing GTAO engine is "done" from their perspective. I'd guess also that the next version of the base engine is in RDR2 or later and doesn't have these issues. But at the same time they likely wouldn't backport the changes for fear of cost overruns.
deleted 6y ago
[deleted]
lucideer 6y ago
> It is absolutely unbelievable [...] that a cash cow [...] has a problem like this Likely it wasn't fixed precisely because it's such a cash cow. "It's making money, don't fuck with it".
uhhhhhhhhhhhhhh 6y ago
Can you quantify the additional profit they would have made if this were fixed N years ago?
grishka 6y ago
> the reverse engineered version of GTA III and Vice City Ohhh. Thank you for telling me about this. I just found a mirror and successfully built it for macOS. Runs so much better than the wine version. But I guess I'll never finish that RC helicopter mission anyway lol
crazygringo 6y ago
I imagine the conversation between the programmer(s) and management went exactly like this: Management: So, what can we do about the loading times? Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all. Management: OK, guess that's that then. Sucks but nothing we can do. Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again." And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers. I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here. But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.
- CountHackulus 6y ago
  I can pretty much guarantee that there was no discussion with management like that. From experience, live ops games are essentially a perpetually broken code base that was rushed into production, then a breakneck release schedule for new features and monetization. I've personally had this conversation a few times: Programmer: Loading times are really slow, I want to look into it next sprint. Management: Feature X is higher priority, put it in the backlog and we'll get to it.
  - sillysaurusx 6y ago
    Former gamedev here. I can vouch for this conversation. The idea that management would ask about slow loading times doesn't seem to be too realistic in my experience.
  - 6y ago
db48x 6y ago
In my experience most engineers have never used a profiler even once. They write the code, and if you're lucky they get it working correctly.
- mikevm 6y ago
  Let's call them "code technicians" instead of engineers, ok? (that's a euphemism for "code monkeys")
Ansil849 6y ago
> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple. It is both believable and - by virtue of the fact that, as you said, the series continues to be a cash cow - is apparently forgivable. Here's the thing: the company has zero reasons to fix this, or other ostensibly egregious affronts like DRM, because gamers keep buying the product. There is literally no economic incentive to 'fix' it.
- _AzMoo 6y ago
  How many players have they lost to excessive load times?
  - astrange 6y ago
    I've lost interest in a lot of online games because the clients always do their mandatory updates on launch, which is exactly the time you want to play the game. (Same thing for websites - they show you all the annoying popup signup sheets/survey questions the instant you load the page.)
    - _AzMoo 6y ago
      For sure. I fit my gaming into about 2 hours a week. If it takes 6 minutes to load there's no way I'm going to play it. I know I'm not the target market but I'm still a missed opportunity.
  - Ansil849 6y ago
    > How many players have they lost to excessive load times? Judging by the number of successful sequels the franchise has spawned, the answer is 'an insignificant number'. The fact of the matter, and the point my comment was trying to make, is that the overwhelming majority of players do not sufficiently care about load times or other complaints to deter them from paying for the game. That is the reality of the situation.
    - 6y ago
sillysaurusx 6y ago
Attitudes like yours are why gamedevs keep to themselves. "Unbelievable" and "unforgivable" eh? It's a greedy attitude. Instead of viewing GTA5 as a success that's brought a lot of people happiness, you view it as a money cow designed to extract every last bit of profit – and time, since this bug caused 70% longer loading times. Perhaps it's both. But you, sitting here behind a keyboard with (correct me if I'm wrong) no gamedev experience, have no idea what it's like on a triple-A gamedev team with various priorities. The fact that the game works at all is a minor miracle, given the sheer complexity of the entire codebase. The fact that someone was able to optimize the obfuscated executable is a wonderful thing. But they weren't a part of the team that shipped GTA 5. If they were, they certainly wouldn't have been able to spend their time on this.
- ryandrake 6y ago
  This kind of excuse making is one of the reasons I got out of software development. It’s not just gamedev. Priorities are way out of wack when you have time to put in binary obfuscation, but no time to fix such a huge performance bottleneck. The idea that “it’s a miracle software works at all” demonstrates the chronic prioritization and project management competence problem in the industry. It’s ok to recognize a thing as a business success but a technical failure. In fact many software projects are business successes despite awful and unforgivable quality compromises. You don’t get to whitewash it just because the thing prints money.
  - systemvoltage 6y ago
    How do we then address chronic incompetence? Never complain about it? This is not small. This kind of incompetency if employed in a different sector such as security would lead to losing personal data of millions. > “it’s a miracle software works at all” This is not the case here. Please re-evaluate your calibration on this topic.
    - sillysaurusx 6y ago
      1. you replied to the wrong person. 2. this kind of incompetence exists in all other sectors. That's why pentests are so crucial, and why they guard the security of millions. 3. we'll have to agree to disagree that it's a minor miracle. Having seen the complexity firsthand, it's quite amazing.
hibikir 6y ago
It's not just believable, but it's normal. I have spent quite a bit of my career maintaining software, and I don't recall one employers where low hanging fruit like this wasn't available everywhere. The problem is not that developers can't optimize things: you will find some developers capable of figuring this problem out anywhere. What makes this low hanging fruit so popular is the fact that we aren't measuring enough, and even when we do, we aren't necessarily prioritizing looking into things that are suspiciously slow. In the case of this example, the issue is also client-side, so it's not as if it's costing CPU time to Rockstar, so it's unlikely you'll have someone who can claim their job description includes wondering if the load times are worth optimizing. When problems like this one get solved is because someone who is very annoyed by the problem and either convinces a developer to even look into the problem. Most of the time, the people that suffer, the people that get to decide how to allocate the time, and the people that are in a position to evaluate the root cause of the problem never even get to talk to each other. It's the price we often pay for specialization and organizations with poor communication. Organizations where the people deciding what has to be done next, and where the company culture dictates that the way forward is to either complete more tickets faster, or find ways to be liked by your manager, are not going to be fostering the kind of thinking that solves a problem like this one, but that's a lot of what you find in many places. A developer with a full plate that is just working on the next feature isn't going to spend their time wondering about load times. But instead we end up blaming the developers themselves, instead of the culture that they swim in.
- smaudet 6y ago
  Hear hear, we should make a punch a dummy manager/BA/Code standards 'lead' day... This code looks like someone with almost no experience hacked it together but because they were an intern and likely Rockstar is a toxic place to work, it never gets prioritized to be fixed. I think if managers prioritized cycle time, metri s more, they'd find that they are encouraging a lot of practices which lead to horrible efficiencies - "measure twice cut once" is a positive mantra which leads to more solid designs with less bugs. Agile sort of addressed this problem but unfortunately only at small size scales. Iteration and story capacity got overprioritized over quality, customer engagement, and self-leading teams. Plus things such as scaled agile suffer from the oxymoron of planning fast iteration - if you have a master plan you lose the ability to respond to change and iterate, or you allow iteration and you must accept if any team iterates the whole team must discard the plan...which at some point means you either accept high cycle times or you figure out a way to decouple functionality to the extent the planning becomes the standard fallacy of waterfall - wasting meeting time to go over a plan that isn't based on anything.
karmasimida 6y ago
It could be that at the time GTA online first publishes, the list as hashmap isn't too much of an issue, due to limited catalog, but get progressively worse as the inventory grows. Ofc this is just a hypothesis, but I see the hesitation to change legacy code if it ain't broken as a wide spread mentality.
- sellyme 6y ago
  > I see the hesitation to change legacy code if it ain't broken as a wide spread mentality. Load times measured in double digit minutes on a significant number of machines meets absolutely every reasonable definition of "broken".
omgJustTest 6y ago
Focus is on the money, not on the technicals, on the production side. It is a game, to entertain and ultimately “waste time” on the consumer side. Also on the topic of parsing, +1-11gbyte/sec is possible if you go binary. Point is: this isn’t a technical problem, it’s a choice.
matheusmoreira 6y ago
I used to play this game a lot on PS4. I actually dropped it due to the ridiculous loading times... I still assumed it was doing something useful though. I can't believe they wasted so much of my time and electricity because of this. Even cheaply-made mobile games don't have bugs like this. > their parent company unjustly DMCA'd re3 Wow, this is EA games level scumbaggery... I don't think I'm gonna buy games from them again.
Scoundreller 6y ago
> obfuscated executable loaded with anti-cheat measures I'm impressed that gamecopyworld.com is still online, updated, and has the same UI that it did in 2003
- lethologica 6y ago
  Whoa, what a (refreshing) blast from the past.
pilif 6y ago
> This online gamemode alone made $1 billion in 2017 alone. which of course goes to show that at least from a business side, this issue is completely inconsequential and all resources should be used to push for more monetization (and thus adding to the problem by adding more items to the JSON file) rather than fixing this issue, because, clearly, people don't seem to mind 6 minutes loading time. I'm being snarky here, yes, but honestly: once you make $1 billion per year with that issue present, do you really think this issue matters at all in reality? Do you think they could make $1+n billion a year with this fixed?
- hderms 6y ago
  The bigger the scale the bigger a few percentage point improvement would be worth. I would generally think if you're at 1bn in revenue you should devote 1%+ percentage points of your workforce towards finding low hanging fruit like this. If 1% of employees deployed to find issues that, when fixed, yield 2% improvement in revenue thats likely a winning scenario
- imtringued 6y ago
  This is losing them money. If they fixed the issue they absolutely would get $1+n billion instead of just $1 billion and that n alone is big enough to pay multiple years worth of 6 digit salaries just to fix this single bug.
serf 6y ago
>It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple. Having played the game, it's not surprising to me in the least. I have never yet encountered another such 'wild-west' online experience. It's the only game that is so un-modereated that i've ever played where the common reaction to meeting a hacker that is interested in griefing you is to call your hacker friend white-knight and ask him to boot the griefer-hacker from the lobby. Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..) On top of all this, Rockstar attempts to have an in-game economy , even selling money outright to players in the form of 'Shark Cards' for real-life currency , while 'modders' (hackers) easily dupe gold for anyone that may ask in a public lobby. This isn't just all coincidence; the game lacks any kind of realistic checks/balances with the server for the sake of latency and interoperability -- but this results in every 13 year old passing around the Cheat Engine structs on game-cheating forums and acting like virtual gods while tossing legitimate players around lobbies like ragdolls -- meanwhile Rockstar continues releasing GTA Online content while ignoring playerbase pleas for supervision. It's truly unique though -- an online battlefield where one can literally watch battles between the metaphorical white hat and black hat hackers; but it's a definite indicator of a poorly ran business when vigilante customers need to replace customer service. Also, an aside, most 'mod-menus' -- the small applets put together using publicly available memory structs for game exploit -- most all have a 'quick connect' feature that allows hackers to join lobbies much faster than the GTA V client usually allows for. This feature has existed for years and years, and I believe it performs tricks similar to those listed in the article.
- imtringued 6y ago
  >Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..) Interesting, back in the day the coolest CTD I did was to simply crank up the combo multiplier in S4 League to crash the game client on every player in the room except mine since that game was peer to peer and thus any form of hacking (teleportation, infinite melee range, instant kill, immortality, etc) was possible. The combo multiplier was set to 256 and thus every single particle was duplicated 256 times and this caused the game to crash.
rustybolt 6y ago
Not very surprising. Twitter doesn't work properly on my desktop, google freezes when showing the cookiewall, github freezes my phone. These are all important projects of billion dollar companies.
xwolfi 6y ago
I work in a large multi billion company and we have people staring at a slow problem for a decade before a noob comes with a profiler and find they browse every key of a Map instead of calling get and such. Or do 1 million db queries on a GUI startup... Not surprised they didn't bother for 6 minutes when it takes us 10 years to fix a 30minutes locked startup.
greggman3 6y ago
Tell that to Valve. The Source engine and all its games (Half Life 1, 2, Portal, Alyx) have horrible load times. They might not be as bad as the GTA example but they're long and extremely frustrating. And yet, no one cares, Those games (and GTA5) all sold millions of copies. The only way this stuff gets fixed is if (a) some programmer takes pride in load times or (b) customers stop buying games with slow load times. (b) never happens. If the game itself is good then people put up with the load times. If the game is bad and it has bad load times they'll point to the load times as a reason it's bad but the truth is it's the game itself that's bad because plenty of popular games have bad load times Also, any game programmer loading and parsing text at runtime by definition, doesn't care about load times. If you want fast load times you setup your data so you can load it directly into memory, fix a few pointers and then use it where it is. If you have to parse text or even parse binary and move things around then you've already failed.
- gwd 6y ago
  I think there may sort of be another thing going on: Basically, that the length of load time is an indicator, "This is a really serious program." I've sort of noticed the same thing with test machines: the more expensive the machine, the longer it takes to actually get to the grub prompt. Six minutes is probably excessive, but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game.
  - ric2b 6y ago
    > but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game. Given that it has been the most common criticism of the game since it launched, I don't think anyone views it as a sign of quality.
- danlugo92 6y ago
  Do you have a link to how one would archiece this data pointer magic? I wouldnt know what to search for.
randomNumber7 6y ago
> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years Agree. I found the slow behavior of sscanf while writing one of my first C programs during an internship^^ You literally just have to google "scanf slow" and find lots of information.
hinkley 6y ago
I've worked a number of places where the engineering culture discourages any sort of fishing expeditions at all, and if I weren't so stubborn everything would take 2-4x as long to run as it does. I say engineering culture, because at 2 of these places everyone was frustrated with engineering because the customers wanted something better, but the engineers would point at flat flame charts, shrug, and say there's nothing that can be done. Bull. Shit. There's plenty that can be done because there are parts of a process that don't deserve 15% of the overall budget. The fact that they are taking 1/6 of the time like 5 other things is a failure, not a hallmark of success. Finding 30% worth of improvements with this perspective is easy. 50% often just takes work, but post-discovery much of it is straightforward, if tedious. My peers are lying with charts to get out of doing "grunt work" when there's a new feature they could be implementing. But performance is a feature.
awpy 6y ago
Maybe it was outsourced. I don't understand how a team could make such an excellent game and fail to resolve such a simple bottleneck.

anoncake 6y ago

I find it absolutely believable that a for-profit company does not prioritize fixing a game that is already a cash cow anyway.

xyst 6y ago

is it really unbelievable? companies this big tend to prioritize hiring a shit ton of middlemen (VPs, project managers, developer managers, offshore managers) in order to avoid paying out for talent to build and constantly maintain the project. I guess paying a shit ton of money to 1 person to manage 10+ poorly paid contractors works out for them, accounting wise. If one really examined the accounting for GTAO, I would bet that most of the billions of dollars that were earned in micro transactions went to marketing, product research, and to middle management in the form of bonuses.

spuz 6y ago
Even if you view this as a business decision rather than a technical one, any smart project manager would realise a 6 minute loading time literally costs the company millions per year in lost revenue. (How many times have you felt like firing up GTA Online only to reconsider due to the agonising load time). I would guess this was simply a case of business folk failing to understand that such a technical issue could be so easily solved plus developers never being allowed the opportunity to understand and fix the issue in their spare time.
- alien_ 6y ago
  The people who observe the slow loading time already paid for the game, so I guess R* won't lose much revenue because of this nasty bug.
  - bcrosby95 6y ago
    The game has microtransactions. Coincidentally, also a large reason why load times were so slow.
    - deleted 6y ago
      [deleted]
  - delecti 6y ago
    The online mode has microtransactions. People not playing anymore aren't paying for microtransactions either.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - AuryGlenz 6y ago
    I didn’t buy the game to play with my friends because I heard of how terrible the loading situation was.
IshKebab 6y ago
It's kind of hard to believe. GTA5's online mode is their cash cow, and 6 minute load times are common?! It's kind of amazing people even play it with those load times. It's such a simple problem that one dev could have found and fixed it within a day.
- toast0 6y ago
  It's not at all hard to believe if you've been playing video games for a while. Everything is getting slower and slower, and nobody cares. When I played the Atari 2600, I had to wait for the TV to warm up, but otherwise there were no games with anything approaching load times (with 128 bytes of RAM in the console, who would know). The NES didn't have much in the way of load times either, but you did have to fiddle with the cartridge slot. SNES and Genesis didn't usually load (Air Buster being a noticeable exception). CD based systems sure did like to load, but that's somewhat understandable. In the mean time, more and longer boot screens. The Switch uses a cartridge (or system flash/SD cards), but it likes to load forever too. PC Gaming has had loading for longer, but it's been getting longer and longer. Some arcade games have lengthy loading sequences, but only when you turn them on while they read their entire storage into RAM so they can be fast for the rest of the time they're on (which in arcades is usually all day).
  - ubercow13 6y ago
    Shorter loading times were one of the main selling points of this console generation.
  - yoz-y 6y ago
    > Everything is getting slower and slower, and nobody cares. It really depends. The latest crop of games I’ve played (Doom Eternal, Cyberpunk) loads way faster than games from a few years back (aforementioned GTA-V, Shadow Warrior 2…). This is also on the same machine, so it’s not the hardware that makes it faster.
    - hnick 6y ago
      Doom Eternal's load times were so good I didn't even bother moving it to my SSD (I junction to a larger HDD by default).
xyzelement 6y ago
I am always amused by comments like this. You have no idea what development practices they follow (neither do I) but it's hilarious to read your tone. GTA has achieved tremendous success both as an entertaining game and as a business. It's enjoyed by millions of people and generates billions in revenue. As per this article, it has startup problems (which don't seem to actually really hurt the overall product but I agree sound annoying) but the bigger picture is: it's a huge success. So - Rockstar has nailed it. What exactly is your platform for analyzing/criticizing their processes or even having a shot of understanding what they are? What have you build that anyone uses? (not saying you haven't, but.. have you been involved with anything remotely close to that scale?) And if not, whence he high horse?
- brokencode 6y ago
  Successful people and businesses can be wrong. You are not making a case for why those development practices are okay, but are simply appealing to authority. I and most other customers would argue that 6 minute loading times are atrocious, and if there is an easy fix like this, it makes me lose a lot of respect for the developer who doesn’t fix it. It maybe would even make me avoid them in the future. A reputation is built over years, but can be lost pretty much instantly. Companies have to continue serving their customers to enjoy ongoing success.
  - hombre_fatal 6y ago
    They didn’t make an appeal to authority but an appeal to commercial success, and they’re right on. The fact that GTAO is so popular should make most HNers rethink what they know about the commercial necessity of optimization vs building a compelling product.
    - incrudible 6y ago
      The loading times were not initially that long and then the slow CPU makes a big difference. This really only goes to show how much you can get away with if you have an outstandingly popular product that has no direct competition. Chances are that your product is not that compelling, if it performs poorly, that will hurt adoption. It will never become outstandingly popular in the first place.

masklinn 6y ago

> The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top. For what it's worth, 10MB of JSON is not much. Duplicating the example entry from the article 63000 times (replacing `key` by a uuid4 for unicity) yields 11.5MB JSON. Deserialising that JSON then inserting each entry in a dict (indexed by key) takes 450ms in Python. But as Bruce Dawson oft notes, quadratic behaviour is the sweet spot because it's "fast enough to go into production, and slow enough to fall over once it gets there". Here odds are there were only dozens or hundreds of items during dev so nobody noticed it would become slow as balls beyond a few thousand items. Plus load times are usually the one thing you start ignoring early on, just start the session, go take a coffee or a piss, and by the time you're back it's loaded. Especially after QA has notified of slow load times half a dozen times, the devs (with fast machines and possibly smaller development dataset) go "works fine", and QA just gives up.

hanniabu 6y ago
That QA bit is too true, "it works for me!" shrug.
ldng 6y ago
But is quadratic the real issue ? Isn't that a developer answer ? The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases. And what is small? Medium? Large? The truth is that there is no one size fits all and assumptions need to be reviewed periodically and adapted accordingly. And they never are... Ask a DBA.
- gridspy 6y ago
  quadratic is a fancy way of saying "this code is super fast with no data, super slow once you have a decent amount" The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list. Why quadruple? Because you end up scanning a list which is twice as long. You have to scan that list twice as many times. 2x2 = 4. The larger list no longer fits in the fast (cache) memory, among other issues. The cache issue alone can add another 10x (or more!) penalty.

0xy 6y ago

Something I've noticed in highly successful companies is that problems never get fixed because the sound of the money printer from the core business is deafening. Our customer portal loads 2 versions of React, Angular, Knockout and jQuery on the same page? Doesn't matter, it's printing billions of dollars. Rockstar's money printer is so loud that they don't care about problems. Same thing for Valve, their money printer is so loud that they barely bother to make games anymore and let the Steam client languish for years (how did they let Discord/Twitch happen?).

LukvonStrom 6y ago
Look at sharepoint. It's a total nightmare of a platform to develop on, but the people just adapted and built their businesses on it
- ip26 6y ago
  I suspect sharepoint is the platform version of excel & VBA. You & I might hate it, but it gets powerful capabilities into the hands of regular people. sharepoint is probably many non-engineer's very first exposure to actual version control with checkouts, checkins, version history, and merge.
  - ant6n 6y ago
    Sharepoint has version control? I thought it´s mostly a poorly functioning Dropbox copy.
    - easton 6y ago
      This comment is absolutely hilarious to me, because the main selling point for SharePoint (for the longest time) was that you could version control Microsoft Office documents. Then intranets happened, someone wanted to build a Turing machine with it, and now it's the monstrosity we know now.
- passivate 6y ago
  What makes it a nightmare? Can you share a few issues you've run into?

tamrix 6y ago

Well this sprint we didn't release any new features but we reduced the load.... Dammit hackernews!

kuroguro 6y ago

Small nitpick: I believe these are items/prices for the in-game currency, not micro-transactions. You can buy in-game currency for real world money tho: https://gta.fandom.com/wiki/Cash_Cards https://gta.fandom.com/wiki/Cash_Cards Not 100% sure, never bought anything.

account42 6y ago
So what you are saying is they are in fact microtransactions.

Thaxll 6y ago

It was probably fast 10years ago when the store had couple of items, the dev back then never thought that it would grow to 60k items. Classic programming right there. As for profiling, Windows Performance Toolkit is the best available no?

trulyme 6y ago
Meh. It's ok to assume low number of items and code accordingly. What is not ok is for the company to ignore such a problem for years, instead if detecting and fixing it.

gameswithgo 6y ago

What is complicated about it is that an online modern 3d game is huge, and there are 95,000 places where a dumb mistake could hurt performance a lot for some customers. You catch 94,999 of them and then "unforgiveable"

Plutoberth 6y ago
If it was that way for a few months and then fixed... still pretty shoddy but sure. However, it has been that way for YEARS, and is one of the most common complaints among players. I wonder how much of the remaining loading time could actually be shaved off if someone with the source code took a crack at it.

moonchild 6y ago

> salty because their parent company unjustly DMCA'd re3 Unjustly, but legally. The people you should be salty at are the lawmakers.

breakingcups 6y ago
That still remains to be seen. A DMCA is not a court order. Anyone can file one and take a repository offline for two weeks.
- moonchild 6y ago
  Yes but the code was clearly derived directly from a decompiled binary; not ‘clean room’ reverse engineering. Hence, illegal, regardless of whether a dmca takedown notice is filed.
  - xyzzy_plugh 6y ago
    That doesn't make it illegal, there are plenty of jurisdictions where non-clean-room reverse engineering is perfectly legal.
    - charcircuit 6y ago
      But distributing it without a license from the copyright holder is.
      - jagger27 6y ago
        Distributing what, exactly? It required you already owned a copy of the game to install/build it.
        charcircuit 6y ago
        Distributing whatever was in the repo. Requiring a copy of the game doesn't magically make it legal. A modded game is a derivative work.

formerly_proven 6y ago

Also: Rockstar being too cheap to implement anti-cheat on the by far most successful online shooter on the planet. Also > I don’t think there’s any easier way out. lmfao

buzzerbetrayed 6y ago
This may be obvious, but is GTAV the most successful online shooter on the planet? (Never played it)
- sickofparadox 6y ago
  Its the second best selling game of all time[1], and because Minecraft doesn't have guns, I suppose that would qualify it as the most successful online shooter (even if I think another genre would be more applicable). [1]https://en.wikipedia.org/wiki/List_of_best-selling_video_games https://en.wikipedia.org/wiki/List_of_best-selling_video_gam...
  - gambiting 6y ago
    Well but that's a bit like saying that since there are technically races in GTA it's also the most successful racing game. It will also be the most successful flying game and the most successful sailing simulator....etc etc. I don't know, even though there is shooting in GTA I don't think I'd call it a shooter.
    - sickofparadox 6y ago
      I agree that there is probably some other genre that would describe it better, but I think it wouldn't be unfair by any means to describe GTA as a third-person shooter.
  - charcircuit 6y ago
    That list is incomplete. CSGO is the most played game on Steam and is now free to play. I would not be surprised in the slightest if more people have CSGO.

AnonsLadder 6y ago

Does anyone have a link to a copy of re3? Iirc, there was a gitrepo that kept a copy of all DMCA'd repos

djmips 6y ago
try the hacker news search (bottom of the page) and you'll find stories on the takedown where there are links to backups posted in the comments.

nikanj 6y ago

The old maxim of "Premature optimization is the root of all evil" has over time evolved to "If you care one iota about performance, you are not a good programmer".

deleted 6y ago
[deleted]
vendiddy 6y ago
And fwiw, the full quote is: We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
- blowski 6y ago
  Also, it’s not “never optimise”. It’s “only optimise once you’ve identified a bottleneck”. I guess in a profit-making business you only care about bottlenecks that are costing money. Perhaps this one isn’t costing money.
  - skzo 6y ago
    ding ding ding ding
  - saagarjha 6y ago
    This one isn’t measured to be costing money. Trying to figure out how much money you’re losing as a result of performance problems is extremely difficult to do.
  - CogitoCogito 6y ago
    Perhaps this one is costing them a lot of money.
  - nmfisher 6y ago
    Precisely. Hasn't GTA Online done over a billion in revenue? Given how incredibly successful it's been, it's conceivable the suits decided the opportunity cost of investing man-hours to fix the issue was too high, and that effort would be better spent elsewhere.

imglorp 6y ago

> This online gamemode alone made $1 billion in 2017 alone. There's the answer right there. They figure it's making $1B/yr, leave it alone. Maintenance? That cuts into the billion. Everyone moved onto the next project.

tyingq 6y ago
Or they fix it, see that their "in game time" average drops, and then back it out...
- themeiguoren 6y ago
  I would not at all be surprised if the long load time made for a sunk cost that kept people playing for longer sessions rather than picking it up for less than half an hour at a time.
  - iknowstuff 6y ago
    I don't know. In GTA Online you encounter loading every 15min.
  - simias 6y ago
    I enjoyed GTA online but haven't touched it in well over a year, and the insane loading times are definitely a big reason why. For those who haven't played the game it's important to emphasize that it's usually in the 5minute range, and even then you'll regularly end up with connectivity issues or other problems that will kick you out of the lobby for yet an other ~5 minute load. When I played it wasn't uncommon to spend 30 minutes mostly looking at the loading screen while you were trying to set up a play session with a couple of friends. If you're an adult with limited playtime it's just a complete dealbreaker. You can't just decide to have a quick 20minute play session if you know that you'll have to spend at least half of it looking at loading screens.
  - antihero 6y ago
    This is actually very possible - if they allowed people to dip in and dip out, they will. I remember when CP2077 came out on GeForce NOW and the wait times were 30+mins. I'd play a 6 hour session until I was booted off simply to make it worth the wait.

neatze 6y ago

I am not saying it is the case, nor understand details of solution in depth to comment on it, but in analogy, this reads to me, like an yelling at person who figure out how to solve rubix cube puzzle steps, because once steps are known solution is simple.

burlesona 6y ago
No, other people have pointed this out, this should have been very easy to recognize as inefficient in the source code. More likely the code was written hastily and only tested against very small inputs, and then nobody ever actually tried to improve the famously long load times.
- wnoise 6y ago
  The sscanf issue was not obvious: it looks linear. And should be, on a better sscanf implementation. The duplicate checking on the other hand is a classic "accidentally quadratic" case that is obvious.

oh_sigh 6y ago

Maybe long load times are advantageous? Because it creates longer user sessions on average? If you devote 10 minutes to loading the game you will probably want to play for at least 30 minutes.

erik_seaberg 6y ago
Wouldn't the most impatient customers be more likely to pay for items, rather than earn them in game?

zionic 6y ago

I stopped playing GTAV online a few years back because of the crazy load times, not only that but you have to go through 6+ minute load screens multiple times in many sessions. This oversight has cost them millions of dollars easy.

wyaeld 6y ago
If it made over $1b in a year previously, and had such insane load times, its very plausible this bad coding has cost them north of another $1b. Probably ranks pretty highly up there in terms of damage to company financials, due to a lack of care.

umanwizard 6y ago

“worth their salt” is doing a lot of work here. No true Scotsman fallacy? I think you might be surprised by how few programmers even know what a profiler is, let alone how to run one.

thrwyoilarticle 6y ago
That seems like a misapplication of the fallacy. If we assume 'worth their salt' is a synonym for 'good', then saying any good developer can operate a profiler is entirely reasonable.

thrwyoilarticle 6y ago

The popular view is that companies who write software know how to prioritise, so if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile. I disagree. If there are no internal incentives for the people who know how to fix this to fix it, or if there's no path from them thinking fixing it could improve revenues to being assigned the ticket, things like this won't get fixed. I can fully believe the load times will result in fewer users and lower expenditure. I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.

saghm 6y ago
> I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. The messenger website has been atrocious for me lately. On my high-powered desktop, it often lags a bit, and on my fairly high-end laptop, it's virtually unusable. I thought it must be something I changed in my setup, but it's oddly comforting to hear that I'm not the only one with such issues.
- baby 6y ago
  For me it’s google maps, it has gotten so freaking slow both on mobile and on desktop. Actually google docs and sheet are the same.
  - amluto 6y ago
    Gmail. Typing in gmail is painfully laggy.
- ianlevesque 6y ago
  Try https://www.messenger.com/desktop https://www.messenger.com/desktop
  - lethologica 6y ago

Scaevolus 6y ago

I suspect that the core engine programmers moved onto other projects long ago, leaving GTA:O running with mostly artists and scenario designers to produce more DLC. This bug wouldn't present in the first couple years with the limited amount of DLC, so by the time it got ridiculous there wasn't anyone left with the confidence to profile the game and optimize it. A junior dev could fix this, but would probably assume that slow loads are a deep complex engine problem that they won't be able to fix. Alternatively, management would declare that there's too much risk doing more technical engine work, and not sign off on any proposed "minor" optimizations because it's too risky.

jorams 6y ago
> This bug wouldn't present in the first couple years with the limited amount of DLC GTA Online loading times have been infamous for a very long time. They were already unjustifiably bad when they released the game for PC, and at that point engine programmers would surely be involved.
- Aeolun 6y ago
  The 1.5 minute default load time is also ridiculous.
anotherfish 6y ago
This is very much the likely scenario. The money is in further DLC. The existing GTAO engine is "done" from their perspective. I'd guess also that the next version of the base engine is in RDR2 or later and doesn't have these issues. But at the same time they likely wouldn't backport the changes for fear of cost overruns.

deleted 6y ago

[deleted]

lucideer 6y ago

> It is absolutely unbelievable [...] that a cash cow [...] has a problem like this Likely it wasn't fixed precisely because it's such a cash cow. "It's making money, don't fuck with it".

uhhhhhhhhhhhhhh 6y ago

Can you quantify the additional profit they would have made if this were fixed N years ago?

grishka 6y ago

> the reverse engineered version of GTA III and Vice City Ohhh. Thank you for telling me about this. I just found a mirror and successfully built it for macOS. Runs so much better than the wine version. But I guess I'll never finish that RC helicopter mission anyway lol

crazygringo 6y ago

I imagine the conversation between the programmer(s) and management went exactly like this: Management: So, what can we do about the loading times? Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all. Management: OK, guess that's that then. Sucks but nothing we can do. Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again." And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers. I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here. But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.

CountHackulus 6y ago
I can pretty much guarantee that there was no discussion with management like that. From experience, live ops games are essentially a perpetually broken code base that was rushed into production, then a breakneck release schedule for new features and monetization. I've personally had this conversation a few times: Programmer: Loading times are really slow, I want to look into it next sprint. Management: Feature X is higher priority, put it in the backlog and we'll get to it.
- sillysaurusx 6y ago
  Former gamedev here. I can vouch for this conversation. The idea that management would ask about slow loading times doesn't seem to be too realistic in my experience.
- 6y ago

db48x 6y ago

In my experience most engineers have never used a profiler even once. They write the code, and if you're lucky they get it working correctly.

mikevm 6y ago
Let's call them "code technicians" instead of engineers, ok? (that's a euphemism for "code monkeys")

Ansil849 6y ago

> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple. It is both believable and - by virtue of the fact that, as you said, the series continues to be a cash cow - is apparently forgivable. Here's the thing: the company has zero reasons to fix this, or other ostensibly egregious affronts like DRM, because gamers keep buying the product. There is literally no economic incentive to 'fix' it.

_AzMoo 6y ago
How many players have they lost to excessive load times?
- astrange 6y ago
  I've lost interest in a lot of online games because the clients always do their mandatory updates on launch, which is exactly the time you want to play the game. (Same thing for websites - they show you all the annoying popup signup sheets/survey questions the instant you load the page.)
  - _AzMoo 6y ago
    For sure. I fit my gaming into about 2 hours a week. If it takes 6 minutes to load there's no way I'm going to play it. I know I'm not the target market but I'm still a missed opportunity.
- Ansil849 6y ago
  > How many players have they lost to excessive load times? Judging by the number of successful sequels the franchise has spawned, the answer is 'an insignificant number'. The fact of the matter, and the point my comment was trying to make, is that the overwhelming majority of players do not sufficiently care about load times or other complaints to deter them from paying for the game. That is the reality of the situation.
  - 6y ago

sillysaurusx 6y ago

Attitudes like yours are why gamedevs keep to themselves. "Unbelievable" and "unforgivable" eh? It's a greedy attitude. Instead of viewing GTA5 as a success that's brought a lot of people happiness, you view it as a money cow designed to extract every last bit of profit – and time, since this bug caused 70% longer loading times. Perhaps it's both. But you, sitting here behind a keyboard with (correct me if I'm wrong) no gamedev experience, have no idea what it's like on a triple-A gamedev team with various priorities. The fact that the game works at all is a minor miracle, given the sheer complexity of the entire codebase. The fact that someone was able to optimize the obfuscated executable is a wonderful thing. But they weren't a part of the team that shipped GTA 5. If they were, they certainly wouldn't have been able to spend their time on this.

ryandrake 6y ago
This kind of excuse making is one of the reasons I got out of software development. It’s not just gamedev. Priorities are way out of wack when you have time to put in binary obfuscation, but no time to fix such a huge performance bottleneck. The idea that “it’s a miracle software works at all” demonstrates the chronic prioritization and project management competence problem in the industry. It’s ok to recognize a thing as a business success but a technical failure. In fact many software projects are business successes despite awful and unforgivable quality compromises. You don’t get to whitewash it just because the thing prints money.
- systemvoltage 6y ago
  How do we then address chronic incompetence? Never complain about it? This is not small. This kind of incompetency if employed in a different sector such as security would lead to losing personal data of millions. > “it’s a miracle software works at all” This is not the case here. Please re-evaluate your calibration on this topic.
  - sillysaurusx 6y ago
    1. you replied to the wrong person. 2. this kind of incompetence exists in all other sectors. That's why pentests are so crucial, and why they guard the security of millions. 3. we'll have to agree to disagree that it's a minor miracle. Having seen the complexity firsthand, it's quite amazing.

hibikir 6y ago

It's not just believable, but it's normal. I have spent quite a bit of my career maintaining software, and I don't recall one employers where low hanging fruit like this wasn't available everywhere. The problem is not that developers can't optimize things: you will find some developers capable of figuring this problem out anywhere. What makes this low hanging fruit so popular is the fact that we aren't measuring enough, and even when we do, we aren't necessarily prioritizing looking into things that are suspiciously slow. In the case of this example, the issue is also client-side, so it's not as if it's costing CPU time to Rockstar, so it's unlikely you'll have someone who can claim their job description includes wondering if the load times are worth optimizing. When problems like this one get solved is because someone who is very annoyed by the problem and either convinces a developer to even look into the problem. Most of the time, the people that suffer, the people that get to decide how to allocate the time, and the people that are in a position to evaluate the root cause of the problem never even get to talk to each other. It's the price we often pay for specialization and organizations with poor communication. Organizations where the people deciding what has to be done next, and where the company culture dictates that the way forward is to either complete more tickets faster, or find ways to be liked by your manager, are not going to be fostering the kind of thinking that solves a problem like this one, but that's a lot of what you find in many places. A developer with a full plate that is just working on the next feature isn't going to spend their time wondering about load times. But instead we end up blaming the developers themselves, instead of the culture that they swim in.

smaudet 6y ago
Hear hear, we should make a punch a dummy manager/BA/Code standards 'lead' day... This code looks like someone with almost no experience hacked it together but because they were an intern and likely Rockstar is a toxic place to work, it never gets prioritized to be fixed. I think if managers prioritized cycle time, metri s more, they'd find that they are encouraging a lot of practices which lead to horrible efficiencies - "measure twice cut once" is a positive mantra which leads to more solid designs with less bugs. Agile sort of addressed this problem but unfortunately only at small size scales. Iteration and story capacity got overprioritized over quality, customer engagement, and self-leading teams. Plus things such as scaled agile suffer from the oxymoron of planning fast iteration - if you have a master plan you lose the ability to respond to change and iterate, or you allow iteration and you must accept if any team iterates the whole team must discard the plan...which at some point means you either accept high cycle times or you figure out a way to decouple functionality to the extent the planning becomes the standard fallacy of waterfall - wasting meeting time to go over a plan that isn't based on anything.

karmasimida 6y ago

It could be that at the time GTA online first publishes, the list as hashmap isn't too much of an issue, due to limited catalog, but get progressively worse as the inventory grows. Ofc this is just a hypothesis, but I see the hesitation to change legacy code if it ain't broken as a wide spread mentality.

sellyme 6y ago
> I see the hesitation to change legacy code if it ain't broken as a wide spread mentality. Load times measured in double digit minutes on a significant number of machines meets absolutely every reasonable definition of "broken".

omgJustTest 6y ago

Focus is on the money, not on the technicals, on the production side. It is a game, to entertain and ultimately “waste time” on the consumer side. Also on the topic of parsing, +1-11gbyte/sec is possible if you go binary. Point is: this isn’t a technical problem, it’s a choice.

matheusmoreira 6y ago

I used to play this game a lot on PS4. I actually dropped it due to the ridiculous loading times... I still assumed it was doing something useful though. I can't believe they wasted so much of my time and electricity because of this. Even cheaply-made mobile games don't have bugs like this. > their parent company unjustly DMCA'd re3 Wow, this is EA games level scumbaggery... I don't think I'm gonna buy games from them again.

Scoundreller 6y ago

> obfuscated executable loaded with anti-cheat measures I'm impressed that gamecopyworld.com is still online, updated, and has the same UI that it did in 2003

lethologica 6y ago
Whoa, what a (refreshing) blast from the past.

pilif 6y ago

> This online gamemode alone made $1 billion in 2017 alone. which of course goes to show that at least from a business side, this issue is completely inconsequential and all resources should be used to push for more monetization (and thus adding to the problem by adding more items to the JSON file) rather than fixing this issue, because, clearly, people don't seem to mind 6 minutes loading time. I'm being snarky here, yes, but honestly: once you make $1 billion per year with that issue present, do you really think this issue matters at all in reality? Do you think they could make $1+n billion a year with this fixed?

hderms 6y ago
The bigger the scale the bigger a few percentage point improvement would be worth. I would generally think if you're at 1bn in revenue you should devote 1%+ percentage points of your workforce towards finding low hanging fruit like this. If 1% of employees deployed to find issues that, when fixed, yield 2% improvement in revenue thats likely a winning scenario
imtringued 6y ago
This is losing them money. If they fixed the issue they absolutely would get $1+n billion instead of just $1 billion and that n alone is big enough to pay multiple years worth of 6 digit salaries just to fix this single bug.

serf 6y ago

>It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple. Having played the game, it's not surprising to me in the least. I have never yet encountered another such 'wild-west' online experience. It's the only game that is so un-modereated that i've ever played where the common reaction to meeting a hacker that is interested in griefing you is to call your hacker friend white-knight and ask him to boot the griefer-hacker from the lobby. Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..) On top of all this, Rockstar attempts to have an in-game economy , even selling money outright to players in the form of 'Shark Cards' for real-life currency , while 'modders' (hackers) easily dupe gold for anyone that may ask in a public lobby. This isn't just all coincidence; the game lacks any kind of realistic checks/balances with the server for the sake of latency and interoperability -- but this results in every 13 year old passing around the Cheat Engine structs on game-cheating forums and acting like virtual gods while tossing legitimate players around lobbies like ragdolls -- meanwhile Rockstar continues releasing GTA Online content while ignoring playerbase pleas for supervision. It's truly unique though -- an online battlefield where one can literally watch battles between the metaphorical white hat and black hat hackers; but it's a definite indicator of a poorly ran business when vigilante customers need to replace customer service. Also, an aside, most 'mod-menus' -- the small applets put together using publicly available memory structs for game exploit -- most all have a 'quick connect' feature that allows hackers to join lobbies much faster than the GTA V client usually allows for. This feature has existed for years and years, and I believe it performs tricks similar to those listed in the article.

imtringued 6y ago
>Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..) Interesting, back in the day the coolest CTD I did was to simply crank up the combo multiplier in S4 League to crash the game client on every player in the room except mine since that game was peer to peer and thus any form of hacking (teleportation, infinite melee range, instant kill, immortality, etc) was possible. The combo multiplier was set to 256 and thus every single particle was duplicated 256 times and this caused the game to crash.

rustybolt 6y ago

Not very surprising. Twitter doesn't work properly on my desktop, google freezes when showing the cookiewall, github freezes my phone. These are all important projects of billion dollar companies.

xwolfi 6y ago

I work in a large multi billion company and we have people staring at a slow problem for a decade before a noob comes with a profiler and find they browse every key of a Map instead of calling get and such. Or do 1 million db queries on a GUI startup... Not surprised they didn't bother for 6 minutes when it takes us 10 years to fix a 30minutes locked startup.

greggman3 6y ago

Tell that to Valve. The Source engine and all its games (Half Life 1, 2, Portal, Alyx) have horrible load times. They might not be as bad as the GTA example but they're long and extremely frustrating. And yet, no one cares, Those games (and GTA5) all sold millions of copies. The only way this stuff gets fixed is if (a) some programmer takes pride in load times or (b) customers stop buying games with slow load times. (b) never happens. If the game itself is good then people put up with the load times. If the game is bad and it has bad load times they'll point to the load times as a reason it's bad but the truth is it's the game itself that's bad because plenty of popular games have bad load times Also, any game programmer loading and parsing text at runtime by definition, doesn't care about load times. If you want fast load times you setup your data so you can load it directly into memory, fix a few pointers and then use it where it is. If you have to parse text or even parse binary and move things around then you've already failed.

gwd 6y ago
I think there may sort of be another thing going on: Basically, that the length of load time is an indicator, "This is a really serious program." I've sort of noticed the same thing with test machines: the more expensive the machine, the longer it takes to actually get to the grub prompt. Six minutes is probably excessive, but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game.
- ric2b 6y ago
  > but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game. Given that it has been the most common criticism of the game since it launched, I don't think anyone views it as a sign of quality.
danlugo92 6y ago
Do you have a link to how one would archiece this data pointer magic? I wouldnt know what to search for.

randomNumber7 6y ago

> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years Agree. I found the slow behavior of sscanf while writing one of my first C programs during an internship^^ You literally just have to google "scanf slow" and find lots of information.

hinkley 6y ago

I've worked a number of places where the engineering culture discourages any sort of fishing expeditions at all, and if I weren't so stubborn everything would take 2-4x as long to run as it does. I say engineering culture, because at 2 of these places everyone was frustrated with engineering because the customers wanted something better, but the engineers would point at flat flame charts, shrug, and say there's nothing that can be done. Bull. Shit. There's plenty that can be done because there are parts of a process that don't deserve 15% of the overall budget. The fact that they are taking 1/6 of the time like 5 other things is a failure, not a hallmark of success. Finding 30% worth of improvements with this perspective is easy. 50% often just takes work, but post-discovery much of it is straightforward, if tedious. My peers are lying with charts to get out of doing "grunt work" when there's a new feature they could be implementing. But performance is a feature.

awpy 6y ago

Maybe it was outsourced. I don't understand how a team could make such an excellent game and fail to resolve such a simple bottleneck.