5 ms·

My takeaway is that it seems like they did NO benchmarking of their own before choosing to do all that duplication. They only talk about performance tradeoff no

• maccard 10mo ago
  I've been involved in decisions like this that seem stupid and obvious. There's a million different things that could/should be fixed, and unless you're monitoring this proactively you're unlikely to know it hsould be changed. I'm not an arrowhead employee, but my guess is at some point in the past, they benchmarked it, got a result, and went with it. And that's about all there is to it.
  • Xelbair 10mo ago
    >These loading time projections were based on industry data - comparing the loading times between SSD and HDD users where data duplication was and was not used. In the worst cases, a 5x difference was reported between instances that used duplication and those that did not. We were being very conservative and doubled that projection again to account for unknown unknowns. >We now know that, contrary to most games, the majority of the loading time in HELLDIVERS 2 is due to level-generation rather than asset loading. This level generation happens in parallel with loading assets from the disk and so is the main determining factor of the loading time. We now know that this is true even for users with mechanical HDDs. they did absolutely zero benchmarking beforehand, just went with industry haresay, and decided to double it just in case.
    • creshal 10mo ago
      "Industry hearsay" in this case was probably Sony telling game devs how awesome the PS5's custom SSD was gonna be, and nobody bothered to check their claims.
      • mary-ext 10mo ago
        the industry hearsay is about concern of HDD load times tho
        • creshal 10mo ago
          HDD load times compared to......?
          • mary-ext 10mo ago
            are we not reading the same post and comments? the "industry hearsay" from two replies above mine is about deliberate data duplication to account for the spinning platters in HDD (which isn't entirely correct, as the team on Helldivers 2 have realized)
      • maccard 10mo ago
        What are you talking about? This has nothing to do with consoles, and only affects PC builds of the game
        • creshal 10mo ago
          HD2 started as playstation exclusive, and was retargeted mid-development for simultaneous release. So the PS5's SSD architecture was what developers were familiar with when they tried to figure out what changes would be needed to make the game work on PC.
          • maccard 10mo ago
            I don't really understand your point. You're making a very definitive statement about how the PS5's SSD architecture is responsible for this issue - when the isssue is on a totally different platform, where they have _already_ attempted (poorly, granted) to handle the different architectures.
            • creshal 10mo ago
    • maccard 10mo ago
      Nowhere in that does it say “we did zero benchmarking and just went with hearsay”. Basing things on industry data is solid - looking at the steam hardware surveys if a good way to figure out the variety of hardware used without commissioning your own reports. Tech choices are no different. Do you benchmark every single decision you make on every system on every project you work on? Do you check that redis operation is actually O(1) or do you rely on hearsay. Do you benchmark every single SQL query, every DTO, the overhead of the DI Framework, connection pooler, json serializer, log formatter? Do you ever rely on your own knowledge without verifying the assumptions? Of course you do - you’re human and we have to make some baseline assumptions, and sometimes they’re wrong.
    • pixl97 10mo ago
      >they did absolutely zero benchmarking beforehand, just went with industry haresay, a https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence It was a real issue in the past with hard drives and small media assets. It's still a real issue even with SSDs. HDD/SSD IOPS are still way slower than contiguous reads when you're dealing with a massive amount of files. At the end of the day it requires testing which requires time at a time you don't have a lot of time.
      • imtringued 10mo ago
        It's not an issue with asynchronous filesystem IO. Again, async file IO should be the default for game engines. It doesn't take a genius to gather a list of assets to load and then wait for the whole list to finish rather than blocking on every tiny file.
        • pixl97 10mo ago
          There are two different things when talking about application behavior versus disk behavior. >wait for the whole list to finish rather than blocking on every tiny file. And this is the point. I can make a test that shows exactly what's going on here. Make a random file generator that generates 100,000 4k files. Now, write them on hard drive with other data and things going on at the same time. Now in another run of the program have it generate 100,000 4k files and put them in a zip. Now, read the set of 100k files from disk and at the same time read the 100k files in a zip.... One finishes in less than a second and one takes anywhere from a few seconds to a few minutes depending on your disk speeds.
      • the_af 10mo ago
        This is not a good invokation of Chesterton's Fence. The Fence is a parable about understanding something that already exists before asking to remove it. If you cannot explain why it exists, you shouldn't ask to remove it. In this case, it wasn't something that already existed in their game. It was something that they read, then followed (without truly understanding whether it applied to their game), and upon re-testing some time later, realized it wasn't needed and caused detrimental side-effects. So it's not Chesterton's Fence. You could argue they followed a videogame industry practice to make a new product, which is reasonable. They just didn't question or test their assumptions that they were within the parameters of said industry practice. I don't think it's a terrible sin, mind you. We all take shortcuts sometimes.
    • FieryMechanic 10mo ago
      They made a decision based on existing data. This isn't unreasonable as you are pretending, especially as PC hardware can be quite diverse. You will be surprised what some people are playing games on. e.g. I know people that still use Windows 7 on a AMD BullDozer rig. Atypical for sure, but not unheard of.
      • red-iron-pine 10mo ago
        i believe it. hell i'm in F500 companies and virtually all of them had some legacy XP / Server 2000 / ancient Solaris box in there. old stuff is common, and doubly so for a lot of the world, which ain't rich and ain't rockin new hardware
        • FieryMechanic 10mo ago
          My PC now is 6 years old and I have no intention of upgrading it soon. My laptop is like 8 years old and it is fine for what I use it for. My monitors are like 10-12 years old (they are early 4k monitors) and they are still good enough. I am primarily using Linux now and the machine will probably last me to 2030 if not longer. Pretending that this is an outrageous decision when the data and the commonly assumed wisdom was that there were still a lot of people using HDDs. They've since rectified this particular issue and there seems to be more criticism of the company after fixing an issue.
  • alias_neo 10mo ago
    They admitted to testing nothing, they just [googled it]. To be fair, the massive install size was probably the least of the problems with the game, it's performance has been atrocious, and when they released for xbox, the update that came with it broke the game entirely for me and was unplayable for a few weeks until they released another update. In their defense, they seem to have been listening to players and have been slowly but steadily improving things. Playing Helldivers 2 is a social thing for me where I get together online with some close friends and family a few times a month and we play some helldivers and have a chat, aside from that period where I couldn't play because it was broken, it's been a pretty good experience playing it on Linux; even better since I switched from nvidia to AMD just over a week ago. I'm glad they reduced the install size and saved me ~130GB, and I only had to download about another 20GB to do it.
  • seg_lol 10mo ago
    Performance profiling should be built into the engine and turned on at all times. Then this telemetry could be streamed into a system that tracks it across all builds, down to a specific scene. It should be possible to click a link on the telemetry server and start the game at that exact point.
    • maccard 10mo ago
      How would that help them diagnose a code path that wasn't ever being run (loading non duplicated assets on HDDs)?
      • seg_lol 10mo ago
        > diagnose a code path that wasn't ever being run
• esrauch 10mo ago
  It's very easy to accidentally get misleading benchmarking results in 100 different ways, I wouldn't assume they did no benchmarking when they did the duplication.
• Hikikomori 10mo ago
  They used industry data to make the decision first to avoid potential multi minute load times for 10% or do of their players, hard to test all kinds of pc configurations. Now they have telemetry showing that it doesn't matter because another parallel task takes about as much time anyway.
  • whywhywhywhy 10mo ago
    Maybe it's changed a lot statistically in the last few years but for long time PC gamers used to have the mantra of small SSD for the OS and large HDD for games if they're price conscious so I could see that being assumed to be much more normal during development.
    • Dylan16807 10mo ago
      It's a shameful tragedy of the commons if you bloat your game 6x because you think your customers don't have enough SSD space for their active games.
  • justsomehnguy 10mo ago
    So they premature optimized for a wrong case. > multi minute load times 23Gb / 100mb / 60s = 3.92m So in the worst case when everything is loaded at once (how on a system with < 32Gb RAM?) it takes 4 minutes. Considering GTA whatever version could sit for 15 minutes at the loading screen because nobody bothered to check why - the industry could really say not to bother.
  • _aavaa_ 10mo ago
    But they did NOT know it would lead to multi-minute load time. They did not measure a baseline. Instead they did blindly did extra work and 6x’ed the storage requirement.
• djmips 10mo ago
  A tale as old as time. Making decisions without actually profiling before, during and after implementing.
• Xelbair 10mo ago
  worse, in their post they basically said: >we looked at industry standard values and decided to double them just in case.
  • functionmouse 10mo ago
    Some kind of evil, dark counterpart to Moore's law in the making
  • red-iron-pine 10mo ago
    this is one of the best selling games in history, and is emently popular across the globe. it had no serious or glaring impact to their bottom line. thus it was the right call, and if they didn't bother to fix it they'd still be rolling in $$$$
    • ycombinatrix 10mo ago
      All companies should also defraud & rug pull their customers. It will make them a lot of money and is thus the right call. Who cares about customers am I right? They'd still be rolling in $$$$.
• wongarsu 10mo ago
  It's pretty standard to do that duplication for games on CD/DVD because seek times are so long. It probably just got carried over as the "obviously correct" way of doing things, since HDDs are like DVDs if you squint a bit
  • jayd16 10mo ago
    The game does ship on disc for console, no?
    • Teknoman117 10mo ago
      The current generation of consoles can’t play games directly off the disk. They have to be installed to local storage first.
  • Arrath 10mo ago
    I had assumed the practice started to die off when installing games became dominant over streaming from the disc even on consoles. Seems I was wrong!
• Hendrikto 10mo ago
  > our worst case projections did not come to pass. These loading time projections were based on industry data - comparing the loading times between SSD and HDD users where data duplication was and was not used. In the worst cases, a 5x difference was reported between instances that used duplication and those that did not. We were being very conservative and doubled that projection again to account for unknown unknowns. They basically just made the numbers up. Wild.
  • rjzzleep 10mo ago
    On the flip side I don't remember who did it, but basically extracting textures on disk fixed all the performance issues UE5 has on some benchmarks(sorry for being vague, but I can't find the source material right now). But their assumption is in fact a sound one.
    • Normal_gaussian 10mo ago
      Yes. Its quite common for games to have mods that repack textures or significantly tweak the UE5 config at the moment - and its very common to see users using it when it doesn't actually affect their use cases. As an aside, I do enjoy the modding community naming over multiple iterations of mods - "better loading" -> "better better loading" -> "best loading" -> "simplified loading" -> "x's simplified loading" -> "y's simplified loading" -> "z's better simplified loading". Where 'better' is often some undisclosed metric based on some untested assumptions.
  • fullstop 10mo ago
    It's like the story of a young couple cooking their first Christmas ham. The wife cuts the end off of the ham before putting it in the oven. The husband, unwise in the ways of cooking, asks her why she does this. "I don't know", says the wife, "I did it because my mom did it." So they call the mom. It turns out that her mother did it, so she did too. The three of them call the grandma and ask "Why did you cut the end off of the ham before cooking it?" The grandma laughs and says "I cut it off because my pan was too small!"
    • 10mo ago
• Pannoniae 10mo ago
  The good old "studios don't play their own games" strikes again :P Games would be much better if all people making them were forced to spend a few days each month playing the game on middle-of-the-road hardware. That will quickly teach them the value of fixing stuff like this and optimising the game in general.
  • Forgeties79 10mo ago
    They could have been lying I guess but I listened to a great podcast about the development of Helldivers 2 (I think it was gamemakers notebook) and one thing that was constantly brought up was as they iterated they forced a huge chunk of the team to sit down and play it. That’s how things like diving from a little bit too high ended up with you faceplanting and rag-dolling, tripping when jet packing over a boulder that you get a little too close to, etc. They found that making it comically realistic in some areas led to more unexpected/emergent gameplay that was way more entertaining. Turrets and such not caring if you’re in the line of fire was brought up I believe. That’s how we wound up with this game where your friends are as much of a liability as your enemies.
  • whizzter 10mo ago
    People literally play the games they work on all the time, it's more or less what most do. Pay 2000$ for indie games so studios could grow up without being beholden to shareholders and we could perhaps get that "perfect" QA,etc. It's a fucking market economy and people aren't making pong level games that can be simply tuned, you really get what you pay for.
  • maccard 10mo ago
    I've worked in games for close to 15 years, and every studio I've worked on we've played the game very regularly. My current team every person plays the game at least once a week, and more often as we get closer to builds. In my last project, the gameplay team played every single day. > Games would be much better if all people making them were forced to spend a few days each month playing the game on middle-of-the-road hardware How would playing on middle of the road hardware have caught this? The fix to this was to benchmark the load time on the absolute bottom end of hardware, with and without the duplicated logic. Which you'd only do once you have a suspicion that it's going to be faster if you change it...
• whizzter 10mo ago
  It's an valid issue, those of us who worked back in the day on GD/DVD,etc games really ran into bad loading walls if we didn't duplicate data for straight streaming. Data-sizes has continued to grow and HDD-seek times haven't gotten better due to physics (even if streaming probably has kept up), the assumption isn't too bad considering history. It's a good that they actually revisited it _when they had time_ because launching a game, especially a multiplayer one, will run into a lot of breaking bugs and this (while a big one, pun intended) is still by most classifications a lower priority issue.
• dwroberts 10mo ago
  It seems plausible to me that this strategy was a holdover from the first game, which shipped for PS4 and XBO I don’t know about the Xbox, but on PS4 the hard drive was definitely not fast at all
• jayd16 10mo ago
  You can't bench your finished game before it exists and you don't really want to rock the boat late in dev, either. It was a fundamentally sound default that they revisited. Then they blogged about the relatively surprising difference it happen to make in their particular game. As it turns out the loading is CPU bound anyway, so while the setting is doing it's job, in the context of the final game, it happens to not be the bottle neck. There's also the movement away from HDD and disc drives in the player base to make that the case as well.