9 ms·
Helldivers 2 devs slash install size from 154GB to 23GB
- easyThrowaway 11mo agoDid the duplicated files were even used on pc? Like, do you even have such low access to the file system that you can deduce which duplicated instance has a faster access time on a mechanical hard drive?
- arghwhat 11mo agoNot sure if this is what they did, but you can just put all the things you need together sequentially into a single file and rely on the filesystem to allocate contiguous blocks where possible (using the appropriate size hints to help). It's trivial unpack at loading time without any performance impact. A filesystem is by itself just one big "file" acting like a file archive.
- tehbeard 11mo agoIt's not which duplicated instance.... Think of it as I have two packs for levels. Creek.level and roboplanet.level Both use the cyborg enemies, by duplicating the cyborg enemy model and texture data across both files, Only the level file needs to be opened to get all nessecary data for a match. Because modern OS will allow you to preallocate contiguous segments and have auto defrag, you can have it read this level file at max speed, rather than having to stop and seek to go find cyborg.model file because it was referenced by the spawn pool. Engine limitations may prevent other optimisations you think up as a thought exercise after reading this. It's similar to how crash bandicoot packed their level data to handle the slow speed of the ps1 disc drive. As to why they had a HDD optimisation in 2024... Shrugs
- embedding-shape 11mo ago> As to why they had a HDD optimisation in 2024... Shrugs Sadly, Valve doesn't include/publish HDD vs SSD in/on their surveys (https://store.steampowered.com/hwsurvey/?platform=combined https://store.steampowered.com/hwsurvey/?platform=combined) but considering the most popular combo seems to be 16GB RAM, 8GB VRAM, 2.3 Ghz to 2.69 Ghz CPU frequency, I'm getting the impression that the average gaming PC machine isn't actually that beefy. If someone told me the most common setup paired with the previous specs was a small SSD drive for the OS and a medium/large-sized HDD for everything else and I would have believed you. I think us as (software/developer/technology) professionals with disposable income to spend on our hobbies forget how things are for the average person out there in the world.
- phatfish 11mo agoSteam has so many users I'm not sure the average says a lot? If you are just playing Hentai games like most Steam users (j/k, probably) you can do that on any device from the last 10 years. More interesting would be to see the specs for users who bought COD (add other popular franchises as you wish) in the last 2 years. That would at least trim the sample set to those who expect to play recent graphics heavy titles.
- rwmj 11mo ago23GB is supposed to be "slim"?!
- VoodooJuJu 11mo ago[dead]
- bilekas 11mo agoIn this day and age it's a gift to only be ~23GB.. I'm reminded of the old days when you literally didn't have the space so had to get creative, now any kind of space optimization isn't even considered.
- 0cf8612b2e1e 11mo agoNot true at all on my PlayStation. Just a few games with 100GB+ install size can quickly put you into a space juggling scenario.
- onli 11mo agoYes. High resolution textures take up a lot of space. Have a look at HD texture mods for skyrim for example. 23GB is more in line with a game from a few years ago, so this really is slim for a modern game with modern graphics.
- mfro 11mo agoHave you played a big budget video game released in the last 10 years? It’s pretty standard to reach upwards of 60GB.
- jwagenet 11mo agoGTAV had a 60GB install size over a decade ago.
- phoronixrly 11mo agoI do love rich soundtracks with high quality compression, and textures that look crisp on 4k. And also games with 100+ hours of single-player campaign.
- rincebrain 11mo agoI've been really curious precisely what changed, and what sort of optimization might have been involved here. Because offhand, I know you could do things like cute optimizations of redundant data to minimize seek time on optical media, but with HDDs, you get no promises about layout to optimize around... The only thing I can think of is if it was literally something as inane as checking the "store deduplicated by hash" option in the build, on a tree with copies of assets scattered everywhere, and it was just nobody had ever checked if the fear around the option was based on outcomes. (I know they said in the original blog post that it was based around fears of client performance impact, but the whole reason I'm staring at that is that if it's just a deduplication table at storage time, the client shouldn't...care? It's not writing to the game data archives, it's just looking stuff up either way...)
- alias_neo 11mo agoI'm not entirely clear what you're trying to say, but, my understanding is that they simply put lots of copies of files in lots of places like games have done for a long time, in the hopes it would lower seek times on HDDs for those players who use them. They realised, after a lot of players asking, that it wasn't necessary, and probably had less of an impact than they thought. They removed the duplicates, and drastically cut the install size. I updated last night, and the update alone was larger than the entire game after this deduplication run, so I'll be opting in to the Beta ASAP. It's been almost a decade since I ran spinning rust in a desktop, and while I admire their efforts to support shitty hardware, who's playing this on a machine good enough to play but can't afford £60 for a basic SSD for their game storage?
- deleted 11mo ago[deleted]
- eska 11mo agoHDDs also have a spinning medium and a read head , so the optimization is similar to optical media like CDs. Let’s say you have UI textures that you always need, common player models and textures, the battle music, but world geometry and monsters change per stage. Create an archive file (pak, wad, …) for each stage, duplicating UI, player and battle music assets into each archive. This makes it so that you fully utilize HDD pages (some small config file won’t fill 4kb filesystem pages or even the smaller disk sectors). All the data of one stage will be read into disk cache in one fell swoop as well. On optical media like CDs one would even put some data closer to the middle or on the outer edge of the disc because the reading speed is different due to the linear velocity. This is an optimization for bandwidth at the cost of size (which often wasn’t a problem because the medium wasn’t filled anyway)
- djmips 11mo agoI did similar work on a game a long time ago and it took over a month to slim it down to 1/4 of the size but in this case 'at runtime' - the producer wasn't impressed. It looked exactly the same. I wonder if they had any pushback.
- geerlingguy 11mo agoPossibly a similar process to when you go into an AWS account, and find dozens of orphaned VMs, a few thousand orphaned disk volumes, etc., saving like $10k/month just deleting unused resources.
- alias_neo 11mo agoWe've all been there Jeff. In this case, I don't think it was forgetfulness; unlike us, they have an excuse and they were trying to optimise for disk seek times. Anyway, I've got a half-dozen cloud accounts I need to go check for unused resources waves.
- perching_aix 11mo agoIt's not a case of forgotten data, it's duplicated for access time reasons, like with optical media. It follows in the footsteps of trading in storage for less compute and/or better performance. An opposite approach in the form of a mod for Monster Hunter: Wilds recently made it possible [0] for end-users to decompress all the game textures ahead of time. This was beneficial there, because GPU decompression was causing stalls, and the trading in of compute for less storage resulted in significantly worse performance. [0] https://youtu.be/AOxLV2US4Ac https://youtu.be/AOxLV2US4Ac
- snet0 11mo ago> With their latest data measurements specific to the game, the developers have confirmed the small number of players (11% last week) using mechanical hard drives will witness mission load times increase by only a few seconds in worst cases. Additionally, the post reads, “the majority of the loading time in Helldivers 2 is due to level-generation rather than asset loading. This level generation happens in parallel with loading assets from the disk and so is the main determining factor of the loading time.” It seems bizarre to me that they'd have accepted such a high cost (150GB+ installation size!) without entirely verifying that it was necessary! I expect it's a story that'll never get told in enough detail to satisfy curiosity, but it certainly seems strange from the outside for this optimisation to be both possible and acceptable.
- jsheard 11mo agoFrom a business perspective the disk footprint is only a high cost if it results in fewer sales, which I doubt it does to any significant degree. It is wasteful, but can see why optimization efforts would get focused elsewhere.
- code_for_monkey 11mo agoI think certain games dont even bother to optimize the install size so that you cant fit other games on the hard drive, I think COD games are regularly hundreds of gigs
- snet0 11mo agoI've often seen people mention that one reason for games like Call of Duty being so enormous is optimising for performance over storage. You'd rather decompress textures/audio files at install-time rather than during run-time, because you download/install so infrequently.
- justsomehnguy 11mo agoIt's amusing we had a 2x-10x compression and negligible perfomance hit on 486 with DBLSPACE yet in 2020+...
- fleabitdev 11mo agoBack of the envelope, in the two years since the game was released, this single bug has wasted at least US$10,000,000 of hardware resources. That's a conservative estimate (20% of people who own the game keep it installed, the marginal cost of wasted SSD storage in a gaming PC is US$2.50 per TB per month, the install base grew linearly over time), so the true number is probably several times higher. In other words, the game studio externalised an eight-figure hardware cost onto their users, to avoid a five-to-six-figure engineering cost on their side. Data duplication can't just be banned by Steam, because it's a legitimate optimisation in some cases. The only safeguard against this sort of waste is a company culture which values software quality. I'm glad the developers fixed this bug, but it should never have been released to users in the first place.
- ga2mer 11mo ago>Data duplication can't just be banned by Steam Steam compresses games as much as possible, so in the case of Helldivers 2, you had to download between ~30 and ~40 GB, which was then unpacked to 150 GB (according to SteamDB[0]) [0] https://steamdb.info/app/553850/depots/ https://steamdb.info/app/553850/depots/
- embedding-shape 11mo agoMakes sense, initial claim was that HD2 size was mainly because of duplicated assets, and any compression worth it's salt would de-duplicate things effectively.
- fleabitdev 11mo agoIn this case, the bug was 131 GB of wasted disk space after installation. Because the waste came from duplicate files, it should have had little impact on download size (unless there's a separate bug in the installer...) This is why the cost of the bug was so easy for the studio to ignore. An extra 131 GB of bandwidth per download would have cost Steam several million dollars over the last two years, so they might have asked the game studio to look into it.
- 11mo ago
- CafeRacer 11mo agoIn other news - "Call of Duty installer now takes additional 131GB of space on the disk"
- dv_dt 11mo agoI stopped playing CoD mainly because I was tired of juggling disk space to try to play it even casually. It's surprising to me that game publishers have ignored this as some checklist requirement to stay below.
- PunchyHamster 11mo agoThey have a big player base that plays just that and not much more
- miohtama 11mo agoAFAIK Helldivers 2 runs some really old engine which was discontinued many years ago. Not "state of the art." It's also a title that shows you can have a really good game without the latest tech.
- thatguy0900 11mo agoYou can, but on the other hand they've been battling bugs from it with every release. The game is notorious for breaking things constantly. I played for quite a while and the sound engine was always awful with things not in your line of sight frequently not making any noise at all, and every month or two a new major bug relating to host client desyncing is found out by the community who then has to have big campaigns badgering the devs to notice and fix it. Very fun game still but if they had started with a supported engine a lot of stuff would probably work way better
- mrguyorama 11mo agoYeah but what evidence is there that what you describe is an engine problem rather than an Arrowhead can't software engineer problem? As others point out, several other good games were made on this engine, and they do not have such abysmal engineering quality. Vermintide 2 is on the same engine and very much doesn't have problems with constant bugs and regressions. Arrowhead struggled to fix the lock on for the missile launcher for months, and then claimed that "raycasts are hard* and I just don't think they should be for a game dev. Several patches seem to show that they just do not test at all the release build, and that they don't seem to have functioning version control to keep from breaking things they've already fixed. >and every month or two a new major bug relating to host client desyncing is found out by the community who then has to have big campaigns badgering the devs to notice and fix it. This is not a game engine problem. Does fire damage properly work across multiplayer yet?
- panja 11mo agoTo be fair, they started working on the game before it was discontinued. I'm sure someone made a decision that it made more sense to continue on without support rather than start from scratch.
- gethly 11mo agoif your game takes 154 GB of space, you should never be able to touch a computer ever again.
- panja 11mo agoI agree more can be done to shave game sizes but space is cheap these days so that's probably why it hasn't been a focus.
- Sohcahtoa82 11mo agoThe same argument gets used when concerning CPU performance when writing software with 10+ levels of abstraction and now your CPU gets pegged when on a semi-active Discord channel. Who cares that it takes a million clock cycles to process a single message? People are running 5 Ghz CPUs, so that's only 200 ms, right?
- cabirum 11mo agoWhat if.. the management made a request to make the game take more space than the previous release? So everyone could see just how much content there is and how much better everything is. I mean, the developers cannot be that incompetent while being able to ship a high quality product.
- deleted 11mo ago[deleted]
- ChrisArchitect 11mo ago[dupe] https://news.ycombinator.com/item?id=46131518 https://news.ycombinator.com/item?id=46131518
- Kiro 11mo agoNever a dupe when you say it is. You need another word.
- ChrisArchitect 11mo agoSource: https://store.steampowered.com/news/app/553850/view/491583942944621371 https://store.steampowered.com/news/app/553850/view/49158394... (https://news.ycombinator.com/item?id=46131518 https://news.ycombinator.com/item?id=46131518)
- haritha-j 11mo agoI recently downloaded Hunt showdown. I think it was around 70 gigs. About a month later, I had to update it. The download was the same size. I think they literally just overrode the entire game because they were too lazy to update it properly.
- zelphirkalt 11mo agoI observed this too with some other games. Really annoying, when you need to redownload tens of gigabytes , because they cannot be arsed to put a proper updater. Things that were solved by most games way before Steam became even big.
- thunderfork 11mo ago[dead]
- darknavi 11mo agoIt's true for most of the modern web. Hardware and internet connections have come reliable and fast enough for many people not to care. Unfortunately there are still people on metered/slow connections that really do care still.
- haritha-j 11mo agoThis is true. I mainly live in a country with unlimited, fast internet but also spend some time in a country with limited and slower internet. But even when i have fast internet, downloading 70 gigs when I just want to play a game for an hour is very disruptive. I think a lot of this download and replace vs efficiently patch argument is predicated on the premise that the update itself was necessary. In many cases such as Forza horizon, I've noticed that the update is just to push a bunch of paywalled content / advertisements for new paywalled content that most people won't even play. I miss the days when I bought the DVD and that was the whole game.
- KomoD 11mo agoThis is why I stopped playing COD Warzone, every time I wanted to play I had to download the entire game again just because of some small update.
- kakacik 11mo agoDevs went full fitgirl (repack site which reduces sizes of cracked releases significantly via similar approaches)
- l33tfr4gg3r 11mo agoWarframe also did this relatively recently, though perhaps not quite as aggressive a reduction as Helldivers, but still. https://www.pcgamer.com/call-of-duty-take-note-warframe-is-reducing-its-install-size-by-up-to-15gb/ https://www.pcgamer.com/call-of-duty-take-note-warframe-is-r...
- donatj 11mo ago> "It's no surprise to see modern AAA games occupying hundreds of gigabytes of storage these days" Is it not? I've genuinely never understood it! I used to do a little bit of level building for IdTech3 games back in the day but it's been 20 years. I'm not totally ignorant of what's involved, just mostly ignorant. I really want to know though, what is all that data!? Textures? In particular I find the massive disparity between decently similar games interesting. Indiana Jones and the Great Circle takes something like 130gb on my Xbox, whereas Robocop: Rogue City takes something like 8gb. They have similar visual fidelity, I would say Robocop might have a little bit of a lead, but Indiana Jones has fancier dynamic lighting. At 130gb though, I almost could have streamed my entire playthrough of the game at 4k and came out on top.
- zelphirkalt 11mo agoThese days I am just assuming it is textures. Textures for graphic quality, that I don't need on my merely full HD screens. Makes me usually not even bother buying, let alone installing such a game. Even 23 GB is still tons. When I compare that with how much fun I can have playing games that focus on gameplay instead of graphics, I would rather safe money, than spending it and having to go through my files and delete things, because I need more space for a game.
- WreckVenom 11mo ago> Even 23 GB is still tons Maybe back in 2007. I have games from that period that would probably be 30GB. 23GB is pretty good. BTW, HellDivers 2 has pretty decent gameplay. Lots of depth to the game. Unfortunately they've decreased the difficulty as of late.
- Sohcahtoa82 11mo ago> I really want to know though, what is all that data!? Textures? Yeah. A single 4096x4096 texture is 16 megapixels. At 8 bits per channel with potentially 4 channels, that's 64 MB for a single texture uncompressed.
- 11mo ago
- marknutter 11mo agoNow all they need to do is remove the kernel-level anti-cheat
- unixnight 11mo agoIt was legit faster to delete and redownload this game than update it since steam considered my SSD too full (WITH 200 GIGS FREE) to download the files to said SSD, instead opting to use my SLOWEST HDD as the cache drive for the download. It would then proceed to download the update in 5 minutes and spend 8 HOURS UPDATING. A full download of the game? 10 minutes. Glad to see this update. I hope more games follow suit
- PunchyHamster 11mo agoI remember how pre-loading some games turned out to be slower for me because downloading it at launch meant decrypting it directly from the network, but decrypting files on drive that are already encrypted doubled the IO and that was too much random access for my hard drive
- Thev00d00 11mo agoOn size limited platforms like steam deck and friends this is a huge W
- nopcode 11mo agoNixxes has such a good reputation in my book that their name immediately removes any porting-fears when I see they are responsible for a release.
- Artoooooor 11mo ago154GB? That's why I almost don't "buy" AAA games. This is much bigger cost than money for that game.
- Forgeties79 11mo agoI don’t disagree that it’s too large at 154GB but the decision (at least to me) also made more sense at a time when storage, especially external storage, had never been cheaper. That only changed like 6-8 weeks ago. Prior to that getting a 4TB external SSD was like what? $150-$200? $120 for a 2TB nvme? You could get 1TB SSD’s for like $65 if you looked for them.
- maccard 11mo ago> This is much bigger cost than money for that game Is it? A 1TB SSD costs about the same as the game does...
- Havoc 11mo agoThey had the size at 7x duplication to save load times for hdds? WTH
- pixl97 11mo agoYou may misunderstand how poor hdd IOPs are and how much latency 4k random reads induce. Random IO can be 100 to 200 times slower than sequential reads on small file workloads.
- Havoc 11mo agoIt literally says in the article the effect was inconsequential > small number of players (11% last week) using mechanical hard drives will witness mission load times increase by only a few seconds in worst cases.
- pixl97 11mo agoIn this workload, it happened to be, in a lot of workloads it is not at all. This is quite likely because over time media sizes have increased greatly. Microsoft research on operating systems shows that contiguous chunks at 64MB don't really show any IOPS penalty in practical usage scenarios. I work on workloads that work with thousands to millions of small files. 1k to 1MB. I can promise you that the IO effect is massive, with shortcuts like working directly on archive files/zips where they are read and written at once saves massively on hardware costs as IOPS tend to be expensive. This is with fast NVMe disks, no spinning platters involved. Also small files massively increase write amplification on SSDs. There probably is significant savings in deduping 'tiny' files, but the moment you get past a particular threshold it drops off, and the vast majority of game assets have increased over that size.
- canucker2016 11mo agoBack in 2014, Titanfall's disk space was 75% UNCOMPRESSED audio (35GB of 48GB) for the gamers with only dual-core CPUs. from https://www.escapistmagazine.com/titanfall-dev-explains-the-games-35-gb-of-uncompressed-audio/ https://www.escapistmagazine.com/titanfall-dev-explains-the-... “On a higher PC it wouldn’t be an issue. On a medium or moderate PC, it wouldn’t be an issue, it’s that on a two-core [machine] with where our min spec is, we couldn’t dedicate those resources to audio.”
- superxpro12 11mo agoIsn't COD infamous for uncompressed assets as well, or am I misremembering?
- bigyabai 11mo agoAh, that explains how the R1Delta guys were able to cut the size on disk to less than 20gb! https://github.com/r1delta/r1delta https://github.com/r1delta/r1delta
- rasz 11mo agoIt was BS considering countless other games having no problem with sound. Decoding something like Opus takes ~30MHz of a single CPU core[1], meaning even an unreasonable situation of decoding 16 simultaneous uninterrupted 128Kbit Stereo streams would only eat half of one core. [1] iPod Classic (1998 era ARM9) decodes 128 kbps stereo Opus at ~150% real time at stock cpu frequency. Opus is not the lightest choice either https://www.rockbox.org/wiki/CodecPerformanceComparison#ARM https://www.rockbox.org/wiki/CodecPerformanceComparison#ARM
- rpmisms 11mo agoI bet this was a checkbox in the Autodesk Stingray engine, and they didn't have the expertise on hand to unfuck the game files when they unchecked it. Looks like it was easier than expected with a couple greybeards consulting. Awesome win for everyone involved.
- tehjoker 11mo agoI wonder if they will port this to PS5 as well. It would be wonderful to recover some space.
- Narishma 11mo agoPS5 version doesn't suffer from this issue. It's already much smaller.
- bgro 11mo agoThis is yet another thing I’ve been saying is technically feasible in a large variety of games immediately since it started happening to game sizes around 2015. I’ve had nothing except people screaming at me that I’m wrong and that this is just how modern games have to be. There’s always some gaslight argument about how I’m wrong and actually the devs are in fact geniuses because of some genius trick about a variety of systems significantly bloating the size. Rather than try to look at the problem, I get continuously attacked with retconned reasons trying to justify every decision going into this bloat. This is exactly the same dev retcon trend train we saw with Cloud (“No, wrong, it’s not computers. It’s servers. Totally different. And it’s not even servers, it’s cloud.”) and MicroServices (“There’s literally no reason to ever not use microservices unless you’re a legacy dev. Especially on The Cloud, which is different and not comparable in any way to a a self hosted server.”) I’m 3/3 so far. Waiting for you guys to still figure out leetcode and how this actually captures the inverse of the thing you’re trying to account for. But I already know you’re going to retcon that argument too and say ackshully leetcode DOES work to hire developers because we want to hire people who are wealthy enough to have time to memorize questions and answers. We always knew developers would have AI (lol) and this paper that retroactively applies a matching hypothesis that trends with leetcode is actually what smart developers like me knew all along.”
- wackget 11mo ago> made possible with support from Nixxes Software, best known for developing high-quality PC ports of Sony’s biggest PlayStation titles Now if only Nixxes would stop using Sony's stupid proprietary audio system in PC games, I'd be happy. Every Sony game causes my sound card to freak out and I lose the ability to control anything - even volume. My surround speakers stop working completely and I'm effectively forced into accepting mono audio.
- wackget 11mo agoI just checked out this game on Steam and was pretty excited to buy it until I read that it requires a kernel-level anticheat which acts as a rootkit: https://en.wikipedia.org/wiki/NProtect_GameGuard https://en.wikipedia.org/wiki/NProtect_GameGuard Why on earth is any self-respecting person installing this on their machine?
- littlekey 11mo agoBecause they want to play the game and don't know or care what a "kernel-level anticheat which acts as a rootkit" is. Which probably makes up 99.9% of their userbase.
- mdni007 11mo agoIt's almost 2026, you're supposed to keep gaming and personal PCs separate.
- braiamp 11mo agoFYI, it works on Linux, so as far as I'm aware, it only seems to care about you flying around.
- user____name 11mo agoPeople still use HDDs in consumer PCs?
- miladyincontrol 11mo agoGreat change, game is loading faster than ever least on ssds. Who knew letting the OS cache files rather than fail juggling several copies of the same data had benefits. Curious some how it affected hdd users with ram in excess beyond the minimum.
- hiccuphippo 11mo agoDoes this mean people with SSDs are wasting space, expensive space, for the sake of people with HDDs? I hope other devs take notice. Maybe they could even keep two separate builds in case the loading time is too much for HDDs.