3 ms·

Arrowhead probably deserves more love for breaking the norm but I think it's overshadowed by people finding out for the first time the reason HDDs are so common

Arrowhead probably deserves more love for breaking the norm but I think it's overshadowed by people finding out for the first time the reason HDDs are so common in gaming setups is companies have been blindly shaving a few seconds off HDD load time off at the cost of 7x the disk space. If it had been more well known this was the cause of game bloat before then this probably would have been better received. Still, Arrowhead deserves more credit both for testing and breaking the norm as well as making it a popular topic.

abtinf 11mo ago
Part of what makes this outrageous is that the install size itself is probably a significant part of the reason to install the game on an HDD. 154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive. Is there a name for the solution to a problem (make size big to help when installed on HDD) in fact being the cause of the problem (game installed on HDD because big) in the first place?
- consp 11mo ago
  Can any games these days be reliably ran on hdd's with max 200mb/s throughout (at best)? Or does everyone get a coffee and some cookies when a new zone loads? Even with this reduction that will take a while. I thought all required ssd's now for "normal" gameplay.
  - kbolino 11mo ago
    Until you get to super-high-res textures and the like, the throughput isn't nearly as important as the latency. At 200 MB/s the way hard drives usually measure it, you're able to read up to 390,625 512-byte blocks in 1 second, or to put it another way, a block that's immediately available under the head can be read in 2.56 microseconds. On the other hand, at 7200 RPM, it takes up to 8.33 milliseconds to wait for the platter to spin around and reach a random block on the same track. Even if these were the only constraints, sequentially arranging data you know you'll need to have available at the same time cuts latency by a factor of about 3000. It's much harder to find precise information about the speed of the head arm, but it also usually takes several milliseconds to move from the innermost track to the outermost track or vice versa. In the worst case, this would double the random seek time, since the platter has to spin around again because the head wasn't in position yet. Also, since hard drives are so large nowadays, the file system allocators actually tend to avoid fragmentation upfront, leading to generally having few fragments for large files (YMMV). So, the latency on a hard drive can be tolerable when optimized for.
    - wtallis 11mo ago
      > On the other hand, at 7200 RPM, it takes up to 138 microseconds to wait for the platter to spin around and reach a random block on the same track. You did the math for 7200 rotations per second, not 7200 rotations per minute = 120 rotations per second. In gaming terms, you get at most one or two disk reads per frame, which effectively means everything has to be carefully prefetched well in advance of being needed. Whereas on a decade-old SATA SSD you get at least dozens of random reads per frame.
      - kbolino 11mo ago
        Fixed!
- jayd16 11mo ago
  "Self fulfilling prophecy" perhaps?
- KronisLV 11mo ago
  > 154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive. I think War Thunder did it the best: * Minimal client 23 GB * Full client 64 GB * Ultra HQ ground models 113 GB * Ultra HQ aircraft 92 GB * Full Ultra HQ 131 GB For example, I will never need anything more than the full client, whereas if I want to play on a laptop, I won't really need more than the minimal client (limited textures and no interiors for planes). The fact that this isn't commonplace in every engine and game out there is crazy. There's no reason why the same approach couldn't also work for DLCs and such. And there's no reason why this couldn't be made easy in every game engine out there (e.g. LOD level 0 goes to HQ content bundle, the lower ones go into the main package). Same for custom packages for like HDDs and such.
nopurpose 11mo ago
My immediate question is that if all of that was on-disk data duplication, why did it affected download size? Can't small download be expanded into optimal layout on the client side?
- ahartmetz 11mo ago
  Sure it can - it would need either special pre- and postprocessing or lrzip ("long range zip") to do it automatically. lrzip should be better known, it often finds significant redundancy in huge archives like VM images.
- braiamp 11mo ago
  It didn't. They downloaded 43 GB instead of 152 GB, according to SteamDB: https://steamdb.info/app/553850/depots/ https://steamdb.info/app/553850/depots/ Now it is 20 GB => 21 GB. Steam is pretty good at deduplicating data in transit from their servers. They are not idiots that will let developers/publishers eat their downstream connection with duplicated data. https://partner.steamgames.com/doc/sdk/uploading#AppStructureBestPractices https://partner.steamgames.com/doc/sdk/uploading#AppStructur...
  - myself248 11mo ago
    Furthermore, this raises the possibility of a "de-debloater" that HDD users could run, which would duplicate the data into its loading-optimized form, if they decided they wanted to spend the space on it. (And a "de-de-debloater" to recover the space when they're not actively playing the game...) The whole industry could benefit from this.
    - nomel 11mo ago
      > to recover the space when they're not actively playing the game This would defeat the purpose. The goal of the duplication is to place the related data physically close, on the disk. Hard links, removing then replacing, etc, wouldn't preserve the physical spacing of the data, meaning the terrible slow read head has to physically sweep around more. I think the sane approach would be to have a HDD/SDD switch for the file lookups, with all the references pointing to the same file, for SDD.

abtinf 11mo ago

Part of what makes this outrageous is that the install size itself is probably a significant part of the reason to install the game on an HDD. 154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive. Is there a name for the solution to a problem (make size big to help when installed on HDD) in fact being the cause of the problem (game installed on HDD because big) in the first place?

consp 11mo ago
Can any games these days be reliably ran on hdd's with max 200mb/s throughout (at best)? Or does everyone get a coffee and some cookies when a new zone loads? Even with this reduction that will take a while. I thought all required ssd's now for "normal" gameplay.
- kbolino 11mo ago
  Until you get to super-high-res textures and the like, the throughput isn't nearly as important as the latency. At 200 MB/s the way hard drives usually measure it, you're able to read up to 390,625 512-byte blocks in 1 second, or to put it another way, a block that's immediately available under the head can be read in 2.56 microseconds. On the other hand, at 7200 RPM, it takes up to 8.33 milliseconds to wait for the platter to spin around and reach a random block on the same track. Even if these were the only constraints, sequentially arranging data you know you'll need to have available at the same time cuts latency by a factor of about 3000. It's much harder to find precise information about the speed of the head arm, but it also usually takes several milliseconds to move from the innermost track to the outermost track or vice versa. In the worst case, this would double the random seek time, since the platter has to spin around again because the head wasn't in position yet. Also, since hard drives are so large nowadays, the file system allocators actually tend to avoid fragmentation upfront, leading to generally having few fragments for large files (YMMV). So, the latency on a hard drive can be tolerable when optimized for.
  - wtallis 11mo ago
    > On the other hand, at 7200 RPM, it takes up to 138 microseconds to wait for the platter to spin around and reach a random block on the same track. You did the math for 7200 rotations per second, not 7200 rotations per minute = 120 rotations per second. In gaming terms, you get at most one or two disk reads per frame, which effectively means everything has to be carefully prefetched well in advance of being needed. Whereas on a decade-old SATA SSD you get at least dozens of random reads per frame.
    - kbolino 11mo ago
      Fixed!
jayd16 11mo ago
"Self fulfilling prophecy" perhaps?
KronisLV 11mo ago
> 154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive. I think War Thunder did it the best: * Minimal client 23 GB * Full client 64 GB * Ultra HQ ground models 113 GB * Ultra HQ aircraft 92 GB * Full Ultra HQ 131 GB For example, I will never need anything more than the full client, whereas if I want to play on a laptop, I won't really need more than the minimal client (limited textures and no interiors for planes). The fact that this isn't commonplace in every engine and game out there is crazy. There's no reason why the same approach couldn't also work for DLCs and such. And there's no reason why this couldn't be made easy in every game engine out there (e.g. LOD level 0 goes to HQ content bundle, the lower ones go into the main package). Same for custom packages for like HDDs and such.

nopurpose 11mo ago

My immediate question is that if all of that was on-disk data duplication, why did it affected download size? Can't small download be expanded into optimal layout on the client side?

ahartmetz 11mo ago
Sure it can - it would need either special pre- and postprocessing or lrzip ("long range zip") to do it automatically. lrzip should be better known, it often finds significant redundancy in huge archives like VM images.
braiamp 11mo ago
It didn't. They downloaded 43 GB instead of 152 GB, according to SteamDB: https://steamdb.info/app/553850/depots/ https://steamdb.info/app/553850/depots/ Now it is 20 GB => 21 GB. Steam is pretty good at deduplicating data in transit from their servers. They are not idiots that will let developers/publishers eat their downstream connection with duplicated data. https://partner.steamgames.com/doc/sdk/uploading#AppStructureBestPractices https://partner.steamgames.com/doc/sdk/uploading#AppStructur...
- myself248 11mo ago
  Furthermore, this raises the possibility of a "de-debloater" that HDD users could run, which would duplicate the data into its loading-optimized form, if they decided they wanted to spend the space on it. (And a "de-de-debloater" to recover the space when they're not actively playing the game...) The whole industry could benefit from this.
  - nomel 11mo ago
    > to recover the space when they're not actively playing the game This would defeat the purpose. The goal of the duplication is to place the related data physically close, on the disk. Hard links, removing then replacing, etc, wouldn't preserve the physical spacing of the data, meaning the terrible slow read head has to physically sweep around more. I think the sane approach would be to have a HDD/SDD switch for the file lookups, with all the references pointing to the same file, for SDD.