9 ms·

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use A

• terafo 3y ago
  And, besides, his solutions aren't really good. Radical solution would be shipping image generating model with the game and generating textures on the fly. That's how all texture-related data goes from being measured in hundreds of gigs to being measured in hundreds of megs. So basically becomes non-issue.
  • circuit10 3y ago
    That would be far, far slower than just downloading the image
    • terafo 3y ago
      8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.
      • circuit10 3y ago
        I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
        • terafo 3y ago
          > I doubt you can do it in 8K at that speed All inference of Stable Diffusion is done in FP16 at best, with multi-step models, and models themselves are highly inefficient. I would be surprised if in May 2024 4090 isn't capable of generating 8k(8192x8192) Stable Diffusion quality image in less than 1 second. And also you wouldn't realistically need to > also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator Because it allows for much more variance in content. I'm a bit sick of seeing tiled textures everywhere TBH. Also virtually all textures would be AI generated in less than 5 years.
          • WithinReason 3y ago
            I think this is exactly right. The next step is generated meshes/textures and generating entire worlds. You can do generation in a smart way so that performance is not an issue. Predict what you will need before you need it. Generally you would almost never need to generate an 8K texture from scratch, you would generate it from a 4K texture you already generated earlier, which would be much faster. You could have specialised asset generation HW on a GPU.
      • dalmo3 3y ago
        > on consumer GPU You make it sound like my electricity bill would be lower by just downloading the bigger files...
• charcircuit 3y ago
  At least for steam all chunks of a game download are compressed and encrypted.
• overthrow 3y ago
  If you compress data, it will be smaller and require less i/o to load, so it will load faster. That's ESPECIALLY true on HDDs, but even on SSDs there are decompression algos that run faster than SSD bandwidth. The uncompressed asset cargo cult is hurting everyone who plays their games, no matter what kind of disk they use.
  • babypuncher 3y ago
    Jedi Survivor isn't shipping on any platforms constrained by an HDD, so it would be odd if they were making any optimizations for spinning rust, especially when they come at the cost of precious SSD space.
    • falcolas 3y ago
      It’s releasing on PC, which could be constrained. It wont be for a majority of players, but HDD setups still exist.
  • deleted 3y ago
    [deleted]
  • blibble 3y ago
    if it's on a modern SSD you can DMA it directly into the GPU no CPU required
    • ByThyGrace 3y ago
      Interesting. How modern are we talking here? Is this a "devil's in the details" kind-of feature?
      • blibble 3y ago
        drives since 2018 or so it's marketed by Microsoft as "direct storage"
    • toast0 3y ago
      Turns out a sata storage controller can do that too, Microsoft has been adding support for that too. GPU decompression is also a thing though, so compression is still a way to save I/O.
• jjoonathan 3y ago
  In 2003, I found out that America's Army (the US Army videogame) had compressed and decompressed all their textures at some point, so they were paying the quality hit for compression and the space hit for RGBA. This was a time when the game still took a day to download, even on a half decent connection, so >200% footprint inflation kinda mattered. The more things change, the more they stay the same!
  • sgtnoodle 3y ago
    I have fond memories of that game. 80% of the time I would play very patiently and carefully. Occasionally, though, I would go Rambo and take out the whole enemy team in 30 seconds.
    • blarg1 3y ago
      I use to pretend to go afk, and while the enemy was doing a little dance in front of me I would shoot them :)
• taeric 3y ago
  This feels wrong. In that, I'd be surprised if they are not using compression. I think folks are just drastically underestimating how many resources are under discussion. And how big they are. Heck, just the geometry is probably far larger than folks anticipate.
  • sudosysgen 3y ago
    It's true. I've seen tons of games use basically uncompressed audio, badly compressed textures, etc... People often repack video games, making programs to compress and decompress them so they are easier to pirate. It's not uncommon to see a 2-3x reduction in size, for bit-identical results.
    • sroussey 3y ago
      Textures are not compressed for size on disk, they are compressed to the format that can be directly loaded into the GPU.
      • sudosysgen 3y ago
        Yes, that's the problem. You can compress them for size at download time, and recompress them to a GPU-optimal format at the time of installation. You may even be able to do it at runtime with modern hardware, shortening loadtimes and sometimes even improving performance indirectly.
        • coffeebeqn 3y ago
          Then you have to have the game reprocess all the assets on the users machine which could easily take an hour
          • sudosysgen 3y ago
            Essentially any modern image compression format is going to be limited, or close to, by storage speed (for decompression). So unless the user has insanely fast internet (and the servers are also able to saturate their connection), it will be faster in the end.
• bentcorner 3y ago
  Agree - OP even links to this[1] PC gamer article which calls out Hitman 3 as having a much smaller size compared to Hitman 2 (and Hitman 3 includes all of Hitman 2). How did they do it? Compression[2]. [1] https://www.pcgamer.com/biggest-game-install-sizes/ https://www.pcgamer.com/biggest-game-install-sizes/ [2] https://www.pcgamer.com/how-hitman-3s-devs-shrank-the-entire-trilogys-install-size-by-over-80gb/ https://www.pcgamer.com/how-hitman-3s-devs-shrank-the-entire...
  • jsheard 3y ago
    There was more to it than just compression. Hitman 2 allowed the player to install/uninstall each level individually, and the way that was implemented meant that each level had its own copy of every asset it contained. Hitman 3 removed that option and simply shipped every level in one unified package, with assets used in multiple levels only stored once.
    • Dylan16807 3y ago
      Hitman 2 had 6-8? locations. They could have pretty easily packed the assets into a hundred files based on use and made each level install a different subset.
• yjftsjthsd-h 3y ago
  > they ACTIVELY duplicate their assets on disk so they load a bit faster on HDD How does that work?
  • sudosysgen 3y ago
    Shorter seek times because the bits are likely to be on the same or nearby tracks.
    • nijave 3y ago
      Yup, basically converting random IO to sequential IO
• babypuncher 3y ago
  There's no excuse for Jedi Survivor not to compress it's assets. Respawn is skipping last-gen consoles with this game. The modern consoles have significantly more CPU power at their disposal, and dedicated hardware for decompressing texture and sound data. There is no excuse for them to be shipping a 150GB game on the PS5 when so many cross-gen games are 40-60% smaller on PS5 than PS4
  • ehaliewicz2 3y ago
    It's easy to say something like that, but as mentioned above, games are generally duct taped together at the literal last minute and it's easy to miss things that, to outsiders, seem obvious. The difference is that we aren't looking at literally every piece of the game and rushing to fix as many bugs as possible, like the devs, we're just looking at the file size.
    • ehaliewicz2 3y ago
      Of course, the other possibility is that they were aware of it, but it was somewhere in a long list of things that they would have liked to fix, but did not have time for.
• ricardobayes 3y ago
  I remember Fallout New Vegas was almost 50 GB because it was a parallel install of all language/localization files. Once you manually deleted the languages you didn't want it went back to like 10 GB.
  • terafo 3y ago
    And there was a hilarious situation with audio in Titanfall, which included 35 gigs of uncompressed audio(whole game took 48 gigs).
    • klodolph 3y ago
      As I recall, this was a purposeful choice. Titanfall came out in 2014. The minimum requirements were a Core 2 Duo at 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on processors dating back to 2006 or so. Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio. I do think they could have figured something out, like compressing the right subset of audio assets, but I just want to give the context for why they made that choice.
      • recursive 3y ago
        That's like 50 hours of uncompressed stereo audio at CD quality. (2×16bit@44.1kHz) Do games really have that much stuff in them these days? I mean, I wouldn't doubt it, but it's surprising.
        • klodolph 3y ago
          Most of it is probably mono, so double the estimate.
        • midnightclubbed 3y ago
          For story based games (with voice acting) absolutely yes. Dozens of story characters, hundreds of supporting npc and all of them talking as either part of your missions or just to fill in the world. Maybe 50 hours is a lot for a non story based game but even then once you factor in all the sounds and all their variations to reduce repetition you might be up there.
• Waterluvian 3y ago
  In terms of attracting engineering talent, how does the video games industry fare? In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it. Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively performant on the hardware they’re running. I can guess what execs will pick when faced with “delay the game or ship a massive bundle.”
  • jjoonathan 3y ago
    It attracts truly extraordinary talent. One of many examples: https://www.youtube.com/watch?v=eviSykqSUUw https://www.youtube.com/watch?v=eviSykqSUUw . There is more technical ambition in a single slide of that project than an entire department at places I've worked. However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep. If I were to dive into the JIRA for any one of these "missing compression" incidents, I wouldn't expect to find an easy win, I would expect to find "missing compression" beneath a pile of much worse bugs like "level 2 crashes on AMD cards." So it goes.
    • Sohcahtoa82 3y ago
      > However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep. Yeah, I've often heard that the advice for people who really want to make games is to get some friends together and try to put out something together as a side gig while you work somewhere boring to pay the bills. SWE at a game studio, especially any AAA game studio, is an awful experience. The expectations are high and the salaries are low because there's a seemingly unlimited bunch of young adults that want to make games. And yes, it takes extraordinary talent. Your code needs to be able to generate a frame in under 16 milliseconds in order to maintain 60 fps. If you want to appease the hardcore players with these crazy 360 hz monitors, you need to get your game loop down to a mere 3 ms. The amount of shortcuts and optimizations needed to pull that off is insane. At that level, you're trying to figure out how to minimize cache misses and branch mispredictions. How many Node devs are thinking about those?
• gabereiser 3y ago
  This is mostly true. The packages of assets for games are compressed but the individual assets are usually not. It’s mostly fancy zip of files with maybe a different header.
  • kevingadd 3y ago
    So they're compressed? What's wrong with compressing packages? That usually produces a better compression ratio by exploiting redundancy cross-file.
    • gabereiser 3y ago
      When optimizing load times, it’s usually console that’s the benchmark. PC is assumed to have faster or equal hardware so they optimize for the lowest denominator. Consoles aren’t PC’s but they are very close. They lack some instructions while gaining others. I can’t speak on current gen consoles but older ones like the Xbox 360, only could handle zip format and at a slow pace. You could show a loading screen while you unpack the data in memory or you can unpack on disk on installation saving some loading time, or it’s all uncompressed on consoles leaving the console to load the data as fast as console do. The issue at hand around install size comes down to the engines, their ability to decompress asset bundles, the fidelity of the assets (8k vs 8k,4k,2k,1024) if no runtime optimization is done. WAV sounds vs OGG vs MP3 vs flac, there’s a lot of shit in those asset packs. You could compress it all like they do, into a folder structure you can index and load from, or you can write that game code to make boss fights cause camera shake, the decisions you make making a game are really all about making the game and little to do with how best to deliver your content. Some texture formats like DXT are already compressed. Raw PNG’s could have compression. All these things are possible if the engine supports it.
      • dontlaugh 3y ago
        It does vary a bit. One exception is that current consoles guarantee fast SSDs and DMA to GPU, which can't be said of PCs.
• kevingadd 3y ago
  Saying "They take hundreds of gigs because they don't use ANY compression." is absurd when you immediately have to caveat that codecs like BC7 are used for texture data, which makes up a large portion of the shipped data. Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it. I don't know why you're trotting out this misinformed BS. You can also simply watch the statistics when Steam is installing a game - the amount written to disk is larger than the amount downloaded over the network, because they also compress assets in flight to reduce bandwidth usage (the most important thing, since raw assets often load faster from an SSD as long as you have available storage.) You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
  • terafo 3y ago
    >immediately have to caveat Yeah, I could've indicated better that I'm talking about on-disk compression of files. >Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it. Not really if we look at 3rd party games. Jedi Survivor is likely not using that since it takes 147 gigs on PS5 and only 139 on Series X. Warzone doesn't use it, Fortnite doesn't use it(despite Epic's acquisition of RAD), Hogwarts Legacy doesn't use it, Elden Ring doesn't use it, FIFA doesn't use it. These are some of the most popular games on the platform. My point was that he is talking about something beyond LZ compression when most games aren't using tools that are available off-the-shelf. >You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them. We have hardware-accelerated decompression in all current gen consoles and GPU decompression on PC. We would be in much better world if everyone used that.
    • bananaboy 3y ago
      > Not really if we look at 3rd party games. I find this extremely unlikely. The packaging tools by default will compress files when building a package. Decompression is transparent and done by the hardware; the game doesn't have to implement anything or link any libraries. You would have to explicitly select "no compression" during packaging. Warzone definitely uses it - I worked on Cold War, Vanguard, and MW2 specifically on streaming and decompression across our supported consoles.
• naikrovek 3y ago
  that IS his point. the problem is not the compression, the problem is the size of the assets. compression can not solve this. alleviate it a bit, yes, but not solve. he's saying that game asset paradigms are headed in the wrong direction rapidly.
  • terafo 3y ago
    The problem is that it wouldn't help. We have good enough solutions right now, but not many are using them. Better compression will be nice, but it will yield less results than adoption of something that is already here.