14 ms·

Faster LZ is not the answer to 150-250 GB video game downloads

191 points by ingve 4y ago

crummy 4y ago
https://acronyms.thefreedictionary.com/LZ https://acronyms.thefreedictionary.com/LZ I think the right definition is this one: Lempel-Ziv (dictionary-based lossless data compression)
- thfuran 4y ago
  It is
adgjlsfhk1 4y ago
I really want to see someone make a renderer that can work on compressed textures. No need to decompress if you can just read the data on the fly.
- aappleby 4y ago
  Renderers have been using compressed textures for literal decades. Not JPG, alas, because the compression formats have to support random access to individual texels.
- corysama 4y ago
  https://www.reedbeta.com/blog/understanding-bcn-texture-compression-formats/ https://www.reedbeta.com/blog/understanding-bcn-texture-comp... BC1-5 have been standard practice since before the original XBox. I think when people here are talking about "uncompressed assets" they mean "not applying variable-rate zip style compression on top of BCn compression."
- tmilard 4y ago
  I use this one. Web compliant. https://www.babylonjs.com/ https://www.babylonjs.com/
bick_nyers 4y ago
Faster internet and cheaper storage is the answer :) Or at the very least just offer two different downloads of the game, one with lossless textures and one with lossy textures...
- topspin 4y ago
  Guild Wars 2 and others solve this with progressive downloads. You may start playing after a fraction of the content is downloaded. There is effectively no downside: new players can't access most of the content anyhow. It also applies to updates: you can start playing after a big update with a partial download. The background process can be safely interrupted and resumed without glitches.
  - arthur2e5 4y ago
    War Thunder kind of has that, except that the "progressive" bit means you need to download the full thing to progress in the game beyond a certain point. Enlisted, which uses the same engine, is fine though.
    - topspin 4y ago
      The design of ArenaNet's GW/GW2 has always impressed me. 99.4% of all of the game is contained in one file: GW2.DAT, which is (currently) 66.7 GB. I/O for updates is extremely efficient. I've seen many games create a vast file system mess, and they're forever plagued by "corruption" or something, complete with "fix" tools, "cache" clearing push-ups and players reinstalling after big updates that don't install correctly for whatever reason. Somehow ArenaNet surrounded this whole problem and solved it optimally.
      - kevingadd 4y ago
        A massive amount of engineering went into making it work and some of the stuff we did was downright evil :) in gw1, a lot of data was compiled into the exe as c constants so that we wouldn't have to do a bunch of file management and I/o to locate and load it. We ended up hitting limitations in the C compiler.
terafo 4y ago
He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't use any compression beyond something like BC7. And some games are not only without any compression, they ACTIVELY duplicate their assets on disk so they load a bit faster on HDD. And these builds end up on modern consoles that have ludicrously fast SSDs.
- terafo 4y ago
  And, besides, his solutions aren't really good. Radical solution would be shipping image generating model with the game and generating textures on the fly. That's how all texture-related data goes from being measured in hundreds of gigs to being measured in hundreds of megs. So basically becomes non-issue.
  - circuit10 4y ago
    That would be far, far slower than just downloading the image
    - terafo 4y ago
      8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.
      - circuit10 4y ago
        I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
rektide 4y ago
Richg42 is great & his pieces are fun to read. I do think in this case he misses a bit of the point, goes too far: > They are designed for fast loading and streaming off SSD's. Who cares about saving off a few hundred ms (or a second) when it takes hours or days to download the product onto the SSD? Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait. But we load textures many times if we keep playing a game. Rich's overall point remains, but I think some balance of optimizing load speed is also very appreciated & very necessary.
- overthrow 4y ago
  That can be solved on first load with a texture cache. Shader caches are pretty universal at this point. Texture caches aren't (afaik), but I think they should be.
  - rektide 4y ago
    So now we would have faster download & faster load, but we'd have significantly bugger hard drive space usage than we have? That also seems like a concern.
- jeroenhd 4y ago
  > Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait. Are there any statistics on that? Because I have deleted and redownloaded games a whole bunch of times because you can't fit more than two or three games on a single SSD anymore. I've played large games off of hard drives because I simply didn't want to delete 240GB of other files off of my main drive. The game experience was clearly much worse than the developers intended, but their space demands are unreasonable. Especially considering I'm only playing at 1080p! Give me a 1080p installer that shaves off 75% of the texture size and you'll win more than a few milliseconds because the stupid thing can actually fit on my drive and doesn't need to come off the internet again.
hellotomyrars 4y ago
Game size is a perennial talking point and some games earn it and some games don’t. There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall) Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think this is the smart way to do it. Examples of games shrinking I are hard to come by but special credit to IO interactive and Hitman 3. The install size with all the content of the previous games included (If you own them) is less than the install size of the first game. That is impressive and commendable. Games are weird and some people will look at the file size and think it means the game is better, just the way some people think games with more hours are better. The psychology of people who play video games is weird. Duplication of assets on disk was a very weird byproduct of HDD file access conventions, but we shouldn’t need that anymore. There is still a shrinking, but valid crossover window between devs being able to assume all storage is solid state. I think things will get better but the debate of file size vs average/expects storage capacity vs Network speed will never end but there is certainly work to be done.
- RajT88 4y ago
  The .kkreiger devs really showed how far you can go to reduce download size: https://en.wikipedia.org/wiki/.kkrieger https://en.wikipedia.org/wiki/.kkrieger https://www.youtube.com/watch?v=ya_MUKc343U https://www.youtube.com/watch?v=ya_MUKc343U It is dated, but looks pretty good by 2004 standards.
  - thequux 4y ago
    While I'm all for reducing download sizes, perhaps lekktor[1] is a bridge too far. [1]: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for-madmen/ https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
    - Dylan16807 4y ago
      Sure. Focus on the procedural textures and see where that gets you.
KMnO4 4y ago
Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper. I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene. Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a lot of the work involved in making things pretty. You don't need to generate or even understand displacement maps if you can just add more polygons to the model. You don't need to billboard the mountain way in the distance if Unreal does it on the fly. Who cares if that means including a 250mb mesh in your assets, versus a 5mb HDRI. So yes, producers have either become lazy or are simply unable to meet the demands of the hardware by coming up with better algorithms. That's unlikely to change until either hardware stops advancing and either A) can't run the latest games or B) games become too large that they're impossible to download. [0]: https://www.gamedeveloper.com/programming/sponsored-feature-light-it-up-quake-wars-gets-ray-traced https://www.gamedeveloper.com/programming/sponsored-feature-... [1]: https://docs.unrealengine.com/5.0/en-US/nanite-virtualized-geometry-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/nanite-virtualized-g... [2]: https://twitter.com/mariobrothblog/status/1636040893764362241 https://twitter.com/mariobrothblog/status/163604089376436224...
- terafo 4y ago
  > and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't work with open-world games, especially if they have different times of day. And it doesn't work in highly dynamic scenes. We basically hit the end of the road for scaling graphics without RT. And now we have hardware that can do realtime path-traced rendering of major AAA titles, and it looks miles better than anything smoke and mirrors could've ever delivered.
LinuxBender 4y ago
I recall a lot of hype more than a decade ago about all the textures going away and being replaced with math formulas to generate everything on the fly. Games were supposed to get dramatically smaller. What ever happened to that?
- db48x 4y ago
  Economics. Supply and demand. To create good procedural art you have to hire software engineers with artistic talent, which are more expensive than the texture and model artists you can find by the hundreds coming out of art schools.
- babypuncher 4y ago
  It is an order of magnitude more expensive to produce art assets that way than to just hire people familiar with photoshop and maya. Those procedurally generated assets in tech demos like .kkrieger were just that: tech demos. Asset production is already the bulk of the cost of AAA video game production.
- GuB-42 4y ago
  And there was also the opposite, John Carmack in particular didn't have a problem with using huge amount of data, "Rage" is an application of this idea. The game uses a single, huge, non-repeating "megatexture", with elaborate caching strategies to keep it playable even on slow media (HDD/optical). The idea was to give complete freedom to artists and not require them to be mathematicians. Further down the line, he even considered using sparse voxel octrees, i.e. even more huge 3D textures. On the opposite end of the spectrum, you have Inigo Quilez ( https://iquilezles.org/ https://iquilezles.org/ ). He literally draws with maths, and he is famous for his 4k demoscene productions. The problem is that about no one else is capable of doing what he does.
  - gatane 4y ago
    I see that Inigo Quilez work is more in the "render a scene not necesarily in real time" realm. Like, you could make a raytracing program, make it render off camera a shiny scene, and only after rendering it shows you the image. Because of this, its very suited to math and procedural generation. You just change some parameters, render and show. Games instead have to do this in real time, with less latency and CPU/GPU usage. And procedural/math generated content tends to be CPU heavy (more usage+latency), if not also on the GPU (cant be cached, etc). Shaders are more on the middle of this, because you want to apply certains transformations to and from rendering data. Just my armchair observation.
MrFoof 4y ago
The Moore's Law is Dead podcast recently did a podcast with an Infinity Ward award-winning effects artist: https://www.youtube.com/watch?v=tmfHxJT1I3I https://www.youtube.com/watch?v=tmfHxJT1I3I. The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets.
- teaearlgraycold 4y ago
  But why?
  - ok_dad 4y ago
    Have you played a modern game? they have amazing levels of detail in the tiniest places. One game I play, Gran Turismo 7, claims that a single car model start to finish takes months to model for a single modeler. Some of the details in those cars that you can see in VR you can absolutely not see in-game normally, such as the door lock controls or the key-hole for the driver's door being exactly like on the real car, even for something so rare as a McLaren F1 longtail. There is so much waste of time and space dedicated to this stuff, but I love it personally!
    - taeric 4y ago
      Gran Turismo in VR is an amazing experience. I'm not even a racing car fan, but holy crap does that game really capture an experience. I'm somewhat afraid to find out just how much data goes into a single model.
  - MrFoof 4y ago
    Listen to the answer to the question, "How long until our game installs take up 500 gigabytes of space": https://youtu.be/tmfHxJT1I3I?t=7152s https://youtu.be/tmfHxJT1I3I?t=7152s The reality is a game has increasingly more content nowadays. Remember, there's a lot of "games as a service". So you have many "seasons" of additional content. You may not play all the content, but someone is. And when people want 4K and 8K textures, different variations so effects don't look the same twice, on top of all the other sizes, plus all the other effects ... it adds up. Never mind huge open worlds, where the asset footprint for just the map might be 100GB. Having been in a few AAA studios myself (doing database stuff) and sitting next to artists and even art directors, the sheer amount of effort, headcount, etc. put into the art pipeline is genuine madness. In a large studio, art is over HALF the production cost, sometimes more towards 60-65%. Heck, one of the things sometimes brought up about Red Dead Redemption II is that there were three full-time employees in which all they were responsible for was horses. 6000 man-hours a year, on just horses. To sum up what Chris said towards the end, "The only thing keeping these numbers down is the compression algorithms. If we didn't have these newer compression algorithms, we'd probably legitimately be at a terabyte. I can actually tell you for a fact, that there are over a terabyte of assets in the COD engine."
charlie0 4y ago
If users hate large file sizes, they should just stop buying them. Unfortunately, hitting the bottom line is really the only way to get this resolved.
- fbdab103 4y ago
  That seems impossibly black or white. I am not sure there is any product which is perfect in all dimensions. At some point you have to compromise with the state of the world and accept some bad with the good.
mrkramer 4y ago
Cloud gaming is.
W4RH4WK55 4y ago
We recently had blog post closely related to the topic: https://ph3at.github.io/posts/Asset-Compression/ https://ph3at.github.io/posts/Asset-Compression/
- arthur2e5 4y ago
  Thing should get its own submission.
  - kristianp 4y ago
    https://news.ycombinator.com/item?id=35662548 https://news.ycombinator.com/item?id=35662548
- kevingadd 4y ago
  Great post that helps people understand why smaller isn't always better.
codedokode 4y ago
Why not simply make several versions? Like 1 Gb, 10 Gb and 100 Gb versions?
- tmilard 4y ago
  good idea : one why chitty etxture quality, one with average, one with quality texture. They have done this on web-videos. Why not on video games files ?
  - kevingadd 4y ago
    They do, the Xbox calls it smart delivery.
    - tmilard 4y ago
      Yeah you are right
- jsheard 4y ago
  Some games do this by splitting off the highest resolution textures into a free "DLC" that the user can opt not to install if they want to save space. It's sometimes done for audio as well, e.g. optional Japanese voices for the English release of a JRPG. Remedy's Quantum Break had an unusual spin on letting users choose the install size, it made heavy use of live action FMVs which could either be downloaded ahead of time or streamed on the fly as they played. Downloading the FMVs more than doubled the games footprint.
tmilard 4y ago
Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size < 1 mega Byte ( really). b) It downloads in 4 seconds. Check it here : https://free-visit.net/fr/demo01 https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?
- terafo 4y ago
  Because PNG isn't a good format for GPU textures. Oodle texture+Kraken+something like BC7 is an optimal solution and is used in many games.
  - tmilard 4y ago
    I think in today's machine (be it PC, Mac or even Smartphones) the reading-and-transform of a png asset for the GPU is hard coded in the processor : so VERY fast. It works well.
    - terafo 4y ago
      There is no hardware acceleration for PNG in any consumer device that I know of. And there is such thing as VRAM usage, if you decompress PNG to bitmap it would use 3x-4x more VRAM than BC7. If you decompress it from PNG and then compress to BC7 it would increase loading times and basically destroys all point of using PNG instead of lossy compression.
- egypturnash 4y ago
  To limit mesh size at less than 300 points for a scene. I am pretty sure most modern video games use more mesh points than that for, like, a hat your character can wear.
  - tmilard 4y ago
    I it is true : I was very agressive on this one. But still, thousands and tousands of triangle feels like so much overweight for me. Specialy for indoor scenes. Quality for the eye is a lot to do with texture, NOT the number of triangle.
mxmlnkn 4y ago
Slow downloads wouldn't be a problem if shipping actually finished games on Blu-rays were still in vogue. And the disc limitation would make some publishers think twice about shipping 250 GB on 3 or more discs.
deleted 4y ago
[deleted]
pornel 4y ago
Compression optimizes for wall clock time which is (roughly speaking) sum of time it takes to get the compressed data and time it takes to decompress it. Downloads over the network are almost always bottlenecked on the network speed, so the heavier compression you have the better. However, when loading data from the local disk, the I/O can be so fast that the decompression time becomes the bottleneck and slows down loading. Because of that optimization for download time and startup time end up optimizing for different things. With loading speeds so different in these two cases, I wonder if the best solution would be to recompress the data with a different compression level during download and installation. For example, download textures compressed with a good but slow and non-random-access image compression format, and then convert them to a lighter GPU-friendly compression format locally.
- kevingadd 4y ago
  Heavy compression for the first install is a common tactic. Steam and itch do it, and brotli transfer compression is widely used on the web.
  - pornel 4y ago
    Lossless transfer compression is good to have, but it's not enough. Lossy compression can give you over 10 times smaller files, but it's format-specific and needs to be carefully applied by the developer.
- langsoul-com 4y ago
  There's a limit to compression. At some point, too much compression takes too long to unpack. The normal person wouldn't really understand that.
- bick_nyers 4y ago
  Recompression is a fantastic idea.
mabbo 4y ago
Games will be as large as they can be until it starts causing people not to buy the game. Before that point, no one will care. And as someone who just got 1.5Gbps internet installed, my tolerance for file size is continually increasing.
- shmde 4y ago
  Your tolerance for file size is increasing but what about your SSD/HDD tolerance for file size ?
  - maccard 4y ago
    A 2TB SSD is available on Amazon for roughly the cost of 2 AAA games.
kevingadd 4y ago
Rich is a compression expert who wrote a good post, and there are some good comments in this thread already, but a few key summary points to help people understand the tradeoffs here: * Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually make this worse because you can't delta compress big archives as well as individual files. * Smaller on disk doesn't mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See https://ph3at.github.io/posts/Asset-Compression/ https://ph3at.github.io/posts/Asset-Compression/ (from elsewhere in this thread) for one example. * For image and geometry data that is headed for the GPU, it's best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I'm considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2. * For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it's less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency. * In some cases the install you get depends on the hardware you have. It's not uncommon for older platforms to see 'high res texture packs' that could be installed optionally. The Xbox platform has something called "smart delivery" where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere. * Ultimately, unless your game is too big to fit on people's hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam/Itch/Xbox/Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn't make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game's framerate instead. * Lots of people decide a game "isn't compressed" because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it. An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there's no point doing it if you can just use the browser's built in brotli decoder instead.
bhouston 4y ago
250gB? No problem with 1gb/S to 3gb/s fiber to the home. At least here in Ottawa, Canada Fiber to the home is prevalent and affordable. But I understand where fiber isn’t available these are large downloads.
- dendriti 4y ago
  Well shit. Does your "Canada Fiber to the home" come with unlimited fast storage for free?
  - bhouston 4y ago
    The article mentioned primarily download sizes and times. Honestly having +1Gb/S download speeds is quite nice. I also run a local 10Gb/S local network and again it makes large files seem small.
  - nottorp 4y ago
    At 1 Gbps unmetered deleting and reinstalling games later is not much of a problem.
- qtzaz 4y ago
  That's what I was thinking. Game size is not the problem, the problem is that you have a shit internet connection and/or a small hard disk.
langsoul-com 4y ago
I think games should have advanced textures optional by default. 4k textures are unnecessary if it results in a 50gig+ download.
- kevingadd 4y ago
  It used to be more common to have optional texture packs like this. It seems to have fallen out of favor, I can offer some informed speculation as to the reasons: * Every additional configuration i.e. 'texture pack vs no texture pack' is more QA workload to verify that everything works as expected. So adding this option is more work for your already-busy testers. * Adding DLCs to deployment systems like Steam or PSN is additional work and then when patching the game you would also need to patch the DLC. I can imagine the coordination overhead here being a pain. * Some users will fail to install the DLC and then complain that the game is ugly or that the screenshots were misleading. This is ultimately a user education problem but it's also the case that the flow for installing DLC on most platforms is awkward. * The DLC install process adds additional points of failure, so you need to handle all of that now too. * Users installing a new graphics card will probably be annoyed to have to suddenly wait for a HD texture pack to download before they can test it out. All pretty minor individually, but it probably adds up to explain why most studios don't do this. I wonder how widespread developer support for Xbox Smart Delivery (which is basically what you describe) actually is at this point?
Animats 4y ago
Meanwhile, over in metaverse land, there are petabytes of content in the big worlds. I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing. All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want to rewrite OpenJPEG in Rust, go for it. You can download part of a JPEG 2000 file and get a lower resolution texture, and that's what's done for distant objects. The general idea is to download textures at a size that gets one texel per screen pixel. (Yes, subpixel textures would be better.) Textures are locally cached to SSD. Background threads are frantically loading and unloading textures in the GPU at different resolutions as the viewpoint moves. The highest resolution is compressed losslessly; the others are lossy. In metaverse systems with user generated content, there's not much instancing. So the texture load is much higher. Wander around Cyberpunk 2077 and you'll see the same railing show up in completely different contexts. This doesn't happen in worlds with user-created content. There's no producer or art director to insist on reusing content. If your game downloads content on the fly, you need substantial content delivery capacity. Second Life front-ends Amazon AWS with Akamai. Open Simulator, on the other hand, often just hosts the content on a single game server, along with the the simulator itself. My own viewer pulls about 200Mb/s from the content servers for about 15 seconds when you first connect. Then it's done until the viewpoint moves.
- Gelenoto 4y ago
  So you compare a highly detailed state of the art game with gigabytes if proper assets to the 'art' dessert of vr?
  - bick_nyers 4y ago
    Second Life has been around since before Oculus was born, it is an OG MMO
- bick_nyers 4y ago
  Check out HTJ2K if you haven't already, it's J2K with SIMD instructions, it will have solid support because the medical industry is (slowly) moving towards adopting it. Sounds like you already saw NVIDIA's J2K decoder. There are FPGA solutions as well that are insanely fast (CAST and some others). HTJ2K has support in OpenJPEG (master branch at least), might need to compile with a specific flag or something I can't remember. I think JPEG-XL is ideal for these types of applications in my opinion, even if you ship the decoder with your client. Their progressive encoding/decoding is much better than J2K in my opinion.
bullen 4y ago
I use zip (in flight) and 2 different lz (on disk) for my 3D action voxel MMO. But the characters are flat shaded so those textures are tiny. You need to look at the data, animation weights compresses 95% with a simple lz. One big dissappointment was half-float support having low performance on the some drivers, that would have saved 50% off the models. One advantage of not compressing is that your disk shows the (V)RAM usage you'll have. I'm aiming for 1GB total (to support 2GB Raspberry 4) and my biggest concern is uncompressed 120Hz animations! (to save CPU) But so many of those can be hot-loaded off disk.
timwaagh 4y ago
I think honestly the problem is small disks. Games are achieving almost photorealism on 4k with maybe 150gb. That's quite honestly not bad. You can't store the universe on a floppy disk.
- bick_nyers 4y ago
  Blu-ray movies are like 50GB after all, and are significantly less hours of entertainment.
cultureswitch 4y ago
Maybe I misunderstand something but to me it seems the premise is wrong. What's most important about game textures is that they can load fast at a quality good enough not to disturb gameplay. Absolutely everything else is a secondary concern.
bno1 4y ago
Saving a few ms during decompression actually matters when the game streams data from the disk in real time. People also hate loading screens and objects popping-in because the streaming pipeline is slow.
sdfjkl 4y ago
Well, whatever the repackers (e.g. Fitgirl) use seems to work a lot better than what the game publishers use (often no compression at all). Original Size: 93.1 GB -> Repack Size: from 35.5 GB [Selective Download] Original Size: 3.1 GB -> Repack Size: 844 MB And these are lossless, bit for bit reproductions. In many cases assets (audio, video, images/textures) could be using lossy compression for even greater reduction in size with no significant loss in perceived quality (but maybe some loss of advertising/bragging rights). The downside is, some effort is required packing it: https://fitgirl-repacks.site/new-compression-algorithm/ https://fitgirl-repacks.site/new-compression-algorithm/
- SSLy 4y ago
  Another downside is obscene time it takes to unpack FG releases.
  - sdfjkl 4y ago
    That depends. If you're on a slow connection, the 40 minutes to unpack something are nothing in comparison to the week it took you to download it. If used by the game publisher, the decompression could happen during download (I believe this is how steam does it). If you're on a metered connection (common in many of the more forgotten parts of the world), the bloated download costs you real money or is simply not possible (or you have to spread it over several periods of data allowance).
AniseAbyss 4y ago
[dead]
cabirum 4y ago
I suspect it's not the devs should be blamed, but marketing and management. "A newer and a better game must also take more space, right?"
faebi 4y ago
Don't want to blame too much, but it seems like your internet got stuck in the past. I have 10gbits. I used to have 1gbits for a long time before. 100mbits is not that much these days. Switzerland is talking about a government mandated minimum speed of 80mbits in 2024. That's a 7 hour download on minimum speed for an unusually large game. Switzerland is not alone in this regard. That could be a view of privilege, but average internet speeds are still progressing forward fast.
- SergeAx 4y ago
  I beleive it is your internet is a bit in a future. Good for you, but not a reason to blame others. See https://www.fastmetrics.com/internet-connection-speed-map-usa.php#average-speeds-usa https://www.fastmetrics.com/internet-connection-speed-map-us... , for example. 30-50 Mbps was median in 2020, even with COVID-induced improvements I doubt things became significantly better today.
- MMAesawy 4y ago
  It is an extremely privileged view. In my home country of Egypt, a population of 100 million lives on an average home internet speed of 30mbps with about 300gb monthly data cap, in 2023. I'm sure most of the world is similar.
- ProtoAES256 4y ago
  You're correct about the internet stucked the past, but for a lot of places in the world even with a lot of money you won't get fast speeds! My hometown is only getting 100/25Mbits max after the previous max 30/5Mbits 2 years ago! It's not like we don't have the money, I bought a 10900k and 2080 super for my brother back then, it's just that the speed is unavailable. A "git gud" simply don't do that justice.
justeleblanc 4y ago
This whole thing is an environmental disaster.
amelius 4y ago
In the future assets will be created using a prompt the size of a tweet. It would be silly to send the pixels instead of the prompt.

191 points by ingve 4y ago

https://acronyms.thefreedictionary.com/LZ https://acronyms.thefreedictionary.com/LZ I think the right definition is this one: Lempel-Ziv (dictionary-based lossless data compression)

thfuran 4y ago
It is

adgjlsfhk1 4y ago

I really want to see someone make a renderer that can work on compressed textures. No need to decompress if you can just read the data on the fly.

aappleby 4y ago
Renderers have been using compressed textures for literal decades. Not JPG, alas, because the compression formats have to support random access to individual texels.
corysama 4y ago
https://www.reedbeta.com/blog/understanding-bcn-texture-compression-formats/ https://www.reedbeta.com/blog/understanding-bcn-texture-comp... BC1-5 have been standard practice since before the original XBox. I think when people here are talking about "uncompressed assets" they mean "not applying variable-rate zip style compression on top of BCn compression."
tmilard 4y ago
I use this one. Web compliant. https://www.babylonjs.com/ https://www.babylonjs.com/

bick_nyers 4y ago

Faster internet and cheaper storage is the answer :) Or at the very least just offer two different downloads of the game, one with lossless textures and one with lossy textures...

topspin 4y ago
Guild Wars 2 and others solve this with progressive downloads. You may start playing after a fraction of the content is downloaded. There is effectively no downside: new players can't access most of the content anyhow. It also applies to updates: you can start playing after a big update with a partial download. The background process can be safely interrupted and resumed without glitches.
- arthur2e5 4y ago
  War Thunder kind of has that, except that the "progressive" bit means you need to download the full thing to progress in the game beyond a certain point. Enlisted, which uses the same engine, is fine though.
  - topspin 4y ago
    The design of ArenaNet's GW/GW2 has always impressed me. 99.4% of all of the game is contained in one file: GW2.DAT, which is (currently) 66.7 GB. I/O for updates is extremely efficient. I've seen many games create a vast file system mess, and they're forever plagued by "corruption" or something, complete with "fix" tools, "cache" clearing push-ups and players reinstalling after big updates that don't install correctly for whatever reason. Somehow ArenaNet surrounded this whole problem and solved it optimally.
    - kevingadd 4y ago
      A massive amount of engineering went into making it work and some of the stuff we did was downright evil :) in gw1, a lot of data was compiled into the exe as c constants so that we wouldn't have to do a bunch of file management and I/o to locate and load it. We ended up hitting limitations in the C compiler.

terafo 4y ago

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't use any compression beyond something like BC7. And some games are not only without any compression, they ACTIVELY duplicate their assets on disk so they load a bit faster on HDD. And these builds end up on modern consoles that have ludicrously fast SSDs.

terafo 4y ago
And, besides, his solutions aren't really good. Radical solution would be shipping image generating model with the game and generating textures on the fly. That's how all texture-related data goes from being measured in hundreds of gigs to being measured in hundreds of megs. So basically becomes non-issue.
- circuit10 4y ago
  That would be far, far slower than just downloading the image
  - terafo 4y ago
    8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.
    - circuit10 4y ago
      I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator

rektide 4y ago

Richg42 is great & his pieces are fun to read. I do think in this case he misses a bit of the point, goes too far: > They are designed for fast loading and streaming off SSD's. Who cares about saving off a few hundred ms (or a second) when it takes hours or days to download the product onto the SSD? Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait. But we load textures many times if we keep playing a game. Rich's overall point remains, but I think some balance of optimizing load speed is also very appreciated & very necessary.

overthrow 4y ago
That can be solved on first load with a texture cache. Shader caches are pretty universal at this point. Texture caches aren't (afaik), but I think they should be.
- rektide 4y ago
  So now we would have faster download & faster load, but we'd have significantly bugger hard drive space usage than we have? That also seems like a concern.
jeroenhd 4y ago
> Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait. Are there any statistics on that? Because I have deleted and redownloaded games a whole bunch of times because you can't fit more than two or three games on a single SSD anymore. I've played large games off of hard drives because I simply didn't want to delete 240GB of other files off of my main drive. The game experience was clearly much worse than the developers intended, but their space demands are unreasonable. Especially considering I'm only playing at 1080p! Give me a 1080p installer that shaves off 75% of the texture size and you'll win more than a few milliseconds because the stupid thing can actually fit on my drive and doesn't need to come off the internet again.

hellotomyrars 4y ago

Game size is a perennial talking point and some games earn it and some games don’t. There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall) Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think this is the smart way to do it. Examples of games shrinking I are hard to come by but special credit to IO interactive and Hitman 3. The install size with all the content of the previous games included (If you own them) is less than the install size of the first game. That is impressive and commendable. Games are weird and some people will look at the file size and think it means the game is better, just the way some people think games with more hours are better. The psychology of people who play video games is weird. Duplication of assets on disk was a very weird byproduct of HDD file access conventions, but we shouldn’t need that anymore. There is still a shrinking, but valid crossover window between devs being able to assume all storage is solid state. I think things will get better but the debate of file size vs average/expects storage capacity vs Network speed will never end but there is certainly work to be done.

RajT88 4y ago
The .kkreiger devs really showed how far you can go to reduce download size: https://en.wikipedia.org/wiki/.kkrieger https://en.wikipedia.org/wiki/.kkrieger https://www.youtube.com/watch?v=ya_MUKc343U https://www.youtube.com/watch?v=ya_MUKc343U It is dated, but looks pretty good by 2004 standards.
- thequux 4y ago
  While I'm all for reducing download sizes, perhaps lekktor[1] is a bridge too far. [1]: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for-madmen/ https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
  - Dylan16807 4y ago
    Sure. Focus on the procedural textures and see where that gets you.

KMnO4 4y ago

Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper. I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene. Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a lot of the work involved in making things pretty. You don't need to generate or even understand displacement maps if you can just add more polygons to the model. You don't need to billboard the mountain way in the distance if Unreal does it on the fly. Who cares if that means including a 250mb mesh in your assets, versus a 5mb HDRI. So yes, producers have either become lazy or are simply unable to meet the demands of the hardware by coming up with better algorithms. That's unlikely to change until either hardware stops advancing and either A) can't run the latest games or B) games become too large that they're impossible to download. [0]: https://www.gamedeveloper.com/programming/sponsored-feature-light-it-up-quake-wars-gets-ray-traced https://www.gamedeveloper.com/programming/sponsored-feature-... [1]: https://docs.unrealengine.com/5.0/en-US/nanite-virtualized-geometry-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/nanite-virtualized-g... [2]: https://twitter.com/mariobrothblog/status/1636040893764362241 https://twitter.com/mariobrothblog/status/163604089376436224...

terafo 4y ago
> and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't work with open-world games, especially if they have different times of day. And it doesn't work in highly dynamic scenes. We basically hit the end of the road for scaling graphics without RT. And now we have hardware that can do realtime path-traced rendering of major AAA titles, and it looks miles better than anything smoke and mirrors could've ever delivered.

LinuxBender 4y ago

I recall a lot of hype more than a decade ago about all the textures going away and being replaced with math formulas to generate everything on the fly. Games were supposed to get dramatically smaller. What ever happened to that?

db48x 4y ago
Economics. Supply and demand. To create good procedural art you have to hire software engineers with artistic talent, which are more expensive than the texture and model artists you can find by the hundreds coming out of art schools.
babypuncher 4y ago
It is an order of magnitude more expensive to produce art assets that way than to just hire people familiar with photoshop and maya. Those procedurally generated assets in tech demos like .kkrieger were just that: tech demos. Asset production is already the bulk of the cost of AAA video game production.
GuB-42 4y ago
And there was also the opposite, John Carmack in particular didn't have a problem with using huge amount of data, "Rage" is an application of this idea. The game uses a single, huge, non-repeating "megatexture", with elaborate caching strategies to keep it playable even on slow media (HDD/optical). The idea was to give complete freedom to artists and not require them to be mathematicians. Further down the line, he even considered using sparse voxel octrees, i.e. even more huge 3D textures. On the opposite end of the spectrum, you have Inigo Quilez ( https://iquilezles.org/ https://iquilezles.org/ ). He literally draws with maths, and he is famous for his 4k demoscene productions. The problem is that about no one else is capable of doing what he does.
- gatane 4y ago
  I see that Inigo Quilez work is more in the "render a scene not necesarily in real time" realm. Like, you could make a raytracing program, make it render off camera a shiny scene, and only after rendering it shows you the image. Because of this, its very suited to math and procedural generation. You just change some parameters, render and show. Games instead have to do this in real time, with less latency and CPU/GPU usage. And procedural/math generated content tends to be CPU heavy (more usage+latency), if not also on the GPU (cant be cached, etc). Shaders are more on the middle of this, because you want to apply certains transformations to and from rendering data. Just my armchair observation.

MrFoof 4y ago

The Moore's Law is Dead podcast recently did a podcast with an Infinity Ward award-winning effects artist: https://www.youtube.com/watch?v=tmfHxJT1I3I https://www.youtube.com/watch?v=tmfHxJT1I3I. The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets.

teaearlgraycold 4y ago
But why?
- ok_dad 4y ago
  Have you played a modern game? they have amazing levels of detail in the tiniest places. One game I play, Gran Turismo 7, claims that a single car model start to finish takes months to model for a single modeler. Some of the details in those cars that you can see in VR you can absolutely not see in-game normally, such as the door lock controls or the key-hole for the driver's door being exactly like on the real car, even for something so rare as a McLaren F1 longtail. There is so much waste of time and space dedicated to this stuff, but I love it personally!
  - taeric 4y ago
    Gran Turismo in VR is an amazing experience. I'm not even a racing car fan, but holy crap does that game really capture an experience. I'm somewhat afraid to find out just how much data goes into a single model.
- MrFoof 4y ago
  Listen to the answer to the question, "How long until our game installs take up 500 gigabytes of space": https://youtu.be/tmfHxJT1I3I?t=7152s https://youtu.be/tmfHxJT1I3I?t=7152s The reality is a game has increasingly more content nowadays. Remember, there's a lot of "games as a service". So you have many "seasons" of additional content. You may not play all the content, but someone is. And when people want 4K and 8K textures, different variations so effects don't look the same twice, on top of all the other sizes, plus all the other effects ... it adds up. Never mind huge open worlds, where the asset footprint for just the map might be 100GB. Having been in a few AAA studios myself (doing database stuff) and sitting next to artists and even art directors, the sheer amount of effort, headcount, etc. put into the art pipeline is genuine madness. In a large studio, art is over HALF the production cost, sometimes more towards 60-65%. Heck, one of the things sometimes brought up about Red Dead Redemption II is that there were three full-time employees in which all they were responsible for was horses. 6000 man-hours a year, on just horses. To sum up what Chris said towards the end, "The only thing keeping these numbers down is the compression algorithms. If we didn't have these newer compression algorithms, we'd probably legitimately be at a terabyte. I can actually tell you for a fact, that there are over a terabyte of assets in the COD engine."

charlie0 4y ago

If users hate large file sizes, they should just stop buying them. Unfortunately, hitting the bottom line is really the only way to get this resolved.

fbdab103 4y ago
That seems impossibly black or white. I am not sure there is any product which is perfect in all dimensions. At some point you have to compromise with the state of the world and accept some bad with the good.

mrkramer 4y ago

Cloud gaming is.

W4RH4WK55 4y ago

We recently had blog post closely related to the topic: https://ph3at.github.io/posts/Asset-Compression/ https://ph3at.github.io/posts/Asset-Compression/

arthur2e5 4y ago
Thing should get its own submission.
- kristianp 4y ago
  https://news.ycombinator.com/item?id=35662548 https://news.ycombinator.com/item?id=35662548
kevingadd 4y ago
Great post that helps people understand why smaller isn't always better.

codedokode 4y ago

Why not simply make several versions? Like 1 Gb, 10 Gb and 100 Gb versions?

tmilard 4y ago
good idea : one why chitty etxture quality, one with average, one with quality texture. They have done this on web-videos. Why not on video games files ?
- kevingadd 4y ago
  They do, the Xbox calls it smart delivery.
  - tmilard 4y ago
    Yeah you are right
jsheard 4y ago
Some games do this by splitting off the highest resolution textures into a free "DLC" that the user can opt not to install if they want to save space. It's sometimes done for audio as well, e.g. optional Japanese voices for the English release of a JRPG. Remedy's Quantum Break had an unusual spin on letting users choose the install size, it made heavy use of live action FMVs which could either be downloaded ahead of time or streamed on the fly as they played. Downloading the FMVs more than doubled the games footprint.

tmilard 4y ago

Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size < 1 mega Byte ( really). b) It downloads in 4 seconds. Check it here : https://free-visit.net/fr/demo01 https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?

terafo 4y ago
Because PNG isn't a good format for GPU textures. Oodle texture+Kraken+something like BC7 is an optimal solution and is used in many games.
- tmilard 4y ago
  I think in today's machine (be it PC, Mac or even Smartphones) the reading-and-transform of a png asset for the GPU is hard coded in the processor : so VERY fast. It works well.
  - terafo 4y ago
    There is no hardware acceleration for PNG in any consumer device that I know of. And there is such thing as VRAM usage, if you decompress PNG to bitmap it would use 3x-4x more VRAM than BC7. If you decompress it from PNG and then compress to BC7 it would increase loading times and basically destroys all point of using PNG instead of lossy compression.
egypturnash 4y ago
To limit mesh size at less than 300 points for a scene. I am pretty sure most modern video games use more mesh points than that for, like, a hat your character can wear.
- tmilard 4y ago
  I it is true : I was very agressive on this one. But still, thousands and tousands of triangle feels like so much overweight for me. Specialy for indoor scenes. Quality for the eye is a lot to do with texture, NOT the number of triangle.

mxmlnkn 4y ago

Slow downloads wouldn't be a problem if shipping actually finished games on Blu-rays were still in vogue. And the disc limitation would make some publishers think twice about shipping 250 GB on 3 or more discs.

deleted 4y ago

[deleted]

pornel 4y ago

Compression optimizes for wall clock time which is (roughly speaking) sum of time it takes to get the compressed data and time it takes to decompress it. Downloads over the network are almost always bottlenecked on the network speed, so the heavier compression you have the better. However, when loading data from the local disk, the I/O can be so fast that the decompression time becomes the bottleneck and slows down loading. Because of that optimization for download time and startup time end up optimizing for different things. With loading speeds so different in these two cases, I wonder if the best solution would be to recompress the data with a different compression level during download and installation. For example, download textures compressed with a good but slow and non-random-access image compression format, and then convert them to a lighter GPU-friendly compression format locally.

kevingadd 4y ago
Heavy compression for the first install is a common tactic. Steam and itch do it, and brotli transfer compression is widely used on the web.
- pornel 4y ago
  Lossless transfer compression is good to have, but it's not enough. Lossy compression can give you over 10 times smaller files, but it's format-specific and needs to be carefully applied by the developer.
langsoul-com 4y ago
There's a limit to compression. At some point, too much compression takes too long to unpack. The normal person wouldn't really understand that.
bick_nyers 4y ago
Recompression is a fantastic idea.

mabbo 4y ago

Games will be as large as they can be until it starts causing people not to buy the game. Before that point, no one will care. And as someone who just got 1.5Gbps internet installed, my tolerance for file size is continually increasing.

shmde 4y ago
Your tolerance for file size is increasing but what about your SSD/HDD tolerance for file size ?
- maccard 4y ago
  A 2TB SSD is available on Amazon for roughly the cost of 2 AAA games.

kevingadd 4y ago

Rich is a compression expert who wrote a good post, and there are some good comments in this thread already, but a few key summary points to help people understand the tradeoffs here: * Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually make this worse because you can't delta compress big archives as well as individual files. * Smaller on disk doesn't mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See https://ph3at.github.io/posts/Asset-Compression/ https://ph3at.github.io/posts/Asset-Compression/ (from elsewhere in this thread) for one example. * For image and geometry data that is headed for the GPU, it's best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I'm considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2. * For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it's less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency. * In some cases the install you get depends on the hardware you have. It's not uncommon for older platforms to see 'high res texture packs' that could be installed optionally. The Xbox platform has something called "smart delivery" where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere. * Ultimately, unless your game is too big to fit on people's hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam/Itch/Xbox/Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn't make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game's framerate instead. * Lots of people decide a game "isn't compressed" because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it. An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there's no point doing it if you can just use the browser's built in brotli decoder instead.

bhouston 4y ago

250gB? No problem with 1gb/S to 3gb/s fiber to the home. At least here in Ottawa, Canada Fiber to the home is prevalent and affordable. But I understand where fiber isn’t available these are large downloads.

dendriti 4y ago
Well shit. Does your "Canada Fiber to the home" come with unlimited fast storage for free?
- bhouston 4y ago
  The article mentioned primarily download sizes and times. Honestly having +1Gb/S download speeds is quite nice. I also run a local 10Gb/S local network and again it makes large files seem small.
- nottorp 4y ago
  At 1 Gbps unmetered deleting and reinstalling games later is not much of a problem.
qtzaz 4y ago
That's what I was thinking. Game size is not the problem, the problem is that you have a shit internet connection and/or a small hard disk.

langsoul-com 4y ago

I think games should have advanced textures optional by default. 4k textures are unnecessary if it results in a 50gig+ download.

kevingadd 4y ago
It used to be more common to have optional texture packs like this. It seems to have fallen out of favor, I can offer some informed speculation as to the reasons: * Every additional configuration i.e. 'texture pack vs no texture pack' is more QA workload to verify that everything works as expected. So adding this option is more work for your already-busy testers. * Adding DLCs to deployment systems like Steam or PSN is additional work and then when patching the game you would also need to patch the DLC. I can imagine the coordination overhead here being a pain. * Some users will fail to install the DLC and then complain that the game is ugly or that the screenshots were misleading. This is ultimately a user education problem but it's also the case that the flow for installing DLC on most platforms is awkward. * The DLC install process adds additional points of failure, so you need to handle all of that now too. * Users installing a new graphics card will probably be annoyed to have to suddenly wait for a HD texture pack to download before they can test it out. All pretty minor individually, but it probably adds up to explain why most studios don't do this. I wonder how widespread developer support for Xbox Smart Delivery (which is basically what you describe) actually is at this point?

Animats 4y ago

Meanwhile, over in metaverse land, there are petabytes of content in the big worlds. I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing. All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want to rewrite OpenJPEG in Rust, go for it. You can download part of a JPEG 2000 file and get a lower resolution texture, and that's what's done for distant objects. The general idea is to download textures at a size that gets one texel per screen pixel. (Yes, subpixel textures would be better.) Textures are locally cached to SSD. Background threads are frantically loading and unloading textures in the GPU at different resolutions as the viewpoint moves. The highest resolution is compressed losslessly; the others are lossy. In metaverse systems with user generated content, there's not much instancing. So the texture load is much higher. Wander around Cyberpunk 2077 and you'll see the same railing show up in completely different contexts. This doesn't happen in worlds with user-created content. There's no producer or art director to insist on reusing content. If your game downloads content on the fly, you need substantial content delivery capacity. Second Life front-ends Amazon AWS with Akamai. Open Simulator, on the other hand, often just hosts the content on a single game server, along with the the simulator itself. My own viewer pulls about 200Mb/s from the content servers for about 15 seconds when you first connect. Then it's done until the viewpoint moves.

Gelenoto 4y ago
So you compare a highly detailed state of the art game with gigabytes if proper assets to the 'art' dessert of vr?
- bick_nyers 4y ago
  Second Life has been around since before Oculus was born, it is an OG MMO
bick_nyers 4y ago
Check out HTJ2K if you haven't already, it's J2K with SIMD instructions, it will have solid support because the medical industry is (slowly) moving towards adopting it. Sounds like you already saw NVIDIA's J2K decoder. There are FPGA solutions as well that are insanely fast (CAST and some others). HTJ2K has support in OpenJPEG (master branch at least), might need to compile with a specific flag or something I can't remember. I think JPEG-XL is ideal for these types of applications in my opinion, even if you ship the decoder with your client. Their progressive encoding/decoding is much better than J2K in my opinion.

bullen 4y ago

I use zip (in flight) and 2 different lz (on disk) for my 3D action voxel MMO. But the characters are flat shaded so those textures are tiny. You need to look at the data, animation weights compresses 95% with a simple lz. One big dissappointment was half-float support having low performance on the some drivers, that would have saved 50% off the models. One advantage of not compressing is that your disk shows the (V)RAM usage you'll have. I'm aiming for 1GB total (to support 2GB Raspberry 4) and my biggest concern is uncompressed 120Hz animations! (to save CPU) But so many of those can be hot-loaded off disk.

timwaagh 4y ago

I think honestly the problem is small disks. Games are achieving almost photorealism on 4k with maybe 150gb. That's quite honestly not bad. You can't store the universe on a floppy disk.

bick_nyers 4y ago
Blu-ray movies are like 50GB after all, and are significantly less hours of entertainment.

cultureswitch 4y ago

Maybe I misunderstand something but to me it seems the premise is wrong. What's most important about game textures is that they can load fast at a quality good enough not to disturb gameplay. Absolutely everything else is a secondary concern.

bno1 4y ago

Saving a few ms during decompression actually matters when the game streams data from the disk in real time. People also hate loading screens and objects popping-in because the streaming pipeline is slow.

sdfjkl 4y ago

Well, whatever the repackers (e.g. Fitgirl) use seems to work a lot better than what the game publishers use (often no compression at all). Original Size: 93.1 GB -> Repack Size: from 35.5 GB [Selective Download] Original Size: 3.1 GB -> Repack Size: 844 MB And these are lossless, bit for bit reproductions. In many cases assets (audio, video, images/textures) could be using lossy compression for even greater reduction in size with no significant loss in perceived quality (but maybe some loss of advertising/bragging rights). The downside is, some effort is required packing it: https://fitgirl-repacks.site/new-compression-algorithm/ https://fitgirl-repacks.site/new-compression-algorithm/

SSLy 4y ago
Another downside is obscene time it takes to unpack FG releases.
- sdfjkl 4y ago
  That depends. If you're on a slow connection, the 40 minutes to unpack something are nothing in comparison to the week it took you to download it. If used by the game publisher, the decompression could happen during download (I believe this is how steam does it). If you're on a metered connection (common in many of the more forgotten parts of the world), the bloated download costs you real money or is simply not possible (or you have to spread it over several periods of data allowance).

AniseAbyss 4y ago

[dead]

cabirum 4y ago

I suspect it's not the devs should be blamed, but marketing and management. "A newer and a better game must also take more space, right?"

faebi 4y ago

Don't want to blame too much, but it seems like your internet got stuck in the past. I have 10gbits. I used to have 1gbits for a long time before. 100mbits is not that much these days. Switzerland is talking about a government mandated minimum speed of 80mbits in 2024. That's a 7 hour download on minimum speed for an unusually large game. Switzerland is not alone in this regard. That could be a view of privilege, but average internet speeds are still progressing forward fast.

SergeAx 4y ago
I beleive it is your internet is a bit in a future. Good for you, but not a reason to blame others. See https://www.fastmetrics.com/internet-connection-speed-map-usa.php#average-speeds-usa https://www.fastmetrics.com/internet-connection-speed-map-us... , for example. 30-50 Mbps was median in 2020, even with COVID-induced improvements I doubt things became significantly better today.
MMAesawy 4y ago
It is an extremely privileged view. In my home country of Egypt, a population of 100 million lives on an average home internet speed of 30mbps with about 300gb monthly data cap, in 2023. I'm sure most of the world is similar.
ProtoAES256 4y ago
You're correct about the internet stucked the past, but for a lot of places in the world even with a lot of money you won't get fast speeds! My hometown is only getting 100/25Mbits max after the previous max 30/5Mbits 2 years ago! It's not like we don't have the money, I bought a 10900k and 2080 super for my brother back then, it's just that the speed is unavailable. A "git gud" simply don't do that justice.

justeleblanc 4y ago

This whole thing is an environmental disaster.

amelius 4y ago

In the future assets will be created using a prompt the size of a tweet. It would be silly to send the pixels instead of the prompt.