3 ms·
We usually don’t compress the data on disk; decompression would make loading and file access slower. Instead, we just pack the uncompressed files together (fr
by mewse 5y ago
We usually don’t compress the data on disk; decompression would make loading and file access slower.
Instead, we just pack the uncompressed files together (frequently using normal zip in a no-compression mode) so that we can avoid needing to ask the OS to open and close files for us or examining the contents of a directory, both of which can be kind of startlingly slow (by video game standards) on some common OSes. Instead, we will generally cache the directory data from the zip file and just use that rather than go to disk.
(of course, the whole download/patch would all be compressed for network transfer, but files would then be decompressed during the installation process)
- maccard 5y ago> We usually don’t compress the data on disk; decompression would make loading and file access slower. _You_ mightn't but the last three AAA games I worked on do/did. PS5 expectes compressed files, and does HW decompression (ahem, mostly) on the fly.
- Danieru 5y agoOn the switch the reads are so slow the fastest loading requires atbleast mild compression. At least it did when I was testing packaging for my latest switch release. Despite the weak cpu. Ps4 also did the compressed packages by default thing if I remember right. The upside there being ample cpu for decompression such that no compression was never fastest.
- sumtechguy 5y agoI could see that keeping one big file would still be advantageous too in that environment. As a fopen on a set of small sized files plus read plus close over and over does add up. Just in cpu time and memory slack. Whereas treating it as one giant packed backing store would have advantages in speed. But at a cost of dev time. Even if you are compressing it as well could be an advantage. But I would expect there to be some spot where it crosses over from being an advantage to a disadvantage. I suspect it would be on small files/objects. But that would just be for speed. You may have to optimize for size in some cases. In the end it is usually better to build some test code and find out. Sometimes the results are what you expect. Sometimes you are spending time on things that do not matter anymore but were a big deal 20 years ago.
- mewse 5y agoOh, definitely! I haven’t worked on anything targetting a console past the PS3 generation and it completely slipped my mind that the latest gen consoles are architected specifically for streaming compressed data. On the Windows/Mac/Linux title I’m working on now, I definitely measure a sizeable improvement to performance when loading from an uncompressed zip rather than from a compressed one. But even that could be down to the particular set of libraries I’m using to handle it.
- PeterisP 5y agoIIRC in a bunch of scenarios compression makes loading and file access faster, as you're I/O limited and it's quicker to read less data and decompress. You do need to choose simple/quick/not-that-much-compressing compression algorithms for that.
- tialaramex 5y ago> We usually don’t compress the data on disk; decompression would make loading and file access slower. Did you actually benchmark this? It probably makes sense in your head, but on any vaguely modern hardware it's very unlikely to actually be true because of how exponential the memory hierarchy is.
- bsenftner 5y agoConsole hardware tends to have fast processors & cache but extremely slow RAM. Benchmarking a console's memory vs cache access tends to be one of the first things a team of principal game devs do and that information becomes bible for their titles.
- greyhair 5y agoI have worked on a number of embedded products which ran from compressed root file system from eMMC. The overhead was a wash because RAM is so much faster than eMMC. What you spent in decompression time was covered by reduced eMMC access time.