4 ms·

I very much agree. My vision was a ZIP file that when selected in an emulator, could be extracted for instant access, and any save RAM, cheat codes, save states

by byuu 7y ago

I very much agree. My vision was a ZIP file that when selected in an emulator, could be extracted for instant access, and any save RAM, cheat codes, save states, etc could be stored alongside the game in said folder. It would be bundled with an extensible markup description of the board, and optional pack-in content like box art and user manuals. For games stored on flash memory that could write to themselves (BS Memory, Neo Geo Pocket), this would also prevent the archived, pristene copy from being damaged. The folder would then act like a physical cartridge, and all your game saves would move along with it. I wrote about my idea here: https://higan.byuu.org/game-paks https://higan.byuu.org/game-paks It ... was not a popular idea, to put it mildly. I further shot myself in the foot by trying to force the idea on people. You will never win people over with an approach like that. Let's avoid mentioning names for obvious reasons, but unfortunately those most capable of adopting and promoting such a format through their database effort are unwilling to use ZIP archives in this way. They will only allow for an uncompressed single-file-per-game approach. As mentioned in my article, this turns into quite the problem. When you take the 8KB program ROM for Galaxian, and append a character ROM for graphics onto it, but your header can only tell you the size of the program ROM in 16KB increments, it means there's no way of specifying where the program ROM ends, and the character ROM begins. Any emulator that supports Galaxian does it via game-specific workarounds like checksum identification.

bane 7y ago
I remember when Nesticle and some of the other viable Nintendo Entertainment System emulators first started showing up. It took a while but eventually everybody just ended up settling on the iNES format by Marat Fayzullin. This was a very useful thing. I've always found it extremely frustrating that later systems had so many competing formats for the roms as you inevitably end up with lots of duplicate games just so different emulators can handle them. It gets even worse for optical media with all the .cue file stuff and distributions that provide the audio tracks outside of the iso in half a dozen different formats. It's almost impossible to assemble a good collection for those kinds of systems. I honestly think a .zip based distribution makes a lot of sense since most emulators of cartridge based systems can open zipped archives and run the first compatible file they find in them. But all the rest of the information for the game could be "built in" to the zip. If you look at the Gamebase projects, there's plenty of hobbyists who are willing to pour thousands of hours into assembling games, metadata, manual scans and other such things into those kinds of projects...but they lack the ability to unify all that ephemera into a single archive per game. If such a thing were to become standard, I'm certain people/groups outside of the "established" database groups would begin to publish collections. I for one would definitely welcome a format with built-in cheats, patches and so on and would love to download a collection of say, Japanese RPGs that come with scanned manuals, translated manuals and can run the game in the original and patched versions....or to try out all the weird and wacky Mario/Sonic patched game variants. I just can't be bothered to go download each patch myself to try it out, but if they were already present in the .zip :) I think even support soundtracks would be awesome and could see people firing up collections of games just to listen to the music, without having to keep track of a separate soundtrack archive. I think this would be an even bigger benefit for modern optical media. The .bin/.cue stuff is honestly terrible. More than half the time I've ended up with a .bin or some kind of .something that I need to convert for some reason, or mount and extract stuff and then handbuild .cue files which don't work half the time. I don't even know if the problem is what I've downloaded, the .cue I made or the emulation compatibility. A standard distribution "format" that just solved all that would be incredibly useful.
- near 7y ago
  At some point, I want to talk about CDs as well. A proper CD archival format would include the lead-in sectors, with which it is absolutely trivial to extract track information from the Q-subchannel of it. Easier than parsing a CUE sheet, or MDF, or CCF, or whatever else have you. I do understand it's hard to rip this data from CDs (usually requires an older Plextor drive), but worst case, as bad as it'd be, generating the TOC from a CUE sheet is very simple, and if it's marked with a special entry to denote it being machine-generated, I feel that would be a good compromise. Then you'd have a single CD image format in a single file that captures all data on CDs. If you wanted to, you could then compress that image with either ZIP, or a custom-purpose tool that combines things like Reed-Solomon prediction, subchannel prediction, FLAC encoding for audio tracks, etc, resulting in images not any larger than ISO+FLAC. But well, https://xkcd.com/927 https://xkcd.com/927 and all, right? > I for one would definitely welcome a format with built-in cheats A thing I do with bsnes is bundle a database with most known cheat codes for all games, so you've got all the cheats nicely labeled and ready to go whenever you play a game. I kinda wish this idea caught on more for all the other systems and emulators, but it hasn't been all that popular.
  - bane 7y ago
    I kind of think that part of the problem is that there needs to be a bit of a network effect here for these kinds of .zip packaged formats to take off: 1) An emulator of sufficient popularity has to fully support it. 2) The format has to provide some kind of benefit to the user such that they're willing to convert/duplicate their entire collection. 3) Somebody has to make and distribute collections of packaged roms that include enough extra stuff to make them worthwhile. To be honest, you might be in a good position w/r to #1 since your SNES emulator is now more or less the default in the community (at least for big multi emulation systems like retroarch). For #2, I think the benefit really derives from what's in the package: manuals, savestates, cheats, soundtracks, playthroughs (that you can jump to in the middle of), patches, variants, meta-data (publication info, credits, etc.) and so on could all be really interesting to lots of people, but really would rely on the emulator to take advantage of in some way. For #3, There's probably already enough information freely available to make a utility that would take somebody's existing ROM collection and yank down all the rest of the stuff from various websites all at once to build up an initial package of stuff. I kind of almost just think you should just go ahead and support something like this and have it slowly percolate through the community. If it achieve some kind of network effect it'll slowly replace the other formats in a de-facto way.
spilk 7y ago
how about MAME, which has a zip-based format containing raw ROM dumps, and the "CHD" format for disk/disc based media? obviously all of the metadata is still contained within MAME itself.
- thristian 7y ago
  MAME is a kind of special case, where the same group of people write the emulator and catalogue the ROM dumps. For most emulated platforms, there's more than one emulator, and often more than one popular emulator, so deciding on a new archival format is a major political endeavour, on top of the existing tricky technical problems. Also, even MAME is not immune to political ROM-format problems. The format MAME used was designed to help people maintain and update arcade boards, which often had the ROM chips socketed so they could be replaced by the end user and so "a set of ROMs for a game" was a reasonable thing to talk about. MAME uses that format for every platform they support, but most home consoles were not designed to be end-user modifiable, and sometimes the same game shipped on different sets of different sizes of ROM chips (say, a 4Mbit chip, or two 2Mbit chips) depending on what was cheapest at the time. MAME as a collective project doesn't have the will to use different formats for different platforms, and home-console emulators aren't interested in following MAME's arcade-based conventions.
- near 7y ago
  MAME follows the internal database approach to emulation. The primary issue with this approach is that it does not support homebrew, fan translations, and ROM hacks. If you modify games, checksum lookups fail. For a ROM hack, you could still boot something where the original was in the database by manually specifying the name and accepting the warning (which is fine), but for a new unlicensed / homebrew title, you could only maybe possibly boot it by faking being another very similar game, which feels quite wrong. I'm not saying the MAME approach is wrong: every approach has its pros and cons, regrettably.
gnud 7y ago
Now, I'm _really_ not an emulator expert, but as I recall both snes9x and zsnes will read rom dumps from zip files. So it seems to be possible to do this without too much cooperation. If archivers started including useful metadata in addition to the rom dumps, I'm sure the emulators would start to use it as well. Or maybe the archivers are the problem?
- byuu 7y ago
  The latter. I support this, but it does no good when no one uses the format. The most prominent databasing effort will not include any additional files. Not even metadata, they're missing the coprocessor firmware required to play games. I wrote about this a few years back here ( https://higan.byuu.org/firmware https://higan.byuu.org/firmware ), but fair warning it's a bit rantier than my recent articles ^^; I've just taken a shotgun approach recently: supporting games split into multiple pieces inside folders, games inside ZIP archives, games in merged files to a single ROM, games with external firmware in a separate subfolder, games played by an internal database, games that have an external manifest to describe the PCBs, games played by heuristic detection, and games that fall back on less accurate high-level emulation instead of not booting at all.
zamadatix 7y ago
I must be missing understanding on some part of the picture of why many were so against it. Unless I'm missing something everything but "decompress the file and load the ROM in it" is completely optional to implement and would be no different than how things work with raw ROMs today, what was the push back on?
- thristian 7y ago
  There were a few points of contention: - for consistency and simplicity, byuu wanted a standard naming scheme for the files inside the archive. The main game data should be "program.rom", the non-volatile storage should be "save.ram", etc. Unfortunately, existing emulators all work by grabbing the first (or last?) file they find whose name ends in .sfc, so that was an incompatibility. - Many people don't pick their own filenames, they use a database like No-Intro or GoodSNES to rename and compress files automatically. Such tools would mistakenly rip apart structured game folders every time they were run. - Many emulators have config settings for "where to store save games" and "where to look for patch files". If a game folder contains a patch file, should it be loaded instead of a patch in the "patch files" directory? If a game folder is loaded, should save games be written to the folder instead of the "save games" directory?
- FitzRoy27 7y ago
  It's not just "the ROM in it", it's "the ROMs in it". Single-rom games don't benefit from containers. The whole point of treating all games as containers is to allow multi-rom games to be free of metadata. 30% of NES games, 1% of SNES games, and most arcade games are multi-rom. In order for the emulator to tell contents apart without metadata, you have to have a naming scheme. So, for example, Galaxian.fc.zip would contain "program.rom" and "character.rom". Currently, those pieces are blobbed together with a header in front. Thus you can't open a .NES file in your file browser and see if it has a character rom, what size it is, what sha256 it has. This causes massive tedium with NES dump verification since you can't assume anything at face value. All parts and values are obscured and need extracted. The other reason metadata is evil is that it's a royal PITA to maintain. It's far easier to update your emulator than thousands of individual files. Do not listen to anyone who tells you that we need to drag licensed games into the metadata mud for the sake of unlicensed games. They are pulling your chain, an unlicensed db for nes is entirely feasible, or even the mapper number applied to the extension ala "Cheetahmen II.224.fc.zip". MAME did this right with a database and NEVER writing inside the container, but only because their hand was forced into it. So many arcade games have 3 or more roms, so they were never tempted to take shortcuts and blob shit together to avoid zip containers. Then they screwed up their own good fortune with some truly bizarre stuff. Google non-merged vs merged and behold threads and threads of confusion. Now weigh that confusion against the supposed benefit.

by byuu 7y ago

bane 7y ago

I remember when Nesticle and some of the other viable Nintendo Entertainment System emulators first started showing up. It took a while but eventually everybody just ended up settling on the iNES format by Marat Fayzullin. This was a very useful thing. I've always found it extremely frustrating that later systems had so many competing formats for the roms as you inevitably end up with lots of duplicate games just so different emulators can handle them. It gets even worse for optical media with all the .cue file stuff and distributions that provide the audio tracks outside of the iso in half a dozen different formats. It's almost impossible to assemble a good collection for those kinds of systems. I honestly think a .zip based distribution makes a lot of sense since most emulators of cartridge based systems can open zipped archives and run the first compatible file they find in them. But all the rest of the information for the game could be "built in" to the zip. If you look at the Gamebase projects, there's plenty of hobbyists who are willing to pour thousands of hours into assembling games, metadata, manual scans and other such things into those kinds of projects...but they lack the ability to unify all that ephemera into a single archive per game. If such a thing were to become standard, I'm certain people/groups outside of the "established" database groups would begin to publish collections. I for one would definitely welcome a format with built-in cheats, patches and so on and would love to download a collection of say, Japanese RPGs that come with scanned manuals, translated manuals and can run the game in the original and patched versions....or to try out all the weird and wacky Mario/Sonic patched game variants. I just can't be bothered to go download each patch myself to try it out, but if they were already present in the .zip :) I think even support soundtracks would be awesome and could see people firing up collections of games just to listen to the music, without having to keep track of a separate soundtrack archive. I think this would be an even bigger benefit for modern optical media. The .bin/.cue stuff is honestly terrible. More than half the time I've ended up with a .bin or some kind of .something that I need to convert for some reason, or mount and extract stuff and then handbuild .cue files which don't work half the time. I don't even know if the problem is what I've downloaded, the .cue I made or the emulation compatibility. A standard distribution "format" that just solved all that would be incredibly useful.

near 7y ago
At some point, I want to talk about CDs as well. A proper CD archival format would include the lead-in sectors, with which it is absolutely trivial to extract track information from the Q-subchannel of it. Easier than parsing a CUE sheet, or MDF, or CCF, or whatever else have you. I do understand it's hard to rip this data from CDs (usually requires an older Plextor drive), but worst case, as bad as it'd be, generating the TOC from a CUE sheet is very simple, and if it's marked with a special entry to denote it being machine-generated, I feel that would be a good compromise. Then you'd have a single CD image format in a single file that captures all data on CDs. If you wanted to, you could then compress that image with either ZIP, or a custom-purpose tool that combines things like Reed-Solomon prediction, subchannel prediction, FLAC encoding for audio tracks, etc, resulting in images not any larger than ISO+FLAC. But well, https://xkcd.com/927 https://xkcd.com/927 and all, right? > I for one would definitely welcome a format with built-in cheats A thing I do with bsnes is bundle a database with most known cheat codes for all games, so you've got all the cheats nicely labeled and ready to go whenever you play a game. I kinda wish this idea caught on more for all the other systems and emulators, but it hasn't been all that popular.
- bane 7y ago
  I kind of think that part of the problem is that there needs to be a bit of a network effect here for these kinds of .zip packaged formats to take off: 1) An emulator of sufficient popularity has to fully support it. 2) The format has to provide some kind of benefit to the user such that they're willing to convert/duplicate their entire collection. 3) Somebody has to make and distribute collections of packaged roms that include enough extra stuff to make them worthwhile. To be honest, you might be in a good position w/r to #1 since your SNES emulator is now more or less the default in the community (at least for big multi emulation systems like retroarch). For #2, I think the benefit really derives from what's in the package: manuals, savestates, cheats, soundtracks, playthroughs (that you can jump to in the middle of), patches, variants, meta-data (publication info, credits, etc.) and so on could all be really interesting to lots of people, but really would rely on the emulator to take advantage of in some way. For #3, There's probably already enough information freely available to make a utility that would take somebody's existing ROM collection and yank down all the rest of the stuff from various websites all at once to build up an initial package of stuff. I kind of almost just think you should just go ahead and support something like this and have it slowly percolate through the community. If it achieve some kind of network effect it'll slowly replace the other formats in a de-facto way.

spilk 7y ago

how about MAME, which has a zip-based format containing raw ROM dumps, and the "CHD" format for disk/disc based media? obviously all of the metadata is still contained within MAME itself.

thristian 7y ago
MAME is a kind of special case, where the same group of people write the emulator and catalogue the ROM dumps. For most emulated platforms, there's more than one emulator, and often more than one popular emulator, so deciding on a new archival format is a major political endeavour, on top of the existing tricky technical problems. Also, even MAME is not immune to political ROM-format problems. The format MAME used was designed to help people maintain and update arcade boards, which often had the ROM chips socketed so they could be replaced by the end user and so "a set of ROMs for a game" was a reasonable thing to talk about. MAME uses that format for every platform they support, but most home consoles were not designed to be end-user modifiable, and sometimes the same game shipped on different sets of different sizes of ROM chips (say, a 4Mbit chip, or two 2Mbit chips) depending on what was cheapest at the time. MAME as a collective project doesn't have the will to use different formats for different platforms, and home-console emulators aren't interested in following MAME's arcade-based conventions.
near 7y ago
MAME follows the internal database approach to emulation. The primary issue with this approach is that it does not support homebrew, fan translations, and ROM hacks. If you modify games, checksum lookups fail. For a ROM hack, you could still boot something where the original was in the database by manually specifying the name and accepting the warning (which is fine), but for a new unlicensed / homebrew title, you could only maybe possibly boot it by faking being another very similar game, which feels quite wrong. I'm not saying the MAME approach is wrong: every approach has its pros and cons, regrettably.

gnud 7y ago

Now, I'm _really_ not an emulator expert, but as I recall both snes9x and zsnes will read rom dumps from zip files. So it seems to be possible to do this without too much cooperation. If archivers started including useful metadata in addition to the rom dumps, I'm sure the emulators would start to use it as well. Or maybe the archivers are the problem?

byuu 7y ago
The latter. I support this, but it does no good when no one uses the format. The most prominent databasing effort will not include any additional files. Not even metadata, they're missing the coprocessor firmware required to play games. I wrote about this a few years back here ( https://higan.byuu.org/firmware https://higan.byuu.org/firmware ), but fair warning it's a bit rantier than my recent articles ^^; I've just taken a shotgun approach recently: supporting games split into multiple pieces inside folders, games inside ZIP archives, games in merged files to a single ROM, games with external firmware in a separate subfolder, games played by an internal database, games that have an external manifest to describe the PCBs, games played by heuristic detection, and games that fall back on less accurate high-level emulation instead of not booting at all.

zamadatix 7y ago

I must be missing understanding on some part of the picture of why many were so against it. Unless I'm missing something everything but "decompress the file and load the ROM in it" is completely optional to implement and would be no different than how things work with raw ROMs today, what was the push back on?

thristian 7y ago
There were a few points of contention: - for consistency and simplicity, byuu wanted a standard naming scheme for the files inside the archive. The main game data should be "program.rom", the non-volatile storage should be "save.ram", etc. Unfortunately, existing emulators all work by grabbing the first (or last?) file they find whose name ends in .sfc, so that was an incompatibility. - Many people don't pick their own filenames, they use a database like No-Intro or GoodSNES to rename and compress files automatically. Such tools would mistakenly rip apart structured game folders every time they were run. - Many emulators have config settings for "where to store save games" and "where to look for patch files". If a game folder contains a patch file, should it be loaded instead of a patch in the "patch files" directory? If a game folder is loaded, should save games be written to the folder instead of the "save games" directory?
FitzRoy27 7y ago
It's not just "the ROM in it", it's "the ROMs in it". Single-rom games don't benefit from containers. The whole point of treating all games as containers is to allow multi-rom games to be free of metadata. 30% of NES games, 1% of SNES games, and most arcade games are multi-rom. In order for the emulator to tell contents apart without metadata, you have to have a naming scheme. So, for example, Galaxian.fc.zip would contain "program.rom" and "character.rom". Currently, those pieces are blobbed together with a header in front. Thus you can't open a .NES file in your file browser and see if it has a character rom, what size it is, what sha256 it has. This causes massive tedium with NES dump verification since you can't assume anything at face value. All parts and values are obscured and need extracted. The other reason metadata is evil is that it's a royal PITA to maintain. It's far easier to update your emulator than thousands of individual files. Do not listen to anyone who tells you that we need to drag licensed games into the metadata mud for the sake of unlicensed games. They are pulling your chain, an unlicensed db for nes is entirely feasible, or even the mapper number applied to the extension ala "Cheetahmen II.224.fc.zip". MAME did this right with a database and NEVER writing inside the container, but only because their hand was forced into it. So many arcade games have 3 or more roms, so they were never tempted to take shortcuts and blob shit together to avoid zip containers. Then they screwed up their own good fortune with some truly bizarre stuff. Google non-merged vs merged and behold threads and threads of confusion. Now weigh that confusion against the supposed benefit.