Teal Deer alert: I know this is too long; didn't read. But there were a lot of posts to reply to since I went to bed.
ibeenew2 wrote:FitzRoy wrote:ibeenew2 wrote:I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database.
But that's inconsiderate to those who want to work with and store this data in its native form without specialty tools.
Anyone working with the data will already need these "specialty tools" to generate the iNES header or IPS patch. If I have a ZapFC game and change one byte, it is no longer valid, so I would need other tools.
But only the
authors of hacks would need such tools: extract the .prg and .chr from the pristine ROM and the .ines from the database, make the hack, and then make and distribute a patch against the .prg and .chr files. I think the idea is that the
end user of a hack obtains the pristine ROM in ZapFC format and then uses in-emulator IPS support to patch the PRG or CHR at runtime.
FitzRoy wrote:If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
That's like saying the NES emulator itself is incomplete by design because it will never emulate clone systems. Define what the NES is as it pertains to preservation. Can you do that for me? Does it include toasters and calculators? What is NES hardware? Everything in the world or something more exclusive?
NES emulators are expected to run games from the original NES console. Your idea rejects hundreds (thousands?) of those games that were released during the NES era. It also rejects all homebrew games being created now. That is what makes your idea incomplete by design.
Emulators would continue to support iNES. They would also support zipped .prg+.chr+.ines, and they would support zipped .prg+chr without .ines for games that are on someone's canonical list of good headers. FitzRoy would make the list of North American NES games with the Official Nintendo Seal. Someone else would make the list of Famicom games using Nintendo boards or from publishers that were licensed in North America (e.g. Konami), and someone else would make the list of North American NES games from the well-known unlicensed publishers (Tengen, Color Dreams, Camerica, AVE).
FitzRoy's big problem appears to be that the proposal for switching from the iNES container to split PRG and CHR in a PKZIP container and the proposal for
allowing games with the Official Nintendo Seal to be distributed without the header have been conflated into a single proposal. One step at a time please.
For the last and final time, put emulators in charge of mapping, not users.
"Put emulators in charge of mapping" gave me an idea for an interesting research project to make an emulator that automatically guesses what board a game expects. For example, if it routinely writes to ROM locations with the same value, it's probably a discrete mapper of some sort. If a game routinely writes to columns in the middle of the screen as the screen scrolls, it needs its mirroring switched to vertical.
That way the entire known game set can be put in the database instead of just a subset.
After the entire known game set is published, someone's going to discover a prototype and the set will become obsolete. FitzRoy wants to start with a subset that won't grow, and then other subsets can be added to it.
FitzRoy wrote:I want CHR and PRG data to be CONTAINED
The CONTAINER is called iNES, a standard and easy container to work with.
But nothing else uses this container. 7-Zip can't open it, nor can File Roller, and libraries to manipulate it don't come with every *n?x distribution. It can't contain a box, label, or manual, or anything else needed to reproduce the original CIB. Nor does iNES even handle the 8 KiB PRG of Galaxian.
FitzRoy wrote:hacks to be patches
Does IPS even handle patching multiple files at once?
No. Nor does it handle expanding the PRG and moving the CHR to compensate. A new binary diff format is needed; I'd recommend a zipped collection of IPS files or (better yet) bsdiff files.
FitzRoy wrote:external mapping files required for unlicensed games only.
Why not external mapping files required for unknown games only, like with a good header database?
"Known games" would start with FitzRoy's set of licensed games. Then you could download someone else's Tengen+Camerica+Color Dreams+AVS set.
koitsu wrote:cpow wrote:Would making it a footer solve the problems? How about a margin note?
Making it a footer would actually make things more difficult and emulator ROM load-time slower.
That was a joke son, but PKZIP does indeed use a footer. But on platforms other than the NES and the GBA, an emulator can load the whole file into memory first.
koitsu wrote:One would have to seek to the end of the file + use ftell() (to find out how big the file is)
By which point the file's extent list or FAT entries are in the operating system's disk cache.
The qualm is that it's a waste of an inode
Switch to a PKZIP container with .prg, .chr, and .ines files, and you don't waste an inode.
ZIP might be the easiest to implement, but there's all sorts of caveats when it comes to the specification (consider 8.3 format vs. long filenames, etc.).
The zipfile libraries in operating systems and programming languages appear to handle these caveats well.
Maybe RAR might work better?
RAR? Hell no. Its license is non-free. If you want to improve on PKZIP, use 7-Zip.
Let's just stay with iNES 2.0 and be done with it
Sure, if you're willing to provide the split and join tools. Yes, join
tool, because it would ignore the file sizes in the provided header and properly double-up Galaxian's PRG.
FitzRoy wrote:tepples wrote:That's why I recommended using a more descriptive name than "Database".
Like header database? What's a header?
Like "board list", as I recommended earlier. Perhaps users might understand if I write the whole troubleshooter entry:
"All NES Game Paks contain printed circuit boards with ROM chips and other circuitry. A game will work only on the right board. Some ROM files come with board information, which may be correct or wrong; others don't have board information at all. $emulatorname uses a board list to determine the correct board to emulate for any game on the list. A board list for all North American NES games with the Official Nintendo Seal is available from FitzRoy's web site."
ZapFC is a double extension format, it already differentiates itself from a regular zip container.
As do .jar, .odt, .smzip, etc. So the big change from .nes to .zapfc is that the container is switched from the ad-hoc iNES to .zip, and an option is available to distribute ROMs without board information, relying on a separately distributed board list. Then you plan to make and distribute a board list for all North American NES games with the Official Nintendo Seal.
"Frankenfile" is a term I use for distinct parts combined in a unique or arbitrary way.
In other words, I now understand that "frankenfile" = a container not widely used. Thank you for clarifying.
frantik wrote:seriously though.. if you have a new container format, release a whole set of roms in that format and also make a great emulator which supports those roms
A tool to convert a whole set of ROMs from iNES to zipped split ROMs would be easy for me to make in Python, and leaving the 16-byte .ines file out of the result would give .zapfc exactly as FitzRoy appears to describe.
FitzRoy wrote:clueless wrote:but possibly with a larger header that would also include things like
1) Dumper (person)
2) Dumper device
3) Date of dump
4) Canonical title of game, its region code, language, etc...
Games should not be archived this way any more than books should be archived with your notes in or on the pages.
Books have a cover, and they have a
title page and verso.
I agree with clueless: make your board list, and patch an emulator to support zipped split ROMs.
3gengames wrote:iNES 2.0 was brought up because of this and it fixes pretty much everything.
Except for the problem of widely used ROM sets with incorrect board information.
byuu wrote:ZIP supports multiple encryption types, there are compressors (such as 7-zip) that can make ZIP files other tools cannot read
Require .zapfc files to use a baseline ZIP format: either uncompressed or DEFLATE with window no bigger than 32 KiB, and no crypto. What other "etc" were you thinking of that interferes with defining baseline ZIP?
ZIP also doesn't define the specs of filename storage, you can't know whether the filename is UTF-8, SJIS, or something else.
That's why zapfc would define a profile of ZIP where all filenames are UTF-8, not shit JIS.
And from a permanent preservation standpoint, there's no telling how long ZIP will remain dominant. Imagine finding a bunch of ARC or LHA files today.
The PKZIP 2 format has been around for 18 years. It's an international standard, part of
ISO/IEC 26300:2006 Open Document Format for Office Applications (OpenDocument) v1.0. It's far less obscure than the iNES container and on the same standardization footing as the ISO 9660 file system used on CDs.
I feel that all containers are simply reinventing the file system.
Different file systems for different purposes. In fact, I invented my own container for files inside a Game Boy Advance game because I had specific needs.
Hell, solid-archive the whole damn set and get some major space savings.
As I understand it, solid archiving saves only when multiple files share data, which is good for region variants but little else. And you need to extract them anyway before playing, as on average half the archive needs to be read to extract one file.
I'm sure many thought MPEG-2/XviD was amazing until H.264 came along. So you never know what can happen.
I know what will happen: MPEG is working on two new standards. One is HEVC (aka H.265), which is expected to run at half the bitrate of AVC (aka H.264) for a given quality level. The other is MPEG Royalty Free, which is expected to compete with Xvid without using any patented techniques that require royalties; this
will begin in March 2011.