ZapFC Headerless Format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

User avatar
FitzRoy
Posts: 144
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy »

byuu wrote:Nor would they be for extra files in folders. I'm not saying it's the sure-proof method here. There is too much inertia in getting the OS to treat a folder like a file.
You responded before I could edit, but I was going to give ease of future validation as the reason tools control distributors and not vice versa. But then I realized, according to my spec, emulators wouldn't play the files either. So there is truly no way unverifiable, broken roms would get distributed.

So if I understand what it is you're saying, you'd go out of spec for the folders because you don't think it would affect distribution. And you'd be right because it would be the least used variant. God that's underhanded.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

Teal Deer alert: I know this is too long; didn't read. But there were a lot of posts to reply to since I went to bed.
ibeenew2 wrote:
FitzRoy wrote:
ibeenew2 wrote:I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database.
But that's inconsiderate to those who want to work with and store this data in its native form without specialty tools.
Anyone working with the data will already need these "specialty tools" to generate the iNES header or IPS patch. If I have a ZapFC game and change one byte, it is no longer valid, so I would need other tools.
But only the authors of hacks would need such tools: extract the .prg and .chr from the pristine ROM and the .ines from the database, make the hack, and then make and distribute a patch against the .prg and .chr files. I think the idea is that the end user of a hack obtains the pristine ROM in ZapFC format and then uses in-emulator IPS support to patch the PRG or CHR at runtime.
FitzRoy wrote:
If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
That's like saying the NES emulator itself is incomplete by design because it will never emulate clone systems. Define what the NES is as it pertains to preservation. Can you do that for me? Does it include toasters and calculators? What is NES hardware? Everything in the world or something more exclusive?
NES emulators are expected to run games from the original NES console. Your idea rejects hundreds (thousands?) of those games that were released during the NES era. It also rejects all homebrew games being created now. That is what makes your idea incomplete by design.
Emulators would continue to support iNES. They would also support zipped .prg+.chr+.ines, and they would support zipped .prg+chr without .ines for games that are on someone's canonical list of good headers. FitzRoy would make the list of North American NES games with the Official Nintendo Seal. Someone else would make the list of Famicom games using Nintendo boards or from publishers that were licensed in North America (e.g. Konami), and someone else would make the list of North American NES games from the well-known unlicensed publishers (Tengen, Color Dreams, Camerica, AVE).

FitzRoy's big problem appears to be that the proposal for switching from the iNES container to split PRG and CHR in a PKZIP container and the proposal for allowing games with the Official Nintendo Seal to be distributed without the header have been conflated into a single proposal. One step at a time please.
For the last and final time, put emulators in charge of mapping, not users.
"Put emulators in charge of mapping" gave me an idea for an interesting research project to make an emulator that automatically guesses what board a game expects. For example, if it routinely writes to ROM locations with the same value, it's probably a discrete mapper of some sort. If a game routinely writes to columns in the middle of the screen as the screen scrolls, it needs its mirroring switched to vertical.
That way the entire known game set can be put in the database instead of just a subset.
After the entire known game set is published, someone's going to discover a prototype and the set will become obsolete. FitzRoy wants to start with a subset that won't grow, and then other subsets can be added to it.
FitzRoy wrote:I want CHR and PRG data to be CONTAINED
The CONTAINER is called iNES, a standard and easy container to work with.
But nothing else uses this container. 7-Zip can't open it, nor can File Roller, and libraries to manipulate it don't come with every *n?x distribution. It can't contain a box, label, or manual, or anything else needed to reproduce the original CIB. Nor does iNES even handle the 8 KiB PRG of Galaxian.
FitzRoy wrote:hacks to be patches
Does IPS even handle patching multiple files at once?
No. Nor does it handle expanding the PRG and moving the CHR to compensate. A new binary diff format is needed; I'd recommend a zipped collection of IPS files or (better yet) bsdiff files.
FitzRoy wrote:external mapping files required for unlicensed games only.
Why not external mapping files required for unknown games only, like with a good header database?
"Known games" would start with FitzRoy's set of licensed games. Then you could download someone else's Tengen+Camerica+Color Dreams+AVS set.
koitsu wrote:
cpow wrote:Would making it a footer solve the problems? How about a margin note?
Making it a footer would actually make things more difficult and emulator ROM load-time slower.
That was a joke son, but PKZIP does indeed use a footer. But on platforms other than the NES and the GBA, an emulator can load the whole file into memory first.
koitsu wrote:One would have to seek to the end of the file + use ftell() (to find out how big the file is)
By which point the file's extent list or FAT entries are in the operating system's disk cache.
The qualm is that it's a waste of an inode
Switch to a PKZIP container with .prg, .chr, and .ines files, and you don't waste an inode.
ZIP might be the easiest to implement, but there's all sorts of caveats when it comes to the specification (consider 8.3 format vs. long filenames, etc.).
The zipfile libraries in operating systems and programming languages appear to handle these caveats well.
Maybe RAR might work better?
RAR? Hell no. Its license is non-free. If you want to improve on PKZIP, use 7-Zip.
Let's just stay with iNES 2.0 and be done with it ;-)
Sure, if you're willing to provide the split and join tools. Yes, join tool, because it would ignore the file sizes in the provided header and properly double-up Galaxian's PRG.
FitzRoy wrote:
tepples wrote:That's why I recommended using a more descriptive name than "Database".
Like header database? What's a header?
Like "board list", as I recommended earlier. Perhaps users might understand if I write the whole troubleshooter entry:

"All NES Game Paks contain printed circuit boards with ROM chips and other circuitry. A game will work only on the right board. Some ROM files come with board information, which may be correct or wrong; others don't have board information at all. $emulatorname uses a board list to determine the correct board to emulate for any game on the list. A board list for all North American NES games with the Official Nintendo Seal is available from FitzRoy's web site."
ZapFC is a double extension format, it already differentiates itself from a regular zip container.
As do .jar, .odt, .smzip, etc. So the big change from .nes to .zapfc is that the container is switched from the ad-hoc iNES to .zip, and an option is available to distribute ROMs without board information, relying on a separately distributed board list. Then you plan to make and distribute a board list for all North American NES games with the Official Nintendo Seal.
"Frankenfile" is a term I use for distinct parts combined in a unique or arbitrary way.
In other words, I now understand that "frankenfile" = a container not widely used. Thank you for clarifying.
frantik wrote:seriously though.. if you have a new container format, release a whole set of roms in that format and also make a great emulator which supports those roms
A tool to convert a whole set of ROMs from iNES to zipped split ROMs would be easy for me to make in Python, and leaving the 16-byte .ines file out of the result would give .zapfc exactly as FitzRoy appears to describe.
FitzRoy wrote:
clueless wrote:but possibly with a larger header that would also include things like

1) Dumper (person)
2) Dumper device
3) Date of dump
4) Canonical title of game, its region code, language, etc...
Games should not be archived this way any more than books should be archived with your notes in or on the pages.
Books have a cover, and they have a title page and verso.

I agree with clueless: make your board list, and patch an emulator to support zipped split ROMs.
3gengames wrote:iNES 2.0 was brought up because of this and it fixes pretty much everything.
Except for the problem of widely used ROM sets with incorrect board information.
byuu wrote:ZIP supports multiple encryption types, there are compressors (such as 7-zip) that can make ZIP files other tools cannot read
Require .zapfc files to use a baseline ZIP format: either uncompressed or DEFLATE with window no bigger than 32 KiB, and no crypto. What other "etc" were you thinking of that interferes with defining baseline ZIP?
ZIP also doesn't define the specs of filename storage, you can't know whether the filename is UTF-8, SJIS, or something else.
That's why zapfc would define a profile of ZIP where all filenames are UTF-8, not shit JIS.
And from a permanent preservation standpoint, there's no telling how long ZIP will remain dominant. Imagine finding a bunch of ARC or LHA files today.
The PKZIP 2 format has been around for 18 years. It's an international standard, part of ISO/IEC 26300:2006 Open Document Format for Office Applications (OpenDocument) v1.0. It's far less obscure than the iNES container and on the same standardization footing as the ISO 9660 file system used on CDs.
I feel that all containers are simply reinventing the file system.
Different file systems for different purposes. In fact, I invented my own container for files inside a Game Boy Advance game because I had specific needs.
Hell, solid-archive the whole damn set and get some major space savings.
As I understand it, solid archiving saves only when multiple files share data, which is good for region variants but little else. And you need to extract them anyway before playing, as on average half the archive needs to be read to extract one file.
I'm sure many thought MPEG-2/XviD was amazing until H.264 came along. So you never know what can happen.
I know what will happen: MPEG is working on two new standards. One is HEVC (aka H.265), which is expected to run at half the bitrate of AVC (aka H.264) for a given quality level. The other is MPEG Royalty Free, which is expected to compete with Xvid without using any patented techniques that require royalties; this will begin in March 2011.
User avatar
Memblers
Site Admin
Posts: 4044
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Post by Memblers »

tepples wrote:
Memblers wrote:My main complaint with ZIP'ed ROMs is because the NES is a computer itself, it seems counter-productive to use a format that the system itself can't work with.
The NES itself doesn't understand the iNES header any more than it understands the PKZIP header. So I guess your complaint boils down to the fact that iNES is implemented in the PowerPak and PKZIP is not.
Sorta yeah, but there is no source release for the PowerPak, so I would still have to implement file unzipping by myself (if the NES could even do it with all available WRAM). And for absolutely no benefit at all to the user (compression doesn't matter for something this small, these aren't floppy disks), I surely wouldn't waste my time on that.
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

I'll write a ROM conjoining tool, later. Heck, if I get really bored and mess with some more C, I'll make a program to remove the headers and make two separate .bin files of the ROMs for you. It'll take a couple minutes if you have all the ROM's in a file and just CTRL+A and then drag them all the the windows application. I'll work on these later maybe. I don't feel like doing anything truthfully. :P
User avatar
clueless
Posts: 496
Joined: Sun Sep 07, 2008 7:27 am
Location: Seatlle, WA, USA

Post by clueless »

3gengames wrote:I'll write a ROM conjoining tool, later. Heck, if I get really bored and mess with some more C, I'll make a program to remove the headers and make two separate .bin files of the ROMs for you. It'll take a couple minutes if you have all the ROM's in a file and just CTRL+A and then drag them all the the windows application. I'll work on these later maybe. I don't feel like doing anything truthfully. :P
I'm just curious. Why write such a one-time use tool as a GUI? This would be ~20 lines or less in perl or python, or ~40 lines of ANSI-C as a command line app that just takes takes filenames in argv[]. Let the shell expand the wildcard glob for you (well, in Windows you still have to do that yourself, but its pretty trivial, 10 lines of code using the win32 api). A gui app w/ drag + drop seems like overkill to me.

I'm not trying to pick a fight / flamewar. I'm just curious about why a GUI app.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

clueless wrote:A gui app w/ drag + drop seems like overkill to me.
I don't think he meant to make a GUI application, since you can drag files and drop them on command line application EXEs, which causes the dropped files to be passed as parameters.
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

Yep, working on it now. C is alot better when you get the hang of it. :D Even if he doesn't use this...I may.
User avatar
Bregalad
Posts: 8055
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Post by Bregalad »

Sorry to sound extremely sarcastic, but just to say I'm laughing loud at you guys doing a 8-page hot flamed debate on a yet another useless proposal for iNES remplacement that nobody will ever use.

MWAHAHAHAHAHA.

(don't tell me that wasn't constructive I know that, I just couldn't resist)
Useless, lumbering half-wits don't scare us.
User avatar
FitzRoy
Posts: 144
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy »

tepples wrote:No. Nor does it handle expanding the PRG and moving the CHR to compensate. A new binary diff format is needed; I'd recommend a zipped collection of IPS files or (better yet) bsdiff files.
In this instance, patch conversion would be far easier than the SNES because headers on ".nes" files were a certainty.
.prg, .chr, and .ines files
When you create an external mapper format for unlicensed games, I would recommend not basing it off iNES, but rather something which allows custom mappers to be created. iNES (any revision) has no mapping guarantee for licensed games and doesn't allow custom mapping for unlicensed. That is the gist of its problems.
.nes to .zapfc
The extension is not .zapfc. It's .fc.zip or .fc.7z. I call the format ZapFC because of how I store the data and calculate the checksums.
Require .zapfc files to use a baseline ZIP format: either uncompressed or DEFLATE with window no bigger than 32 KiB, and no crypto. What other "etc" were you thinking of that interferes with defining baseline ZIP?
Aren't those requirements implied by the fact that ZapManager will only validate such a zip? If you can't find a verifiable source set, you went terribly wrong somewhere.
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

Well until I can name a file from char pointer, (ROM+variable+.BIN) in C, it won't happen. Can't get multi file names so whatever. I give up. And the new custom mapper is easy to add to your own dev emulator and header, then get it filed later to the official iNES mapper sheet. Not like there's millions of new mappers being made anyway.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

FitzRoy wrote:When you create an external mapper format for unlicensed games, I would recommend not basing it off iNES, but rather something which allows custom mappers to be created. iNES (any revision) has no mapping guarantee for licensed games and doesn't allow custom mapping for unlicensed. That is the gist of its problems.
How would one represent a custom mapper other than a netlist with the NES on one side and ROMs and RAMs on the other, which would have to be simulated every CPU cycle (1.8 MHz) and every PPU fetch (2.7 MHz)? Feel free to implement your idea in an emulator.
The extension is not .zapfc. It's .fc.zip or .fc.7z.
Even worse. Windows Explorer decides what program to start based solely on the characters after the last dot. You'll have to associate .zip and .7z files with a workaround program that starts your compression program if no ROMs are found inside.
Require [fc.zip] files to use a baseline ZIP format: [etc]
Aren't those requirements implied by the fact that ZapManager will only validate such a zip?
I was pointing out to others what a "profile of the ZIP format" might mean. And for which platforms will ZapManager be made available?
3gengames wrote:Well until I can name a file from char pointer, (ROM+variable+.BIN) in C, it won't happen.
I don't understand what you mean by this, other than snprintf.
frantik
Posts: 377
Joined: Tue Mar 03, 2009 3:56 pm

Post by frantik »

Bregalad wrote:Sorry to sound extremely sarcastic, but just to say I'm laughing loud at you guys doing a 8-page hot flamed debate on a yet another useless proposal for iNES remplacement that nobody will ever use.

MWAHAHAHAHAHA.

(don't tell me that wasn't constructive I know that, I just couldn't resist)
yes i keep coming back to this thread just to see how how manages to keep going. Tepples breathed a lot of life into it with that huge post though :-p
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

I need to make files that say ROM#.bin, and the variable part can't happen I think? Not from what I tell. :/ Oh well....
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

Do you mean you don't know how to make strings that say "ROM0.bin", "ROM1.bin", etc.? If so:

Code: Select all

char filename[16];
snprintf(filename, sizeof(filename),
         "ROM%d.bin", filenumber);
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

Can that be put into a fopen somehow? I am needing to name all the files in that order, then a master text file will tell what NES rom is equal to what bin files for this, so they're seperated.
Post Reply