ZapFC Headerless Format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Mon Feb 28, 2011 11:24 pm

FitzRoy wrote:Although this may not have been true for the NSS, many SNES games that initially shipped between several PRG roms were later manufactured on one. Same data. So that should tell you that the PRG data was whole to begin with. To avoid the distribution of arbitrary variants, that's how it should probably be distributed.
and why does it seem to hurt everyone supporting two variants of a game, even if the content is the same? oh, maybe the waste of HD space... well, I'll go deleting a single one of the tons of .avi I have to make space for all the existing variants of NES and SNES carts...
FitzRoy wrote:There were also a few NES games that did this with the CHR data.
yes. e.g. Excitebike and Pinball came in two variants. they both are supported by MESS. once you use xml and split prg/chr, it costs 0 lines of codes to support different ways to split prg or chr, as long as the offset you have to load files to is present in the xml.
FitzRoy wrote:And then their hardware will eventually die anyway, so why does emulation need to concern itself with how many ways the data could have been split depending on rom costs?
and who are you to decide they should not be able to repair their hardware when they still can?



anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...

User avatar
FitzRoy
Posts: 125
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy » Mon Feb 28, 2011 11:55 pm

etabeta wrote:and who are you to decide they should not be able to repair their hardware when they still can?
All these people really need is documentation. It doesn't require thousands of people storing redundant data to give a few people the knowledge to repair doomed hardware.
anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...
It might be a few weeks before I flesh that out.

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Tue Mar 01, 2011 12:33 am

FitzRoy wrote:
anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...
It might be a few weeks before I flesh that out.
ah ok. good luck.

it is a bit tricky: basically the board is always the same (and it's basically a mapper 3 pcb) but there are two pins that can be set to open or close in the factory. the game code at start goes to read the correspondent addresses and if it finds the wrong value, it does not start properly, because it thinks you are pirating the cart or flashing the game on a different pcb.

anyway, bootgod sorted it out (as described in the posts I linked) and we verified the settings for all the games using that pcb type

the same solution you might find in your format for this could also be of use to handle VRC2-VRC4-VRC6 variants which are currently spread across different mappers and require ad hoc address masks to work in emulator.

bootgod found out that the difference between the models are related to different wiring of specific pcb lines, so that different wirings result in different bits being responsible of the prg/chr bankswitch & mirroring mechanism.
hence, you can decide to support 6 mappers with address masks, or you can decide to emulate these as three pcb types only, by setting the proper address lines at start on a game-dependent base.

both NEStopia and MESS follow the latter approach, instead of supporting separately mappers 21, 22, 23, 24, 25, 26.
the only problem is that different games use different lines, so that it can be tricky to cover combinations all in your format (VRC2 has 3 pin connections to be checked, while VRC4 and VRC6 have only two, but they could in principle be connected to any of the 8 prg + 8chr address lines, even if only a few combinations have been found so far).

let me know if you need any test cases for these or for mapper 185 games :)




[1] actually, if we would like to be uber-picky, mapper 3 and 185 could be merged (given that they are the same except for these pins), and pins should be set to open/open or close/close (I don't remember which one of the two) for mapper 3 games which do not check those addresses... however, not a priority.

User avatar
Memblers
Site Admin
Posts: 3799
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Post by Memblers » Tue Mar 01, 2011 8:17 am

FitzRoy wrote:
zamenus wrote:Apologies if I've overlooked something, but how is this format proposal better than Nestopia's "NES XML" format? As long as you supply the headerless/split ROM with an XML file that has the cartridge information, you should have the functionality that you want*.
I did it to reduce the filesize by 2/3 and use a stronger hash.
Reduce the filesize, why does that matter? Is that also an unspoken reason you want to use PKZIP format? Even though seemingly everyone here who works with the actual NES hardware is opposed to it? It's not 1997, when I used to store all my ROMs in ZIP files with a front-end to unpack them. Now I've got a 1.5TB hard drive. Why in the world would I, or anyone else in the future, care about filesize for something this small?

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Tue Mar 01, 2011 9:11 am

Memblers wrote: Is that also an unspoken reason you want to use PKZIP format?
are you suggesting to keep the files unzipped or to use different compression formats?

FWIW, MAME/MESS treats zips like folders: you can have prg/chr either in gamename.zip or uncompressed in a gamename/ folder. zip is only used to rapidly verify checksums, so that you can store files inside the zip even with the wrong filename and they will be loaded (if you keep file unzipped, their name must match MAME/MESS internal filenames)

that's why we have never cared about .ace, .rar or .7z. zip already gives all we need :)

User avatar
kevtris
Posts: 504
Joined: Sat Oct 29, 2005 2:09 am
Location: Indianapolis
Contact:

Post by kevtris » Tue Mar 01, 2011 9:36 am

etabeta wrote:
FitzRoy wrote:And then their hardware will eventually die anyway, so why does emulation need to concern itself with how many ways the data could have been split depending on rom costs?
and who are you to decide they should not be able to repair their hardware when they still can?
This is one of the bigger lols to me when it comes to storing data as a single block or split up into chip size chunks. The reason I find it so funny is that most licensed NES games were stored on ROMs that are not pin compatible with EPROMs or any other programmable logic device. Arcade boards on the other hand, often DO have EPROMs or JEDEC pinout ROMs.

Nintendo was fond of using non-JEDEC pinout ROMs for everything NES, so it's impossible to fix broken cartridges without mucho rewiring. In which case, the person doing the rewiring is going to be savvy enough to be able to figure out how to cut out the chunk of data needed.

Not to mention that there's just no way to easily replace those 128K byte 28 pin ROMs that were very popular for awhile. If one of those is bad you might as well just buy another cartridge unless you like to do lots of rewiring.

(As an aside, I noticed that Nintendo wasn't so averse to using JEDEC pinout stuff on the Famicom cartridges though. My cart of Just Breed has a JEDEC pinout 512K ROM on it which I socketed. I am not sure about their earlier releases though, if they were JEDEC or not.)

I've always had an idle curiosity over why Nintendo decided to use two different kinds of ROMs for the PRG and CHR on cartridges that don't have compatible pinouts. Either it was speed issues, or just stubbornness, or it was to try and slow down piracy.
/* this is a comment */

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Tue Mar 01, 2011 9:40 am

we were talking about SNES, which I thought it uses pretty standard chips. However, I did not know about the pin incompatibility: were they incompatible in PC10 and VS arcade pcbs too? it does not sound a really smart move, unless their repairing service was super-efficient with arcades...

User avatar
clueless
Posts: 498
Joined: Sun Sep 07, 2008 7:27 am
Location: Seatlle, WA, USA

Post by clueless » Tue Mar 01, 2011 9:40 am

Keep the files unzipped so that they are easier to manipulate with low-level C or ASM code. Flash carts that run on the NES use the NES's CPU to drive all the logic. Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.

Reading a file format like iNES or UNIF is fairly easy in C and ASM. Any compression container adds a lot of overhead for such small systems and even PC based utilities. I know that others have voiced opposition to TAR, but I find TAR really easy to code to. The "unix baggage" only matters if one is using TAR to archive file attributes for future recovery operations.

tepples
Posts: 21875
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples » Tue Mar 01, 2011 9:51 am

clueless wrote:Keep the files unzipped so that they are easier to manipulate with low-level C or ASM code. Flash carts that run on the NES use the NES's CPU to drive all the logic.
If they're based on NAND+RAM like a PowerPak (NES) or SuperCard (GBA). NOR flash-based cards like Blargg's Munchausen cart (NES) or FireLinker (GBA) do the logic in PC-side flash-programming software.
Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.
The PowerPak has 512 KiB of PRG RAM dedicated to emulating PRG ROM. Why can't the unzipper use that?
I know that others have voiced opposition to TAR, but I find TAR really easy to code to.
I looked at tar early in my GBA era (2001), but I discovered that searching for a file by name involves a linear search through the whole file. At the time, I wanted something where I could binary search through only the directory, and I came up with a simple uncompressed archive format that I named GBFS. Now GBFS tools are included with every copy of devkitARM.

User avatar
clueless
Posts: 498
Joined: Sun Sep 07, 2008 7:27 am
Location: Seatlle, WA, USA

Post by clueless » Tue Mar 01, 2011 10:22 am

tepples wrote:
clueless wrote:Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.
The PowerPak has 512 KiB of PRG RAM dedicated to emulating PRG ROM. Why can't the unzipper use that?
I did not consider that possibility. But what happens when you _WANT_ to decompress a PROG-ROM that big?
tepples wrote:
I know that others have voiced opposition to TAR, but I find TAR really easy to code to.
I looked at tar early in my GBA era (2001), but I discovered that searching for a file by name involves a linear search through the whole file. At the time, I wanted something where I could binary search through only the directory, and I came up with a simple uncompressed archive format that I named GBFS. Now GBFS tools are included with every copy of devkitARM.
But in this case, the "container" only holds at most three objects: prog-rom, char-rom and the "board" file.

A few years ago I wrote a game that stored all of its graphics assets as PNGs inside a single TAR file. My game would mmap() (or the win32 equiv on the windows build) the TAR file read-only, scan through it once making an index (in memory). The code would then directly used the mapped bytes, instead of reading them into a buffer. on huge files, memory mapped IO is more efficient than normal file IO, especially on a Windows NT based kernel (which implements file reading as memory maps under the hood anyway). Since then, I've regarded TAR as a viable container format, depending on the application. TAR headers are fairly easy to parse. The use of octal digit encoding is really odd, but hey, tar was invented in the 70s. They smoked a lot of odd shit back then...
HISTORY
A tar command appeared in Seventh Edition Unix, which was released in January, 1979. It replaced the tp program from Fourth Edition Unix which in turn replaced the tap program from First Edition Unix. John Gilmore's pdtar public-domain implementation (circa 1987) was highly influential and formed the basis of GNU tar. Joerg Shilling's star archiver is another open-source (GPL) archiver (originally developed circa 1985) which features complete support for pax interchange format.
from http://www.freebsd.org/cgi/man.cgi?quer ... +8-current
Last edited by clueless on Wed Mar 02, 2011 6:33 am, edited 1 time in total.

User avatar
Memblers
Site Admin
Posts: 3799
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Post by Memblers » Tue Mar 01, 2011 10:52 am

etabeta wrote: are you suggesting to keep the files unzipped or to use different compression formats?
Unzipped, absolutely. It seems fine to me that an emulator would load a ROM set from a zip file (in fact, many of them already do for individual zipped iNES files). But it seems totally counter-productive to make the compression enshrined in the file format, because then the NES can't even use it's own ROM format.. Yeah the NES is a "console", but it was designed as computer originally, even the name Famicom stands for 'Family Computer'.

So the compression pretty much seems to make some people's life harder, while lots of users would see no change (and wouldn't care one bit what format it's in). I know if I release a game, I sure wouldn't use this format if I have to tell the users "keep it zipped if you use this device, unzip it if you use this device, sorry you can't rezip it on your own because the algorithm isn't supported on this device, but it is on that one", etc. Much better to just make it an uncompressed format, then just let the emulators load from ZIP files if they want (like they already do).

BTW, PC10 carts, I have 10 of them and most of them do have EPROMs. Some of them have MaskROMs though, but it also seems that some of the boards conveniently have solder jumpers you can change to select either type of chip. Of course that's different from NES carts, where you'd have to cut traces and rewire. I'd bet VS games were all EPROM, because of the code modification needed for coin-op. But I'm not sure. The VS mainboard takes EPROMs, but some games were released on daughterboards, so anything is possible there.

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Tue Mar 01, 2011 2:35 pm

Memblers wrote:Unzipped, absolutely. It seems fine to me that an emulator would load a ROM set from a zip file (in fact, many of them already do for individual zipped iNES files). But it seems totally counter-productive to make the compression enshrined in the file format, because then the NES can't even use it's own ROM format..
agreed. it's the emulator which should take care of compression/decompression if needed. I had missed this particular aspect of the format (14 pages read in a morning were a bit too much, sorry)
Memblers wrote:BTW, PC10 carts...
very interesting info, thanks. that's probably where I got the (wrong) idea that also PRG/CHR were using standard ROM chips. now I have a clearer picture.

User avatar
FitzRoy
Posts: 125
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy » Tue Mar 01, 2011 4:16 pm

Decompressing the set to ".fc" folders treated as files by the flash cart is probably better suited for stuff like flash carts. The format only enforces ZIP as the default because folders can't be download links and flash carts are less than 1% of users.

panzeroceania
Posts: 8
Joined: Mon Feb 28, 2011 4:09 am

Post by panzeroceania » Tue Mar 01, 2011 7:54 pm

yeah, as I understood it, it's more about using it as a container and a download link than to save space right?

As stated, the Famicom Rom Set is not that big by today's standards.

etabeta
Posts: 109
Joined: Wed Nov 29, 2006 10:11 am
Location: Trieste, Italy

Post by etabeta » Tue Mar 01, 2011 11:51 pm

FitzRoy wrote:Decompressing the set to ".fc" folders treated as files by the flash cart is probably better suited for stuff like flash carts. The format only enforces ZIP as the default because folders can't be download links and flash carts are less than 1% of users.
but this in my opinion bloats a bit the whole idea: why do you need .fc folders and zip container for download? why not plain zip and extension-less folders?

when you already have a database of hashes to match the files with, what is the advantage of a prescribed extension like .fc? you can open all directories and simply refuse to load files which do not match your database...

or am I missing anything?

Post Reply