New File Format

[mattmatteh]

if its only once then a convert utility would probably work better anyway. and use a script or util to change many roms in a folder.

matt

[tokumaru]

but then you have this higher nibble lower nibble rubbish

This just makes the value a bit more difficult to read. If a person is too lazy to to form a byte out of 2 nibbles this does not justify a new file format.

followed by eight bytes or so of total and utter crap, namely someones name which would confuse an emulator if we implemented the unused bytes

A second identifier can prevent this. If this ID is not present (wich would be the case if crap was written in those 8 bytes), the emulator should just consider this an old format iNES file, and ignore those 8 bytes, just as it is today.

What's wrong with my previous post which has a diagram of the new format. Not only is it very similar to iNES, it is also very easy to implement into ROMs.

It makes no sense to come up with a format that is so similar to iNES. The point here is not that iNES is good. The only reason we're trying to expand iNES (as opposed to comming up with something new) is compatibility. If we were to come up with something new (and kill compatibility altogether), I'm sure it could be better than iNES or the format you proposed.

You don't need a convert utility, only a HEX editor, and you can do the whole thing by hand.

Again, laziness from the part of the programmer does not justify a new format.

[blargg]

Bregalad, your summary is a good idea. Helps get people on the same page of the debate.

Well, IMO the ROMs should be a digital copy of the cartridge itself, and having the SRAM inside of the file is a good idea.

I do agree that a .nes file is not just a ROM, rather a virtual NES cartridge. This isn't a technical reason for having SRAM inside the file, more a common-sense reason.

Someone said that people running ROMs off of CDs couldn't save their SRAM data. Well, you still couldn't save your SRAM data anyway, even if it wasn't in the file. It would have to be directed to the hard drive which would defeat the whole point of having the ROM on a CD.

What if a person wants to keep all umpteen-thousand games on a CD-ROM and their save states on the hard drive? Or keep the game files on a directory for which they have read-only access, in order to prevent rogue programs from wiping out their game collection? Or they want to avoid having their backup program re-copy entire game files every time they play, rather than just the smaller save state file?

a lot of online communities encourage trading of SRAM files, save states, and movies but not the ROM files themselves due to copyright.

That ends the debate about save states in game files in my mind: due to this reason alone, it is just not viable.

Code: Select all

4E 45 53 02 01 00 01 00 
|  |  |  |  |  |  |  | 
|  |  |  |  |  |  |  +- (to make it 8 bytes so debuggers look ok) 
|  |  |  |  |  |  +- Mirroring etc. 
|  |  |  |  |  +- Mapper/Board 
|  |  |  | +- CHR Banks 
|  |  | +- PRG Banks 
|  |  +- S 
|  +- E 
+- N

All I see is a mere rearrangement of bytes, with even less room for expandability. What the heck is the gain over iNES?

I agree that crazy ideas won't help here. As I see it, we do not need a "revolution".

Would a Wii in November do?

[tokumaru]

All I see is a mere rearrangement of bytes, with even less room for expandability. What the heck is the gain over iNES?

Exactly. It's like he just doesn't like to mess with nibbles. Even though this is a thing that the emulator author has to do only once...

Would a Wii in November do?

I don't need a revolution, nor a Wii. Or a PS3. Or an XBOX 360. The PS2 I have at home is already too much. 3D sucks (words from a guy who is trying to make a "3D" raycaster for the NES! O.o).

[Zepper]

Wii sucks. Honestly, I enjoy the NES. ^_^;;..

About the "new" format, well...
The iNES seems deprecated because of arcaic times of emulation, do you remember? Yes, lots of "l33t" offering iNES ROM images and "signing them" right in the header. The most (un)famous is DiskDude junk that makes a lot of ROMs to fail when loaded in an emulator.

Now, you CANNOT suggest an "updated" iNES thing by cruching it. Plus, any format must need free space to be expanded somewhat. You should discuss what information is REALLY required to get the game EMULATED or RUNNING into a NES. This is the point, which UNIF seems to choose the 2nd option...

[blargg]

The most (un)famous is DiskDude junk that makes a lot of ROMs to fail when loaded in an emulator.

If emulator finds 16th byte of iNES header is non-zero, clear 8th through 16th bytes to zero before parsing. Problem solved?

[Bregalad]

You mean 15th byte, right ?
Anyway, the current iNES specification DOES SAY all bytes 8-15 must be zeros. All ROMs having something in there aren't valid iNES ROMs, so they won't be valid on a updated iNES either.
However, a converting/fixing tool like good NES could fix all the ROMs with junk in their header. However, this shouldn't have anything to do with the FORMAT itself.

EDIT : About recent console, I don't need any PS3/XBC360/Wii either. 3D sucks, except if it is really well done. The only 3D graphics I ever loved are in Final Fantasy X. (Dragon Quest VIII is also have fair landscapes and characters, but it still hurt eyes a bit).
And I'm against 3D in the NES more than anything else. However, I'm very pro 2D (or 3D isometric) recent games, even if there is REALLY a few titles here.

[tokumaru]

If emulator finds 16th byte of iNES header is non-zero, clear 8th through 16th bytes to zero before parsing. Problem solved?

You mean 15th byte, right ?

Byte 15 is the 16th one, because 0 is the 1st. I think that blargg meant "clear 9th through 16th bytes" though, since the 8th byte is byte 7 and it is used by iNES.

[WedNESday]

Code: Select all

4E 45 53 02 01 00 01 00 
|  |  |  |  |  |  |  | 
|  |  |  |  |  |  |  +- (to make it 8 bytes so debuggers look ok) 
|  |  |  |  |  |  +- Mirroring etc. 
|  |  |  |  |  +- Mapper/Board 
|  |  |  | +- CHR Banks 
|  |  | +- PRG Banks 
|  |  +- S 
|  +- E 
+- N

All I see is a mere rearrangement of bytes, with even less room for expandability. What the heck is the gain over iNES?

Firstly, then information is better provided and easier to read. I know that someone has said that programmer laziness is no need for a new header, I disagree. I do agree that no extra information is needed other than to run the ROM (i.e. checksums, ROM name etc). As for there being no room for expansion, iNES has never needed those eight extra bytes, and they were just abused. With the given format, what more could we possibly need?

Code: Select all

4E 45 53 02 01 00 01 00 
|  |  |  |  |  |  |  | 
|  |  |  |  |  |  |  +- (to make it 8 bytes so debuggers look ok) 
|  |  |  |  |  |  +- Mirroring etc. (see below)
|  |  |  |  |  +- Mapper/Board 
|  |  |  | +- CHR Banks 
|  |  | +- PRG Banks 
|  |  +- S 
|  +- E 
+- N

Byte 7 (06h) (maybe this order)

bit 0 - Horizontal/Vertical Mirroring
bit 1 - SRAM
bit 2 - 50/60hz
bit 3 - 4 Screen VRAM
bit 4 - Trainer
bit 5 - 1 Screen Mirroring (overrides bit 0)
bit 6 - 0 (reserved)
bit 7 - 0 (reserved)

As far as the mapper/board number goes, can we stop talking about why this format will be no good and actually start brainstorming. We will stick with the mapper numbers or move on to boards instead?

[CaH4e3]

0..255 is not enough to some multigame carts with maximum capacity... Anyway, mostly maximum number is 256, so at least it needs to hold hi bit of each PRG and CHR banks number along with common flags or something... But easier way is to assign 2byte values to bank numbers...

Mirroring is better to represent with two bits:

00 - one-page mirroring
01 - Vertical mirroring
10 - Horizontal mirroring
11 - Four screen mirroring...

But actually there is hardware mapper mirroring so all this bits is unnecessary... So probably there is need one another bit, if it 1, then hardware used and first two ignored, else selected as above...

Code: Select all

As far as the mapper/board number goes, can we stop talking about why this format will be no good and actually start brainstorming. We will stick with the mapper numbers or move on to boards instead?

One more thin you should do before brainstorming. Rearraging mappers information... It needs collecting all mappers info such banking registers, irq handlers, mirrorings and memory ranges... Then you will probably find out that many mappers is duplicated or just have smal difference between each others, but many mappers contain two or three emulated hardware board in one... First should be merged, other ones should be splitted. I'm trying to do something, but anyway I've made many dupes for mappers and boards.

[Bregalad]

I don't think more mirroring bits than iNES are needed. There is no problem with mirroring support now. The only mappers that use one-screen mirroring are mapper-controlled, there is NEVER hardwired one-screen mirroring, because the fact that you couldn't switch between both nametable banks for all 4 nametables mapped in PPU would loose all the interest of this mirroring mode.
The H/V mirroring bit is only here for harwired mirroring mappers, and ignored for all other mappers. The four-screen bit is here for several MMC3 games that use 4-screen mirroring. I don't think there is more than a couple of game, tough.

Again, the structure of iNES WON'T BE CHANGED for reasons that already have been mentionned : Noone (from "Joe Gamer" to myself) will care about any new format. Only a update of iNES would be of any pratical use. It is a simple of this.

[kevtris]

I was waiting awhile before I brought this up, but I had an idea for a slightly modified .NES header format myself.

I have gone through over 4000 ROMs, doing various testing on my FPGA NES console now, and I believe I'm in a fairly good position to recommend some changes.

The goals of my changes are thus:

1) retain absolute backwards compatibility (for existing ROMs)

2) this format must be "future proof"

3) the changes I make must be carefully documented and must make sense.

4) The changes must make "sense" from both a hardware and software standpoint.

This thread has kind of made me post my ideas early before I've had a chance to fully revise and pull together exactly what I want to do, but you should be able to get a good idea for what I wanted to do.

So, without further ado...

As mentioned before, the format will be retained as it is. For a recap, here is the existing format:

byte 0-3: 'NES<eof>'
byte 4: # of PRG pages
byte 5: # of CHR pages
byte 6: flags byte 0
byte 7: flags byte 1
byte 8-15: not used

There are two free bits on flags byte 1.

I propose the following:

D3 set and D2 clear of flags byte 1 will indicate that this is an extended header. DiskDude! in the header will have D2 set and D3 clear, so this will prevent interference with those old dirty ROMs. (thanks to Q for this suggestion)

If D3/D2 are not set and clear respectively, this is not a "Version 2" header. This should be a pretty definitive way to prevent confusion. Again, if the emulator does not support the extended header, the game loads like normal and the emulator is none the wiser.

Now that we know this is an extended ROM, the following things were concidered as problem areas for .NES

1) Vs. Unisystem.

The Vs. Unisystem has no less than 11 different PPUs and various protection schemes on some of the Namco/Atari games (such as RBI baseball). Some way has to be found to accomodate this.

2) PRG ROM in excess of 2Mbytes, CHR ROM in excess of 1Mbyte.

This has already occured, and has been causing trouble for some
ROMs. So far, the hack has been to set PRG ROM to 00h to indicate 4Mbytes of ROM (since FFh is 16K short of 4Mbytes), and in the case of
exceeding the 2Mbyte-8K CHR barrier, ROMs have been allocating the CHR in the PRG space, and the emulator has to sort this out. Very messy.

3) "sub mappers".

Some of the allocated mappers are actually multiple mappers with 1
number. The only way to sort this mess is to use CRC checks in the emulator. This of course is messy, since when a new ROM comes along that isn't in the CRC table, things break.

4) mapper numbers.

Face it, we're running out of mapper numbers. 256 was alot at the start (heck, 16 even was alot to begin with!) but we're filling the space up.

5) WRAM.

Not all carts that support WRAM support 8K of it. Some support less, some support more, and some even have EEPROM!

So, after testing nearly every ROM in goodNES, those are the trouble areas I have found.

The proposed solution is thus:

Code: Select all

byte8: 

7        0
---------
SSSS xxxM


S: sub-mapper number 

This specifies the sub-mapper for this ROM. If the ROM has no
sub-mappers, this field shall be 0000b.

M: mapper bits 8.

This specifies 1 more mapper number bit, which extend the total
number of possible mappers to 512.  The other three bits are
earmarked for more mappers- up to 4096 total if needed- but I wish to stress that we should NOT just willy-nilly allocate mapper numbers greater than 0ffh until it is absolutely required.  See below for more on this.  The "x" bits shall thusly be clear at all times.


byte9:

7       0
---------
CCCC PPPP

C: 4 more CHR ROM size bits
P: 4 more PRG ROM size bits

These combine with the existing 8 bits of each to form 12 bits total for the number of PRG and CHR banks... this is enough for 64Mbytes-16K of PRG data and 32Mbytes-8K of CHR data.  

byte10:
7       0
---------
CCCC PPPP

C: quantity of CHR RAM present which is not battery backed
P: quantity of WRAM present which is not battery backed

each nybble:

0 - no RAM of this type is present.
1 - 128 bytes of RAM
2 - 256 bytes of RAM
3 - 512 bytes of RAM
4 - 1K of RAM
5 - 2K of RAM
6 - 4K of RAM
7 - 8K of RAM
8 - 16K of RAM
9 - 32K of RAM
10 - 64K of RAM
11 - 128K of RAM
12 - 256K of RAM
13 - 512K of RAM
14 - 1M of RAM
15 - reserved, do not use

byte11:
7       0
---------
CCCC PPPP

C: amount of battery backed CHR RAM (yes, carts exist with this!!!)
P: amount of battery backed WRAM or EEPROM.

Certain Famicom cartridges use an EEPROM for storing their game data, instead of a static RAM. 

Like above, these values follow the size table listed.


byte12:
7       0
---------
xxxx xxBP

P: this is a PAL ROM.  when set, indicates PAL mode.

B: when set, indicates this ROM works on both PAL and NTSC machines (some of the Codemasters games actually will adjust the game depending on if it detects you running on a PAL or NTSC machine.  It adjusts the timing of the game, and fixes the music).   

Not many games would have this B flag set.  

x: this bit is not used yet.  They shall be maintained clear.

byte13:
7       0
---------
UUUU PPPP

This byte is reserved for the Vs. Unisystem only.  If this is not a Vs. Unisystem ROM, then this byte shall be all 0's.

P: PPU type.  This specifies the type of PPU used for this game.   I have a list of this, but I have not processed it yet.  There's around 11 different PPU's that exist.

U: various Unisystem flag bits.  Again, I am working hard on this stuff so it's not fully complete.

I am working on buying 1 of every Unisystem game so I can study the hardware and come up with a coherent methodology.  If anyone has any Unisystem games that I don't have, I could REALLY use them.

So far, for the Unisystem flags I can come up with, there's a special copy protection chip or chips that exist on games such as RBI Baseball.  I totally RE'd RBI baseball's chip, but I know at least 2 other namco/atari games use similar but different chips... since running those games with my clone of the chip doesn't work.

Well, that in a nutshell is my proposal. It has been extensively thought out, future proofed, and carefully designed to fit in with existing emulators and ROMs. In fact, these additions would probably only affect around 5% of the existing ROMs, but they would banish CRC and mapper hacks forever.

I propose making a "fixup" program for existing ROMs which need it, and CAREFULLY documenting all the updates needed in 1 central location so that the existing mapper issues cannot occur again.

Also on the mappers issue, I have made an absolutely comprehensive mapper guide vs. mapper number which I will be posting to my page at some point. It lists every single implemented mapper number, where it can be found, and what it is composed of. Everything on it has been tested and revised, since it was produced when I made the FPGA NES.

And as such, the extra mapper bits should NOT be used until we run out of existing mapper numbers to prevent confusion and needless trouble.

[kevtris]

Here's my Vs. byte proposal. I still need to figure out what to do about the dual Vs. games such as Baseball... I will probably allocate a bit or two on the flags byte for those... anyways, here is what I have.

Code: Select all

                         Vs. Unisystem Byte Proposal
                         ---------------------------

09/15/06
K.Horton
----


This is my proposal for the Vs. Unisystem byte on my proposed
.NES header expansion.  It takes into account all known aspects
of the Vs. Unisystem.


There are three different methods of making the Vs. Unisystem games
hard to copy for arcade operators.  

1) Different PPU's.   This was done to prevent operators from
   buying 1 conversion kit and then burning 9 copies of the EPROMs
   for other machines.  

2) Different control layouts.  Again, this was done to prevent
   operators from just burning up N copies of the EPROMs for their
   machines.  Not many games use a munged layout... I suspect the
   number is around 5 to 7, with maybe 4 or 5 different pinouts.

3) Custom chips.  Only Namco/Atari appears to have done this.  There
   only seems to be four different games which use one of these things.
   RBI Baseball, TKO Boxing, Super Xevious, and possibly 1 other
   Japan-only game.  This chip is designed of course to prevent an
   operator from burning another set of ROMs for a different game.
 

The proposed byte:

7       0
---------
MMMM PPPP



P: PPU type.

 0 - RP2C03B
 1 - RP2C03G   
 2 - PR2C04-0001 
 3 - RP2C04-0002
 4 - RP2C04-0003
 5 - RP2C04-0004
 6 - RC2C03B
 7 - RC2C03C
 8 - RC2C05-01
 9 - RC2C05-02
10 - RC2C05-03
11 - RC2C05-04
12 - RC2C05-05
13 - not defined
14 - not defined
15 - not defined


I have dumped the palettes from ALL of these PPUs, and have exact bit for
bit copies of them.  The last 5 PPUs (RC2C05) have the standard NES 
palette in them, however they return a specific word in the lower 5 bits
of 2002h, and registers 2000h and 2001h are flipped around.  I'm fairly
certain that these are all the PPU's that exist.  I have a good cross
section of games now.

M: Vs. mode.

 0 - Normal- no special mode(s)
 1 - RBI Baseball
 2 - TKO Boxing
 3 - Super Xevious
 4 - ...

The rest of the mode settings are undefined at this time- I am hard at work
coming up with a complete set of Vs. stuff.  It's just taking awhile since I
have to buy the stuff on ebay when it comes through.  If anyone has any
Vs. stuff they can let me borrow, let me know what you have and we will
go from there.  I have around 15 games so far, and I have most of the
esoteric stuff which used the add-on boards.  I'm in most interested in
Atari/Tengen/Namco games such as TKO boxing.

(thanks Q for the corrections/additions)

[Bregalad]

Wow ! Your new format looks pretty decent.
However, I've some minor problems with it :

- What on earth are those PPU types ? I've never heard of that. Normally, a NES game should work with all PPU revisions, huh ?

- Why CHR-RAM could be battery backed ? I don't see much interest with that. What game could have any use of this ? To save space, a game would write all its save to the CHRRAM ? That make no sense to me, while technically possible.

- Why CHR-RAM/SRAM size could be less than 8kb (exept 0kb) ? I doubt any 128-byte RAM chips are used in any NES game. I think less bits would do for the RAM size.

- More info on what exactly "submappers" would look like would be welcome.

- Why don't put a bit that tells if the cartridge is american on japanese ? Both are NTSC and there isn't much difference in emulation issues, but I guess it would still be cool to have that.

Something like 3 bits :
000 -> Japan, NTSC
001 -> America, NTSC
010 -> Europe, PAL
011 -> (Australia ?), PAL
100 -> Pirate/Homebrew/Unlicenced, NTSC
110 -> Pirate/Homebrew/Unlicenced, PAL
111 -> Pirate/Homebrew/Unlicenced, support both video standards

Emulators could just look the middle bit to know if they must emulate PAL or NTSC, or have a more complex system to emulate all regions... ?

Other than this, it looks great.

EDIT : Totally outtopic, but I'm very curious to know wich games use EEPROM instead of RAM+Battery to hold saves. Wich mapper support that ? Are those licenced games or only pirate games ?

[Marty]

Excellent proposal. If everyone else agrees and once all the details are in place, I'd like to get started right away.

- What on earth are those PPU types ? I've never heard of that. Normally, a NES game should work with all PPU revisions, huh ?

Probably, but there might differences we don't know yet. Besides, knowing the PPU type is a must if we want to display the VS games in correct colors.

- Why CHR-RAM/SRAM size could be less than 8kb (exept 0kb) ? I doubt any 128-byte RAM chips are used in any NES game. I think less bits would do for the RAM size.

Then what about the Bandai mapper 16/157 EEPROMS? Mapper 16 may have a 24C01 (128x8 bits) or a 24C02 (256x8 bits). Mapper 157 (Datach Joint System) may have both a 24C01 and a 24C02 (384x8 bits). MMC6 has 1k RAM only and some unlicensed carts may also have less than 8k.

- More info on what exactly "submappers" would look like would be welcome.

A few examples on how it could be laid out:

mapper 4 sub-mappers: MMC3A, MMC3B, MMC3C, MMC6, MIMIC-1, NAMCOT 108/109/119
mapper 33 sub-mappers: TC0190, TC0350
mapper 69 sub-mappers: FME-7, SUNSOFT5A, SUNSOFT5B