It is currently Tue Dec 12, 2017 2:38 pm

All times are UTC - 7 hours





Post new topic Reply to topic  [ 78 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next
Author Message
PostPosted: Mon Sep 18, 2006 10:11 pm 
Offline
User avatar

Joined: Sat Oct 29, 2005 2:09 am
Posts: 502
Location: Indianapolis
I completed my initial "official" spec for the new addition. This week I will start in on a few of the basic submapper things.

Code:
                     NES 2.0 File Addition Specification
                     -----------------------------------


Version 1.00
09/18/06
Written by K.Horton
Thanks to Quietust for ideas and proofing and help
----


This is the tentative addition to the standard .NES file format that
most emulators use.  This addition is designed to disambiguate certain
ROMs that currently can only be discerned via a CRC-32 or similar hash
check.  Naturally, this causes problems for new ROMs that are not in the
database, but need special handling.

There are four goals for this specification.

1) Retain 100% backwards compatibility with existing emulators/ROMs/etc.
   (*this includes "dirty ROMs" with crap such as "diskdude!!!" in the header
   and other atrocities*)

2) The format must be "future proof".

3) The changes made must be VERY CAREFULLY documented and make sense.

4) Said changes must make sense from both a hardware and software
   standpoint.

* * *

The standard iNES specification is presented below:

 0-3: string    "NES"<EOF>
   4: byte      Number of 16K byte program ROM pages
   5: byte      Number of 8K character ROM pages (0 indicates CHR RAM)
   6: bitfield  flags byte 0
   7: bitfield  flags byte 1
8-15: byte      These bytes are not used, and should be 00h.

Flags byte 0:

7       0
---------
NNNN FTBM

N: Lower 4 bits of the mapper number
F: Four screen mode. 0 = no, 1 = yes. (When set, the M bit has no effect)
T: Trainer.  0 = no trainer present, 1 = 512 byte trainer at 7000-71FFh
B: SRAM at 6000-7FFFh battery backed.  0= no, 1 = yes
M: Mirroring.  0 = horizontal, 1 = vertical.

Flags byte 1:

7       0
---------
NNNN xxPV

N: Upper 4 bits of the mapper number
P: Playchoice 10.  When set, this is a PC-10 game
V: Vs. Unisystem.  When set, this is a Vs. game
x: these bits are not used.

* * *

For NES 2.0, none of the above will be changing, except for the two unused
bits on flags byte 1.

To indicate that this is a NES 2.0 file, bit 2 will be clear, and bit 3 will
be set.   i.e.

7       0
---------
NNNN 10PV

Same as above, but the 1 and 0 pattern will denote an NES 2.0 file.  This
neatly gets around the "diskdude!!!" problem, because those headers do not
have the correct bits set on this byte, and will thus be treated as a
regular iNES file.

That is the "how" of determining whether we are dealing with a valid NES 2.0
file.  Now that that is done, the desired aspects of such a thing need to
be considered.  I have tested over 4000 ROMs and have dumped at least a
thousand more, and reverse engineered probably 60-70 different mappers.
This has given me a front row seat into the shortcomings of the original,
and a good idea of where additional information is needed.

* * *

The new things we need to know are thusly:

1) Vs. Unisystem

The Vs. Unisystem is one of the two Nintendo arcade machine series produced
which use "mostly NES/famicom" hardware.  These games will run fine on
emulators if a little extra things are stored in the header.

Nintendo wanted to make life difficult for arcade operators when it came to
copy protection.  Three main schemes were devised. (See the "Vs. system byte"
description below for a detailed analysis)

2) PRG ROM in excess of 2Mbytes, CHR ROM in excess of 1Mbyte

This has already occured, and has been causing trouble for some ROMs. So
far, the hack has been to set PRG ROM to 00h to indicate 4Mbytes of ROM
(since FFh is 16K short of 4Mbytes), and in the case of exceeding the
2Mbyte-8K CHR barrier, ROMs have been allocating the CHR in the PRG space,
and the emulator has to sort this out. Very messy.

3) "Sub-mappers"

Some of the allocated mappers are actually multiple mappers with 1 number.
Examples include mapper 34 (Nina/BNROM), mapper 83 (two styles of CHR ROM
banking), mapper 1 (various ROM/RAM hacks), mapper 16 (EEPROM/WRAM/light pen/
etc).

Currently, the only fix for this is to CRC the games and then hack the mapper
if the CRC or other hash matches.  This of course fails if the game is not
in the database.

4) Mapper numbers

Face it, we're running out of mapper numbers.  16 seemed like it would be
enough, but they were quickly exhausted.  Then 256 mappers seemed like a
vast space to work on.  But now, we are getting near the end of the line,
and running out of mapper numbers.  I have personally assigned at least 50
or 60 of them, which is almost 1/4th of the total mapper space.

5) WRAM

Not all carts that support WRAM support 8K of it. Some support less,
some support more, and some even have EEPROM!   Heck, some carts even
battery backed the stupid CHR RAM.  This last one was a very recent
find and goes to show that a workable extention needs to reasonably
cover all possible bases.

* * *

The proposed solution:

Byte 8:

7       0
---------
SSSS xxxM

S: Sub-mapper number.

This specifies the submapper for this ROM.  If no submapper mode is
needed, set this nybble to 0000b.

M: Mapper number extension.

CAUTION: DO NOT USE THIS BIT YET.  There are still some existing
numbers left in the current iNES mapper space. (Around 30-40 or so by
my last count).  The other three bits marked with "x" are also slated
to be used if more than 512 mappers are required.  This would allow
support for 4096 mappers all together.  This should hold us until the
next iceage.  I repeat: do not designate mappers above 256 yet.

--

Byte 9:

7       0
---------
CCCC PPPP

C: 4 more CHR ROM size bits
P: 4 more PRG ROM size bits

These combine with the existing 8 bits of each to form 12 bits total
for the number of PRG and CHR banks... this is enough for 64Mbytes-16K
of PRG data and 32Mbytes-8K of CHR data.

--

For the following two bytes, this table defines the size of the RAM
segments:

Each Nybble:
------------
0 - no RAM of this type is present.
1 - 128 bytes of RAM
2 - 256 bytes of RAM
3 - 512 bytes of RAM
4 - 1K of RAM
5 - 2K of RAM
6 - 4K of RAM
7 - 8K of RAM
8 - 16K of RAM
9 - 32K of RAM
10 - 64K of RAM
11 - 128K of RAM
12 - 256K of RAM
13 - 512K of RAM
14 - 1M of RAM
15 - reserved, do not use


Byte 10:
7       0
---------
pppp PPPP

p: Quantity of PRG RAM which is battery backed (or serial EEPROM, see below)
P: Quantity of PRG RAM which is NOT battery backed

--

Byte 11:
7       0
---------
cccc CCCC

c: Quantity of CHR RAM which is battery backed (yes it exists! see below)
C: Quantity of CHR RAM which is NOT battery backed

--

A note about serial EEPROMs and battery backed CHR RAM...
Some mapper 16 (Bandai) games use serial EEPROMs to store the game data,
rather than a battery backed SRAM.  These can be as small as 128 bytes or
as large as 512 bytes.  They tended to use 24C01 (128 bytes) 24C02 (256
bytes) or another semicustom chip I cannot find a workalike for.  The
interface for the 24Cxx parts is I^2E which is a Philips specification.
The workalike chip is very similar, but the address and data are clocked
in backwards from the I^2C parts.

As for battery backed CHR RAM, the Racermate cartridge has 64K of CHR RAM
total:  32K is battery backed, and 32K of it is not.  They store all the
stats and such in it.  Why you would do such a thing, I do not have a
clue... but they did!  I traced out the circuit myself and couldn't
believe it.

--

Byte 12:
7       0
---------
xxxx xxBP

P: This is a PAL ROM.  When set, indicates PAL mode.
B: When set, indicates this ROM works on both PAL and NTSC machines.
Some of the Codemasters games actually will adjust the game depending
on if it detects you running on a PAL or NTSC machine - it adjusts the
timing of the game, and fixes the music.

Not many games would have this B flag set.

x: These bits are not used yet.  They shall be maintained clear.


byte 13:
7       0
---------
MMMM PPPP

This byte is reserved for the Vs. Unisystem only.  If this is not a Vs.
Unisystem ROM, then this byte shall be all 0's.

P: PPU.  There are 13 Vs. PPUs that can be found on the games:

 0 - RP2C03B     (bog standard RGB palette)
 1 - RP2C03G     (similar pallete to above, might have 1 changed colour)
 2 - PR2C04-0001 (scrambled palette + new colours)
 3 - RP2C04-0002 (same as above, different scrambling, diff new colours)
 4 - RP2C04-0003 (similar to above)
 5 - RP2C04-0004 (similar to above)
 6 - RC2C03B     (bog standard palette, seems identical to RP2C03B)
 7 - RC2C03C     (similar to above, but with 1 changed colour or so)
 8 - RC2C05-01   (all five of these have the normal palette...
 9 - RC2C05-02   (...but with different bits returned on 2002)
10 - RC2C05-03
11 - RC2C05-04
12 - RC2C05-05
13 - not defined (do not use these)
14 - not defined
15 - not defined


I have dumped the palettes from ALL of these PPUs, and have exact bit for
bit copies of them.  The last 5 PPUs (RC2C05) have the standard NES
palette in them, however they return a specific word in the lower 5 bits
of 2002h, and registers 2000h and 2001h are flipped around.  I'm fairly
certain that these are all the PPU's that exist.  I have a good cross
section of games now.


M: Vs. mode:

 0 - Normal- no special mode(s)
 1 - RBI Baseball
 2 - TKO Boxing
 3 - Super Xevious
 4 - ...


This section is a tad bare right now... I'm still trying to figure out
exactly how to flesh this out.  This should be a good start, however.

If anyone is interested in the things nintendo did to make your life
difficult as an arcade operator, here it is:

 a) Different PPUs.  There are 13 different PPU chips made that you
    can find on Vs. arcade boards.

 b) Different controller pinouts.  Some games came with new control panels
    you had to install with the game.  This was pretty basic stuff and just
    remapped a few of the buttons.

 c) Atari/Namco/Tengen came up with at least three different protection chips
    which map in the 5000-5FFFh area that the game checks. If the chip
    does not return the correct data, the game hangs or fails to start.

_________________
/* this is a comment */


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 1:23 am 
Offline

Joined: Thu Sep 15, 2005 9:23 am
Posts: 1194
Location: Behind you with a knife!
What about games that use $6000-7FFF but are not battery backed?

_________________
http://www.jamesturner.de/


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 1:51 am 
Offline
User avatar

Joined: Sat Oct 29, 2005 2:09 am
Posts: 502
Location: Indianapolis
WedNESday wrote:
What about games that use $6000-7FFF but are not battery backed?


That's what the "Battery" bit is for in the header. You clear said bit. This exists in the traditional iNES header.

_________________
/* this is a comment */


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 10:08 am 
Offline

Joined: Wed Mar 09, 2005 9:08 am
Posts: 348
I really the "RAM size" bits are a bad idea. To know how to handle the RAM, you need to know the board's wiring anyway, except for the standard case of 8kB RAM at $6000. So the RAM size should be fully determined by the sub-mapper.

For instance, how would you handle contradictions between the sub-mapper and the RAM size bits? Having the same information in two different places in the file sounds like a bad idea to me. Which information source is the more trustworthy of the two?...


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 10:19 am 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10164
Location: Rio de Janeiro - Brazil
Bananmos wrote:
Which information source is the more trustworthy of the two?...

The one that was defined for the 2.0 version. The other just remains there for compatibility reasons. If old information contradicts new information, go with the newer one.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 10:45 am 
Offline
User avatar

Joined: Sun Sep 19, 2004 10:59 pm
Posts: 1393
tokumaru wrote:
Bananmos wrote:
Which information source is the more trustworthy of the two?...

The one that was defined for the 2.0 version. The other just remains there for compatibility reasons. If old information contradicts new information, go with the newer one.


Actually, both of those fields are new.

One argument I have in favor of indicating both is that it allows reading the RAM sizes without having to look up the mapper+submapper - if you want to, say, filter your ROMs by the amount of save RAM they use, it would be rather annoying to have to look up all of the mappers and submappers every single time (not to mention requiring the program itself to be updated every time a new mapper/submapper is allocated).

Besides, there could potentially be places where separate submappers for different RAM sizes would not be optimal - the mapper logic might not differ substantially, or there might not be room for enough submappers to distinguish every variant.

You see it as redundant and a potential for contradictions. I see it as an additional useful way of presenting information and the ability to override values (in this case, I would say that the RAM counts override the submapper, since the submapper can distinguish far more than RAM sizes).

_________________
Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 10:48 am 
Offline

Joined: Wed Mar 22, 2006 8:00 am
Posts: 354
If nothing else, please provide a version number somewhere in the spec. That is the one thing Marat failed to include that led to all the problems we have now. An emu MUST know which revision it is looking at. What happens if some special-case cartridge comes up where the info in this revision isn't sufficient? Extendibility requires a method to know whether or not the extension should be acknowledged by the emulator. Yes, the two bits in the eighth byte are good enough (in at least most cases, which is about all you can expect anymore) to distinguish this revision from the previous one, but how do we distinguish this revision from the next one (assuming there is)? I did not see anything in the spec that would allow for this possibility.

If INES originally had a version number, we wouldn't have to search the header for DiskDuede or anything before acknowledging the eighth byte - if it was version 0, the second flags byte would have been disregarded, while version 1 would have allowed the upper four bits to be used, and version 2 the lower 2 bits (VS and PC10). An emu who only knew version 0 would give a warning ("Image may require a feature I don't know about. Proceed anyawy?") if it saw a version 1 or 2 image. Problem solved. But alas, those days cannot be relived, but at least we can try to avoid a similar problem in the future (assuming this standard goes anywhere).

One more thing. If a game has battery-backed RAM and non-battery-backed RAM, how does an emulator know which segment should be loaded first (e.g. is it bank 0 or bank 1 that gets saved)?


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 11:10 am 
Offline
User avatar

Joined: Fri Nov 12, 2004 2:49 pm
Posts: 7314
Location: Chexbres, VD, Switzerland
Banamnos looks like thinking the exact opposite as I do.
Anyway, most people seems to have adopted the submapper idea, so I don't have to do anything agains it, but I think submappers are only needed on a few rare mappers. They aren't needed on standard cards, because the SRAM size bits tells us everything.

A few improvements need to be done, because I think dvdmth is right. Also I'm sad that my region idea wasn't taken in account.

_________________
Life is complex: it has both real and imaginary components.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 11:28 am 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10164
Location: Rio de Janeiro - Brazil
Quietust wrote:
Actually, both of those fields are new.

Oh yeah! My bad! Sorry! =)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 12:00 pm 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
Imagine a file format consisting of the signature, a version number, then data that's version-specific. At some point, a new version is created that completely changes the format of the data after the version. For all intents and purposes, it's a new file format. Since iNES only has 16 bytes in the header, that's about the only approach you can take with a version number.

If iNES were a block-oriented format, you could just add your new information as a new block type that would be ignored by older programs. Presumably it would only add to the information about the file. With the 16 byte header of iNES, the only way to add new information without removing the old is to define more of the unused bytes.

In other words, I'm not imagining scenarios where a version number would help.

A related idea that I can imagine uses for are some mapper+submapper combinations making use of otherwise-unused bytes. This also relates to the 4-bit fields that specify various memory sizes. I think their value might be understood by rewinding a bit in the extension of iNES. Think of taking the current iNES and adding a new sub-mapper field. You'd find that many variants were simply specifying different memory sizes, and probably end up with lots of similar-but-subtly-different code in various mappers that decodes these fields.

Rather than do this, you have dedicated memory size fields to go along with the sub-mapper bits. This makes the meaning of each more consistent, leaving the sub-mapper bits for unique features of the main mapper number. The issue of memory size being insufficient because it could be wired different ways would be handled with a sub-mapper bit, since this is truely unique information.

The issue of potential inconsistent information is an important one to consider. In this case, the mapper itself defines what additional fields are even relevant. There's no way around this, as even the most basic options like "WRAM is battery-backed" and "H/V mirroring" don't apply to some boards. So the memory sizes are meaningful only in the context of a particular mapper. I'd think that one important guideline for consistency would be to avoid using the sub-mapper field if one of the memory size fields were sufficient.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 4:41 pm 
Offline
User avatar

Joined: Sat Oct 29, 2005 2:09 am
Posts: 502
Location: Indianapolis
blargg wrote:
Imagine a file format consisting of the signature, a version number, then data that's version-specific. At some point, a new version is created that completely changes the format of the data after the version. For all intents and purposes, it's a new file format. Since iNES only has 16 bytes in the header, that's about the only approach you can take with a version number.

<snip>

A related idea that I can imagine uses for are some mapper+submapper combinations making use of otherwise-unused bytes. This also relates to the 4-bit fields that specify various memory sizes. I think their value might be understood by rewinding a bit in the extension of iNES. Think of taking the current iNES and adding a new sub-mapper field. You'd find that many variants were simply specifying different memory sizes, and probably end up with lots of similar-but-subtly-different code in various mappers that decodes these fields.

Rather than do this, you have dedicated memory size fields to go along with the sub-mapper bits. This makes the meaning of each more consistent, leaving the sub-mapper bits for unique features of the main mapper number. The issue of memory size being insufficient because it could be wired different ways would be handled with a sub-mapper bit, since this is truely unique information.

The issue of potential inconsistent information is an important one to consider. In this case, the mapper itself defines what additional fields are even relevant. There's no way around this, as even the most basic options like "WRAM is battery-backed" and "H/V mirroring" don't apply to some boards. So the memory sizes are meaningful only in the context of a particular mapper. I'd think that one important guideline for consistency would be to avoid using the sub-mapper field if one of the memory size fields were sufficient.


Yeah, I should've stressed that before about the submappers... they are pretty much a "last resort". MMC5 is a good example of where the WRAM sizes are very useful- there are at least 4 kinds of MMC5 WRAM setups, while the MMC5 part itself does not change. Here are a few I know of:

0K (no RAM at all)
8K (battery backed, or not battery backed)
16K (8K battery backed, 8K not battery backed)
32K (battery backed or not)

This is the ideal problem the WRAM size fields were designed to solve.

As for the sub-mapper info, as Blargg explains, it's for hardware hookup differences that you may not be able to suss out with the information given in the header, such as the afore-mentioned mapper 16 and mapper 83 stuff.

_________________
/* this is a comment */


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 5:27 pm 
Online

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19335
Location: NE Indiana, USA (NTSC)
blargg wrote:
If iNES were a block-oriented format, you could just add your new information as a new block type that would be ignored by older programs. Presumably it would only add to the information about the file. With the 16 byte header of iNES, the only way to add new information without removing the old is to define more of the unused bytes.

Or to define a footer that always occurs 16 + nPRGBanks*16384 + nCHRBanks*8192 bytes into the file. Make this footer block-oriented, and you have my iNIF suggestion.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 19, 2006 8:18 pm 
Offline
User avatar

Joined: Sun Sep 19, 2004 10:59 pm
Posts: 1393
I've thrown together a Windows app to edit .NES headers with 'NES 2.0' support:
http://www.qmtpro.com/~nes/nes2edit.exe

_________________
Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another.


Last edited by Quietust on Fri Jan 23, 2009 7:41 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 20, 2006 12:00 am 
Offline
User avatar

Joined: Sat Oct 29, 2005 2:09 am
Posts: 502
Location: Indianapolis
Quietust wrote:
I've thrown together a Windows app to edit .NES headers with 'NES 2.0' support:
http://www.qmtpro.com/~nes/nes2edit.exe


And with this, I have started implementing it in the FPGA NES. I will have some more substantial submapper setups soon. I have "large ROM" support so far added, and I ripped out and re-did my loading routines to follow the spec. Once finished, I will have a GoodNES style tool that will fix all the broken ROM headers for all current ROMs, and full documentation on the submappers and such. Stay tuned.

_________________
/* this is a comment */


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 20, 2006 8:23 am 
Offline
User avatar

Joined: Fri Nov 12, 2004 2:49 pm
Posts: 7314
Location: Chexbres, VD, Switzerland
Oh, I really thank you for listening to other people ideas....

_________________
Life is complex: it has both real and imaginary components.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 78 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group