It is currently Tue Dec 18, 2018 6:27 am

All times are UTC - 7 hours





Post new topic Reply to topic  [ 10 posts ] 
Author Message
PostPosted: Wed Nov 21, 2018 1:11 pm 
Offline

Joined: Mon Nov 11, 2013 2:55 pm
Posts: 47
Location: Minsk, Belarus
I had proposed to add Dendy support to the NES 2.0 format in 2016, here. Unfortunately, there was no active discussion. But now, in 2018, two important things happened. NSFe format now has Dendy support (in the regn chunk), and there is an another effort to add Dendy support to the NES 2.0 format, here. I have already participated in the latter topic, but now I go further and propose to extend all the NES-related formats in the same way. It is still possible to do, because all the formats are still have a very similar byte for this purpose. That's why I have created a new topic. I'm sorry that it is literally the third topic where I had written about the subject, but it is the first (and the last) dedicated one.

Overview of current situation.

iNES

original specification states:
Code:
Byte     Contents
---------------------------------------------------------------------------
9        bit 0     1 for PAL cartridges, otherwise assume NTSC.
         bit 1-7   Reserved, must be zeroes!
---------------------------------------------------------------------------


UNIF

original specification states:
Code:
ID: [TVCI]
Length: BYTE
Revision: 6
Description:  Television Standards Compatability Information set to:
0- Originally NTSC cartridge
1- Originally PAL cartridge
2- Does not matter
NOTE: ALL North American carts that are dumps of the North American
Version are NTSC. All licensed famicom games are NTSC.
It is important to mention that unlike iNES, it treats this byte not as a bitfield, but as an enum. First two values effectively are the same as the bit 0 from the byte 9 of the iNES header, but the third value effectively means that "both systems are supported".

NES 2.0

The byte was moved and extended a bit, original specification states:
Code:
Byte 12:
7       0
---------
xxxx xxBP

P: This is a PAL ROM.  when set, indicates PAL mode.
B: When set, indicates this ROM works on both PAL and NTSC machines
(some of the Codemasters games actually will adjust the game depending
on if it detects you running on a PAL or NTSC machine.  It adjusts the
timing of the game, and fixes the music).   

Not many games would have this B flag set. 

x: These bits are not used yet.  They shall be maintained clear.
So, it is a mixture of what we have seen in the iNES and UNIF. This byte is a bitfield, and the spec clearly states that bit 0 set means "indicates PAL mode" (like in iNES), and bit 1 set means "it supports both NTSC and PAL". Unfortunately, description of these bits is ambiguous (it is not described how to treat the new bit 1 in combination with the old bit 0). That's why it is treated in different ways by different emulators. FCEUX: 00 - NTSC, 01 - PAL, 10 - NTSC, 11 - NTSC. Mesen: 00 - NTSC, 01 - PAL, 10 - NTSC, 11 - PAL. Nintendulator: 00 - NTSC, 01 - PAL, 10 - leave as is, 11 - leave as is. IMHO, Mesen has the most logical behavior, just because it does not contradict to the clear statement about meaning of the bit 0 which was borrowed from the iNES.

NSF

The header has this byte:
Code:
offset  # of bytes   Function
----------------------------
$07A    1   BYTE    PAL/NTSC bits
                bit 0: if clear, this is an NTSC tune
                bit 0: if set, this is a PAL tune
                bit 1: if set, this is a dual PAL/NTSC tune
                bits 2-7: not used. they *must* be 0
Similar to NES 2.0. Ambiguous.

NSFe

The mandatory INFO chunk has this byte:
Code:
offset  # of bytes   Function
----------------------------
$0006   1   BYTE    PAL/NTSC bits
                bit 0: if clear, this is an NTSC tune
                bit 0: if set, this is a PAL tune
                bit 1: if set, this is a dual PAL/NTSC tune
                bits 2-7: not used. they *must* be 0
Similar to NES 2.0. Ambiguous.

But it has a very flexible and unambiguous extension, the regn chunk:
Code:
offset  # of bytes   Function
----------------------------
$0000   1   BYTE    Bitfield of supported regions.
                bit 0: NTSC
                bit 1: PAL
                bit 2: Dendy
                bit 3-7: Reserved (always clear)
$0001   1   BYTE    Specifies preferred region if multiple regions are supported. (Optional)
                0 - NTSC
                1 - PAL
                2 - Dendy

This chunk adds support for the Dendy region, and the ability to specify which region is preferred (for players that can support multiple regions). This chunk should appear after the INFO chunk, and overrides its region data (byte 6).
So, it overrides byte 6 if is present. It is aware of existence of Dendy systems. Also it specifies both list of supported systems and a preferred system, so an emulator have enough information about how this ROM can be executed properly.

Time to action

It is time to remove the ambiguity and to add feature parity to all the NES formats. All the above formats can be extended the same way, with preserving compatibility where it wasn't ambiguous before:
Code:
Byte 9 in iNES, or Byte 12 in NES 2.0, or TVCI byte in UNIF, or byte $7A in NSF:
7       0
---------
xdpn xxBP

The byte consists of two nibbles. The higher nibble describes which systems are supported, the lower nibble describes which system is preferred.

If higher nibble is 0, use legacy interpretation:
0000 0000 - NTSC only
0000 0001 - PAL only
0000 0010 - Supports NTSC and PAL, but NTSC is preferred
0000 0011 - Supports NTSC and PAL, but PAL is preferred

Otherwise, higher nibble describes supported systems:
d - Dendy
p - PAL
n - NTSC

Lower nibble (bits BP) describes preferred system:
00 - NTSC (a legacy emulator will use NTSC)
01 - PAL (a legacy emulator will use PAL)
10 - Dendy with fallback to NTSC (a legacy emulator will use NTSC)
11 - Dendy with fallback to PAL (a legacy emulator will use PAL)

If the lower nibble chooses the system which is not marked as a supported system in the higher nibble, fallback to the legacy behavior. Bits 2,3 and 7 should be ignored by emulators, and should be 0 in ROMs.
It describes both how to treat original format with two bits unambiguously, and how to treat extended format. It will be possible to use exactly the same code for extracting information from this byte for all the NES formats.

There is a list of supported systems by a ROM, and a preferred system selected in the ROM. A user of an emulator can choose a preferred system in the settings of the emulator: Auto, NTSC, PAL, Dendy, Force NTSC, Force PAL, Force Dendy. If Auto is specified, emulator will always use the preferred system specified in the ROM itself. If the user chooses NTSC/PAL/Dendy as preferred, the emulator checks if the preferred by user system is supported by the ROM, and if it is, it should use the preferred by user system. Otherwise, it should use preferred by ROM system. And if the user chooses to force NTSC/PAL/Dendy, the emulator just ignores the byte in the header and always uses the forced system.


Last edited by VEG on Wed Nov 21, 2018 3:48 pm, edited 2 times in total.

Top
 Profile  
 
PostPosted: Wed Nov 21, 2018 3:32 pm 
Offline

Joined: Mon Nov 11, 2013 2:55 pm
Posts: 47
Location: Minsk, Belarus
There is another proposal from the NewRisingSun:
Code:
Byte 12: TV System
       7654 3210
       ---------
       .... ..TT
              ++- Frame timing
                  0: RP2C02 (NTSC)
                  1: RP2C07 (PAL licensed)
                  2: RP2C02 and RP2C07 (game self-adjusting or doesn't matter)
                  3: UMC 6527P (PAL with NTSC-like timing, also known as "Dendy" or "Micro Genius" mode)
But it has important disadvantages (NewRisingSun, sorry for criticism, I'm trying to be objective):
- It effectively changes the bitfield from the NES 2.0 to an enum. It seems like an extension not to NES 2.0, but to UNIF. If an emulator follows the original NES 2.0 spec where bit 0 is clearly stated as "this is a PAL ROM", it will always run Dendy ROMs in PAL mode, what is not good, because Dendy was created to be compatible with NTSC games, so it is closer to NTSC.
- It has no feature parity with NSFe format, where it is already possible to specify list of supported and a preferred system.
- It has a strange asymmetry: it is possible to specify that NTSC and PAL are supported, but it is still not possible to specify that the ROM supports NTSC and Dendy, for example. Only one combination is avalable.
- It sill has no proper disambiguation about "what to do when more than one system type is supported" even though it allows to specify that a ROM supports both NTSC and PAL.
This proposal was already adopted by the Mesen lately. But the proposal didn't have enough time to be adopted by ROM makers. It wasn't published anywhere except of that topic, and most people just don't know about it. I believe that it is OK if Mesen changes how this byte is treated one more time. After all, its original treatment of the byte was similar to the unambiguous treatment which I propose for legacy two bits.

To keep everything related about the matter, I'd like to provide NewRisingSun's objections to my proposal with some comments.
NewRisingSun wrote:
I oppose, because of one criterion: the header should be unambiguous; every game should have one header with values that are correct for that game, with any other being incorrect. That allows "good" ROMs to have one unambiguous CRC32 including the header. All the current and proposed fields meet that criterion, including the input device field as previously described.

Existence NES 2.0 itself breaks this argument. For example, a ROM of the Duck Hunt can have 3 different headers variations:
- iNES header
- NES 2.0 header
- NES 2.0 header with Zapper controller set
- etc.
All of them are correct. Just different extensions are used. New extensions can't make proper old files invalid.

NewRisingSun wrote:
Denoting what is preferred on the other hand is inherently subjective and therefore ambiguous. You could have three times the same ROM image with three different headers because somebody prefers NTSC/PAL/Dendy, and have all three being correct according to the definition.

It is not about preference of a user. It is the most suitable system for the ROM, even when 2 or 3 systems are supported. There is always the most suitable system, just because it is not possible to make a game which behaves exactly the same on all possible systems. There is always a slight difference, even when the game tries to adjust itself to different systems. So, one system is the perfect match, but others are also acceptable.
For example, Unchained Nostalgia. In case of PAL you will have a slightly wrong sound on noise channel (because it is not possible to make it absolutely the same on three systems), in case of NTSC you may notice artifacts of speed adoption (it skips every 6th frame on NTSC systems to have the same speed as on Dendy). So, it is perfect only on Dendy, but it also works nicely on PAL and NTSC systems. That's why Dendy is preferred, but NTSC and PAL also are allowed. This bitfield is not for storing some settings by a user. It is intended to describe what is supported by the ROM, and what is the best option. An emulator can take advantage of this information.


Top
 Profile  
 
PostPosted: Tue Dec 04, 2018 9:30 am 
Offline

Joined: Tue Nov 23, 2004 9:35 pm
Posts: 713
NewRisingSun wrote:
I oppose, because of one criterion: the header should be unambiguous; every game should have one header with values that are correct for that game, with any other being incorrect. That allows "good" ROMs to have one unambiguous CRC32 including the header. All the current and proposed fields meet that criterion, including the input device field as previously described.

VEG wrote:
Existence NES 2.0 itself breaks this argument. For example, a ROM of the Duck Hunt can have 3 different headers variations:
- iNES header
- NES 2.0 header
- NES 2.0 header with Zapper controller set
- etc.
All of them are correct. Just different extensions are used. New extensions can't make proper old files invalid.


No, only the third is correct for NES2.0. The regular iNES header (first example) will fail a CRC check that expects the NES2.0 header. The second example will fail to pass the CRC check because it does not have the Zapper controller bit set, which is the canonical header. To play Duck Hunt, you must have a Zapper connected to $4017 D3 & D4. Whatever else you may have plugged into the system is not critical.

_________________
Nerdly Pleasures - My Vintage Video Game & Computing Blog


Top
 Profile  
 
PostPosted: Thu Dec 06, 2018 11:49 am 
Offline

Joined: Mon Nov 11, 2013 2:55 pm
Posts: 47
Location: Minsk, Belarus
Great Hierophant wrote:
The regular iNES header (first example) will fail a CRC check that expects the NES2.0 header.
iNES header is still a valid one. You can't force everybody to use NES 2.0. A general set of ROMs consists of iNES files for 99%, and probably a few of NES 2.0 if any. It is stupid to expect that a ROM is NES 2.0 only because it is barely used in reality and it will never be as popular as iNES.
Great Hierophant wrote:
The second example will fail to pass the CRC check because it does not have the Zapper controller bit set, which is the canonical header.
The "Input/expansion port device" byte is still not a part of the NES 2.0 spec, so even if the Duck Hunt NES 2.0 file exists, it doesn't have this byte set. And it is a perfectly valid NES 2.0 file. And even after the extension which defines the "Input/expansion port device" byte is added, the old NES 2.0 Duck Hunt ROM will be a valid file. Just because the "Input/expansion port device" byte is an extension, it shouldn't break old files or make them invalid.

As for me, I would prefer iNES header for all Nintendo licensed games, just because NES 2.0 is ugly and overcomplicated without any good reason for it. UNIF and NES 2.0 are just for very special cases, they are useful for less than 1% of times. The proposed "Input/expansion port device" byte is useful, but it could be added into iNES also.


Top
 Profile  
 
PostPosted: Thu Dec 06, 2018 12:58 pm 
Offline

Joined: Sun Apr 13, 2008 11:12 am
Posts: 7842
Location: Seattle
VEG wrote:
iNES header is still a valid one.
This has never been about what is a valid header.

A mapper 66 file holding a mapper 0 game is a valid header.

This is about canonical headers. Given a specific PRG and CHR, what is the single correct 16-byte header that it should have.

Quote:
The proposed "Input/expansion port device" byte is useful, but it could be added into iNES also.
Adding it to NES2.0 is adding it to iNES.

iNES has a problem with dirty headers; that's the entire reason NES2.0 exists. You can't stuff more data into the iNES1 header as a result.


Top
 Profile  
 
PostPosted: Thu Dec 06, 2018 1:32 pm 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 709
Man, I am really looking forward to the day when Sour returns from his vacation.


Top
 Profile  
 
PostPosted: Fri Dec 07, 2018 1:42 am 
Offline

Joined: Mon Nov 11, 2013 2:55 pm
Posts: 47
Location: Minsk, Belarus
lidnariq wrote:
Adding it to NES2.0 is adding it to iNES.
These formats are different. For example, TV System byte is byte 9 for iNES and byte 12 for NES 2.0.
lidnariq wrote:
This is about canonical headers. Given a specific PRG and CHR, what is the single correct 16-byte header that it should have.
I believe that the iNES version of the header should be canonical for Nintendo licensed games. Even Nintendo itself adopted this format for NES Classic.


Last edited by VEG on Fri Dec 07, 2018 2:07 am, edited 1 time in total.

Top
 Profile  
 
PostPosted: Fri Dec 07, 2018 1:50 am 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7026
Location: Canada
VEG wrote:
lidnariq wrote:
Adding it to NES2.0 is adding it to iNES.
These formats are different. For example, TV System byte is byte 9 for iNES and byte 12 for NES 2.0.

In practice, the TV System byte is nonexistent for iNES 1. Rippers don't use it. Emulators don't use it. It was a late extension that never took hold.


Top
 Profile  
 
PostPosted: Fri Dec 07, 2018 1:59 am 
Offline

Joined: Mon Nov 11, 2013 2:55 pm
Posts: 47
Location: Minsk, Belarus
rainwarrior wrote:
In practice, the TV System byte is nonexistent for iNES 1. Rippers don't use it. Emulators don't use it. It was a late extension that never took hold.
This byte is a part of the iNES spec for ages (since 2005). Mesen understands this byte, I guess that iNES also understands it, so emulators use it. I haven't checked other emulators, but I don't see any reason to not support it. It is a nice addition to the canonical header from the original author of the format.


Top
 Profile  
 
PostPosted: Fri Dec 07, 2018 2:39 am 
Offline

Joined: Sun Apr 13, 2008 11:12 am
Posts: 7842
Location: Seattle
We can't extend iNES1 because of dirty headers: there is no way to know whether the contents of bytes 7-15 are meaningful or garbage.
In order to address dirty headers, there to be some specific indicator that the rest of the contents are valid.
This specific indicator that the rest of the contents are valid is NES2.0


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group