TV System byte extension in iNES, NES 2.0, UNIF, NSF, NSFe

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

VEG
Posts: 50
Joined: Mon Nov 11, 2013 2:55 pm
Location: Minsk, Belarus

TV System byte extension in iNES, NES 2.0, UNIF, NSF, NSFe

Post by VEG » Wed Nov 21, 2018 1:11 pm

I had proposed to add Dendy support to the NES 2.0 format in 2016, here. Unfortunately, there was no active discussion. But now, in 2018, two important things happened. NSFe format now has Dendy support (in the regn chunk), and there is an another effort to add Dendy support to the NES 2.0 format, here. I have already participated in the latter topic, but now I go further and propose to extend all the NES-related formats in the same way. It is still possible to do, because all the formats are still have a very similar byte for this purpose. That's why I have created a new topic. I'm sorry that it is literally the third topic where I had written about the subject, but it is the first (and the last) dedicated one.

Overview of current situation.

iNES

original specification states:

Code: Select all

Byte     Contents
---------------------------------------------------------------------------
9        bit 0     1 for PAL cartridges, otherwise assume NTSC.
         bit 1-7   Reserved, must be zeroes!
---------------------------------------------------------------------------
UNIF

original specification states:

Code: Select all

ID: [TVCI]
Length: BYTE
Revision: 6
Description:  Television Standards Compatability Information set to: 
0- Originally NTSC cartridge
1- Originally PAL cartridge
2- Does not matter
NOTE: ALL North American carts that are dumps of the North American
Version are NTSC. All licensed famicom games are NTSC.
It is important to mention that unlike iNES, it treats this byte not as a bitfield, but as an enum. First two values effectively are the same as the bit 0 from the byte 9 of the iNES header, but the third value effectively means that "both systems are supported".

NES 2.0

The byte was moved and extended a bit, original specification states:

Code: Select all

Byte 12: 
7       0 
--------- 
xxxx xxBP 

P: This is a PAL ROM.  when set, indicates PAL mode. 
B: When set, indicates this ROM works on both PAL and NTSC machines 
(some of the Codemasters games actually will adjust the game depending 
on if it detects you running on a PAL or NTSC machine.  It adjusts the 
timing of the game, and fixes the music).    

Not many games would have this B flag set.  

x: These bits are not used yet.  They shall be maintained clear.
So, it is a mixture of what we have seen in the iNES and UNIF. This byte is a bitfield, and the spec clearly states that bit 0 set means "indicates PAL mode" (like in iNES), and bit 1 set means "it supports both NTSC and PAL". Unfortunately, description of these bits is ambiguous (it is not described how to treat the new bit 1 in combination with the old bit 0). That's why it is treated in different ways by different emulators. FCEUX: 00 - NTSC, 01 - PAL, 10 - NTSC, 11 - NTSC. Mesen: 00 - NTSC, 01 - PAL, 10 - NTSC, 11 - PAL. Nintendulator: 00 - NTSC, 01 - PAL, 10 - leave as is, 11 - leave as is. IMHO, Mesen has the most logical behavior, just because it does not contradict to the clear statement about meaning of the bit 0 which was borrowed from the iNES.

NSF

The header has this byte:

Code: Select all

offset  # of bytes   Function
----------------------------
$07A    1   BYTE    PAL/NTSC bits
                bit 0: if clear, this is an NTSC tune
                bit 0: if set, this is a PAL tune
                bit 1: if set, this is a dual PAL/NTSC tune
                bits 2-7: not used. they *must* be 0
Similar to NES 2.0. Ambiguous.

NSFe

The mandatory INFO chunk has this byte:

Code: Select all

offset  # of bytes   Function
----------------------------
$0006   1   BYTE    PAL/NTSC bits
                bit 0: if clear, this is an NTSC tune
                bit 0: if set, this is a PAL tune
                bit 1: if set, this is a dual PAL/NTSC tune
                bits 2-7: not used. they *must* be 0
Similar to NES 2.0. Ambiguous.

But it has a very flexible and unambiguous extension, the regn chunk:

Code: Select all

offset  # of bytes   Function
----------------------------
$0000   1   BYTE    Bitfield of supported regions.
                bit 0: NTSC
                bit 1: PAL
                bit 2: Dendy
                bit 3-7: Reserved (always clear)
$0001   1   BYTE    Specifies preferred region if multiple regions are supported. (Optional)
                0 - NTSC
                1 - PAL
                2 - Dendy

This chunk adds support for the Dendy region, and the ability to specify which region is preferred (for players that can support multiple regions). This chunk should appear after the INFO chunk, and overrides its region data (byte 6).
So, it overrides byte 6 if is present. It is aware of existence of Dendy systems. Also it specifies both list of supported systems and a preferred system, so an emulator have enough information about how this ROM can be executed properly.

Time to action

It is time to remove the ambiguity and to add feature parity to all the NES formats. All the above formats can be extended the same way, with preserving compatibility where it wasn't ambiguous before:

Code: Select all

Byte 9 in iNES, or Byte 12 in NES 2.0, or TVCI byte in UNIF, or byte $7A in NSF:
7       0
---------
xdpn xxBP

The byte consists of two nibbles. The higher nibble describes which systems are supported, the lower nibble describes which system is preferred.

If higher nibble is 0, use legacy interpretation:
0000 0000 - NTSC only
0000 0001 - PAL only
0000 0010 - Supports NTSC and PAL, but NTSC is preferred
0000 0011 - Supports NTSC and PAL, but PAL is preferred

Otherwise, higher nibble describes supported systems:
d - Dendy
p - PAL
n - NTSC

Lower nibble (bits BP) describes preferred system:
00 - NTSC (a legacy emulator will use NTSC)
01 - PAL (a legacy emulator will use PAL)
10 - Dendy with fallback to NTSC (a legacy emulator will use NTSC)
11 - Dendy with fallback to PAL (a legacy emulator will use PAL)

If the lower nibble chooses the system which is not marked as a supported system in the higher nibble, fallback to the legacy behavior. Bits 2,3 and 7 should be ignored by emulators, and should be 0 in ROMs.
It describes both how to treat original format with two bits unambiguously, and how to treat extended format. It will be possible to use exactly the same code for extracting information from this byte for all the NES formats.

There is a list of supported systems by a ROM, and a preferred system selected in the ROM. A user of an emulator can choose a preferred system in the settings of the emulator: Auto, NTSC, PAL, Dendy, Force NTSC, Force PAL, Force Dendy. If Auto is specified, emulator will always use the preferred system specified in the ROM itself. If the user chooses NTSC/PAL/Dendy as preferred, the emulator checks if the preferred by user system is supported by the ROM, and if it is, it should use the preferred by user system. Otherwise, it should use preferred by ROM system. And if the user chooses to force NTSC/PAL/Dendy, the emulator just ignores the byte in the header and always uses the forced system.
Last edited by VEG on Wed Nov 21, 2018 3:48 pm, edited 2 times in total.

VEG
Posts: 50
Joined: Mon Nov 11, 2013 2:55 pm
Location: Minsk, Belarus

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by VEG » Wed Nov 21, 2018 3:32 pm

There is another proposal from the NewRisingSun:

Code: Select all

Byte 12: TV System
       7654 3210
       ---------
       .... ..TT
              ++- Frame timing
                  0: RP2C02 (NTSC)
                  1: RP2C07 (PAL licensed)
                  2: RP2C02 and RP2C07 (game self-adjusting or doesn't matter)
                  3: UMC 6527P (PAL with NTSC-like timing, also known as "Dendy" or "Micro Genius" mode)
But it has important disadvantages (NewRisingSun, sorry for criticism, I'm trying to be objective):
- It effectively changes the bitfield from the NES 2.0 to an enum. It seems like an extension not to NES 2.0, but to UNIF. If an emulator follows the original NES 2.0 spec where bit 0 is clearly stated as "this is a PAL ROM", it will always run Dendy ROMs in PAL mode, what is not good, because Dendy was created to be compatible with NTSC games, so it is closer to NTSC.
- It has no feature parity with NSFe format, where it is already possible to specify list of supported and a preferred system.
- It has a strange asymmetry: it is possible to specify that NTSC and PAL are supported, but it is still not possible to specify that the ROM supports NTSC and Dendy, for example. Only one combination is avalable.
- It sill has no proper disambiguation about "what to do when more than one system type is supported" even though it allows to specify that a ROM supports both NTSC and PAL.
This proposal was already adopted by the Mesen lately. But the proposal didn't have enough time to be adopted by ROM makers. It wasn't published anywhere except of that topic, and most people just don't know about it. I believe that it is OK if Mesen changes how this byte is treated one more time. After all, its original treatment of the byte was similar to the unambiguous treatment which I propose for legacy two bits.

To keep everything related about the matter, I'd like to provide NewRisingSun's objections to my proposal with some comments.
NewRisingSun wrote:I oppose, because of one criterion: the header should be unambiguous; every game should have one header with values that are correct for that game, with any other being incorrect. That allows "good" ROMs to have one unambiguous CRC32 including the header. All the current and proposed fields meet that criterion, including the input device field as previously described.
Existence NES 2.0 itself breaks this argument. For example, a ROM of the Duck Hunt can have 3 different headers variations:
- iNES header
- NES 2.0 header
- NES 2.0 header with Zapper controller set
- etc.
All of them are correct. Just different extensions are used. New extensions can't make proper old files invalid.
NewRisingSun wrote:Denoting what is preferred on the other hand is inherently subjective and therefore ambiguous. You could have three times the same ROM image with three different headers because somebody prefers NTSC/PAL/Dendy, and have all three being correct according to the definition.
It is not about preference of a user. It is the most suitable system for the ROM, even when 2 or 3 systems are supported. There is always the most suitable system, just because it is not possible to make a game which behaves exactly the same on all possible systems. There is always a slight difference, even when the game tries to adjust itself to different systems. So, one system is the perfect match, but others are also acceptable.
For example, Unchained Nostalgia. In case of PAL you will have a slightly wrong sound on noise channel (because it is not possible to make it absolutely the same on three systems), in case of NTSC you may notice artifacts of speed adoption (it skips every 6th frame on NTSC systems to have the same speed as on Dendy). So, it is perfect only on Dendy, but it also works nicely on PAL and NTSC systems. That's why Dendy is preferred, but NTSC and PAL also are allowed. This bitfield is not for storing some settings by a user. It is intended to describe what is supported by the ROM, and what is the best option. An emulator can take advantage of this information.

Great Hierophant
Posts: 745
Joined: Tue Nov 23, 2004 9:35 pm

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by Great Hierophant » Tue Dec 04, 2018 9:30 am

NewRisingSun wrote:I oppose, because of one criterion: the header should be unambiguous; every game should have one header with values that are correct for that game, with any other being incorrect. That allows "good" ROMs to have one unambiguous CRC32 including the header. All the current and proposed fields meet that criterion, including the input device field as previously described.
VEG wrote: Existence NES 2.0 itself breaks this argument. For example, a ROM of the Duck Hunt can have 3 different headers variations:
- iNES header
- NES 2.0 header
- NES 2.0 header with Zapper controller set
- etc.
All of them are correct. Just different extensions are used. New extensions can't make proper old files invalid.
No, only the third is correct for NES2.0. The regular iNES header (first example) will fail a CRC check that expects the NES2.0 header. The second example will fail to pass the CRC check because it does not have the Zapper controller bit set, which is the canonical header. To play Duck Hunt, you must have a Zapper connected to $4017 D3 & D4. Whatever else you may have plugged into the system is not critical.

VEG
Posts: 50
Joined: Mon Nov 11, 2013 2:55 pm
Location: Minsk, Belarus

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by VEG » Thu Dec 06, 2018 11:49 am

Great Hierophant wrote:The regular iNES header (first example) will fail a CRC check that expects the NES2.0 header.
iNES header is still a valid one. You can't force everybody to use NES 2.0. A general set of ROMs consists of iNES files for 99%, and probably a few of NES 2.0 if any. It is stupid to expect that a ROM is NES 2.0 only because it is barely used in reality and it will never be as popular as iNES.
Great Hierophant wrote:The second example will fail to pass the CRC check because it does not have the Zapper controller bit set, which is the canonical header.
The "Input/expansion port device" byte is still not a part of the NES 2.0 spec, so even if the Duck Hunt NES 2.0 file exists, it doesn't have this byte set. And it is a perfectly valid NES 2.0 file. And even after the extension which defines the "Input/expansion port device" byte is added, the old NES 2.0 Duck Hunt ROM will be a valid file. Just because the "Input/expansion port device" byte is an extension, it shouldn't break old files or make them invalid.

As for me, I would prefer iNES header for all Nintendo licensed games, just because NES 2.0 is ugly and overcomplicated without any good reason for it. UNIF and NES 2.0 are just for very special cases, they are useful for less than 1% of times. The proposed "Input/expansion port device" byte is useful, but it could be added into iNES also.

lidnariq
Posts: 8787
Joined: Sun Apr 13, 2008 11:12 am
Location: Seattle

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by lidnariq » Thu Dec 06, 2018 12:58 pm

VEG wrote:iNES header is still a valid one.
This has never been about what is a valid header.

A mapper 66 file holding a mapper 0 game is a valid header.

This is about canonical headers. Given a specific PRG and CHR, what is the single correct 16-byte header that it should have.
The proposed "Input/expansion port device" byte is useful, but it could be added into iNES also.
Adding it to NES2.0 is adding it to iNES.

iNES has a problem with dirty headers; that's the entire reason NES2.0 exists. You can't stuff more data into the iNES1 header as a result.

NewRisingSun
Posts: 1062
Joined: Thu May 19, 2005 11:30 am

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by NewRisingSun » Thu Dec 06, 2018 1:32 pm

Man, I am really looking forward to the day when Sour returns from his vacation.

VEG
Posts: 50
Joined: Mon Nov 11, 2013 2:55 pm
Location: Minsk, Belarus

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by VEG » Fri Dec 07, 2018 1:42 am

lidnariq wrote:Adding it to NES2.0 is adding it to iNES.
These formats are different. For example, TV System byte is byte 9 for iNES and byte 12 for NES 2.0.
lidnariq wrote:This is about canonical headers. Given a specific PRG and CHR, what is the single correct 16-byte header that it should have.
I believe that the iNES version of the header should be canonical for Nintendo licensed games. Even Nintendo itself adopted this format for NES Classic.
Last edited by VEG on Fri Dec 07, 2018 2:07 am, edited 1 time in total.

User avatar
rainwarrior
Posts: 7680
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by rainwarrior » Fri Dec 07, 2018 1:50 am

VEG wrote:
lidnariq wrote:Adding it to NES2.0 is adding it to iNES.
These formats are different. For example, TV System byte is byte 9 for iNES and byte 12 for NES 2.0.
In practice, the TV System byte is nonexistent for iNES 1. Rippers don't use it. Emulators don't use it. It was a late extension that never took hold.

VEG
Posts: 50
Joined: Mon Nov 11, 2013 2:55 pm
Location: Minsk, Belarus

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by VEG » Fri Dec 07, 2018 1:59 am

rainwarrior wrote:In practice, the TV System byte is nonexistent for iNES 1. Rippers don't use it. Emulators don't use it. It was a late extension that never took hold.
This byte is a part of the iNES spec for ages (since 2005). Mesen understands this byte, I guess that iNES also understands it, so emulators use it. I haven't checked other emulators, but I don't see any reason to not support it. It is a nice addition to the canonical header from the original author of the format.

lidnariq
Posts: 8787
Joined: Sun Apr 13, 2008 11:12 am
Location: Seattle

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by lidnariq » Fri Dec 07, 2018 2:39 am

We can't extend iNES1 because of dirty headers: there is no way to know whether the contents of bytes 7-15 are meaningful or garbage.
In order to address dirty headers, there to be some specific indicator that the rest of the contents are valid.
This specific indicator that the rest of the contents are valid is NES2.0

Sour
Posts: 739
Joined: Sun Feb 07, 2016 6:16 pm

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by Sour » Fri Dec 21, 2018 5:42 am

(Just giving my thoughts on this, since someone asked me to)

I'm not sure I see a need of changing UNIF at all - that format should just be left as is and people should use NES 2.0 if they require its features. The same could arguably be said about NSF - why change its specs when we can just add optional chunks to NSFe? If someone needs/wants to specify Dendy timings in a new NSF, they can use NSFe.

As for NES 2.0, I'm not sure we need extra features beyond what NewRisingSun's proposal offered. e.g: What's the benefit "Dendy with NTSC fallback"? Emulators will all just use Dendy in this scenario. The rom should just specify the "best" way to use it and NTSC should probably being preferred at this point whenever it works, since its quirks are probably better emulated than PAL/Dendy's. If users want to select another region, it can be left up to them (all emulators already allow this, and I doubt any emulator would block out specific regions based on the header saying the rom doesn't support it...)

tepples
Posts: 21755
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by tepples » Fri Dec 21, 2018 6:23 am

Future famiclones that support multiple TV systems on HDMI output could treat it as "use Dendy if the connected display supports either 50 Hz or FreeSync; otherwise use NTSC"

NewRisingSun
Posts: 1062
Joined: Thu May 19, 2005 11:30 am

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by NewRisingSun » Fri Dec 21, 2018 6:42 am

Emulating Dendy-targeted games as standard NTSC will break several Chinese games that do a lot of things in their NMI handler, such as Waixing's Journey to the West II. I have made good experiences implementing a "Dendy at 60 Hz" mode, which basically speeds up the entire emulated Dendy system (i.e. with 312 lines) by 20% except for the APU.

With Sour's opinion now in, I shall keep my current proposal and copy it to the wiki, making only one change where I find myself somewhat agreeing with VEB: Value $02 should no longer say "Game adjusts itself for RP2C02 or RP2C07", but merely "Game is self-adjusting or region is unknown", with the implementation advice that emulators should not enforce any particular region for value $02, but either apply a user-defined "Default Region", which will be the same region that is used for iNES 1.0 ROMs, or just keep the last selected region.

User avatar
tokumaru
Posts: 11467
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by tokumaru » Fri Dec 21, 2018 8:59 am

NewRisingSun wrote:Emulating Dendy-targeted games as standard NTSC will break several Chinese games that do a lot of things in their NMI handler
I though that NMIs fired at the end of vblank on Dendy, leaving as much time left for vblank operations as on NTSC consoles.

NewRisingSun
Posts: 1062
Joined: Thu May 19, 2005 11:30 am

Re: TV System byte extension in iNES, NES 2.0, UNIF, NSF, NS

Post by NewRisingSun » Fri Dec 21, 2018 9:36 am

Correct, but that affects the timing of NMI vs. beginning of the next frame only.

These Chinese games for Dendy-like famiclones fail on NTSC for a different reason: Having disabled rendering but not NMI generation, their NMI handler takes more than 29780.5 CPU cycles to finish. On NTSC, this leads to the NMI handler being re-entered before it has finished, in turn causing the game to freeze. On Dendy, even though the NMI is generated at the end of vertical blanking for compatibility with NTSC as you pointed out, the game's NMI handler has 35464 CPU cycles until the next NMI is generated. The problem does not occur during normal gameplay, but mainly during screen/level transitions while the memory data for a new level is being generated. (This is a part of why I dislike games that have their entire game logic in their NMI handler.)

These games can be patched to work on NTSC by either disabling/enabling $2000.7 at the beginning/end of the NMI handler, or setting/clearing a "busy" flag at the beginning/end of the NMI handler.

Post Reply