NES 2.0 expansion for maskROM data

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Great Hierophant
Posts: 780
Joined: Tue Nov 23, 2004 9:35 pm

Re: NES 2.0 expansion for maskROM data

Post by Great Hierophant »

I don't think that even the creator of the A Winner is You 64MB homebrew cart had vocals.

I think that tacking on the extra data to the end of the .nes file is probably best way to go. The situation is not unique, PC-10 ROMs have an extra 8KB on the end of their files that emulators ignore unless they support the PC-10 (few do).
User avatar
B00daW
Posts: 586
Joined: Thu Jan 03, 2008 1:48 pm

Re: NES 2.0 expansion for maskROM data

Post by B00daW »

Another example of an ADPCM decoder is City Fighter IV.
Image

I don't think it's emulated properly since CaH4e3 just hooks the data to $4011

Code: Select all

static void UNLCITYFIGHTPower(void) {
	prg_reg = 0;
	Sync();
	pcmwrite = GetWriteHandler(0x4011);
	SetReadHandler(0x8000, 0xFFFF, CartBR);
	SetWriteHandler(0x8000, 0xFFFF, UNLCITYFIGHTWrite);
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NES 2.0 expansion for maskROM data

Post by lidnariq »

rainwarrior wrote:I wasn't trying to do something "clever", I was trying to suggest something "extensible".
[...]
...except with an easily extensible format? It's incredibly possible with chunks.
[...]
An extensible format means it's possible for people to add new bits of data they have a use for without having to play some political game of deciding whether it's important to be part of the format or not. Anybody that thinks it's important can pick it up and add it, anyone that doesn't can ignore it without changing a single line of code in their emulator.
[...]
All of those format arguments indicate a large desire to include other information in the file that maybe you think isn't important, but other people do. The whole reason I suggested chunks was to avoid argument and arbitration about what can go in the file-- it makes it very easy to ignore anything you don't care about!
[...]
Anyhow, the reason I suggested it is that it's very easy to implement, simple, practical solution to a general problem that seems to continually arise. This thread is maybe narrowly about one specific case of that problem, but I'm certain it will keep coming up. I don't really understand the downside to it (?), so it's a serious suggestion on my part.
Thank you for recapitulating part of the MAME/UNIF/byuu's format argument. I asked you to not.

The thread started specifically with the question of
What do we do as far as NES2.0 headers for files with extra data

What you said was
If you used a chunked format instead you wouldn't need to worry about this

Surely you see how this failed to answer the question.

What you seem to have meant to say was
Maybe this isn't appropriate to add to NES2.0 headers

Which, well, fine, but advocating for a new (or new old) format is an extremely divisive point in this forum. You're welcome to do as you wish, but please don't bring it here.
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 expansion for maskROM data

Post by rainwarrior »

lidnariq wrote:Thank you for recapitulating part of the MAME/UNIF/byuu's format argument. I asked you to not.
I don't seem to know what "MAME/UNIF/byuu's format argument" is the way you're phrasing this, nor do I understand that you asked me to not bring it up.

I said exactly what I said as a good faith suggestion in response to the problem that was brought up in this thread. It's what I'd probably want to do if I was implementing this.
lidnariq wrote:Surely you see how this failed to answer the question.
No, I don't, and I feel quite insulted by this response.
Last edited by rainwarrior on Tue Jun 20, 2017 10:54 pm, edited 2 times in total.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NES 2.0 expansion for maskROM data

Post by lidnariq »

rainwarrior wrote:
lidnariq wrote:Surely you see how this failed to answer the question.
No, I don't, and I feel quite insulted by this response.
"How do I X?"
"Y (which is wholly unrelated to X) solves the underlying question I think you meant to ask."

Do you really not understand how that's not an answer? Do you not understand how it's denying the asked question?

Nevermind that your phrasing was extremely obtuse ("Y solves the problem" (but doesn't involve X at all)), leading to all this extra discussion when your entire point is really just "Maybe you shouldn't want to X"

Anyway, if you have any point pertaining to putting any such metadata in a NES2.0-encoded file, please continue. Otherwise, please open a new thread instead.
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 expansion for maskROM data

Post by rainwarrior »

Edit: removed unnecessary and off topic respose.
Last edited by rainwarrior on Wed Jun 21, 2017 5:11 pm, edited 2 times in total.
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 expansion for maskROM data

Post by rainwarrior »

Edit: removed unnecessary and off topic respose.
Last edited by rainwarrior on Wed Jun 21, 2017 5:11 pm, edited 1 time in total.
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: NES 2.0 expansion for maskROM data

Post by Myask »

Wondered how long that would take. Problem dumps finally coming in, I guess.

calm down. Lidnariq's right in that "use chunks" is an odd answer to "how should we extend our binary-blob format"…even if it's not asked that way. (And we have had the holywar argument a fair few times, with myself and… zzo? vs koitsu et al, usually…) And rainwarrior's right in that chunks are more extensible. Should really actually obey IFF constraints if one goes to chunked, though; leading with NES[$1A] isn't compliant, it's not FORM, LIST, nor "CAT ".
the only parts used were the M50805, µPD7755, and µPD7756. These would correspond to 960B, 12KiB, and 32KiB.
Which used the 7755? Terao no Dosukoi Oozumou, fixed. also, aren't those numbers wr- grumble grumble Bootgod specifies those particular chips in bits, not Bytes.

Which makes it annoying, as that means that mapper 18 has three separate soundchip options (incl. none). Submapper time?

Also, SND-ROM after INST-ROM…arbitrarily because INST-ROM was in the specification first. Which would leave the order:
Header [Trainer] PRG [CHR] [INST] [PC-PROM] [SND] [Title]
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES 2.0 expansion for maskROM data

Post by tepples »

rainwarrior wrote:
lidnariq wrote:I don't think it's a clever idea to append a chunked format to the existing iNES format.
I wasn't trying to do something "clever", I was trying to suggest something "extensible".
But unless the extensibility is carefully planned out, it isn't going to be easy to support on an NES flash cart such as PowerPak or EverDrive. Is it practical to make a UNIF loader in 1K of code on a 6502? Because that's all you get for each overlay.
rainwarrior wrote:
lidnariq wrote:It's impossible to support every single possible way someone could inject audio data...
...except with an easily extensible format? It's incredibly possible with chunks.
And as I mentioned two years ago, chunked NES ROMs are going to be incredibly difficult to patch with things like IPS or xdelta3. They'd need a special-purpose patcher that might be slow to reach minority computing platforms.
rainwarrior wrote:Similarly, if you don't want a ROM with MP3s in it, that's no problem for you... you don't have to download that ROM! Someone else would probably care about it though!
That's been true for only a few weeks. Before May 11, 2017, including support for MP3 in an emulator would have incurred patent liability.
AWJ
Posts: 433
Joined: Mon Nov 10, 2008 3:09 pm

Re: NES 2.0 expansion for maskROM data

Post by AWJ »

I think using a NES 2.0 submapper for each combination of mapper and ADPCM chip that's verified to exist as real hardware makes more sense than adding entirely new header fields just for them. Whatever type of sound chip is used, its registers have to be mapped in a way that doesn't clash with mapper registers or with SRAM--you can't just stick a 7756 onto any arbitrary cartridge PCB--so tying them to mappers seems reasonable to me. Let's not recreate the NSF lunacy where people can and do compose "NES music" that uses one of every cartridge sound chip all at once.

Once the type of ADPCM chip used by a cartridge is specified, I think it's fine to say that the ADPCM data goes after the end of the CHR data (or after the PC10 Z80 data in the entirely hypothetical case of a PC10 cartridge with an ADPCM chip) and its size is dictated by the chip type. We don't use a header field to specify the size of the EXRAM in MMC5 cartridges, after all.
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 expansion for maskROM data

Post by rainwarrior »

Myask wrote:Lidnariq's right in that "use chunks" is an odd answer to "how should we extend our binary-blob format"…even if it's not asked that way. (And we have had the holywar argument a fair few times, with myself and… zzo? vs koitsu et al, usually…) And rainwarrior's right in that chunks are more extensible. Should really actually obey IFF constraints if one goes to chunked, though; leading with NES[$1A] isn't compliant, it's not FORM, LIST, nor "CAT ".
I mentioned RIFF as an example of a format that accomplishes the chunk idea, I wasn't making any argument to use IFF standards.

The view I have is just that "if the mapper number is X then the meaning of data appended after PRG + CHR changes" is very fragile. The length of the file provides the minimum amount of information to include just that, but any further discovered data needs become extremely difficult to accommodate. Lidnariq made a comment about ordering of PC-10 data vs this hypothetical extension, which isn't specifically a problem for anybody, but it was a small representation of the kind of problems that can easily arise: multiple appended blocks are likely to require some data about their size.

Yes, my suggestion was to build that information into the appended data, instead of putting it into the header. Yes, as an answer it was a little bit oblique to the letter of B00daW's question, if you really want to interpret it that way, but I did honestly feel it was relevant to a larger context here. Sorry, if this is not what lidnariq wanted to talk about, but his response felt bafflingly and unnecessarily hostile toward it. I didn't initially want to make a big thing about it either, but he said some things I thought were worth responding to.

Yes also, the particular problem of these jaleco carts doesn't have this case, but I thought it was reasonable to suggest something that might ease the pain of having to renegotiate the format every time something new comes out. In particular the argument to restrict it to commercial era games only...!!! (I'm pretty certain B00daW is interested in being able to emulate new homebrew hardware ideas, and so are a lot of people.)
tepples wrote:But unless the extensibility is carefully planned out, it isn't going to be easy to support on an NES flash cart such as PowerPak or EverDrive. Is it practical to make a UNIF loader in 1K of code on a 6502? Because that's all you get for each overlay.
I wasn't advocating UNIF, or even thinking about UNIF, though looking at its format now I am reminded that it does use a chunked format.

I don't think PowerPak's lack of support for UNIF has anything to do with the chunks. The hardest thing that would be hard to fit for code size is the insane number of unique board names instead of an iNES mapper number, but even that has feasible solutions, IMO. PowerPak doesn't have a UNIF loader because few people use it and bunnyboy or other capable people just aren't interested in it. Chunks themselves aren't hard or impractical to parse at all.

In this case though I was only suggesting chunks for appended metadata. Not UNIF, not MAME (yes, ZIP compression is a pretty significant problem), just what to do with some extra data which has hitherto been unavailable and thus ignored by emulators. The Jaleco data seems very "optional" to me, and chunks are easy to optionally ignore. (The current emulator solutions seem to be simulation via a collection of WAV files in the same directory.)
tepples wrote:And as I mentioned two years ago, chunked NES ROMs are going to be incredibly difficult to patch with things like IPS or xdelta3. They'd need a special-purpose patcher that might be slow to reach minority computing platforms.
Again, I wasn't advocating UNIF. If you want IPS compatibility, the PRG and CHR are still where they always were. We've been discussing how inadequate IPS is for a long time, though.

So... sure IPS wouldn't work on the appended chunks, if there's many versions of the ROM out there with different orderings (though this is not much different from the PRG variation problems we already have due to lack of a CRC check), but as a spitball idea the entire problem might also be solved with a "patch" chunk just appended to the file too; the patching process wouldn't need any special tool, just a simple concatenation!
tepples wrote:
rainwarrior wrote:Similarly, if you don't want a ROM with MP3s in it, that's no problem for you... you don't have to download that ROM! Someone else would probably care about it though!
That's been true for only a few weeks. Before May 11, 2017, including support for MP3 in an emulator would have incurred patent liability.
MP3 was just a hypothetical example for the purposes of arguing about whether wild homebrew hardware ideas should be accommodated. I don't personally feel there's a reason to decide they should never be, or that they can't necessarily be planned for in advance, and that seemed to be the argument that Lidnariq was making.

The suggestion was really that I think it would be worthwhile to continue the iNES format in a more extensible direction if possible. Instead of having arguments about what the minimum possible amount of data we need to represent is, make it possible for users to add the things they need.


Anyhow, I'm not exactly trying to say that chunks are the only way to go, I really just wanted to float the idea while we're discussing possibilities. I do think something belongs somewhere in the file to indicate how much extra data follows; even reserving 1 bit in the header for "has something appended, see mapper definition for how to interpret" would be better than trying to use the length of the file itself, I think. Or if not in the header, some signature just before the appended data to verify that "yes, this is the new stuff".
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: NES 2.0 expansion for maskROM data

Post by Myask »

I expect lidnariq just wants to avoid the "if we're revising it at all, let's overhaul it" argument.
I mentioned RIFF as an example of a format that accomplishes the chunk idea, I wasn't making any argument to use IFF standards.
Oh, I know that. I was making said argument, albeit as an aside.
bit in the header for "has something appended, see mapper definition for how to interpret" would be better than trying to use the length of the file itself, I think.
Yes, because of the optional final title chunk field that can be 128/127 bytes, and the PC-PROM 16(or was ti 32) bytes…So, back to NES 2.1 proposals.

Code: Select all

Byte 12
AAAA.DNP
A: audio/(or other auxiliary) ROM size (as per bytes 10-11)
D: Dendy mode supported
N: NTSC mode supported
P: PAL mode supported
We haven't grown a third address bus to need another "battery-backed RAM" field.
Header [Trainer] PRG [CHR] [INST] [PC-PROM] [AUD] [Title] still sound like a good location for audio? The rest are already specified.
We could specify PRGbanks=0 to be PRGbanks=$1000, while we're at it.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NES 2.0 expansion for maskROM data

Post by lidnariq »

I owe Rainwarrior an apology.

Not only was I horrifically rude, but I strongly suspect we were violently agreeing: NES2.0 is an inappropriate container for arbitrary extra data to be determined in the future.

(To make it perfectly clear, I would have no problem with someone putting an MP3 player inside a Famicom cart, and then someone wanting to dump that and play it in an emulator. I just don't see any practical way to package that in a NES2.0 file.

That's what I meant when I said "I object": with only 20ish bits remaining, there isn't enough state left to describe arbitrary variable-length binary attachments. The µPD775X ROM actually includes a table-of-contents inside it, to let the decoder know where of the up-to-256 different playback entries start. Something similar would be required for anything else. I assume that some of the bytes inside the M50805 also serve a similar function.

"Hilariously", ogg files actually support enough internal metadata to be able to scan a file to find each of the separate audio streams inside, while MP3 doesn't.)

The entire original question then got bogged down into ... at least, my misunderstanding, and perhaps also Rainwarrior's.

So: I'm sorry.
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 expansion for maskROM data

Post by rainwarrior »

Myask wrote:
bit in the header for "has something appended, see mapper definition for how to interpret" would be better than trying to use the length of the file itself, I think.
Yes, because of the optional final title chunk field that can be 128/127 bytes, and the PC-PROM 16(or was ti 32) bytes…So, back to NES 2.1 proposals.

Code: Select all

Byte 12
AAAA.DNP
A: audio/(or other auxiliary) ROM size (as per bytes 10-11)
Hmm, I don't think 4-bits is a very good size identifier for a generic auxiliary blob. Lots of things aren't power-of-two sized, and that 1MB limit is really low. If there's a blob of data whose length is not fixed, just put that length info into the blob? For consistency, always leading that block with its total 4-byte size might be prudent, to facilitate the option of skipping an unknown blob, though probably some blobs wouldn't be as optional as extra audio data. (I just think it's valuable if the structure of the file is still parsable even if the meaning of the data isn't known; e.g. still being able to analyze the NES file even if you don't know anything about a new mapper's definition-- there are useful things you can do with NES files besides emulate them.)

That's why I suggested at most 1 bit in the header to indicate its presence or non-presence, or alternatively just some kind of signature at its expected position, though a signature followed by a size is effectively a "chunk" ;P
Myask wrote:We could specify PRGbanks=0 to be PRGbanks=$1000, while we're at it.
I am not sure what's best to do with PRG of 0, though I do think it bigger PRG support would be a better use for some of the remaining bits than an auxiliary blob size. (Ha ha, as a silly thought you could set PRG size as 0 and have a mapper whose real PRG goes in the aux blob.)
Great Hierophant
Posts: 780
Joined: Tue Nov 23, 2004 9:35 pm

Re: NES 2.0 expansion for maskROM data

Post by Great Hierophant »

As I understand it, there are three approaches to packaging a rom in terms of popularity. The first and most popular is the concatenated approach of iNES where you join header + prg-rom + chr-rom in one file. The second method is the split directory/zip approach, where every rom is its own file and all the roms needed to make a game run are contained in a single directory or zip file. This is the approach taken by MESS and higan/nSide. The final approach is the UNIF approach where you have one file with a chunk format which structures the ROMs and configuration information in the file with defined labels. It does seem easier to use either the second or third approach to add new data to existing ROMs, but given their relative lack of popularity, the tacking on approach of iNES will probably prevail.
Post Reply