A note to all NES-emulator authors: ROM header wishlist

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: A note to all NES-emulator authors..

Post by lidnariq »

James wrote:So would you dedicate some of that space to defining different VRC4 revisions? How about something like whether or not expansion audio is used and how it's mixed on the cart?
Submappers. Already done for VRC4. I'm tentatively working through the N163, but I don't know enough to make an official stance. I'm not certain what the point of explicitly marking a "no sound" version of VRC7 for TTA2 (separate from the question of A3 vs A4 for the register selection).
byuu wrote:Map database entries to SHA256 hashes. Barring a future intentional exploit that allows SHA256 hash collisions (which we don't have yet, and if there ever is one, we can use stronger hashes), there are more possible ROM combinations than there are atoms in the universe.
I don't understand why keeping people from intentionally making their ROM collide with something else is useful. The ability to make a collision even seems useful to me, if you have a game that isn't describable using the conventional headers. The bit that started this entire thread is for exactly this reason: iNES1 headers are insufficient, and overly-precise hashes prevent useful intentional collisions.

I guess the answer is: because there's two different objectives: One is denoting "This is the Real Original Game" and for that you want to prevent people from distributing broken things (But is that a real threat, or just an imagined one? CRC32 is enough to prevent almost any accidental collision). The other is "use this data as though it were on this PCB" and that's theoretically what the headers are for.
zzo38
Posts: 1096
Joined: Mon Feb 07, 2011 12:46 pm

Re: A note to all NES-emulator authors..

Post by zzo38 »

byuu wrote:> and optionally implement .NES.INI too Ooooh, another new format. And one that doesn't support trees. Don't forget to also include a MESS internal database entry, a Nestopia XML file, and a higan manifest.bml as well. Is anyone using JSON or YAML yet? Gotta be somebody out there.... The idea would be that absolutely anything and everything would go into it. If anyone finds certain info relevant, we could add it. Peoples' derivations could discard what they consider unimportant. Not only could you generate XML/JSON/YAML/BML, you could also generate iNES 1.0/UNIF/iNES 2.0 headers from the information. ...
No. .NES.INI is a separate file for each game, not a database (this is why trees aren't needed). Things needed for emulation should be in the NES 2.0 headers; if the header is bad, it should be corrected with an external tool rather than within the emulator itself. The emulator itself shouldn't do cheating stuff like checksums and so on; it should use NES 2.0 headers if available. The .NES.INI is supposed to contain extra data which usually isn't needed for the game to work (although the format allows such things to be provided in case some emulator needs it).

However, a SQL database can be made containing game checksums and then given the PRG ROM and CHR ROM files (and possibly other data), can map this data to a .NES.INI and to a NES 2.0 header. But this should be an external tool, not part of the emulator. I know about programming in SQLite so I can know how such thing can be done. You could have seven fields: game title, PRG ROM checksum, PRG ROM size, CHR ROM checksum, CHR ROM size, NES 2.0 header, and .NES.INI data.

I don't know what XML format Nestopia uses, or what it uses it for.

For the emulator to use hashes instead of the actual data in the header, is a klugy way which I think is no good (for several reasons). Sometimes it might be necessary, but usually the header should be corrected before loading in the emulator!!!!!!!! Nevertheless, a few emulators have some features that are badly written and need a checksum to know how to run them (ideally, no emulator should require this, but sometimes there is no choice); for these things, .NES.INI does provide a solution (if the .NES.INI exists and contains a bootgod ID of the game, that should be used instead of the checksum). However, if the header is simply incorrect, you should not do this; in fact, the .NES.INI specification specifically says that you shouldn't correct the iNES header based on the bootgod ID.
tepples wrote:The NES 2.0 file represents the function of everything inside the Game Pak shell. Things like controllers and overscan go in the manual. (Yes, a few NES era manuals mentioned overscan. I can provide references if you want.) So what you need is a machine-readable version of things commonly found in the manual.
That is precisely what the .NES.INI format is designed for; things like controllers and overscan. It is not intended to correct bad headers. The NES 2.0 header should represent the function of the cartridge and should do so accurately (if it is wrong, it should be corrected externally and not within the emulator).

It is not intended that any emulator must support the .NES.INI format; it is a completely optional file (although I provide it in my own software for the Famicom, despite not yet being supported by anything; emulators will just read the .NES and work fine, although you might need to configure the input device manually).
Last edited by zzo38 on Fri Jun 21, 2013 7:06 pm, edited 1 time in total.
(Free Hero Mesh - FOSS puzzle game engine)
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A note to all NES-emulator authors..

Post by James »

lidnariq wrote:I'm not certain what the point of explicitly marking a "no sound" version of VRC7 for TTA2 (separate from the question of A3 vs A4 for the register selection).
On "no sound" Famicom cartridges, 2A03 audio is simply passed through untouched (pins 45 and 46 bridged together), so audio output is 1.0 * 2A03 output. On cartridges with expansion audio, 2A03 output is mixed with expansion audio at some ratio, so audio output will be x * 2A03 + y * expansion. I'm unsure if the values of x and y differ for boards with the same mapper (e.g., is Just Breed audio mixed in differently than Metal Slader Glory audio is?).

In any event, knowing how audio is passed through the cartridge is necessary to accurately emulate audio output.
get nemulator
http://nemulator.com
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A note to all NES-emulator authors..

Post by James »

byuu wrote:Map database entries to SHA256 hashes.
Upon further reflection, I think you're right about this. The iNES format is sufficient for the vast majority of ROMs. There is little incentive for users to switch formats or distribute new ROMs, so this has to be handled without any header changes.

Though I think it should be CRC-64 instead of SHA-256. Mainly because the implementation is easier, but a little bit because we can't agree on everything here. :lol:

Seriously, though, let's just do this. How do we get started?
get nemulator
http://nemulator.com
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Re: A note to all NES-emulator authors..

Post by Near »

> What do you do with homebrew and hacks? Fallback to iNES/other header and hope for the best?

Add them to the database.

> The NES 2.0 file represents the function of everything inside the Game Pak shell. Things like controllers and overscan go in the manual.

Every PCB attribute known so far. Who knows what Hong Kong will come out with next.

By having an infinitely extensible format, you never have to worry about running out of bits. After that, you could argue binary vs text, embedded in the ROM vs separate.

You're already using submappers to denote VRC pinouts, so there's no longer even a 1:1 correspondence of bits to functionality. What the bits mean change based on other bits. And you already have a pseudo-database inside the emulator to tell you that mapper #(rand % 256) == VRC4 anyway. Codification into a database really isn't much more work at this point.

> I don't understand why keeping people from intentionally making their ROM collide with something else is useful.

Mainly for preservation. I am trying to document every licensed SNES image, and for that, I need hashes that verify images are pristine originals.

> But is that a real threat, or just an imagined one?

You'd have to ask historians. We're probably somewhat well off in that even if a fake did appear in the timespan between a hashing algorithm being broken and a stronger one appearing to replace it, the real one would presumably have more copies around anyway, so in theory it should be easy to spot.

Still, no harm in some due diligence. I have my own master set, and will rehash the originals with SHA3+.

> No. .NES.INI is a separate file for each game, not a database (this is why trees aren't needed)

My BML files are separate for each game as well, and can also be in a database.

Trees are useful for a single game as well. They help show parent-child relationships.

Here's a template for a DSP-1 board in my format:

Code: Select all

board type=1K1B revision=01
  rom name={rom.name} size={rom.size}
  ram name={ram.name} size={ram.size}
  map id=rom address=00-3f,80-bf:8000-ffff
  map id=rom address=40-7d,c0-ff:0000-ffff
  map id=ram address=20-3f,a0-bf:6000-7fff mask=0xe000
  necdsp model=uPD7725 frequency=8000000
    rom name={rom[1].name} size={rom[1].size}
    rom name={rom[2].name} size={rom[2].size}
    ram size={ram[1].size}
    map id=io address=00-1f,80-9f:6000-7fff select=0x1000
In an actual generated manifest, there would be a separate tree for the information (preferred controllers, catalog name (for sorting), actual name (in native language), publisher, serial#, etc.

> Though I think it should be CRC-64 instead of SHA-256.

The beauty of this is that we could store as many hash formats as we wanted, and the emulator author could pluck out the fields they wanted. Bonus points if the emulator has a "sync with online database" option in the menu for updates.
Last edited by Near on Fri Jun 21, 2013 9:31 pm, edited 2 times in total.
zzo38
Posts: 1096
Joined: Mon Feb 07, 2011 12:46 pm

Re: A note to all NES-emulator authors..

Post by zzo38 »

byuu wrote:> What do you do with homebrew and hacks? Fallback to iNES/other header and hope for the best?

Add them to the database.
I think this is stupid to ever use a database for the emulator to decide. Use the NES 2.0 headers; if they aren't available, use an external program to convert them.
(Free Hero Mesh - FOSS puzzle game engine)
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Re: A note to all NES-emulator authors..

Post by 3gengames »

byuu....you have the best emu nobody uses because it doesn't play any roms. I know I've downloaded meaning to convert ROMS and just left ti in the dust. But really, CRC crap and it telling you what ROM version, good bad, or nonexistent...who really gives 2 craps? Especially when it annoys you. It wouldn't be too bad if it was ALL in the background, and didn't have to annoy the piss out of you with the message you have to load 5 times to read it all. Headers are the absolute best all-in-one solution. If I'm on a desert island with my laptop and want to play games, fuck CRC checks and fuck databases. Self contained formats are the only ones worth 2 damns, end of it.
User avatar
blargg
Posts: 3715
Joined: Mon Sep 27, 2004 8:33 am
Location: Central Texas, USA
Contact:

Re: A note to all NES-emulator authors..

Post by blargg »

tepples wrote:@byuu
The NES 2.0 file represents the function of everything inside the Game Pak shell. Things like controllers and overscan go in the manual. (Yes, a few NES era manuals mentioned overscan. I can provide references if you want.) So what you need is a machine-readable version of things commonly found in the manual.
Nice. Taking this further, settings like per-game overscan and turbo rate are like emulating a separate game room for each game. There you can adjust the TV's internal controls and have a NES advantage with the turbo controls set just how you like it for that game, and not have them disturbed when playing another game in a different game room.
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Re: A note to all NES-emulator authors..

Post by Near »

3gengames, just a friendly reminder, I never read any of your posts:

Image

If it wasn't about disparaging me (which would be a first), then please accept my apologies and ignore this message.
If it was, you're wasting your effort here. Might I recommend you try 4chan? There's a much more receptive audience to trolling there.

> I think this is stupid to ever use a database for the emulator to decide. Use the NES 2.0 headers; if they aren't available, use an external program to convert them.

To each their own. By the way, if you develop an emulator, and if it gets popular enough, I'll save you the trouble of learning this lesson yourself:
People went crazy when I made my emulator require an external program that translated games to the format I wanted. You're suggesting the same thing, just with iNES2 instead of BML.
I ended up integrating the conversion tool right back into the emulator anyway. And now everyone pretends that said integration isn't there, because it was missing from a single release.

> Nice. Taking this further, settings like per-game overscan and turbo rate are like emulating a separate game room for each game.

There's always RetroCopy. You can have your own 3D rendered video game movie theater room to play your games on.

Anyway, I like the idea of per-game overscan. I don't want to see artifacts on vertical scrolling in Mega Man, but cutting off 16 pixels from the top of the screen looks terrible in horizontal scrolling games. I wouldn't recommend including subjective data into binary headers inside games themselves, but it's a nice addition to extensible metadata in a database. Let each emulator author choose if they want to act on it or not.
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Re: A note to all NES-emulator authors..

Post by 3gengames »

byuu wrote:3gengames, just a friendly reminder, I never read any of your posts:

Image

Thanks, then we don't have anybody justifying garbage. You call it trolling, I call it being the only person who will call you out on your emulator forcing the worst standards ever seen in any emulator, failed or not. :)
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: A note to all NES-emulator authors..

Post by Sik »

byuu wrote:I want to know what controllers a game supports, and which ones are the preferred input methods, if applicable.
Am I the only one who thinks that should be left out of the ROM and instead just make the user select the peripherals? I know this is what's done on Mega Drive emulators, no idea why it can't happen elsewhere. I mean, on real hardware you can be using the wrong set of peripherals and nothing would stop you if you forget to swap them out...

Something that could be done is store the peripherals used when the ROM is closed and restore them when the ROM is reopened again (which in fact will ensure that what the user decides is the ideal is used, not what the emulator thinks it is). Of course make this a setting. Keeping track by filename should be enough, this is already done for savegames and savestates anyway.
lidnariq wrote:I don't understand why keeping people from intentionally making their ROM collide with something else is useful. The ability to make a collision even seems useful to me, if you have a game that isn't describable using the conventional headers. The bit that started this entire thread is for exactly this reason: iNES1 headers are insufficient, and overly-precise hashes prevent useful intentional collisions.
For the only thing intentional collisions are useful is to make it harder for emulators to attempt to emulate the game correctly (as they can't tell what they have to emulate, at least without trial and error). I guess this makes sense if you want to ensure nobody can play it anywhere except real hardware, but eventually emulator authors will find a way around it.

EDIT: typo
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Re: A note to all NES-emulator authors..

Post by Near »

> Am I the only one who thinks that should be left out of the ROM and instead just make the user select the peripherals?

I think it should absolutely be left out of the ROM.

But inside some sort of information markup or database, it's a nice idea.
This is just something that makes the emulator more friendly. Duck Hunt only works with the Zapper. Why make your user select the Zapper when they load Duck Hunt, and then swap back to a controller when they load Mario Bros? I picked that example intentionally, as there's the infamous SMB/DH combo. So when the user goes to choose a controller for SMB/DH, why show them the Oeka Kids tablet? What if on a whim they played Oeka Kids, but didn't know it used the tablet? What if they play Tokimeki Memorial and don't even know it supports a mouse? They may not have a copy of the manual, as very few people are scanning those in. Yeah, their fault, but still. We can help them with that.

Same reason I include a cheat code database with my emulator. No reason to make users go on Google looking for codes that may be for a different country or ROM revision, or just may not work at all.

We can make users do lots of work. Usually users tend to like having to do less work.

> I mean, on real hardware you can be using the wrong set of peripherals and nothing would stop you if you forget to swap them out...

Sure, and you can run NTSC games on PAL (maybe with an adapter) and vice versa to get lovely error messages. And some people want to see those errors for some reason, and so we should allow them to if they want that.

> Something that could be done is store the peripherals used when the ROM is closed and restore them when the ROM is reopened again

Exactly. And you could also include overscan clipping, preferred video filter (Zelda 3 looks great with HQ2x, DKC looks terrible with it), custom controller bindings, cheat codes, etc.

Now what if, the very first time a user loaded a game, we looked up the best default settings for them, and then let them configure the settings from there?

> but eventually emulator authors will find a way around it.

Custom polynomials for a per-emulator-build checksum routine would do quite nicely ;)
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: A note to all NES-emulator authors..

Post by tokumaru »

Sik wrote:Am I the only one who thinks that should be left out of the ROM and instead just make the user select the peripherals?
I used to think that this information was completely unnecessary, but after giving some thought to it, emulators are often about enhancing the experience. We get savestates, video filters, cheats... luxuries we can't possibly have on the actual console. So why would automatic controller configuration be any different?

I do agree that this should be left out of the ROM though (it should be in an external file, a database somewhere, etc.), and be completely optional and overridable. After all, there are legitimate uses for unusual controller configurations, like enabling cheats and so on.
Something that could be done is store the peripherals used when the ROM is closed and restore them when the ROM is reopened again (which in fact will ensure that what the user decides is the ideal is used, not what the emulator thinks it is). Of course make this a setting.
Perfect solution, IMO. I see no problem in loading the initial settings from a database though, if you enable this option.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: A note to all NES-emulator authors..

Post by tepples »

byuu wrote:What if they play Tokimeki Memorial and don't even know it supports a mouse?
There's a mouse, and then there's a mouse. The Super NES mouse and the Subor mouse have different protocols, and I've had to reply to e-mails from players who couldn't get the mouse to work in an emulator because Thwaite supports only the Super NES mouse. ("Try it in Nintendulator." "It worked; thank you.") How many mice are there?
We can make users do lots of work. Usually users tend to like having to do less work.
And sometimes you have to make users do work to satisfy the requirements of the law.
Sure, and you can run NTSC games on PAL (maybe with an adapter) and vice versa to get lovely error messages. And some people want to see those errors for some reason, and so we should allow them to if they want that.
Especially homebrew developers testing their work. A game that I developed in 2009 had separate builds for 50 Hz and 60 Hz machines and would freeze on a "REGION CODING FAILED!" screen if the TV doesn't match to discourage people from cheating by playing the NTSC version on a PAL NES or Dendy to get 20% more time to react. Another game that I developed two years later autodetects the TV system but has speed correction in the same build.
Now what if, the very first time a user loaded a game, we looked up the best default settings for them, and then let them configure the settings from there?
Then you might be infringing a Nintendo patent.
zzo38
Posts: 1096
Joined: Mon Feb 07, 2011 12:46 pm

Re: A note to all NES-emulator authors..

Post by zzo38 »

byuu wrote:Here's a template for a DSP-1 board in my format:

Code: Select all

board type=1K1B revision=01
  rom name={rom.name} size={rom.size}
  ram name={ram.name} size={ram.size}
  map id=rom address=00-3f,80-bf:8000-ffff
  map id=rom address=40-7d,c0-ff:0000-ffff
  map id=ram address=20-3f,a0-bf:6000-7fff mask=0xe000
  necdsp model=uPD7725 frequency=8000000
    rom name={rom[1].name} size={rom[1].size}
    rom name={rom[2].name} size={rom[2].size}
    ram size={ram[1].size}
    map id=io address=00-1f,80-9f:6000-7fff select=0x1000
This looks like it might be something similar to the mapper 768.0 which I have defined, but I don't really know if that is it.
tepples wrote:
byuu wrote:What if they play Tokimeki Memorial and don't even know it supports a mouse?
There's a mouse, and then there's a mouse. The Super NES mouse and the Subor mouse have different protocols, and I've had to reply to e-mails from players who couldn't get the mouse to work in an emulator because Thwaite supports only the Super NES mouse. ("Try it in Nintendulator." "It worked; thank you.") How many mice are there?
I don't know how many; could you add information of Subor mouse into wiki? (I recommend the use of the Famicom keyboard and SNES mouse in new projects (these are the official Nintendo protocols), although it would be useful to have the other information too.)
(Free Hero Mesh - FOSS puzzle game engine)
Post Reply