NES 2.0 XML Database

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

NES 2.0 XML Database

Post by NewRisingSun » Fri Apr 17, 2020 12:43 pm

Here is a complete (see note 1) database of NES 2.0 headers in XML format. The fields should be self-explanatory, for the most part:
  • Just "rom" is the entirety of the ROM file without header, including PRG-, CHR- and Misc ROM. It's what you look up when being confronted with a headerless ROM file (see note 2).
  • "prgram","prgnvram","chrram","chrnvram","chrrom","miscrom" tags are only present if the respective memory type is present.
  • "sum16" is a simple sum of all bytes in a particular ROM type, truncated to 16 bits; this value can be found in Nintendo's leaked internal spreadsheet.
  • "console type", "console region", "expansion type", "vs hardware" and "vs ppu" are just the direct values defined for the NES 2.0 header; I did not want to use text strings that have to be looked up.
  • "mirroring" can be:
    • For normal mappers:
      • "H", meaning that byte 6's LSB nibble is set to 0..0
      • "V", meaning that byte 6's LSB nibble is set to 0..1
      • "4", meaning that byte 6's LSB nibble is set to 1..0
    • For mapper 030's idiosyncratic use of the NES header's mirroring bits:
      • "H", meaning that byte 6's LSB nibble is set to 0..0
        "V", meaning that byte 6's LSB nibble is set to 0..1
        "1", meaning that byte 6's LSB nibble is set to 1..0
        "4", meaning that byte 6's LSB nibble is set to 1..1
    • For mapper 218's idiosyncratic use of the NES header's mirroring bits:
      • "H", meaning that byte 6's LSB nibble is set to 0..0
        "V", meaning that byte 6's LSB nibble is set to 0..1
        "0", meaning that byte 6's LSB nibble is set to 1..0
        "1", meaning that byte 6's LSB nibble is set to 1..1
Notes:
  1. The database should be complete for all dumped licensed and original unlicensed games, and will be continuously updated to include more homebrew games/hacks/translations, pirate dumps and plug-and-play consoles, as well as corrections. As it is created automatically from a master set of headered ROM files, I cannot process pull requests on the XML text data; instead, simply point out errors or omissions by posting replies to this thread.
  2. When comparing file hashes against the database's full "rom" hash, be sure to compute two hashes:
    • one that trusts and uses an existing header's ROM size fields to know how many bytes to hash,
    • one that ignores any existing header completely and just hashes the entire file sans header.
    Searching for the first hash value will fail if an otherwise complete ROM has a header with incorrect size fields. Searching for the second hash value will fail if the file has common trailing garbage at the end, such as "This file downloaded from website x". Searching for both hashes will maximize the chances of identifying ROM files with bad headers or trailing garbage bytes. ROM files that have simultaneously trailing garbage and incorrect sizes specified in the header, or no header at all and trailing garbage, cannot be identified without human intervention. ;)
Attachments
nes20db_2020-06-28.7z
(546.59 KiB) Downloaded 102 times
Last edited by NewRisingSun on Mon Jun 08, 2020 12:54 am, edited 4 times in total.

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

Re: NES 2.0 XML Database

Post by NewRisingSun » Sun Apr 19, 2020 1:33 pm

Update 2020-04-19 based on feedback. :)

User avatar
Dwedit
Posts: 4306
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: NES 2.0 XML Database

Post by Dwedit » Sun Apr 19, 2020 1:47 pm

What does Expansion type= mean?
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!

User avatar
kitrinx
Posts: 2
Joined: Mon Feb 04, 2019 10:28 pm

Re: NES 2.0 XML Database

Post by kitrinx » Sun Apr 19, 2020 1:49 pm

Thank you for this NRS. This will make it so much easier to batch repair collections of ROMs with janky headers.

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

Re: NES 2.0 XML Database

Post by NewRisingSun » Sun Apr 19, 2020 1:51 pm

Dwedit wrote:
Sun Apr 19, 2020 1:47 pm
What does Expansion type= mean?
The value of this field.

Great Hierophant
Posts: 766
Joined: Tue Nov 23, 2004 9:35 pm

Re: NES 2.0 XML Database

Post by Great Hierophant » Mon Apr 20, 2020 8:57 am

I now defer to this database, FWIW.

talbin
Posts: 4
Joined: Sat Apr 12, 2014 9:24 am

Re: NES 2.0 XML Database

Post by talbin » Mon Apr 20, 2020 9:58 am

What is the best way to apply this to a set? Is there also something like this for standard ines headers?

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

Re: NES 2.0 XML Database

Post by NewRisingSun » Tue Apr 21, 2020 1:36 pm

A Python script has been written (not by me) for that purpose. As NES 2.0 is designed to be backwards-compatible to standard iNES headers, there is no separate database for that purpose.

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

Re: NES 2.0 XML Database

Post by NewRisingSun » Tue Apr 21, 2020 3:04 pm

The "name" tag has been removed, and my original filenames only included as XML comments.

User avatar
rainwarrior
Posts: 7812
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES 2.0 XML Database

Post by rainwarrior » Tue Apr 21, 2020 7:33 pm

Thank you for doing this good work.

Ice Man
Posts: 530
Joined: Fri Jul 04, 2014 2:34 pm

Re: NES 2.0 XML Database

Post by Ice Man » Wed Apr 22, 2020 2:12 am

Awesome. Thanks alot!

KickMeElmo
Posts: 6
Joined: Fri Apr 24, 2020 2:24 am

Re: NES 2.0 XML Database

Post by KickMeElmo » Fri Apr 24, 2020 2:33 am

For your own information, I applied the 2020-04-22 xml to a 2020-04-22 no-intro set using the python script, and I've recorded the names of the unmatched entries in a text file. In addition to those, "Bible Adventures (USA) (v1.5) (Unl)" wasn't in the xml. Hope this helps.

https://telegra.ph/nes20db-misses-2020-04-22-04-24

Great Hierophant
Posts: 766
Joined: Tue Nov 23, 2004 9:35 pm

Re: NES 2.0 XML Database

Post by Great Hierophant » Fri Apr 24, 2020 7:52 am

KickMeElmo wrote:
Fri Apr 24, 2020 2:33 am
For your own information, I applied the 2020-04-22 xml to a 2020-04-22 no-intro set using the python script, and I've recorded the names of the unmatched entries in a text file. In addition to those, "Bible Adventures (USA) (v1.5) (Unl)" wasn't in the xml. Hope this helps.

https://telegra.ph/nes20db-misses-2020-04-22-04-24
For the (Asia) ROMs not in the database, the attached list should cover their header information for the meantime. For the Virtual Console and the Collection ROMs, I'm sure they can be figured out on their own.

Edit : Eliminated unused values for Poke Block.
Attachments
list.txt
(2.55 KiB) Downloaded 48 times

NewRisingSun
Posts: 1187
Joined: Thu May 19, 2005 11:30 am

Re: NES 2.0 XML Database

Post by NewRisingSun » Sun Apr 26, 2020 12:28 pm

Added many homebrew ROMs as well as some ROM hacks, and corrected the mirroring information on the Oeka Kids games. (NesCartDB incorrectly claims that mirroring is mapper-controlled, but it's not.)

Amended the description of the "mirroring" field to include the meanings in the two mappers 30 and 218 that use the NES header's mirroring bits idiosyncratically. Added a note on the pitfalls of hashing ROM files found "in the wild" with trailing garbage.

talbin
Posts: 4
Joined: Sat Apr 12, 2014 9:24 am

Re: NES 2.0 XML Database

Post by talbin » Sun Apr 26, 2020 2:51 pm

Thanks for this great work! These Vs. came up as missing for me. Does it also make sense to add FDS?

ROM not found in database. SHA1: 611322F482CA236C42C7325D540DF7C906C1C330 File: Vs. Excitebike (set EB4-3 X).nes
ROM not found in database. SHA1: EC1BEA24B1F7E5A86006760EDF12B00A9922B3F5 File: Vs. Soccer (set SC4-3 X).nes

Post Reply