New File Format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

New File Format

Post by WedNESday »

I think that it's time someone proposed a new file format. I have been rewriting WedNESday recently and I have found the iNES format to be old fashioned and open to all kinds of problems (like people putting their names in the header).

Here is a basic makeup of the file.

PRG-ROM Banks
CHR-ROM Banks
SRAM Bank
Header

Here is the header itself.

00h - N
01h - E
02h - S
03h - PRG-ROM Banks
04h - CHR-ROM Banks
05h - Cartridge Board
06h - Mirroring etc.

Here is an example;

Super Mario Bros.

0000-8000h 2 PRG-ROM Banks
8000-A000h 1 CHR-ROM Bank
N/A
Header (see below)

4E 45 43 02 01 00 01

Please let me know your thoughts, opinions etc. I'll edit this post when I make editions to the format. Now I know that I am not too educated about the ins and outs of certain aspects of NES emulation (esp. cartridge boards), but this is something that I want everyone to contribute to. I am also interested in throwing in some checksums. I'm also very interested in discussing what could go into byte 06h along with the mirroring, trainer, VRAM, SRAM etc.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

I don't know man. People have been talking about UNIF for a while and that never really caught, so I don't see why your simplistic new header would either. I have more faith in simplistic headers than UNIF, though.

I don't think checksums are the way to go. Emulators should be able to emulate a game without knowing wich one it is, 'cause it might well not be a commercial game.

A header should be used to identify specific hardware details that you don't have from just looking at the ROM. I think that by now everyone has a pretty good idea of all the different combinations possible for these details, and a new format without the limitations of INES, and not as complicated as UNIF could be made.

Well, I'm not an emulator programmer, I'm a game programmer. So maybe I shouldn't say anything about this.
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Post by Bregalad »

Something retro-compatible with iNES would be fine, but hard to put in correctly. Again, I'm not an emulator programmer, but a game programmer.
It would be nice to have all emulators emulating things like MMC5 SRAM size correctly, because currently, EACH emulator will come with something different.
Useless, lumbering half-wits don't scare us.
User avatar
Memblers
Site Admin
Posts: 4044
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Post by Memblers »

It's looking a lot like iNES already. Which isn't too bad, because I like simplicity. If I can make/edit the header for a ROM easily in a hex editor, I'm happy. But I'm pretty sure that one byte isn't enough to define a board type. Actually I like how UNIF spells out the board name.

I don't think a new format will catch on at all. I really think UNIF could, if there was some big archive up for download that included the box/cart hi-res artwork, manuals and stuff inside the ROMs. Most users won't care about the details, they'll just care if an emulator runs their favorite ROM or not.

What I've kinda wondered about before, is why not a footer? Is there already something defined that can be at the end of an iNES file? I'd imagine emulators just ignore whatever is after the ROM data. If it's not too hacky, one could put anything there I guess.
User avatar
Disch
Posts: 1848
Joined: Wed Nov 10, 2004 6:47 pm

Post by Disch »

I suggested something like this a little while back:

http://nesdev.com/bbs/viewtopic.php?t=1809

Only I find it's much more feasable to update iNES to make a new version that's backwards compatible, rather than simply making a new format from scratch and it having zero support in 99% of emulators.

Ultimately the plan was more or less shot down. I would still be in favor of such an approach... but really... so far your new format seems to be a re-arranged iNES that's just harder to load.

I would say:

1) keep it backwards compatible as much as possible.

2) have the header at the head (start) of the ROM. Putting it at the end is criminal and makes for MUCH harder file loading, especially in an emu which supports multiple file types and/or zipped ROM loading. An additional footer after the PRG/CHR/whatever else is one thing... but the format should at least give the program a heads up as to what file type this is and the PRG/CHR/whatever else sizes before anything else.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

OK, lets say everyone decides to update iNES in a backwards compatible way. The unused bytes in iNES would be used to further describe the game that follows, right?

There were some ROM's out there with garbage on the unused bytes of iNES, weren't there? These would probably break the game in emulators that support the newer format, right? How could this be prevented? How to verify the the information contained in there is valid (if not it won't be used)?

BTW, I'm all for the updating of iNES. I don't know what I could do to help, though.
User avatar
Disch
Posts: 1848
Joined: Wed Nov 10, 2004 6:47 pm

Post by Disch »

tokumaru wrote:There were some ROM's out there with garbage on the unused bytes of iNES, weren't there? These would probably break the game in emulators that support the newer format, right?
There could be ways to differentiate between standard iNES and the update by having a certain thing in the header. Perhaps having an short ID string in the last 3 or 4 bytes in the header. These will go ignored in most emulators which only support standard iNES, but emus which support both will be able to look at it to know what format to use.

Obviously 100% backwards compatibility isn't possible (if it were, a new format wouldn't be needed, right?) The idea is to make older emus support the updated format to a reasonable extent... but ultimately, an emu which does not actually support the update won't be able to run all the ROMs in that format.
mattmatteh
Posts: 345
Joined: Fri Jul 29, 2005 3:40 pm
Location: near chicago

Post by mattmatteh »

i dont think we should update ines. there are still alot of roms with strange stuff in the header as it is now, adding more would really make a mess. ines is flawed from the begining.

since unif was supposed to replace ines, i think we should find out why and fix that. i think quietust said there were a few games that didnt work with it? then unif needs a new chunk so that it does ? but something that doesnt have confliting data either.

i am thinking about removing ines support from my emulator; change them all to unif.

matt
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

mattmatteh wrote:i am thinking about removing ines support from my emulator;
But then most people won't use your emulator, since it won't play 99% of the ROM files out there. I don't know what your goal with your emulator is, though. Maybe you have a small target audience, that doesn't mind loosing iNES support, I don't know.
User avatar
kyuusaku
Posts: 1665
Joined: Mon Sep 27, 2004 2:13 pm

Post by kyuusaku »

How about:

Signature (4 bytes) "NES",0x1B?

Game type in release order (4 bytes) - first/second bytes are strict logical mapper (which is more like board type, 512K MMC1 would be different mapper than normal 256K mapper), third/fourth is mirroring (V, H, mapper, none), CHR ROM/RAM, WRAM is V/NV, WRAM size, bus conflicts, extra misc ROM presence after CHR ROM such as expansion audio ROM or NT-ROM.

PRG data size (2 bytes in 8k blocks which is the smallest Nintendo maskrom)
CHR data size (2 bytes in 8k blocks, CHR RAM would be defined as 1 block)

Optional dumper (person, not hardware) signature (4 bytes) --not to be indexed to a username!

followed by SRAM, PRG, CHR, other data specified in game type.

Note: NO controller ID or ambiguous boardname. I thought the high byte of the mapper could be for unstandardized mapper numbers. By incorporating SRAM in the "ROM", it'll reduce file clutter and unofficially allow trainers to be incorporated into SRAM loading as well as make ROMs act more like their original cartridge) Emulators should then have an import/export SRAM option.
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Post by Bregalad »

I'm pro to upload iNES. There is 8 free bytes and several more bits in theory, if those bastards didn't input "DiskDude" in it on several roms.
I think emus should either detect the "DiskDude" and interpret it as a old iNES rom, or simply do the smart trick that kyu said above, replace the mark with $1b so thing would be clear about wich version of iNES is done.
Here you are my opinion :

What could be added in iNES2 :


- SRAM size specification (including no SRAM), so all games that rely on open bus at $6000-$7fff will work fine, all games reliying on standard 8kb at the same place will work fine too, and MMC5 games doesn't have SRAM bankswitching problems any longer

- Fix all mirroring and SRAM wrong flags on some ROMs

- Add a byte that tells the "version/revision" of the ROM for commercial ROMs (or even homebrew), so that difference between FC, USA NES and PAL NES can be done, along with the different version (PRG0, PRG1, a1, a2, etc....) of each game

- Some system that auto-detect and fixes the horribly popular [hM02] versions of several japanese and fan-translated games (Final Fantasy II, DQ3 and others). The emulator could detect the chains of "bit $xxxx" (where xxxx is greather than $8000) and replace it with "sta $8xxx", and change the mapper to MMC1 and that would be done.

- I'm unsure if ANY mappers bit have to be added, as long as SRAM size is. SUROM can be any MMC1 pack with more than 256 kb of ROM for example. SOROM can be detected with SRAM size.

What shouldn't be added in iNES2 :


- Bus conflicts. No game rely on them, this is just a homebrew things. Emulators definitely can or cannot do something with them, but emulating no bus conflicts will always work. Some authors could just have the pleasure to have the [hM02] hacks not work, but actually just read above.

- Dumper hardware or person, nor footers. I think the goal of a ROM is to contain what the cartridge contains, with a few more information bytes. No person or hardware should be involved, and footers would be a bad idea as there is largly enough free space in 8 empty bytes.
Mabybe company name would be okay, but that wouldn't be of much use.

- Box or manual images. Because this is the main con by downloading a game instead of buying it (and that should remain the con I think, else what interest in buying a rare game ?) and you can just found pictures on the NES on sites like Moby Games.
Useless, lumbering half-wits don't scare us.
User avatar
Disch
Posts: 1848
Joined: Wed Nov 10, 2004 6:47 pm

Post by Disch »

kyuusaku wrote:Signature (4 bytes) "NES",0x1B?
Changing the file signature will kill backward compatibility.

If we want the ROM to load in older emus, it must look just like a regular iNES file.

I really think a secondary signature is the way to go here.
By incorporating SRAM in the "ROM", it'll reduce file clutter and unofficially allow trainers to be incorporated into SRAM loading
I don't really like the idea of "SRAM ROM" or trainers. RAM on the cartridge is just that -- RAM. Putting RAM in a ROM image doesn't really make a lot of sense. And ROM for iNES trainers aren't even on the cartridge and probably should be excluded all around.
User avatar
Memblers
Site Admin
Posts: 4044
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Post by Memblers »

Hm, speaking of trainers, since they apparantly have no legit use (cartridge-wise), couldn't that also be hijacked for the new header info? Just say there's a trainer on the ROM, and have the header (with signature) in there. Or is that hacky/annoying/incompatible?

Just throwing out ideas.
Bregalad wrote: - Add a byte that tells the "version/revision" of the ROM for commercial ROMs (or even homebrew), so that difference between FC, USA NES and PAL NES can be done, along with the different version (PRG0, PRG1, a1, a2, etc....) of each game
I like this idea.
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

Thanks for all of your replies, please do keep them coming. If you believe that checksums are a waste of time, then what is the point of providing two bytes that tell you how many banks there are? The only reason that I can think of is that you can compare them with the file size to see if there is any junk at the end of the file. I am not aware of how many boards there are so if there are more than 256 I would provide an additional byte. I don't think that the following is very wise to do, though;

00h AAROM
01h BBROM
02h CCROM
...

I am much in favour of providing text for board names instead. However, if some boards behave in a similar fashion, then they should have the same byte number. How many boards are there? And can anyone provide me with a list?

I am very interested in a checksum because it means that people can't hack ROMs for there own needs. GameBoy and SNES ROMs have them and most emulators ignore them, but I like the idea of having a clean ROM image.

Why would having the header at the end of the file be a pain in the *** for? It would make bankswitching and file access easier as you don't have to add 10h every time.

I think that having the SRAM in the ROM file is an excellent idea. Not only is it very realistic, EVERY emulator can access it. None of that save state incompatibility rubbish.

I think that the reason that UNIF never took off is that the whole concept of having data chunks and so forth was just a bad idea. Plus if there is no one around to champion the idea then yes it is going to die on its arse. In terms of old emulators not being able to read the ROM? Good. Old emulators have been surpassed by the new active ones and frankly no one uses them anymore. Either comply, or die!
User avatar
Disch
Posts: 1848
Joined: Wed Nov 10, 2004 6:47 pm

Post by Disch »

WedNESday wrote:what is the point of providing two bytes that tell you how many banks there are?
So you know how much PRG/CHR there is. How can you load a ROM unless you know how big it is? Seeking to EOF and backtracking is a luxury that isn't always all that easily performed -- ideally, the ROM should be loadable with a single parse, rather than the emu having to seek around and jump through hoops.
I am much in favour of providing text for board names instead.
I could go either way... though I prefer a nice simple single number. Text strings are typically more complicated... and having the board name isn't really any more descriptive than a properly designated mapper number.

Not to mention there aren't board names for some games -- or two boards with the same name might have different mappers (one of the real big problems with UNIF).
Why would having the header at the end of the file be a pain in the *** for?
Seeking around the file is a pain. If you know everything up front loading is easier since you can load the whole file in one pass.
It would make bankswitching and file access easier as you don't have to add 10h every time.
Or you could do things the "normal" way and just load PRG/CHR into buffers when you load the ROM, then close the file and never have to deal with it again once you load it. Don't have to add anything for the header -- both PRG and CHR start at offset 0 of their respective buffers.
I think that having the SRAM in the ROM file is an excellent idea. Not only is it very realistic
RAM usually doesn't exist in ROM.
EVERY emulator can access it.
Every emulator can read an 8k .sav file. Cross-emu SRAM support was never an issue... in fact I'm sure every NES emulator in existence uses the same 8k dump (it's not even really a file format... just a dump).
None of that save state incompatibility rubbish.
Save state have nothing to do with the file format. Nothing you can put in file specs will change how an emu emulates.
I think that the reason that UNIF never took off is that the whole concept of having data chunks and so forth was just a bad idea.
It's actually superior in many ways. It's much more flexible, and much easier to expand. Chunk/block formats are actually pretty common and once you're used to them they're easy to work with.

I opted for a block-style savestate format for my emu -- works great.
In terms of old emulators not being able to read the ROM? Good. Old emulators have been surpassed by the new active ones and frankly no one uses them anymore. Either comply, or die!
No one will want to use your format if they can't play the games in their favorite emulator. Joe Gamer doesn't care how accurate the file format is as long as it works when he runs it.

Killing old emulator support is pretty much a guarantee that this format will never get off the ground.
Post Reply