New File Format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

User avatar
blargg
Posts: 3715
Joined: Mon Sep 27, 2004 8:33 am
Location: Central Texas, USA
Contact:

Post by blargg »

BootGod wrote:I understand what you mean, but I disagree, I think it's totally on-topic. Otherwise you have one group going in the fix INES direction and another going in the UNIF direction. Which is counter-productive IMO.
This thread is for people who want to take the iNES-style direction, as the original post outlined. I think it's productive to let people go the direction they want, and having people try both approaches is better than attempting to steer one against its will. What you end up with is incoherence, since the two approaches are fundamentally different. "Format wars" can suck, but not as much as forced monoculture.
Someone once mentioned a hybrid format, where you have your normal iNES file, and then just slap UNIF chunks to the end of the file. That seems reasonable to me.
I rather like this idea too. Sounds like something to discuss in a UNIF revival thread. :)
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

I do have another suggestion and that is, each of our emulators has a database (the same database). Two bytes are used to identify a ROM in the database (LSB first).

4E 45 53 12 05

Example;

ROM # 1298 (0512h)
Title: Super Mario Bros.
PRG: 2
CHR: 1
etc.

That way we can make all kinds of additions and removals etc. Personally, I am quite excited by this idea.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

A numbered database. That's what Pocket Heaven uses to identify ROM releases that have a release group's intro animation appended to them. It also makes it difficult for an emulator to support new releases, such as newly dumped prototypes or homebrew.
User avatar
kyuusaku
Posts: 1665
Joined: Mon Sep 27, 2004 2:13 pm

Post by kyuusaku »

tepples wrote:The 1A is there for a reason, the same reason that PNG uses it: to stop MS-DOS TYPE from continuing to read the file.
Who uses type? :) Are there any other escape characters that could be used to do essentially the same as EOF?
Because the SRAM can be cleanly erased without affecting the ROM.
But why shouldn't they be packaged together? If the SRAM is chunked or in a predictable place, everyone capable of using a hex editor can clean or remove it. Having it packaged would also allow people to use SRAM between emulators without having to reconfigure the SRAM directories unless SRAM is defaulted to the ROM directory which makes clutter. It would also allow people to distribute extra code in SRAM (sorta like the GAR.) If people really wanted to store ROMs as read only archive files, they could have the emulator redirect SRAM to a file.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

kyuusaku wrote:
tepples wrote:The 1A is there for a reason, the same reason that PNG uses it: to stop MS-DOS TYPE from continuing to read the file.
Who uses type? :)
I don't know, but a lot of C standard libraries under DOS and Windows stop before the first ^Z when reading files where fopen() specified text mode. Given that some FTP programs behave the same way, it's also useful to make sure the file was transferred in binary mode.
Are there any other escape characters that could be used to do essentially the same as EOF?
In CP/M, ^Z is end of file. MS-DOS is Microsoft's fork of QDOS, a clone of CP/M. I know of no other character that MS-DOS file readers use as end-of-file.
Because the SRAM can be cleanly erased without affecting the ROM.
But why shouldn't they be packaged together? If the SRAM is chunked or in a predictable place, everyone capable of using a hex editor can clean or remove it.
But buggy implementations may overwrite the ROM. In general, applications are not as well tested as an operating system's file system code. Putting the responsibility to keep the data streams separate on the file system will result in less chance of data loss.
Having it packaged would also allow people to use SRAM between emulators without having to reconfigure the SRAM directories unless SRAM is defaulted to the ROM directory which makes clutter.
Storing SRAM in the ROM file also clutters up the Last Modified dates. And would you also want to store the save states for a game in the ROM file itself? This would make it more difficult to trade save states, as a lot of online communities encourage trading of SRAM files, save states, and movies but not the ROM files themselves due to copyright.
It would also allow people to distribute extra code in SRAM (sorta like the GAR.) If people really wanted to store ROMs as read only archive files, they could have the emulator redirect SRAM to a file.
Which reintroduces having to reconfigure the SRAM directories.
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

I've decided to put the header at the start of the file after all. I think that using "NES" as an identifier is the best solution. Including data like ROM name and information like cover art and such is such a waste of time. I am not trying to implement a ROM database here, just a header.

As for the SRAM being in the ROM? Well, IMO the ROMs should be a digital copy of the cartridge itself, and having the SRAM inside of the file is a good idea. Someone said that people running ROMs off of CDs couldn't save their SRAM data. Well, you still couldn't save your SRAM data anyway, even if it wasn't in the file. It would have to be directed to the hard drive which would defeat the whole point of having the ROM on a CD.

So something like;

Code: Select all

4E 45 53 02 01 00 01 00
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  +- (to make it 8 bytes so debuggers look ok)
|  |  |  |  |  |  +- Mirroring etc.
|  |  |  |  |  +- Mapper/Board
|  |  |  | +- CHR Banks
|  |  | +- PRG Banks
|  |  +- S
|  +- E
+- N
What more could we possibly want?

BTW What's the biggest ROM possible? Is it 512k PRG and 512k CHR and 1024k of each?
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Post by Bregalad »

Now, several points have to be made up :
- UNIF is totally useless and nobody will ever use it aside of a few people that seems very involved to it, but this remain very few people
- iNES is incomplete and should be updated to have less problem with the support of some technically complex ROMs that causes problems with the current format
- iNES is a standard format and cannot be altered like that
- Most emulators support only iNES and more accurate emus now use crazy checksum techniques to bypass techincal problems by the few ROMs that causes problems with the current format.
- iNES have still several reserved bytes for the purpose of any uploads

What should be done in my opinion :

- Create a programm that have a large programm in and that can easily convert all old iNES ROMs to new iNES format without killing them, and have an easy to use interface (just like Nestopia's internal database)

- Create a system that can detect old and new iNES ROM format

- Create a system to definitely detect region and video standard

- Create a system to found out SRAM sizes.

- Don't alter the current stucture of an iNES ROM; aka it stills remain 16 byte header, PRG data and CHR Data. Battery-backed data is in a sparate 8kb .sav file, that all emulators can read

- Implement the new format in emulators and force the user to convert its roms using either a link to the programm in point 1, or have a built-in converter inside the emulator.
Useless, lumbering half-wits don't scare us.
mattmatteh
Posts: 345
Joined: Fri Jul 29, 2005 3:40 pm
Location: near chicago

Post by mattmatteh »

instead of using program banks, you could use an unsigned int 32, for teh total program size. i dont see how the mapper/board could be only 1 byte, then you are limited to 256 and will have the same problem as ines.

matt
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Post by Bregalad »

Why ? The current banking count works just fine.
The current mapper numbering woks just fine as well, as far as I know. There is no problem with the 256 limit (btw this isn't one byte, but two separate nybbles). More bits could be added anytime anyway.
However, the only problem with iNES is all about SRAM (open bus or more than 8kb), and with territories.
Please stop this flow of crazy ideas, and just apply the actual fixes, and idea to make them popular ! Flowing crazy ideas will help noone !
Useless, lumbering half-wits don't scare us.
Roth
Posts: 401
Joined: Wed Aug 03, 2005 3:15 pm
Contact:

Post by Roth »

Bregalad wrote:The current mapper numbering woks just fine as well, as far as I know.
I was thinking that at first also. But then I started to think about those who are making new mappers (ie Quietust, and maybe Brian Provincio?). If people make new mappers that will work on the NES, there'll be need for more room to identify in the header of the file system. I'm not sure if there'll be alot of new mappers made, but it's better safe than sorry.
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

Bregalad wrote:Flowing crazy ideas will help noone !
No true. We are all brainstorming, and some excellent points have been raised.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

I agree that crazy ideas won't help here. As I see it, we do not need a "revolution". We need to expand what we have to overcome some difficulties.

Creating a new, ideal format sounds just great, but it's just too difficult to do by now. It has been tried before, and always failed.

Updating iNES sounds much more feasible, and could possibly solve all compatibility issues. It's not like we'd be forcing anyone to update their ROM files, but if they want to get those weird games to work, they may have to update the headers and grab newer emulators.

Come on, guys, a revolution is not the way to go. There is no way everyone would agree on something new, and even if they did, implementing it (having everything comply to the new format) would be a bitch. Just improve iNES already, and people who are not concerned about this will not even notice something changed.
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

tokumaru wrote:It's not like we'd be forcing anyone to update their ROM files, but if they want to get those weird games to work, they may have to update the headers and grab newer emulators.
Well, if they are going to update it then they should be prepared to convert it.

Let's say that iNES has been around for 10 years. Whatever we introduce now, could last from now to eternity, so it is never too late. Staying compatbile with the iNES format is something I don't think is possible. The first six bytes are ok, but then you have this higher nibble lower nibble rubbish followed by eight bytes or so of total and utter crap, namely someones name which would confuse an emulator if we implemented the unused bytes but bad ROMs still had bad headers. What's wrong with my previous post which has a diagram of the new format. Not only is it very similar to iNES, it is also very easy to implement into ROMs. You don't need a convert utility, only a HEX editor, and you can do the whole thing by hand.
mattmatteh
Posts: 345
Joined: Fri Jul 29, 2005 3:40 pm
Location: near chicago

Post by mattmatteh »

how often do you need to change a header on a game ?
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Post by WedNESday »

mattmatteh wrote:how often do you need to change a header on a game ?
Well if we were to implement a new format, then only once.
Post Reply