Planned update to UNIF

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
proxy
Posts: 68
Joined: Tue Mar 08, 2011 9:45 am
Contact:

Planned update to UNIF

Post by proxy »

Hey guys.

A while ago, Tennessee asked me to be the maintainer of UNIF so I am happily hosting the standard at both my personal site (http://codef00.com/unif_cur.txt) and at the libunif google code project (https://code.google.com/p/libunif/downloads/list).

Over time, it has become apparent that UNIF has some shortcomings (http://wiki.nesdev.com/w/index.php/UNIF#Shortcomings). While I know that UNIF isn't quite as widely adopted as some might have hoped, I do think it is worth updating the standard to at least address some of the issues it has in case people want to support it.

So here's what I've got in mind so far:

* NULL terminated text fields will be UTF-8, not ASCII. This is 100% backwards compatible, and allows the original name to be placed in the NAME block. It also, allows PCB's with non-US characters in them to be used correctly in the MAPR field. Almost all of the time, whatever is reading the image doesn't need to be UTF-8 aware, it can just do an ordinary memcmp on the byte string. Of course, it should be displayed as unicode so the user sees the real name :-).

* There is a single UNIF image in the GoodNES DB which has the "WRTR" block (Korean Igo (Unl) [!].unf). This was created by uCON64. I am undecided about whether or not to make it official. My gut says no since the spec already says "skip non-understood blocks". I don't like images having proprietary blocks, but such is life.

* VROR will be more specific. I think the intended interpretation was "if VROR is present, ignore any CHR-ROM blocks and instead provide CHR-RAM".

* VROR's byte value will specify how much CHR-RAM to provide. I am thinking that amount of CHR-RAM will be

Code: Select all

(2 ^ value)
. With the special case of 0 meaning "8K" since most (all?) UNIF images out there have a 0 in this block.

* I want to add a new SYST, block meaning "SYSTEM". This would be 1 byte big and allow specifying "NES", "Playchoice 10", "Vs. System", etc.

* I think it worth have a "LANG" block which would contain an IETF language tag, such as "en-US" and so forth.

* A cool feature I am toying with (I am unsure if it would be worth while), would be to have a new "XLAT" block. This block could be specified 0 or more times. And would allow the image to specify that it is the same as another game, but differs in language. For example the UNIF image for castlevania 3 might have an "XLAT" block which has something like this.

Code: Select all

struct XLAT {
    char lang[10] = "ja"
    char name[] = "Akumajou Densetsu"
}
or something to that effect. Would be interesting to better deal with multi-lingual collections, but may not provide much actual value in practice (would the emulator search your entire rom collection for the translation? that would be a big time waster). So I could go either way with this one.

* I am open to suggestions on how to better address the CTRL and BATR blocks.

* I am also (hesitantly) considering an "INES" block, which would 16 bytes and contain the "known good' iNES 2.0 header. Even though UNIF was designed as a "superior" replacement to INES, it is clear the INES is here to stay, so the standard may as well embrace that offer a means to store that type of information. Though I am very hesitant to include this type of block, since it would perhaps encourage (lazy) people to ignore a lot of the other blocks :-/

* Finally, it seems that there should be some blocks to specify PRG-RAM, though I haven't put much thought into the details of this.

Thoughts?
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Re: Planned update to UNIF

Post by 3gengames »

If iNES is here to stay and is the standard, why do we want UNIF in any way shape or form? We should want to get rid of all these unused formats. I mean, I haven't ever even seen a UNIF file, so what does this standard even matter for? Nobody uses it now, and if nobody will use it in the future, why bother wasting time on it?

But still, IMO, any human readable format is just a parsing nightmare with NO reason. To relate mappers between the two would also be difficult and just a pain. It's even a pain to have these formats still trying to be continued to be used with one another.

I like a header'd standard. Which is 110% adequate...provided we give it enough resources from the get go. -rolls eyes at iNES 1.0, heh.-
User avatar
Quietust
Posts: 1918
Joined: Sun Sep 19, 2004 10:59 pm
Contact:

Re: Planned update to UNIF

Post by Quietust »

One of the minor things that bothered me was the fact that there are multiple PRG/CHR blocks but no way to indicate the address space covered by them, but that's just because my emulator's PRG/CHR switching logic doesn't handle chip numbers (I concatenate them when loading, then my mappers adjust bank numbers on a per-game basis to insert open bus between chips) and it's not worth it for me to fix it (I support 4 different file types [NES, UNIF, FDS, NSF], and only one of them has a notion of chip numbers, so I'd have to update all of the non-UNIF code to specify "chip 0" and make all of the shared code work either way).

Oh, and the fact that the MIRR block's value "3. Vcc (one-screen B)" exists at all (if it's hardwired 1-screen mirroring, it doesn't matter which bank it is)...

I think that VROR was meant to indicate "this game does not have any CHRn blocks, so forcibly use CHR RAM instead even though the board wasn't designed to do it", but that isn't really useful because it's redundant (you can just count how many CHRn blocks you find, then go into "VROR mode" if that count turns out to be zero); changing it to specify the size of CHR RAM would certainly give it some meaning, but I don't know how many games even use that block to begin with.

Had it been done correctly, it could have served its purpose and made NES 2.0 unnecessary, but the fact stands that it was poorly designed and never really caught on; the fact that it never caught on also means that very few emulators even support it in the first place (and some of those probably aren't being maintained anymore, so the number of emulators to support updated UNIF would be even fewer).
Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another.
proxy
Posts: 68
Joined: Tue Mar 08, 2011 9:45 am
Contact:

Re: Planned update to UNIF

Post by proxy »

I do agree in concept that INES is the standard, and you make some good points. I do appreciate your input. It may not be worth while updating UNIF.

As for why spend time on it. Well, firstly, I have seen UNIF images in the wild. Heck there's 153 of them in the GoodNES DB alone, probably others out there. Secondly, the standard does exist and it may as well have the minor shortcomings addressed. So if someone chooses to use it, it is adequate.

Also, as far as it being a "parsing nightmare". My UNIF parsing code is somewhat simpler than my INES parsing code. At the very least, it's easier to understand. UNIF isn't "human readable", it does have some text in it. But those parts aren't so much "parsed" as they are compared or displayed.

Finally, while in concept a header'd format is enough. INES is a mess :-/. The various bit fields are annoying as hell to work with. Tons of ROMs in the wild STILL have "DiskDude" in them. And honestly, shoving all that information in 16-bytes while impressive is unfortunately a pain in the ass.

While you mention it, one thing I'm not sure I ever really understood is: why is iNES 2.0 determined to be backwards compatible with iNES 1.0? This strikes me an an unnecessary restriction. I mean, if a file needs the 2.0 information, it likely is not going to work correctly in an older emulator anyway. So why not take the opportunity to improve things? Why not do something like "If both bytes 4 and 5 (PRG-ROM and CHR-ROm sizes) are 0's then the rest of the file follows INES 2.0 format".

In reality, a lot of the better emulators are ignoring INES too. They just have a DB of game hashes mapped with what they believe to be correct meta-data. So in that case, why even update INES at all?

I dunno, I don't see a huge benefit in making ROM files which won't play correctly in older emulators, but are "loadable" by them.
lidnariq
Posts: 11429
Joined: Sun Apr 13, 2008 11:12 am

Re: Planned update to UNIF

Post by lidnariq »

UNIF files can be thought of as a particularly braindead version of CPIO or TAR; they only encode file size, contents, and restrict filename length to 1 to 4 bytes. (I've even written a program to treat them as such).

I do think defining the use of UTF-8 is a good choice, but otherwise it seems best to not bother having the "how I encode NES ROM" argument again.

3gg: Nantettatte!! Baseball's expansions don't fit well in iNES / NES2.0, and there are only awful ways to make them fit. Naruko suggested repeating the 16KiB expansion ROM 8 times at the beginning of the image, and allocating a submapper; or maybe instead we could put the expansion ROM at the beginning or end and storing the game size as 9×16KiB. Games smaller than 16KiB don't fit nicely, this either leaves a lot of padding or mirroring (Magic Floor uses 4KiB out of 16KiB, test roms are often even worse): OTOH you can't really buy new parallel memories smaller than 8KiB.

Many of the rest of UNIF's shortcomings are shared with iNES (mapper numbers and mapper strings both encourage false combining and then have unintuitive separations when it's later discovered necessary) or are things iNES doesn't even try to do (CTRL). Otherwise, I think the only real things worth trying to fix are borrowing NES2.0's PRG/CHR RAM/NVRAM specifying.

Definitely avoid the 'INES' block, I agree that's dumb.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Planned update to UNIF

Post by tepples »

Quietust wrote:Oh, and the fact that the MIRR block's value "3. Vcc (one-screen B)" exists at all (if it's hardwired 1-screen mirroring, it doesn't matter which bank it is)...
That or repurpose the value for "1-screen at CIRAM $400" for the Magic Floor mapper that uses CIRAM as CHR RAM.
Quietust wrote:I think that VROR was meant to indicate "this game does not have any CHRn blocks, so forcibly use CHR RAM instead even though the board wasn't designed to do it", but that isn't really useful because it's redundant (you can just count how many CHRn blocks you find, then go into "VROR mode" if that count turns out to be zero); changing it to specify the size of CHR RAM would certainly give it some meaning, but I don't know how many games even use that block to begin with.
How would you express Chinese TQROM-like boards, which have both CHR ROM and CHR RAM? Do these Chinese boards even have names?
proxy wrote:In reality, a lot of the better emulators are ignoring INES too. They just have a DB of game hashes mapped with what they believe to be correct meta-data. So in that case, why even update INES at all?
When a homebrew developer releases a new ROM, how does he get the metadata out to emulator developers for inclusion in the next version of the emulator? And how do emulator users install the new version of the emulator? The operating system publisher's update mechanism may either ban emulators entirely (e.g. Fedora) or only take snapshots every two years (e.g. Ubuntu).
lidnariq wrote:UNIF files can be thought of as a particularly braindead version of CPIO or TAR; they only encode file size, contents, and restrict filename length to 1 to 4 bytes. (I've even written a program to treat them as such).
Is PNG likewise braindead?
lidnariq wrote:Nantettatte!! Baseball's expansions don't fit well in iNES / NES2.0, and there are only awful ways to make them fit.
FDS images don't fit inside the Famicom Disk System BIOS ROM. Nor should Double Cassette System images fit inside the Nantettatte Baseball ROM.
lidnariq wrote:Magic Floor uses 4KiB out of 16KiB, test roms are often even worse
Hence why Zap Ruder includes several tests in one ROM, and why Action 53 allows multiple activities in one ROM. Even Nintendo had the same problem back in the day: Clay Shooting and Duck Hunt were sold in one ROM.
WedNESday
Posts: 1284
Joined: Thu Sep 15, 2005 9:23 am
Location: Berlin, Germany
Contact:

Re: Planned update to UNIF

Post by WedNESday »

3gengames wrote:If iNES is here to stay and is the standard, why do we want UNIF in any way shape or form? We should want to get rid of all these unused formats. I mean, I haven't ever even seen a UNIF file, so what does this standard even matter for? Nobody uses it now, and if nobody will use it in the future, why bother wasting time on it?
This.

No offense, but iNES is the one and only header for NES ROMs and to be honest some emus like mine don't even bother with it anymore. Its frankly too late now to try to introduce any other header.
User avatar
Dwedit
Posts: 4921
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: Planned update to UNIF

Post by Dwedit »

I've seen UNIF format roms renamed to .nes before. FCEUX plays them fine.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!
lidnariq
Posts: 11429
Joined: Sun Apr 13, 2008 11:12 am

Re: Planned update to UNIF

Post by lidnariq »

tepples wrote:Is PNG likewise braindead?
Touché.
tepples wrote:
lidnariq wrote:Nantettatte!! Baseball's expansions don't fit well in iNES / NES2.0, and there are only awful ways to make them fit.
FDS images don't fit inside the Famicom Disk System BIOS ROM. Nor should Double Cassette System images fit inside the Nantettatte Baseball ROM.
I'm unclear what your point is? Either emulators would need a special UI for loading only this game's DCS expansions, which I suppose might be bare binaries, since they seem to contain nothing but data, or the expansion data needs to be bundled into the distributed image: either goofily (iNES) or less so (UNIF).
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Planned update to UNIF

Post by tepples »

lidnariq wrote:
tepples wrote:FDS images don't fit inside the Famicom Disk System BIOS ROM. Nor should Double Cassette System images fit inside the Nantettatte Baseball ROM.
I'm unclear what your point is? Either emulators would need a special UI for loading only this game's DCS expansions
A "special UI" for loading a second cartridge wouldn't be just for DCS expansions to one game, especially if your emulator is multi-system. The FDS system card has expansions, namely game disks. So do Wide Boy and Super Game Boy. The Nintendo DS and DS Lite have two slots, one for the DS game and one for a GBA cartridge that some DS games can use. Some games use the GBA slot to migrate your old save files, some to unlock hidden content, and a couple as a separate controller (Guitar Hero On Tour).
User avatar
rainwarrior
Posts: 8731
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Planned update to UNIF

Post by rainwarrior »

If the expansions are just add-on ROMs they could easily be stored in fixed-size headerless files (since the content specs are already known), or even in little iNES files, I suppose. You could add something to an existing format, but like zzo38's Quick Disk proposal, it does seem this particular problem is more about emulator support than the format. The game should be assigned a unique mapper number within iNES because of this capability; it's just a matter of making add-on data part of what that mapper does in your emulator.
Post Reply