NSF specification for FDS memory

Discuss NSF files, FamiTracker, MML tools, or anything else related to NES music.

Moderator: Moderators

User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

NSF specification for FDS memory

Post by rainwarrior »

I think the FDS memory specification needs a bit more definition.


1. When using bankswitching, does writing to RAM ($6000-DFFF) replace values in the original NSF ROM image? i.e. if you write to an area, bankswitch it out, and bankswitch it back in, should the written values persist?

I think the answer to this one should be "yes" for a few reasons.

For rips of actual FDS games, a yes or no seems to be irrelevant, as bankswitching during playback isn't really expected. I'm not certain, but after a short review of available source code this seems to be the dominant behaviour among NSF players, and this does not appear to be causing FDS incompatibility.

For hardware players, I think this generally simplifies the implementation, since the "no" behaviour alternative is copying over the old (untouched) block every time it is bankswitched in. Instead you For software players it is also simpler for the same reasons, though they aren't as much of a problem (no need to store the original ROM copy, FDS bankswitching does not have to be a memcpy operation). I believe this simplicity is why it's the usual implementation in NSF players. What does the PowerPak do?

This behaviour does come with a warning: any writes to the image may or may not be persistent across songs, depending on whether the player will reload between songs. Some software players reload for every track, but I wouldn't expect a hardware player to. This likely won't affect any existing FDS NSFs, but it's something that should be clarified.


2. When using multi-expansion, there are potential write conflicts.

This has been discussed over at the wiki, and there is already a recommendation there, but I'll outline it here in case anyone has comment. ( http://wiki.nesdev.com/w/index.php/NSF#Multi-chip_tunes )

This one is a little bit trickier. VRC6/VRC7/5B writes overlap with FDS RAM areas, so there is a high potential that memory gets corrupted when using FDS with these expansions. Since some NSF players do not seem to prevent this conflict, it's necessary to warn against using FDS with these other expansions, but I think it's worth making a recommendation for implementers.

Since it is normal practice in NSF implementations to ignore the mirror aliases for expansion registers and expect only the address in the NSF spec, blocking RAM writes at all active expansion addresses seems to be the lowest impact solution. The other easy solution I think is to disable all writing to FDS RAM during multi-expansion, but I don't like this as much since it takes away a useful property of the FDS. (Particularly, someone can procedurally generate DPCM samples in the $C000-DFFF region with FDS.)

This one is not particularly important to address for hardware implementations, since multi-expansion hardware has plenty of its own practical design constraints, but I think it would be good to recommend a "standard" behaviour for software implementers. This is another one where I'm curious, what does the PowerPak do?

Finally, NSF makers who want greater compatibility among players can be carefuly to put dummy data areas at the conflicting expansion addresses.



Anyhow, share your thoughts.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

I have found four FDS rips that have a problem with mirrored writes due to bankswitching RAM, and after discovering these I've realized several players have different workarounds for this.

NEZPlug: memcpy instead of mirroring for FDS banks at $6000-7FFF.
FCEUX: memcpy instead of mirroring for all FDS banks.
NSFPlay: does not use bankswitch 6/7 to setup $6000-7FFF.

* Ai Senshi Nicol (FDS)(1987)(Konami).nsf
* Esper Dream (FDS)(1987)(Konami).nsf
* Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito).nsf
* Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack).nsf

All of these make writes to $6000-7FFF which are set by the NSF bankswitch values 6/7 to use the same bank as code elsewhere in memory. Ai Senshi Nicol, Esper Dream, and Seiken Psycho Calibur crash the PowerPak, which I presume uses a bankswitched RAM implementation, since memcpy is impractical. Kiki Kaikai seems to run fine on the PowerPak, strangely. I'm not sure how it survives.

I dislike the workarounds I've seen, especially since memcpy-style FDS bankswitching doesn't really seem feasible for hardware players like the PowerPak. FCEUX's approach is the most sensible, I think, in that it's consistent, and very much like what the FDS does, but I don't think this is the right approach. Why make a special case for the FDS? Bankswitched RAM works well for all the rest, and it could be made to work fine with all existing FDS games?

Instead of applying these workarounds, all of these can be easily "fixed" in the NSF by adding a 4/8k block of 0s and using those as the 6/7 banks. I think it's more appropriate to allow writes to mirror. Does anyone disagree?


I've quickly tested all the FDS NSFs in GilGalad's archive, and these ones were the only ones that seemed to have this problem. I guess most FDS NSFs don't write to $6000-7FFF, or if they do, don't mirror them anywhere except the unused $E000-FFFF range, so it isn't an issue most of the time? Was using bankswitch 6/7 as setup for $6000-7FFF a later addition to the NSF spec?
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NSF specification for FDS memory

Post by lidnariq »

This feels like it's treading dangerously close to the flame wars that the famitracker boards at least used to see regularly (about how permissible is multi-chip).

My personal intuition in the case of multi-expansion is: You should expect it to screw everything up. If you want FDS in your multi-expansion, it should be RAM, it should persist, all mirrors should be present i.e. identical to hardware. This means you should expect maximal badness, and the NSF's playback engine should be written defensively to that end. (This basically means no naive writes to (depending on expansion) $9000..$dfff; they always should affect all hardware that is nominally writable at that address)

This is basically your last suggestion.

I think this approximately agrees with your second post.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

I'm not really opposed to multi-expansion stuff, and I don't actually think it's a big pain to support in software (there are several fairly easy/simple solutions outlined on the NSF wiki page). If you're going to attempt it in hardware, well, it's fair to expect the person who attempts this to know how to recompile their own music code. I haven't actually heard of any hardware multi-expansion attempts use the FDS (excepting the PowerPak's emulated version). The use of FDS RAM is so rare in modern composed NSFs that even the simple solution of disallowing FDS RAM for multi-expansion would be rather effective. Anyhow, I'm not so concerned with there being an "official" way to deal with FDS multi-expansion.


The interaction of bankswitching and FDS, however, is giving me a big pain as I'm looking at it. It really bothers me that these NSF rips have this problem that every emulator seems to have its own workaround for instead of fixing the spec to disallow or alleviate the problem.

The following NSFs use the same bank number for multiple banks and then write to one of them (i.e. mirrored writes, if you're using bankswitched RAM). For most of them this doesn't seem to be causing problems, but the four I listed in my previous post do have a noticeable problem as a result. (I have excluded writes that mirror to a bank in $E000-FFFF, since that's presumed irrelevant to the NSF playback anyway.)

* 19 Neunzehn (FDS)(1988)(Soft Pro).nsf
* Ai Senshi Nicol (FDS)(1987)(Konami).nsf
* Big Challenge! Gun Fighter (FDS)(1989)(-)(Jaleco).nsf
* Cleopatra no Mahou (FDS)(1987)(Square)(DOG).nsf
* Esper Dream (FDS)(1987)(Konami).nsf
* Fire Bam (FDS)(1988)(HAL Laboratory, Live Planning)(HAL Laboratory).nsf
* Hao-kun no Fushigi na Tabi (FDS)(1987)(Carry Laboratory)(DOG).nsf
* Idol Hotline - Nakayama Miho no Tokimeki High School (FDS)(1987)(Nintendo R&D1)(Nintendo).nsf
* Kaettekita Mario Bros. (FDS)(1988)(Intelligent Systems, Nintendo R&D1)(Nintendo).nsf
* Kidou Keisatsu Patlabor - Dai 2 Shoutai Shutsudou Seyo! (FDS)(1989)(-)(Bandai).nsf
* Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito).nsf
* Maerchen Veil (FDS)(1987)(System Sacom)(Sunsoft)(Sunsoft).nsf
* Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack).nsf

A little later I'll probably write a simple tool to fix the problem ones, unless someone can make a compelling argument for one of the memcpy-bankswitch workarounds.

I think most of these are just the result of using 00 for all "unused" bank values even if they're being written to as RAM.

(edit: fixed the list, there were a few false-positives due to a bug in my detection code)
User avatar
Gil-Galad
Posts: 321
Joined: Sat Nov 13, 2004 9:43 pm
Location: Ohio, USA
Contact:

Re: NSF specification for FDS memory

Post by Gil-Galad »

1. When using bankswitching, does writing to RAM ($6000-DFFF) replace values in the original NSF ROM image? i.e. if you write to an area, bankswitch it out, and bankswitch it back in, should the written values persist?
I'd have to say no.

NSF is a ROM based format. Once you load a bank into a certain address range, it overwrites that address range. No values remain from the previous bank or written values in case of FDS RAM. This applies to FDS rips as well.

For FDS rips, when you rip a NSF from a FDS game there are many steps you have to take. You have files instead of banks. Those need to be converted and merged together as banks.

Once a bank is loaded, $6000 - $DFFF is then treated as RAM. Anything can be written there, more or less self-modifying code/data. During play, you don't have to worry about it much as the main code is taking care of reading/writing to RAM areas(sometimes it needs to be modified). As far as the initialization is concerned, a ripper needs to account for any writes to any address no matter if it's in the FDS RAM area or not. If it's not done this way, it may not work on the real hardware. Not to mention there will be incompatibility between other NSF players. Because every programmer implements NSF emulation a bit differently in some respects.

If I ever wanted to change anything, it's allowing $6000 as a load starting point for FDS rips. Some players don't support it, some do. I start all mine at $8000 for compatibility reasons. Then manually load those banks in the address range $6000 - $7FFF. And/or use the bankswitch values in the header as well.

Those NSFs that don't work right simply need to be fixed and fixed for compatibility between software and hardware players. Although, I always try to make sure a NSF works on the real hardware. I have a PowerPak to test my NSFs.

I do realize that the PowerPak also emulates mapper controllers and some sound expansion chips. Other than how the PowerPak initializes, everything else is NES.

Well, just one thing at a time. You wrote about lots of different things. I'll answer this one for now. You may or may not agree with me. I don't emulate, I'm a ripper.

One last thing. The NSF file should NOT be modified in any way outside of loaded memory. The FDS Disk I/O functions should be completely disabled. And hacked out of a NSF.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

Hmm, I didn't notice load addresses in the $6000 range were disallowed in the spec. This seems like an oversight to me, since without bankswitching turned on it seems some FDS games should require it. I know that NotSoFatso does not allow it, but are there any other players that do not? (NSFPlay, NEZPlay, FooGEP, Nestopia, FCEUX, VirtuaNES, VirtuaNSF are all fine with it.)


I understand that the FDS does not really have banks, it just has non-bankswitched RAM. The problem is that NSF as a format forces all memory configurations into 4k banks. Banks are normally just manipulated high address lines, which can create mirrors. I think it's more consistent to implement all NSF bankswitching this way (not to mention easier in hardware emulation), and not have a special case for FDS; it's entirely adequate for FDS rips, but they have to be made with this in mind.

The specific problem is this: when the bankswitch block for and FDS rip contains mirrors, the ripper must account for writes to mirrored addresses. If you don't care what goes in $6000-7FFF, don't mirror code/data in those areas. In the four problem NSFs I've identified, the bankswitch block looks like this:

01 02 00 00 00 00 00 00

The problem is the two 00 00 on the end. These rips tread $6000-7FFF as RAM, and do not read any code/data from this region. The ripper has chosen a "default" bank of 00 to put in this area instead of creating a new initialized bank. This kind of thing is outside the territory of just setting it up like the original FDS version and presuming the emulation is accurate; this is now a deliberate statement that the same data should be put in places it wouldn't normally be.

Now, areas that are empty or only used for RAM need to have -some- value in them, but what I'm arguing is it's not acceptable to just stick a randomly chosen bank in them (like 00) because it has a convenient number. If $6000-7FFF is not supposed to be filled with unique data from the disk image, and is just treated like RAM, it should be explicitly initialized. Write areas should not be mirrored.

i.e. the bank table should look like this instead:

01 02 00 00 00 00 03 04

I already outlined three different workarounds to this problem, and I think they're all inadequate. NSFPlay ignores the spec and doesn't use the last two bankswitch bytes to set up $6000-7FFF. NEZPlay treats bankswitching in $6000-7FFF as copy-to-RAM but allows mirroring in the rest. FCEUX treats all FDS bankswitching as copy-to-RAM (admirable, but I think unnecessary). All of these just hide the problem and prevent practical implementation on bankswitched RAM hardware.

So anyhow, what bothers me is that the behaviour of banks when FDS is enabled is underdefined by the spec, and we have 4 different implementations (none of which have much to do with authentic emulation of the FDS) trying to account for they grey area. The definition I'm suggesting is one that would not break playback for any of these player implementations, and is the only one that doesn't break playback on the PowerPak.


I guess there is another side of this is that I neglected to state, but it should be forbidden to take advantage of mirrored writes as well (i.e. write once, read it from two different locations), since they are impossible on the FDS, and will break some of the workarounds I mentioned.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

So, at the moment I've included this information as guidelines in the wiki:

http://wiki.nesdev.com/w/index.php/NSF# ... kswitching

As you can see, I've not made it a revision to the specification, just included it as advice for someone trying to make an FDS NSF that is compatible as possible. Maybe that's enough to deal with the underdefinition in the spec itself.

I'll fix those four NSFs and post them here in a little bit.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

Here are fixed versions of the 4 problem NSFs:

Ai Senshi Nicol (FDS)(1987)(Konami) a.nsf (attached)
Esper Dream (FDS)(1987)(Konami) a.nsf (attached)
Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito) b.nsf (attached)
Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack) b.nsf (link to attachment below)

All of these are now free of mirrored write problems. Seiken Psycho Calibur still will not run on a PowerPak for reasons I haven't yet determined.

I made these fixes with these python programs:
fix_fds_nsf_a.py (attached)
fix_fds_nsf_b.py (attached)

The a version creates two 0-filled banks for $6000-7FFF. Suitable if all mirrored writes are in the $6000-7FFF area and it's acceptable to initiallize that area to 0. Adds ~8k to the file.

The b version is more thorough, expanding the file to a full 32k of duplicated banks. This fixes mirrored writes in all memory regions, and doesn't change the initialization values.

There are 9 other NSFs that I have seen mirrored writes in (see above), though I had not identified a playback problem with them when listening. I will test them on the PowerPak.

Edit: all of the listed NSFs have problems on the PowerPak, except Fire Bam. Fire Bam mirrors writes from $6000 to $8000, but by stroke of luck, all the writes to the $6000 region ($60CC-67FF to be precise) are copying bytes from the mirrored location in $8000, so no problem is created.

Edit 2: attaching files (dropbox links now broken)

Edit 3: Python scripts were later disallowed on this forum. Uploading a ZIP with what I think was the original script.
Attachments
fix_fds_nsf.py.zip
(2.35 KiB) Downloaded 485 times
Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito) b.nsf
(32.13 KiB) Downloaded 500 times
Esper Dream (FDS)(1987)(Konami) a.nsf
(20.13 KiB) Downloaded 502 times
Ai Senshi Nicol (FDS)(1987)(Konami) a.nsf
(20.13 KiB) Downloaded 492 times
Last edited by rainwarrior on Tue Jun 12, 2018 12:42 pm, edited 3 times in total.
User avatar
Gil-Galad
Posts: 321
Joined: Sat Nov 13, 2004 9:43 pm
Location: Ohio, USA
Contact:

Re: NSF specification for FDS memory

Post by Gil-Galad »

Hmm, I didn't notice load addresses in the $6000 range were disallowed in the spec. This seems like an oversight to me, since without bankswitching turned on it seems some FDS games should require it. I know that NotSoFatso does not allow it, but are there any other players that do not? (NSFPlay, NEZPlay, FooGEP, Nestopia, FCEUX, VirtuaNES, VirtuaNSF are all fine with it.)
FDS support was added to the spec later on after the spec had already had it's basic design. As you can see from the updates. I have no clue what was going on during the design phase but something had to be done and that's how it ended up. Those players support loading at $6000 now. Some of them did not through the years and so a few of them are left with this issue. Hence setting load at $8000 to remain compatible with those older players and those that still don't support loading at $6000. I talked to kevtris about it today and he said that $6000 is a valid load address. Even though it doesn't say it in the spec.

V1.61 - 06/27/2000 Updated spec a bit
V1.60 - 06/01/2000 Updated Sunsoft, MMC5, and Namco chip information
V1.50 - 05/28/2000 Updated FDS, added Sunsoft and Namco chips
V1.32 - 11/27/1999 Added MMC5 register locations
V1.30 - 11/14/1999 Added MMC5 audio bit, added some register info
V1.20 - 09/12/1999 VRC and FDS prelim sound info added
V1.00 - 05/11/1999 First official NSF specification file

I also talked to him about expanding the header bytes that tell which initial banks are loaded when the NSF is loaded. Probably too late to do anything about it since the spec is set in stone now.
User avatar
Gil-Galad
Posts: 321
Joined: Sat Nov 13, 2004 9:43 pm
Location: Ohio, USA
Contact:

Re: NSF specification for FDS memory

Post by Gil-Galad »

Seiken Psycho Calibur still will not run on a PowerPak for reasons I haven't yet determined.
That's happened to me a few times. Sometimes the NSF needs to be ripped again from scratch. That often solves the problem if done right. I very seldom work with an existing NSF.

There could also be a memory conflict, which is one good reason why. Any of the main three addresses could be off. Anything really. Just need to track it down. You could disassemble the PowerPak NSF ROM and look at that. I've done it.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

Loopy did post the source for the NSF ROM here: http://3dscapture.com/NES/

I don't know what goes on inside the mapper, though.

I've been trying to find writes to weird addresses, or reads in the $4000-5FFF area in the FDS NSFs that crash, but haven't spotted anything unusual yet.

I found Namida no Soukoban Special (FDS)(1986)(ASCII).nsf, which reads $4015 and appears to rely on correct behaviour from this register (actually exposed a bug in NSFPlay that I will have fixed in the next version), but this works fine on the PowerPak.
Last edited by rainwarrior on Mon Aug 07, 2017 3:50 pm, edited 1 time in total.
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

Attaching Seiken Psycho Calibur NSF to update broken dropbox link above.
Attachments
Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack) b.nsf
(30.33 KiB) Downloaded 801 times
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: NSF specification for FDS memory

Post by Myask »

Well, it'd be much harder to do what Gil-Galad says in hardware, as that *requires* a memcpy, than to just treat it as bankswitched RAM, since it's just tweaking those high address lines in normal mappings.
Loopy did post the source for the NSF ROM here: http://home.comcast.net/~olimar/NES/
dead link?
User avatar
rainwarrior
Posts: 8733
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NSF specification for FDS memory

Post by rainwarrior »

Myask wrote:Well, it'd be much harder to do what Gil-Galad says in hardware, as that *requires* a memcpy, than to just treat it as bankswitched RAM, since it's just tweaking those high address lines in normal mappings.
A real FDS rip will only need to bankswitch in INIT, which is generally kind of agnostic as to whether that's a copy or not. However, the actual "bankswitch" here is a BIOS call which has been replaced by an NSF bank register write. There's no mechanism for actually doing the hardware copy here, so the ripped version should not expect a copy operation, IMO.

Homebrew NSF becomes a problem for this, as it generally just expects bankswitching RAM semantics, and then multi-chip NSFs are an even worse problem where you should probably write protect the FDS RAM to keep it from getting clobbered by other expansion audio register writes. It's a big mess.

PowerPak and probably any "normal" hardware implementation is just going to use bankswitched RAM, so reused banks should mirror writes to match that, IMO. It was unfortunately never codified in the NSF specs how it should behave, though.
Myask wrote:dead link?
Updated.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NSF specification for FDS memory

Post by lidnariq »

It almost seems like it would make sense to add a bit that specifies whether FDS RAM is writeable or write-protected...
Post Reply