It is currently Fri Mar 22, 2019 5:22 pm

All times are UTC - 7 hours





Post new topic Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
PostPosted: Sat Mar 02, 2019 6:31 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
NewRisingSun wrote:
rainwarrior wrote:
I have not implemented the VRC7 chunk, so it is (correctly) unable to play the file.
It seemed to me that it's complaining because of the version number in the header though, not because of the VRC7 chunk. I base on this on the observation that changing the version number byte to 0x01 loads the file, just plays it incorrectly with VRC7 instead of YM2413, which is what I had expected.

The header you put on it is correct. It goes like this:

Version 2 = intepret $7C
$7C bit 7 = there is a mandatory (uppercase) chunk in the NSFe suffix
'VRC7' chunk is currently unknown by the player

If you set it to version 1, it ignores byte $7C, so even though it will parse the NSFe suffix it will not consider anything in the stuffix to be mandatory, so it will load despite the unknown chunk.

Sorry there isn't any descriptive information about why a file fails to load. That's another thing on my to-do list, along with other NSF validation tools. The only way to determine how it's failing currently is by using a debugger on the loading.


Maybe better behaviour, besides just reporting the error better, would be to play it anyway. The header correctly makes it clear that there's something important in the suffix without which this NSF can't be played properly, though. Like some visible warning but an attempted play would probably be better than refusing it outright, but that's more thought in the validation tools category I guess.


Top
 Profile  
 
PostPosted: Sat Mar 02, 2019 10:43 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
I've finished updating the wiki with the specification.
http://wiki.nesdev.com/w/index.php/NSF2


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 1:46 am 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
So byte $7C bit 7 is now described as "if set, the appended NSFe metadata may contain a mandatory chunk required for playback". Before that, it was just "if set, a metadata block follows the NSF data". I understand this to mean that I need to change all my Namco 163 .NSF files with the mixe chunk to clear bit 7, because the mixe chunk is not a "required chunk". Is that correct?


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 2:31 am 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
NewRisingSun wrote:
So byte $7C bit 7 is now described as "if set, the appended NSFe metadata may contain a mandatory chunk required for playback". Before that, it was just "if set, a metadata block follows the NSF data". I understand this to mean that I need to change all my Namco 163 .NSF files with the mixe chunk to clear bit 7, because the mixe chunk is not a "required chunk". Is that correct?

Sorry, but there is no "before", but I realize it was confusing that someone put it up there on the Wiki for so long. That spec was never implemented by anybody anywhere, and there was no firm definition ever given for the metadata it referred to. When I defined it as referring to NSFe metadata in this thread, I outlined the same usage as it has now.

However, yes, I don't think bit 7 should be set for N163 mixing, but if you set bit 7 and then don't add any allcaps fourCC chunks it will have no effect. 'mixe' being lowercase doesn't count as mandatory, in the same way that, e.g. 'tlbl' doesn't count as mandatory. So... you can change them if you want, but it doesn't actually have an effect unless you used an allcaps fourCC for something in there.

NewRisingSun wrote:
I am not happy with the statement "The older NSF1 had no specification for string encoding. UTF-8 is the standard for NSF2, but for backward compatibility only ASCII should be used". It allows but discourages the use of Kanji and Hanzi characters, so I either have to renounce backward compatibility, or never use Chinese or Japanese characters.

I don't see how I could say anything else about this. Old players use native codepage, so ASCII is the only option for backward compatibility. I'm not happy about it either, but it's still the truth.

To clarify, though, I added to that line: "...should be used for the fields in the header. An NSFe 'auth' chunk can be used to override the header's title/author/copyright with UTF-8 strings."

All NSFe fields are explicitly UTF-8 encoded, this was assumed some years ago (since all existing NSFe up to that point were ASCII, mostly due to NSFe being an obscure format), though I'm not sure if any existing player actually supports it. NSFPlay doesn't support UTF-8 yet, it is non-compilant in that respect, and unfortunately that's not going to change until NSFPlay 3 with a complete rewrite of the UI. (NotSoFatso, where NSFe came from is way too old of a program to have had UTF-8 in it.)

NewRisingSun wrote:
define "foreign script" text fields (ftex for text flbl for tlbl, ftau for taut, faut for auth), that are explicitly UTF-8-encoded and contain the same content but in the original script. For example, auth would contain in its first string "Zelda no Densetsu", while faut would contain "ゼルダの伝説". This is the way things are handled in the .VGM format's [url]GD3 tag[/url] --- both a Latin and a "Japanese" field (the latter also used for Chinese and Korean script), with the player allowing the user to switch between the display of the two, and it has worked very well for them. (This also implies that only the NSFe chunks could have non-latin characters.)

I never liked VGM's explicit preference for Japanese, but sure, multiple language set chunks are a possibility, maybe with a 16-bit enum to specify which language at the start of the chunk?

This isn't something I'm prepared to work on until I am also ready to support UTF-8 properly. It also really needs playlist UI to be able to support it, and that's planned, just not in the immediate future. So... it's something I will keep in mind when I get around to designing that.

FWIW the NSFe spec is pretty easy-going because of the chunks. At least adding new stuff doesn't tend to break old things or compete for the same space as other stuff. However, people aren't really even using NSFe that much yet. I am hoping my merging it into NSF will help increase its popularity though. Some others seemed to be providing the metadata through M3U playlists and stuff like that instead of using NSFe. If something starts supporting other chunks in the mean time, at least if they're non-mandatory they'll still play. If someone implements new stuff ahead of me I'll catch up to it eventually.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 2:56 am 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
Here's a thought, though, instead of defining new chunks for different languages...

Maybe just a 'lang' chunk that contains a single utf-8 string for the language name, and this sets a language group that all subsequent text chunks will belong to. (Order is already important for some existing chunks, so it doesn't seem too out of line with the existing stuff to me.)

So you could have 'lang' 'tlbl' 'lang' 'tlbl' and that would create 2 different 'tlbl' sets belonging to each language.

It also works for just clarifying what language a single-language NSFe is using. (Otherwise I guess it gets assigned to an unspecified/default text group.)


If there's a preferred language, I would guess that it would be best to put it last, for the sake of older players... depends on whether a second tlbl will replace an earlier one or be ignored, but I expect the latter behaviour would be more compatible. (I don't really know of many NSFe players anyway.) Should check later and see what happens with NotSoFatso.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 6:30 am 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
Not languages, scripts. "Zelda no Densetsu" and "ゼルダの伝説" are the same language, one in romanized script, one in original script. That would limit us to only two cases that need to be distinguished. One can indeed use the same NSFe text fields twice with some kind of distinguishing tag, or "switching chunk", in-between, as you proposed, or one could use a second set of chunks, which was my idea.

A third possibility, which I would also like very much, would be to put both in the same field and include some form of in-field separator, for example a pipe character (|) such as "Zelda no Densetsu|ゼルダの伝説". Romanized text would go before, original script text would go after, the separator. No extra fields or chunks would be necessary, and it would backward-compatible in the sense that legacy software would simply display the entire string with the pipe character, while separator-aware software would offer the user an option to indicate a preference for romanized or original script forms; strings without a pipe character would be displayed in their entirety as before.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 8:46 am 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
Also, I noticed something odd: on NSF1 files, or NSF2 files with $7C bit 5 not set, the first few milliseconds of a file are cut off in the sound output. Setting the version number to $02 and setting $7C bit 5 to 1 makes them no longer cut off. I have attached both files; just listen to the beginning of the first song.
Attachment:
CutoffComparison.7z [4.77 KiB]
Downloaded 22 times


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 5:06 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
NewRisingSun wrote:
Not languages, scripts. "Zelda no Densetsu" and "ゼルダの伝説" are the same language, one in romanized script, one in original script. That would limit us to only two cases that need to be distinguished. One can indeed use the same NSFe text fields twice with some kind of distinguishing tag, or "switching chunk", in-between, as you proposed, or one could use a second set of chunks, which was my idea.

I think having multiple languages is a valid need, though. Two scripts for the same language can fit into this too, but I don't think that should be the only thing serviced.

What about a file containing...
Code:
'lang' 'Japanese' 'tlbl' ...
'lang' '日本語' 'tlbl' ...
'lang' 'English' 'tlbl' ...

That could accommodate both needs. The ripper can include as many languages and scripts as they want.

NewRisingSun wrote:
both in the same field and include some form of in-field separator ... backward-compatible in the sense that legacy software would simply display the entire string with the pipe character

I think that would be backward compatible only in a very awkward way, like just technically readable I guess. I'd assume that a player subject to backward compatibility here probably doesn't have unicode support anyway, so this would also dump a bunch of mojibake into their fields.

I don't think backward compatibility is super critical for this feature anyway. The number of affected players is so small. It'd be nice if it's reasonable to do, but maybe not a big deal either way?

NewRisingSun wrote:
separator-aware software would offer the user an option to indicate a preference for romanized or original script forms

Having a language preference would apply to any implementation of this, but yes this necessitates having a standardized set of language designators, and allowing the user to rank languages by descriptor.

An English default ranking might have { 'English', 'Japanese' }.
A Japanese default ranking might have { '日本語', 'Japanese' }.

Anything not on the list would just fall back to file's default order. At least, that's how I imagine I'd want to make this preference look to the user in NSFPlay.

On the format side, we just need to keep a list of the language designators people are using on the Wiki. My suggestion to use a single string to designate it was that the string can be displayed to the user, and the player program doesn't really have to know anything about languages, just unicode.

Finally a specified default order would either prefer the firstmost or lastmost language as a fallback where the user doesn't have a preference that applies. (I was suggesting lastmost because that would offer a backward compatibility fallback for NSFPlay, but I need to test NotSoFatso and Mesen and any other current NSFe players.)

NewRisingSun wrote:
Also, I noticed something odd: on NSF1 files, or NSF2 files with $7C bit 5 not set, the first few milliseconds of a file are cut off in the sound output. Setting the version number to $02 and setting $7C bit 5 to 1 makes them no longer cut off.

When you set that bit, INIT is called twice.

In this case you get a sequence of:
  • INIT #1 sets a byte at $F0 to restart the song at next PLAY.
  • NMI is enabled.
  • INIT #2 begins but is immediately interrupted by NMI
  • PLAY #1 begins.
  • INIT #2 resumes, sets $F0 again.
  • PLAY #2 begins, restarts the song.

You can't just enable non-returning INIT on old rips, you need to write a two-stage INIT suitable for this feature.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 5:23 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 21209
Location: NE Indiana, USA (NTSC)
ISO 15924 defines four-character codes for scripts. In an IETF language tag, an ISO 15924 script code can be used after a two- or three-character language code from ISO 639. Some examples:

  • English written with Latin characters is en-Latn
  • Croatian (Serbo-Croatian written with Latin characters) is sr-Latn
  • Serbian (Serbo-Croatian written with Latin characters) is sr-Cyrl
  • Japanese is ja-Jpan
  • Japanese without kanji is ja-Hrkt
  • Romanized Japanese is ja-Latn
  • Simplified Chinese is zh-Hans
  • Traditional Chinese is zh-Hant
  • Chinese Pinyin is zh-Latn
  • Chinese Zhuyin is zh-Bopo

Use of these codes would save a parser from having to treat "Japanese" and "Japonais" as equivalent codes.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 5:42 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
I strongly feel that that is not problem worth encumbering a player with having to implement this massive ISO spec.

Those codes aren't human readable. They're maybe human guessable, but at the very least the player would have to provide a big list of strings to map these technical codes to a more suitable language name to show the user. Then of course you'd ask them to do it in every possible language. Maybe there's a library that does this... but then you're asking the player to depend on a library...

...to solve an insignificant problem. Just let the ripper put the language name in the file, show that name to the user. Simple. We're not the unicode consortium here, we're just annotating some video game music files.

If you wanna bikeshed about the best name for a language/script combo, go ahead I guess, but I'm just trying to implement a player and format, and this ISO spec does not belong here. Rippers who want to make multi-language NSFe annotations can collectively decide on language strings to use, I don't need to participate in this.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 11:06 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
HA HA HA so after checking every NSFe player I could find (there were a few more than I expected), all but one of them will replace 'auth'/'tlbl' when the chunk is reached, so last one prevails...

...but unfortunately the one player that does something different is NotSoFatso, and what it does is fail the whole file if a second 'auth'/'tlbl' is found! If it was a more obscure player I'd maybe not care, but NotSoFatso is the original NSFe player, and I don't think I should go against it.

So... can't just reuse 'auth' or 'tlbl'. Multiples of these produce an invalid ffile. The other text chunks ('taut', 'text') are newer than NotSoFatso and AFAIK NSFPlay is the only player that listens for them, so they aren't a problem if duplicated. (I don't know if anyone has ever used them in an NSFe except me.)

We're back to having new chunks for additional 'auth' or 'tlbl' at least. This does mean that 'auth' and 'tlbl' will specify the default language set, so I guess that question is answered. Might as well just say that the first language set is the default, and that it should use 'auth'/'tlbl' and not 'autx'/'tlbx' for that first one. While 'taut'/'text' don't technically need a new chunk ('taux','texx'?) the might as well get equivalent ones anyway. In a parser they can probably be easy aliases to implement. I guess it should also be specified that 'auth'/'tlbl'/'taut'/'text' must all belong to the same language group, and be the first language group, whether named by a preceding 'lang' or not.

Otherwise, I still favour a language/script state chunk ('lang') as described above, since this can optionally add a language descriptor to the default set too, and avoids having to add extra information to the other text chunks.


Last edited by rainwarrior on Sun Mar 03, 2019 11:27 pm, edited 1 time in total.

Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 11:10 pm 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
rainwarrior wrote:
You can't just enable non-returning INIT on old rips, you need to write a two-stage INIT suitable for this feature.
Explain (again) what the point of the double-init is? That means I definitely have to write a different INIT routine for NSF1 and NSF2, which is a bit annoying.

rainwarrior wrote:
When you set that bit, INIT is called twice.
Okay, but why are the initial milliseconds cut off (on all files)? I find that sounding quite objectionable.


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 11:36 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 7328
Location: Canada
NewRisingSun wrote:
rainwarrior wrote:
You can't just enable non-returning INIT on old rips, you need to write a two-stage INIT suitable for this feature.
Explain (again) what the point of the double-init is? That means I definitely have to write a different INIT routine for NSF1 and NSF2, which is a bit annoying.

The end of the first INIT is to give the player a signal that it is ready to begin PLAY (i.e. turn on NMI). The point is that from the second INIT you don't have to return. The proposal was for modelling things like Battletoads' intro where it loops samples in the main thread, but interrupts with music playback via NMI.

Of course, between the time it was proposed and now someone found a better way to support Battletoads (though not all NSF1 players support it):
https://forums.nesdev.com/viewtopic.php?p=198697#p198697

Battletoads itself is just one example, though. This has other uses. But... if you don't need this, you shouldn't use the flag. It is not backward compatible, it's a whole new behaviour that is entirely optional. There is no reason to write two different INIT routines.

NewRisingSun wrote:
rainwarrior wrote:
When you set that bit, INIT is called twice.
Okay, but why are the initial milliseconds cut off? I find that sounding quite objectionable.

Nothing is cut off. In the example you gave the song is restarted one frame after beginning. If you compare you will see that the restarted song is playing at a one frame delay, after the initial first frame is messed up by the restart.


Last edited by rainwarrior on Sun Mar 03, 2019 11:38 pm, edited 1 time in total.

Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 11:37 pm 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
Yes, it is cut off, on a all files. Let me post a few examples (later).


Top
 Profile  
 
PostPosted: Sun Mar 03, 2019 11:41 pm 
Offline

Joined: Thu May 19, 2005 11:30 am
Posts: 880
From the original NSF file (not the NSF2 hack). First is NintendulatorNRS, second ins NSFPlug. Notice how the initial parts of the square wave envelope are cut off in NSFPlug.


Attachments:
Zelda no Densetsu Comparison.7z [555.02 KiB]
Downloaded 23 times
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: Yave and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group