It is currently Sun Jun 25, 2017 12:06 pm

All times are UTC - 7 hours



Forum rules


Related:



Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
PostPosted: Tue Mar 29, 2016 12:02 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1277
Hope no one gets mad about a second post; but the first one was too long.

> Any player's that don't recognize it can simply skip over it.

And again, it's not a perfect state capture then for any that are missing this new RIFF section. And it's not a perfect playback for any players that don't support this new RIFF format.

And you may end up with competing emulators implementing competing RIFF sections, and now you're back to "why can metadata be stored in plain-text or binary?? Why have both?"; only for much more important data.

A perfect format needs to be universal.

> Obviously, standardizing the fields would be the REAL solution -- but who's going to get all the SNES APU emulator developers together to conform on that level?

The person that defines the file spec needs to set everything in stone. It is their responsibility.

> How many different major APU emulators in use are there?

There's absolute piss-poor garbage based on things like OpenSPC, older Snes9X, etc. And then there's blargg's DSP which mine is 100% compatible with.

> Could you imagine getting these developers to discuss together the kinds of additional state their emulators would use for a proper SPC save-state?

blargg is all but gone.

> It is possible we could identify similar components and then give them official names to include in a newer format.

Sure. How'd that work out for 'illegal' 6502 instructions? How about for Cx4 instruction names? :P

Something something herding cats. Try and debate standardizing on formats and they'll argue with you for pages and pages over the most asinine edge case minor details, and then block you on Twitter for your trouble :P

> But that sounds like so much work, while at the same time - all of these emulators seem to play regular SPC without the additional state just fine.

If "good enough" is good enough for you, then stick with SPC :P

What's "good enough" for us today may not be for people in the future. I don't want to kick the can down the road, I want to kick it all the way to the finish line.

> and yet, standardization in the SNES community?!?! haha -- yeah right. I mean, it's do-able. But not on my watch.

I know. This is why I gave up on trying to standardize things. I'll just do my own thing, and complain bitterly all the time about it on forums instead :P

I'm very grateful that NESdev is basically the only emulation/ROM hacking forum on the internet I participate on that isn't my own and that doesn't ban people for disagreeing with them.

> Are the errors you talk about in Blargg's core documented? Can you provide reference to an error free core.

I didn't bother to correct the myriad errors in his SMP core. I wrote a new core for Snes9X to replace it. But higan/sfc/smp is what you're looking for. The only limitation of it is that I don't emulate two of the eight TEST register bits (there is no game ever observed that uses this register at all.)

> The thing I like about Blargg's core is that it also has a "Fast" aka "good-for-slower-computers" implementation - I would hope a referenced core does as well.

Even a $5 computer (RPi Zero) can play back SFM files with bsnes' SMP+DSP cores in isolation.

I guess if you're worried about 80 hours of battery life on a Pebble Watch, then okay, go for it. blargg has a fast DSP core, and I gave a fast SMP core to Snes9X (which is in the unreleased v1.54)

> I don't care if anyone adopts my format. Having created an answer to "SPC is crap" is enough for me.

Now that's the spirit! :D

> What is the minimum version of bsnes I can work with??

I don't have any idea when the last SMP/DSP fix went in. I always recommend the latest version in the accuracy profile.

> You can't be suggesting to compress single files as zip as well?

For distribution? Why not? I get what you're saying, people want to store BPS/SFM uncompressed and have them be optimally space efficient, but too bad. Writing an effective compressor/decompressor is very hard work. Writing a toy one like RLE is just going to make the files *bigger* when they are inevitably compressed anyway to distribute online.

> I'm thinking to use RLE for single files, and RAR for archives like they are already. Don't you agree?

I'm not going to say 'absolutely not'. BPS, as a side effect of delta-encoding where you specify a pointer that overlaps not yet decoded data, supports RLE.

It's your format though, do whatever you like :D


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 2:12 pm 
Offline

Joined: Fri Jul 04, 2014 9:31 pm
Posts: 723
I didn't mean to seriously suggest using RLE. I just meant that by its nature it negates my objection to read logging as opposed to write logging, since the major difference is that the SPC700 might read the same value several times before it changes. Any improvement from using a more advanced scheme is thus gravy on top of that.


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 2:53 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 18522
Location: NE Indiana, USA (NTSC)
byuu wrote:
> It is possible we could identify similar components and then give them official names to include in a newer format.

Sure. How'd that work out for 'illegal' 6502 instructions?

As far as I can tell, we standardized on the mnemonics used by ca65's 6502X mode.

Quote:
I'm very grateful that NESdev is basically the only emulation/ROM hacking forum on the internet I participate on that isn't my own and that doesn't ban people for disagreeing with them.

You're welcome. I've tried my best to apply civility guidelines like those of Wikipedia (as written, not as misinterpreted by rules lawyers).

Quote:
> You can't be suggesting to compress single files as zip as well?

For distribution? Why not? I get what you're saying, people want to store BPS/SFM uncompressed and have them be optimally space efficient, but too bad.

Ultimately the problem is that Windows Explorer does not support stacked file extensions such as bps.zip or sfm.7z. This is why people have to rename .spc.rar to .rsn to get them to open in a Winamp plug-in (as opposed to), or .s3m.zip to .s3z to get them to open in ModPlug Player (as opposed to Compressed Folders).

Quote:
Writing an effective compressor/decompressor is very hard work.

Then embed Info-ZIP's effective library. File formats used by Java (.jar), Android (.apk), Winamp (.wsz), StepMania (.smzip), ODF (.odt), and OOXML (.docx) are all built on Zip.

Quote:
Writing a toy one like RLE is just going to make the files *bigger* when they are inevitably compressed anyway to distribute online.

Not necessarily. DEFLATE compression used by the .zip format has a 32K history, and preprocessing with RLE may allow larger matches to fit into 32K. Have you tested this?

Quote:
> I'm thinking to use RLE for single files, and RAR for archives like they are already. Don't you agree?

I'm not going to say 'absolutely not'. BPS, as a side effect of delta-encoding where you specify a pointer that overlaps not yet decoded data, supports RLE.

Plus RAR is a non-free format. The unrar license forbids understanding the file format, making it incompatible with the Debian Free Software Guidelines, the Open Source Definition, and all copyleft free software licenses.


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 3:57 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1277
> As far as I can tell, we standardized on the mnemonics used by ca65's 6502X mode.

Oh, neat! That will be nice when I get back to NES core work again. Been wanting to emulate all the extra opcodes for a while now.

> Ultimately the problem is that Windows Explorer does not support stacked file extensions such as bps.zip or sfm.7z.

Yeah, it's also shit with extensionless files like Makefile and .htaccess (won't even allow you to create the latter, at least from Explorer itself.)

kaijuu gets around this nicely, but I don't expect people to install that :/

> Then embed Info-ZIP's effective library. File formats used by Java (.jar), Android (.apk), Winamp (.wsz), StepMania (.smzip), ODF (.odt), and OOXML (.docx) are all built on Zip.

The point is, I wouldn't mandate the compression in the format. Because then it can't be changed. If you want to use a tiny ZIP library, sure. My library, nall, has an 8KiB inflate implementation that would be perfect for that.

But, look at .tar.gz. Imagine if that was it. Instead, when bz2 came along, compressed sizes went down a lot. And then .xz came out and things went down a lot again. I like the separation of concerns, the Unix model, as it were.

> Not necessarily. DEFLATE compression used by the .zip format has a 32K history, and preprocessing with RLE may allow larger matches to fit into 32K. Have you tested this?

Actually, yes. We tested BPS with a lot of variations. We found BPS raw + 7-zip produced the smallest patch sizes. The BPS files were bigger once decompressed, obviously. The smaller we got the BPS patches with more efficient encodings and through using RLE, the less effective the 7-zip pass was. While the BPS patches were smaller on their own, the bps.7z files were larger this way. And this trend further extended over to the Xdelta format we were testing against (Xdelta3 also has -0 through -9 compression stages of its own. But even at -9, it benefits from a separate 7-zip pass.)

> Plus RAR is a non-free format.

RAR is garbage, yeah. Surpassed by 7-zip a long time ago. But a solid archiver will net you huge gains with all the repetition between tracks in a music album.


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 4:10 pm 
Online

Joined: Sat Apr 25, 2015 1:47 pm
Posts: 282
Location: FL
tepples wrote:
byuu wrote:
> It is possible we could identify similar components and then give them official names to include in a newer format.

Sure. How'd that work out for 'illegal' 6502 instructions?

As far as I can tell, we standardized on the mnemonics used by ca65's 6502X mode.


I was about to point out the illegal instruction list on Oxyron's 6502 page, but it turns out ca65 already uses it as a reference. Neat!

(That entire series is a pretty useful condensed 65xx series reference, especially the 6502 and 65816 pages.)


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 4:22 pm 
Online

Joined: Wed Jul 09, 2008 8:46 pm
Posts: 233
Amusingly, I almost pulled off a successful circumnavigation of a sound driver that streams the note data with one of the games (Paperboy 2, to be exact)... by outright implanting the music data from the SNES ROM and attempting to code the SNES portion in SPC700 code. You can find my WIP SPCs here.
The main problem is currently dealing with hiccups with reading the music data with regards to delay, which I copied straight off of the ROM.

Plus, this solution doesn't work for all games due to memory limitations (NBA Jam '96 and '97 are two such games: the memory consumption alone on the sample data is way too much for my taste, even though I have at least partially reverse engineered the music format). For games that have sample swapping (but not streaming in most cases... with the Visual Concepts games, in regards to streaming samples that play pieces of fully digital musical samples, I have circumnavigated that one personally... see NBA Jam '97 for an example), I have been successful in circumnavigating them, especially Lion King. For Lion King, I discovered that there was data that was not properly copied over (and I realized I had to overwrite SFX samples in some cases), and I personally took care of the problem by analyzing every which samples were being swapped out.

Obviously a custom file format that handles command IDs coming from the SNES would work much better than what I'm trying to do, but until that comes along, I'm attempting the impossible with the SPC format.


Top
 Profile  
 
PostPosted: Tue Mar 29, 2016 4:43 pm 
Offline

Joined: Wed May 19, 2010 6:12 pm
Posts: 2170
I would like to see a file that is just the contents of audio ram from $0200-$ffff and has the program counter start at $0200.


Top
 Profile  
 
PostPosted: Wed Mar 30, 2016 9:14 am 
Offline
User avatar

Joined: Fri Sep 02, 2011 8:34 pm
Posts: 476
byuu wrote:
A perfect format needs to be universal.


Agreed

byuu wrote:
> How many different major APU emulators in use are there?

There's absolute piss-poor garbage based on things like OpenSPC, older Snes9X, etc. And then there's blargg's DSP which mine is 100% compatible with.


OK, so then the only APU emulators that have a future are blargg's and yours - and they are compatible (which, could you please expound on what that means? ie Does this mean I could literally replace DSP files.. or what?) -- with yours in 1st place since you are active. Sounds like standardizing an optional RIFF chunk for "perfect state" between 2 of the best cycle-accurate APU emulators shouldn't be too hard. It sounds like a great place to start.

The "piss-poor" emulators can at the very least, rely on the traditional SPC state that will be present in the newer format -- and that's even if there are still active maintainers of those APU emulators or active developers of applications that use those emulators and want to update to support parsing the newer format. Of course, they can also try incorporating the newer "perfect state" attributes if they desire. However, you (byuu) stated that cycle-accurateness will be required for proper streaming-playback of the new format -- so these "piss-poor" emulators may simply skip processing the streaming chunk, declaring inability to stream (for the few spc that do stream). Of course, they could attempt to stream if they want to give it a shot!

byuu wrote:
> Could you imagine getting these developers to discuss together the kinds of additional state their emulators would use for a proper SPC save-state?

blargg is all but gone.


Since when? I remember talking to him about a year ago.

Blargg's APU emulator does have a "native state saving" feature -- we could look at the kinds of variables it uses and compare with your emulator and deduce a common-terminology, hopefully .. I couldn't do this without you though, otherwise my work would be exponentially more difficult, since I never designed a DSP emulator, and don't really understand the kind of state/variables involved. I could probably do it if I really tried, but I hope that you would just get involved. I am sure it would be a much simpler task for you.

_________________
SNES Tutorials (WLA DX)
SNES Memory Mapping Tutorial (Universal / LoROM) -- By Universal I introduce how memory mapping works, rather than just provide a LoROM map.
SNES Tracker (WIP) - Music/SFX composition tool / SPC Debugger


Top
 Profile  
 
PostPosted: Wed Mar 30, 2016 11:08 am 
Offline
User avatar

Joined: Sun Jul 01, 2012 6:44 am
Posts: 334
Location: Lion's den :3
byuu wrote:
Ramsis wrote:
I know a lot of ROM hackers (myself included) who were early adopters of UPS/BPS, headerless ROM images, game folders etc. I admit those people may still be a minority to this day, but the numbers are clearly increasing.

Not the ones I've seen.

You might want to look harder. :wink:

But never mind. I'm going to keep using your format anyway, and recommend it to fellow ROM hackers like a mantra (with success), even though from your point of view,

byuu wrote:
no amount of improvements will ever convince anyone to switch.

(Yeah, right. :P )

_________________
Some of my projects:
Furry RPG!
Unofficial SNES PowerPak firmware
(See my GitHub profile for more)


Top
 Profile  
 
PostPosted: Wed Mar 30, 2016 12:28 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1277
Quote:
OK, so then the only APU emulators that have a future are blargg's and yours - and they are compatible (which, could you please expound on what that means?)


It means my DSP core is basically a plagiarism of blargg's, but with his full permission including the right to license mine as I want.

I removed all of his optimizations with slower but more obvious code, but all of the operations and even variable names are identical. (I hid one in the modulo array, but I'll probably remove that too. I really doubt a few divisions per sample are that big of a deal.)

I've always wanted to restructure it, but I've never found a nicer way to describe it. The inner loop of the SNES DSP is a chaotic, blended mess.

> Since when? I remember talking to him about a year ago.

search.php?author_id=17&sr=posts

The very first page goes all the way back to February 2014.

I used to talk to him on a daily basis.

So many amazing people have come and gone. anomie, neviksti, blargg, TRAC, etc, etc. I miss talking to them all.

> But never mind. I'm going to keep using your format anyway, and recommend it to fellow ROM hackers like a mantra (with success)

Thank you very much :D

It's not even that I want my format to win, it's just that I want IPS to die.

Best we can do right now is deflate the sole false argument that BPS can't handle multi-patching. As you can see, the opponents were looking for any tiny sliver of a regression from IPS to dismiss it entirely, no matter how inconsequential. That's why it was so critical to ensure there were none, and there aren't, any negatives.

> (Yeah, right. :P )

A bit hyperbolic, but that's how I roll :P


Top
 Profile  
 
PostPosted: Wed Mar 30, 2016 6:05 pm 
Offline
User avatar

Joined: Fri Sep 02, 2011 8:34 pm
Posts: 476
byuu wrote:

So many amazing people have come and gone. anomie, neviksti, blargg, TRAC, etc, etc. I miss talking to them all.


<333

*long moment of nostalgia*
(I learned DP addressing from TRAC in #snesdev ~10 years ago <3)
(Neviksti, bright, helpful, and generous, the word "legend" comes to mind, and pcx2snes aha, and his SNES-Starterkit, and all of the other countless projects he was involved in (Tototek SuperFlash reverse engineering, X-Band, WLA-DX, etc. etc.)
anomie's document(s) always had a strong of impression of "THESE ARE THE BEST TECHNICAL DOCS" -- but I never got to meet this person.
blargg, I only recently started talking to -- and I was so obsessed with making quick progress on SNES Tracker that I overwhelmed him lol. For the brief time we worked together it was wonderful.

------

Byuu, I think you realize I expected more response from you based on my previos post. Particularly about creating universal terminology based on these 2 super-similar emulator cores that appear to be the leaders in SNES APU emulation -- which are apparently so similar that it might be as simple as identifying the terms in only 1 emulator! Do you agree?

I'm at my tipping point whether I'm going to even bother with these "universal" perfect state fields -- so please, this is your chance to get involved on that. Again, I'd think this would be easy for you -- but no pressure, we'll let this be. I don't want to see you stressed or overwhelmed because of some stupid fields.

_________________
SNES Tutorials (WLA DX)
SNES Memory Mapping Tutorial (Universal / LoROM) -- By Universal I introduce how memory mapping works, rather than just provide a LoROM map.
SNES Tracker (WIP) - Music/SFX composition tool / SPC Debugger


Top
 Profile  
 
PostPosted: Fri Jun 17, 2016 3:21 pm 
Offline

Joined: Fri Aug 26, 2011 3:20 am
Posts: 39
I'm hoping somebody here can help me on this subject, because I gotta say I'm VERY annoyed with there being no good solution to this:

I recently obtained a TOSLINK-modded SNES and I can capture the 32Khz audio from it directly into my computer. I wanted to make direct-hardware ripped soundtracks, and heard there was an SPC-to-ROM tool that could turn SPC files into ROMs to be used on real hardware. Problem is, either the tool or the SPC files SUCK. Most don't even work on real hardware at all when I convert them with the tool, and others only partially work with missing and/or glitched channels. These files exhibit the exact same behavior when loaded into byuu's emulator. This is simply absurd that this is the state of soundtrack ripping on the SNES. I'd like to find out if any of the following is available or being worked on:

1. A SNES sound file format that can actually be run as a ROM CORRECTLY on real hardware.
2. If not, then an SPC player that uses the same sound core as byuu's emulator.

Thanks for your help on this. I don't know much about the subject except that I'm getting very frustrated with SPC thus far.


Top
 Profile  
 
PostPosted: Fri Jun 17, 2016 3:38 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 18522
Location: NE Indiana, USA (NTSC)
Which flash cart are you using? My SNES PowerPak (with aftermarket "MUFASA" firmware) has an SPC player.


Top
 Profile  
 
PostPosted: Fri Jun 17, 2016 3:51 pm 
Online

Joined: Sat Apr 25, 2015 1:47 pm
Posts: 282
Location: FL
It could be an issue with the player rather than with the files - SPC files are just an image of the sound processor's state, and should (hopefully) sound exactly as intended on the actual hardware as long as you're able to put the SMP and DSP back into the same state that the SPC file describes (but that's a bit easier said than done.)

Of course the SPC format does have some inherent lack of state that was mentioned previously in the thread but I don't know if it would cause issues as blatant as what you're mentioning.

The sd2snes also has a built-in SPC player that I've had good results with. I can test it with anything you'd like to hear from it.


Top
 Profile  
 
PostPosted: Fri Jun 17, 2016 6:02 pm 
Offline

Joined: Fri Aug 26, 2011 3:20 am
Posts: 39
tepples wrote:
Which flash cart are you using? My SNES PowerPak (with aftermarket "MUFASA" firmware) has an SPC player.


I've got the SD2SNES. I didn't even know it plays SPC files directly, so I will give that a try next. As I said, I'm new to all this, so my apologies for being ignorant of the capabilities of these flash carts.

Edit: Tried the internal SD2SNES SPC player and it works great! Problem solved!

Thanks!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: HihiDanni, KungFuFurby, Revenant and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group