Mesen - NES Emulator

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

tepples wrote:I forget: Are USB "composite devices" allowed to have two devices of the same class?
If I understand the question, then yes: you can specify the number of interfaces in the configuration descriptor. Keyboards and mice tend to do this.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

Sour wrote:test build link
Let me know if that fixes it.
Executable doesn't run: missing MesenCore.dll.
Sour
Posts: 890
Joined: Sun Feb 07, 2016 6:16 pm

Re: Mesen - NES Emulator

Post by Sour »

My bad - that's what I get for making a build right after making a lot of changes to add Linux support in the past 2 days.
Fixed the link, this build should actually run!
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

Yup, confirmed -- that fixes it. :-) I tested with two joypads connected to the same device, ensuring it differentiates between both. One more bug fixed, woot!
Sour
Posts: 890
Joined: Sun Feb 07, 2016 6:16 pm

Re: Mesen - NES Emulator

Post by Sour »

Great! Thanks for reporting this and helping me debug it!
User avatar
Eugene.S
Posts: 317
Joined: Sat Apr 18, 2009 4:36 am
Location: UTC+3
Contact:

Re: Mesen - NES Emulator

Post by Eugene.S »

What do you think about new advanced integer-math NTSC algorhitm?

Blargg's NTSC filter is good, but bisqwit's filter have some advantages over it, including:
- moire is more accurate
- notch is more accurate
- integer math gives good speed
- and main: PAL-Filter can be easily done over this algorithm by slightly modifing it.

Feos tried to implement own PAL-filter to FCEUX, but failed because it was non-integer and have poor performance.
Bisqwit's algorhitm have great potential. Will be nice to see it in Mesen.
tomaitheous
Posts: 592
Joined: Thu Aug 28, 2008 1:17 am
Contact:

Re: Mesen - NES Emulator

Post by tomaitheous »

I have no idea if this is the right place for ask for features, so I apologize in advance for any annoyance.

Would it be possible to add a feature to the debugger where you could get the rate of occurence of each opcode? Similar to the CDL, but record the number of times all instructions are executed. Vsauce's zipf theory youtube video kinda got me thinking, but I think it would be valuable information in profiling how games use opcodes, their occurrence rate, and deviations between 65x variants (other 65x CPUs).

I'm not sure the best way store or display this data, though. Overall/absolute? Per second instances, with an average across 60 frames? Maybe even have a paired or triplet option; the occurrence of an opcide followed by one or two other opcodes (since the opcodes on the 65x tend to be simplistic, there are probably regular paired and grouped pattern occurrences; I'm interested in the ones other than the most common expected clc+opcode, etc).
__________________________
http://pcedev.wordpress.com
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

tomaitheous wrote:Would it be possible to add a feature to the debugger where you could get the rate of occurence of each opcode? Similar to the CDL, but record the number of times all instructions are executed. Vsauce's zipf theory youtube video kinda got me thinking, but I think it would be valuable information in profiling how games use opcodes, their occurrence rate, and deviations between 65x variants (other 65x CPUs).
I'm really not sure how this information is going to be useful without extremely deep or complex analysis (often having to be done by a human anyway). All you're going to be able to get at a basic level is a) what opcode "class" is being utilised (load accum vs. load X vs. store accum vs. no-op vs. branch), or b) what specific opcodes (which would include addressing mode, ex. A9 (LDA imm) vs. A4 (LDY zp) are being called at what amount/rate).

How would this information actually help you in any practical manner? "I think it would be valuable information" begs the question.

As for "deviations between 65x variants", this isn't relevant at all unless I'm missing something obvious? There is only one 6502 in the NES and its counterparts; there aren't going to be "deterministic through profiling" variances between, say, a Dendy CPU and a NES CPU. If this was a multi-platform emulator, then I could see what you're getting at (distantly, bordering on "not really but kinda"), but it's not.

If you're interested in this type of extremely low-level performance profiling (in general), I strongly recommend you look into hardware performance counters on Intel x86/x64 CPUs. I have no familiarity with it, but here you go (and this too).

If there was a feature desired in a NES debugger relating to performance/profiling, I would have to say that being able to represent "amounts of time" (CPU cycles) spent in something (could be a subroutine, could be between address X and Y), as part of either VBlank or non-VBlank. How this is done now is a bit painstaking; rainwarrior mentions it this Lizard Kickstarter update/blog (tweaking the R/G/B intensity bits based on start/end of routines, but there's also Lua if you're into that). If you ask me, I'd rather this be done through actual opcodes in a program (i.e. BRK $40 might start the tracking of cycles for performance counter #0 (say, support up to 16 of them), BRK $80 might turn it off, and then correlate that visually on-screen in some way -- allowing the programmer to move the BRK statements around to "narrow down" what takes up the most time in their routine). Again, this would be super useful for VBlank.
Last edited by koitsu on Thu Dec 15, 2016 3:15 pm, edited 1 time in total.
lidnariq
Posts: 11430
Joined: Sun Apr 13, 2008 11:12 am

Re: Mesen - NES Emulator

Post by lidnariq »

Statistics about byte frequency are good enough to identify whether a given chunk of data is x86 code, or 6502 code, or zlib-compressed, or just encrypted. (q.v.)

I think it would be perfectly plausible that statistics about opcodes, and/or opcode bigrams, could distinguish between different code generators and/or different tasks.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

lidnariq wrote:Statistics about byte frequency are good enough to identify whether a given chunk of data is x86 code, or 6502 code, or zlib-compressed, or just encrypted.
The previous post, if I understand correctly, was talking not about data (i.e. addresses or data being accessed by a program running on the CPU), but rather the regularity/count of opcodes themselves. I don't see how this is practically helpful. How I see it: you now know within, say, 20000 cycles, 9730 of them were spent doing lda, 7000 were spent doing sta, and 3000 were spent in (general classification) branch statements, with random leftovers. What does this tell you? To me it says nothing and teaches me nothing.

As for data access and heuristic analysis from that: different subject, where I can see the usefulness. I have absolutely no idea how such heuristics would work (especially being able to somehow figure out the difference between x86 and 6502 -- but more importantly I don't see how that's relevant inside of a NES emulator? The code running is all 6502, but I digress). I also don't know how something would be able to determine whether or not something was zlib-compressed or encrypted based upon "how often" (?) some data was being accessed. Some kind of "pattern" analysis, yes, I can see that being relevant I suppose, but it seems like a massive CPU-churning time sink (hopefully such a thing would default to off). This is where things are WAY outside my pay grade (one of several trade-offs of not going to college/uni I suppose), which is why I'm not going into too much detail. Do we really have encryption going on in NES games? Let's be practical please. Compression, absolutely. Unique data structures (for pretty much anything), absolutely. Encryption? Come on.

This is a feature/subject that's like opening Pandora's Box. It's guaranteed to never be enough for what people want. A programmer's nightmare.
Sour
Posts: 890
Joined: Sun Feb 07, 2016 6:16 pm

Re: Mesen - NES Emulator

Post by Sour »

Eugene.S wrote:What do you think about new advanced integer-math NTSC algorhitm?
I'm currently busy working on the Linux port, but I'll try implementing this after.
Mesen runs video filters in another thread, so it should still get decent performance whether the filter is fast or somewhat slow (so long as the algorithm can convert 60 frames per second on a single core)
tomaitheous wrote:...
Isn't most/all of this achievable by parsing a trace log file? It contains all the information you would need to get those statistics. I'd imagine writing a script in python or anything else to parse the output would be fairly easy. Of course, if you wanted to analyze the execution over a long period of time, the trace log could become large relatively quickly, though.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

Sour wrote:Isn't most/all of this achievable by parsing a trace log file? It contains all the information you would need to get those statistics. I'd imagine writing a script in python or anything else to parse the output would be fairly easy. Of course, if you wanted to analyze the execution over a long period of time, the trace log could become large relatively quickly, though.
That's what I'd advocate doing too: parsing the trace log.

And yes, trace log analysis is pretty much what most of us assembly hackers (rephrased: "the romhacker guys who do assembly and are thus highly sought after") end up sifting through. It's par for the course when it comes to reverse-engineering something. On several projects, I've sifted through a few hundred megabytes of trace logs to work out details, often on paper. It's how I was able to decode some of the compressed graphics/tiles in Otogirisou (SFC game) (fucking ChunSoft, my god, code from the bowels of hell -- meaning it's complex/hard for me to understand). I then write down details (incl. code) on pieces of paper and work out the details (esp. if graphical) on graph paper, as well as alongside a .txt file (more common with NES/FC titles; SNES/SFC are often too complex for this (matter of style)). Some examples attached (of a 10 or 12 page pamphlet of my notes)

My point being: yeah, trace logs. :-)
Attachments
IMG_20161215_144639176.jpg
IMG_20161215_144654483.jpg
IMG_20161215_144708935.jpg
lidnariq
Posts: 11430
Joined: Sun Apr 13, 2008 11:12 am

Re: Mesen - NES Emulator

Post by lidnariq »

koitsu wrote:9730 of them were spent doing lda, 7000 were spent doing sta, and 3000 were spent in (general classification) branch statements, with random leftovers. What does this tell you? To me it says nothing and teaches me nothing.
Because you're entirely missing the point.

It's not that there were N LDAs, P STAs, and "random leftovers". It's that this time, there were X% more LDAs, Y% fewer STAs, and the exact shape of the "random leftovers" tell you whether what's happening is the same or not. So, yes, this should be visualized, not tabular.

Think of it as being identical to spectroscopy (mass spec or IR spec, either). You don't care that exactly X% of the light at 10µm was transmitted: at best that tells you concentration and at worst it tells you nothing. But you do care that the total shape has a set of spikes that correspond to a C=O double bond.
As for data access and heuristic analysis from that: different subject, where I can see the usefulness. I have absolutely no idea how such heuristics would work (especially being able to somehow figure out the difference between x86 and 6502 -- but more importantly I don't see how that's relevant inside of a NES emulator? The code running is all 6502, but I digress).
You got distracted. That was an example of a parallel: I can identify whether a chunk of binary is 6502 or x86 by looking at the relative frequency of bytes or of byte bigrams.

Because I have seen first-hand examples of how these statistics are useful for static analysis, it makes sense to me that they could be useful for dynamic analysis.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Mesen - NES Emulator

Post by koitsu »

Yeah, sorry, I still don't see the applicability (and I read your explanation 3 times). I'm caught somewhere between "I think I just got shat upon by someone with a college degree; am I reading someone's thesis?" and "Why would there be chunks of x86 binary in a NES game?"

I think where I said "this is where things are WAY outside my pay grade (one of several trade-offs of not going to college/uni I suppose)" applies greatly -- if it didn't before, it absolutely does now.

Edit: A colleague of mine has read this recent feature request/subject and can explain to me the usefulness/applicability of said feature. I'm getting an explanation now off-thread.
User avatar
rainwarrior
Posts: 8731
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Mesen - NES Emulator

Post by rainwarrior »

lidnariq wrote:I can identify whether a chunk of binary is 6502 or x86 by looking at the relative frequency of bytes or of byte bigrams.
Could you, really? Wouldn't the noise of data and operand-data completely drown the opcode signal?

Like, the request seems to want to identify just the opcodes, which are already being sorted out by the emulator.
koitsu wrote:How this is done now is a bit painstaking; rainwarrior mentions it this Lizard Kickstarter update/blog (tweaking the R/G/B intensity bits based on start/end of routines, but there's also Lua if you're into that). If you ask me, I'd rather this be done through actual opcodes in a program (i.e. BRK $40 might start the tracking of cycles for performance counter #0 (say, support up to 16 of them), BRK $80 might turn it off, and then correlate that visually on-screen in some way -- allowing the programmer to move the BRK statements around to "narrow down" what takes up the most time in their routine). Again, this would be super useful for VBlank.
"STA $2001" is using an "actual opcode". ;P I don't really understand the advantage of BRK here (it saves 3 bytes but requires an IRQ handler?).

If you're trying to time within VBlank then you're not doing something you can see in the NES visual output anyway (though you could use an oscilloscope with $4011, $4016, etc. to get a signal across in an alternate way). I was demonstrating the use of $2001 there specifically because it makes visual output on the target hardware.

If you want to time code within an emulator there's a lot of ways to do it. You can use breakpoints. You can trigger LUA from execution points, write instructions (including $2001), or various other triggers, and then use it to gather/process/output your statistics. There's also thefox's custom build of Nintendulator that adds profiling registers at $4020-$403F.

I'll get to trace logs in a moment, but I actually wrote a trace log parser for Lizard, and it's one of the things I "really" use when I want to solve a specific performance problem. The visual check through $2001 is just a rough thing that I can have on while playing through to identify problem spots. When I need more detailed info I use an emulator and other tools. (Breakpoints, trace logs, LUA, etc.)
Sour wrote:Isn't most/all of this achievable by parsing a trace log file? It contains all the information you would need to get those statistics.
...
I'd imagine writing a script in python or anything else to parse the output would be fairly easy. Of course, if you wanted to analyze the execution over a long period of time, the trace log could become large relatively quickly, though.
Yes this is doable through trace logs, though the size of the output will become astronomical in fairly short period of time. It would be much more practical to do like a code data log and just have a 256 entry table that increments the current opcode whenever an instruction executed. This should be easy to add to any open source emulator, but I don't really see at this point why it would be feature needed in any release version of an emulator.
Post Reply