Automated testing techniques?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

User avatar
rainwarrior
Posts: 8731
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Automated testing techniques?

Post by rainwarrior »

Probably also helped a lot that he was a one man programming team. ;) Nobody else involved to have conflicting ideas about how/whether to make it frame-deterministic, and I'm sure it was a design goal from the beginning for him.

FWIW plenty of emulator authors have achieved perfect determinism in their game applications (i.e. see any emulator with a TAS). It's very much doable if you plan for it.

I don't think his estimates for it being useful in less deterministic software are very realistic. Divergence tends to be exponentially cumulative over time. I forget what he said, but it was something like "if it only works 25% of the time it's still useful", but that seems extremely optimistic to me as far as typical nondeterministic simulations go. :P It'd probably either be close to 100% for a while, or close to 0% very quickly, if I may over-generalize.
Oziphantom
Posts: 1565
Joined: Tue Feb 07, 2017 2:03 am

Re: Automated testing techniques?

Post by Oziphantom »

well sure the emulator runs a fixed pixel pipeline that is not synchronised from the actual hardware it is running on, allowing you to single step and warp time as mutch as you want while keep the output identical. Making it invariant to the issues and resources of the host machine.(until you get to the point you need frame skip ;) ) Even something like the "random" number generator in a SID chip is a LFSR with known constants allowing it to be fixed and consistent, as the emulator can set it to always start on boot Cycle X and make sure their is no variance in the initial bits. Also you don't emulate temperature impact on the RC reset circuit, or variance in crystals, or even drive rotation speed/heed seek times.

Again I'm not saying its impossible(any game with multiplayer basically has to do it, and anybody who has done multiplayer knows the pain ;D ), but guy at GDC talking to people who make AAA PS4 games saying, "yo just record the input and play it back on your game, it makes it heaps easy to track bugs - see I did it on my single player "SNES" level game that I have total control over and there is so much free time per frame that having a Cache miss is neither here nor there in the playback" is a bit on the nose ;)

Using "movies" recorded in emulators to track down bugs that happen on somebody else's machine is one of the great things about doing 8/16bit work ;) That and having Cpuhistory with value highlighting nectar of the gods!
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Automated testing techniques?

Post by tepples »

Oziphantom wrote:What he failed to mention was how he got his game running 100% the same every single time from just input data..(across multiple machines even) because that is a real pain in the neck to get right.
Either you don't do anything random at all, i.e make sure you random is 100% fixed order with a 100% guaranteed result.
That was mentioned. He uses an internal PRNG for things that affect game logic instead of that provided by the platform.
Oziphantom wrote:Make sure you sync every timer, every value is initialized correctly at the same time, and there are no variables like loading times off a disk
In environments with asynchronous loading visible to the game engine, the mock for the platform pretends that they're synchronous, and the developer lives with the 99 percent coverage.
Oziphantom wrote:or OS stealing CPU time causing some event to just slip to the next frame
That was mentioned. In engines with a variable time delta, time deltas are considered input and stored as such. Otherwise, in a fixed time step game like RCR, the game logic computation during each time step produces the same result no matter how much wall time it takes, and slowdown handling just means running multiple time steps of game logic before running the graphics engine once.
Oziphantom wrote:or if you are using C# or some other managed language making sure it just doesn't decide to garbage collect.
If the game's state differs across GC, then either the GC is broken or you're hitting finalizers of platform-provided non-memory resources that should be mocked.
Oziphantom wrote:This of cause gets worse when you do Debug -> Development -> Release -> Gold Master -> Retail builds as that will all change the timings and memory layout.
The point of a deterministic game engine is that its result is oblivious to wall time and memory layout. A uint32_t is a uint32_t, no matter how much padding the compiler inserts before it or what cache line it falls on.
Oziphantom wrote:having seen games that the demo mode works fine on Gold Master and then watch it fail horribly in the in game demo from retail
I've seen one. It's called Super Mario Bros. An oversight in its initialization causes its attract mode to desync half the time, where Mario gets stuck against a pipe and doesn't take any damage. Most other bugs that Brian ended up finding in RCR were traced to similar initialization oversights.
User avatar
gauauu
Posts: 779
Joined: Sat Jan 09, 2016 9:21 pm
Location: Central Illinois, USA
Contact:

Re: Automated testing techniques?

Post by gauauu »

Sour wrote:
gauauu wrote:Yeah, I the existing lua support plus the "terminate via lua" option would be all the features I'd really need, although I think I'd want to build up a few lua routines to make it all easier (ie assertEquals(addressLabel, value)). But a headless run mode (which could run at 1000% speed) would be awesome also, for running a suite of tests quickly.
Tinkering a bit with Mesen's code, I'm able to get it to run a rom+lua script, and have the script call emu.stop([exitcode]) (this doesn't exist in the Lua API at the moment) and get that exit code as the executable's exit code in the command line (which should be enough to run it via make?)

At the moment it's part of the C# executable (no window ever gets shown, though), which might not be ideal for startup performance, but would allow me to ship this as a feature pretty easily. I guess performance would really depend on whether or not you would be planning on running dozens vs hundreds/thousands of tests (where each test would be a different Lua script). And of course, Mesen being Mesen, it will probably only run games at like 150-250 FPS with a Lua script loaded.
That's awesome! 150-250 fps is plenty fast (and honestly, I can't imagine having more than a dozen tests ... it's mostly things that I test because they've broken more than once and I'm tired of messing with them)
Post Reply