Anyone doing automated regression testing?

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Anyone doing automated regression testing?

Post by tepples »

creaothceann wrote: Sat Jun 03, 2023 2:20 am Supporting thousands of SNES titles is only doable in a few months/years because you just have to perfectly emulate a few components. IIRC back when bsnes/higan was developed, even testing if every single title reached the title screen took several hours/days.
For comparison, the SameBoy release process includes an automated regression test, letting each of several hundred Game Boy games run in fast forward for a few seconds and then alternating Start and A presses. To speed up test runs, it runs one emulator on each of 24 cores of a Mac. SameBoy automation results include a grid of screenshots.

Do any NES emulators have this sort of regression testing?
User avatar
dougeff
Posts: 3078
Joined: Fri May 08, 2015 7:17 pm

Re: Anyone doing automated regression testing?

Post by dougeff »

testing if every single title reached the title screen took several hours/days.
I believe there's even a game that crashes on the very last level if something is wrong in the emulation. Which would require you to play through the entire game to test it properly.

I think it was Speedy Gonzales, but I can't find it here ??

https://snes.nesdev.org/wiki/Tricky-to-emulate_games


can someone add Speedy Gonzales to this list?
Last edited by dougeff on Sat Jun 03, 2023 1:28 pm, edited 1 time in total.
nesdoug.com -- blog/tutorial on programming for the NES
User avatar
Individualised
Posts: 309
Joined: Mon Sep 05, 2022 6:46 am

Re: Anyone doing automated regression testing?

Post by Individualised »

It's a random level in the middle of the game, not the last level. In the spaceship-themed Zone (it's a Sonic clone) I believe.
creaothceann
Posts: 610
Joined: Mon Jan 23, 2006 7:47 am
Location: Germany
Contact:

Re: Anyone doing automated regression testing?

Post by creaothceann »

tepples wrote: Sat Jun 03, 2023 11:23 am For comparison, the SameBoy release process includes an automated regression test, letting each game run in fast forward for a few seconds and then alternating Start and A presses. To speed up test runs, it runs one emulator on each of 24 cores of a Mac. SameBoy automation results include a grid of screenshots.
Interesting.
Personally I would also include savestates with the screenshots (when the emulator development has become stable enough) to detect non-visual regressions...

tepples wrote: Sat Jun 03, 2023 11:23 am Do any NES emulators have this sort of regression testing?
I only knew about Dolphin's.

dougeff wrote: Sat Jun 03, 2023 1:24 pm I think it was Speedy Gonzales
Yeah:
https://arstechnica.com/gaming/2011/08/ ... -emulator/
https://board.zsnes.com/phpBB3/viewtopic.php?p=160699
https://www.spriters-resource.com/snes/speedygonzales/
https://github.com/snes9xgit/snes9x/issues/280
viewtopic.php?p=237148&hilit=speedy+gonzales#p237148

Individualised wrote: Sat Jun 03, 2023 1:26 pm It's a random level in the middle of the game, not the last level. In the spaceship-themed Zone (it's a Sonic clone) I believe.
It certainly was the last level on most emulators :)
My current setup:
Super Famicom ("2/1/3" SNS-CPU-GPM-02) → SCART → OSSC → StarTech USB3HDCAP → AmaRecTV 3.10
Drag
Posts: 1615
Joined: Mon Sep 27, 2004 2:57 pm
Contact:

Re: Anyone doing automated regression testing?

Post by Drag »

I think a saved state plus playback of scripted button presses (i.e., a TAS movie) would be sufficient to hit those trouble spots that occur towards the middle or end of a game, as long as the saved state represents a completely valid game state.
Post Reply