Defining a cross-emulator save state format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
tepples
Posts: 21710
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Defining a cross-emulator save state format

Post by tepples » Thu May 16, 2019 7:26 pm

In this post, mkwong proposed designing a save state format that is portable across different NES emulators. Let me describe some snags that an effort to design such a format spec would encounter:

How are you going to define a representation for the internal state of hundreds of mappers? That'd be a lot of work.

As we come to understand the NES better, we may discover more state that we need to store. Either it'll have to be a name-value pair format (except for bulk RAM contents) or there'll need to be some sort of version number for the Control Deck's state and the mapper's state.

User avatar
Quietust
Posts: 1488
Joined: Sun Sep 19, 2004 10:59 pm
Contact:

Re: Defining a cross-emulator save state format

Post by Quietust » Thu May 16, 2019 7:37 pm

Cross-emulator save state formats have been tried in the past. They don't really work.
Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another.

User avatar
koitsu
Posts: 4214
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Defining a cross-emulator save state format

Post by koitsu » Thu May 16, 2019 10:30 pm

A Q post from 2005 which I wholeheartedly agree with.

mkwong98
Posts: 221
Joined: Mon May 30, 2011 9:01 pm

Re: Defining a cross-emulator save state format

Post by mkwong98 » Fri May 17, 2019 2:52 am

tepples wrote:How are you going to define a representation for the internal state of hundreds of mappers? That'd be a lot of work.
Yes, that is definitely a lot of work. But as all authors of emulators with save state functionality have done it already, why not spare newcomers from reinventing the wheel?

User avatar
Dwedit
Posts: 4234
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: Defining a cross-emulator save state format

Post by Dwedit » Fri May 17, 2019 11:08 am

Well, if nothing else, attempt to export and import savestates from the "popular" open-source emulators. Maybe FCEUX, Nestopia, Nintendulator, etc...

You might not have complete information needed for another emulator, but some things can be extrapolated.
For example, if you have the current CPU or PPU memory map, you can fill in what the values for mapper latches should be.
If you know the PPU raster position that the save state would happen at, you can ensure that the save state is made at that time.

Even if you can't get a 100% accurate state conversion that would survive TAS-like validation, you might still get something that can let you proceed to play the game without a crash.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!

User avatar
rainwarrior
Posts: 7639
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Defining a cross-emulator save state format

Post by rainwarrior » Fri May 17, 2019 11:48 am

mkwong98 wrote:
tepples wrote:How are you going to define a representation for the internal state of hundreds of mappers? That'd be a lot of work.
Yes, that is definitely a lot of work. But as all authors of emulators with save state functionality have done it already, why not spare newcomers from reinventing the wheel?
Sparing them from reinventing the wheel? Why exactly are they making a new emulator for a system that has hundreds? :P
Dwedit wrote:Well, if nothing else, attempt to export and import savestates from the "popular" open-source emulators. Maybe FCEUX, Nestopia, Nintendulator, etc...
I think the idea of a savestate converter tool might be fun.

You could do this as its own separate utility without trying to burden any emulator authors with it. It might also become a nice point of reference for the actual savestate binary formats used by various emulators, since that stuff tends not to be well documented at all.

(On the other hand, the benefits of having such a thing are probably so small that nobody would ever want to undertake a project like this.)

mkwong98
Posts: 221
Joined: Mon May 30, 2011 9:01 pm

Re: Defining a cross-emulator save state format

Post by mkwong98 » Fri May 17, 2019 10:35 pm

Dwedit wrote:You might not have complete information needed for another emulator, but some things can be extrapolated.
For example, if you have the current CPU or PPU memory map, you can fill in what the values for mapper latches should be.
If you know the PPU raster position that the save state would happen at, you can ensure that the save state is made at that time.

Even if you can't get a 100% accurate state conversion that would survive TAS-like validation, you might still get something that can let you proceed to play the game without a crash.
My idea is similar to yours. A minimum amount of info required for a save state is defined. Beyond that, values are defined with a default value.
rainwarrior wrote:Sparing them from reinventing the wheel? Why exactly are they making a new emulator for a system that has hundreds? :P
Because they want to. :wink:
Anyway we can't ignore the possibility that there are people out there with new ideas and ability to create a good emulator. Mesen is a very good example of a relatively new emulator when there are already hundreds.
rainwarrior wrote:I think the idea of a savestate converter tool might be fun.

You could do this as its own separate utility without trying to burden any emulator authors with it. It might also become a nice point of reference for the actual savestate binary formats used by various emulators, since that stuff tends not to be well documented at all.

(On the other hand, the benefits of having such a thing are probably so small that nobody would ever want to undertake a project like this.)
If we have a save state converter tool, the internal state of the tool itself can be used as the base of a cross-emulator save state format. Then we can change this tool into a lib for emulators to use and we are done! :beer:

User avatar
Zepper
Formerly Fx3
Posts: 3190
Joined: Fri Nov 12, 2004 4:59 pm
Location: Brazil
Contact:

Re: Defining a cross-emulator save state format

Post by Zepper » Tue May 28, 2019 9:18 am

The basic NES memory can't be different, along the CPU state. The problem is all about mappers, I suppose.

tepples
Posts: 21710
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Defining a cross-emulator save state format

Post by tepples » Tue May 28, 2019 9:44 am

Different emulators use different models of PPU state, which may cause part of the problem even apart from mappers. That's why I recommended requiring portable save states to be taken in the post-render line, as it's when the PPU state is least in flux.

supercat
Posts: 161
Joined: Thu Apr 18, 2019 9:13 am

Re: Defining a cross-emulator save state format

Post by supercat » Tue May 28, 2019 12:28 pm

tepples wrote:Different emulators use different models of PPU state, which may cause part of the problem even apart from mappers. That's why I recommended requiring portable save states to be taken in the post-render line, as it's when the PPU state is least in flux.
Emulators can be used for at least four different purposes:
  • Running as efficiently as possible a game that's known to use only a certain set of hardware features and programming techniques.
  • Validating that a game only relies upon a particular set of hardware features and programming techniques.
  • Running a game that uses an unknown set of hardware features and programming techniques.
  • Experimenting with new programming techniques.
Given that some behaviors can vary among different real systems, no emulator can be ideal for all purposes. An emulator that is focused on the first two purposes may be much faster than one which needs to be suitable for the fourth, or needs to accomplish the third without trial and error (typically, one could accomplish the third by making a guess about features and then testing to see if one's guess was correct).

When running programs which don't exploit too many corner cases, an APU/PPU emulator which is written in an interpreted language but skips time-consuming corner cases could probably be faster than a more "accurate" emulator written in native code. Among other things, if the interpreted language included "sync" instructions to indicate when an action by the main CPU might have side effects that are observable on the PPU or vice versa, and if such side effects are rare, the emulator could reduce the rate at which it has to switch between processing CPU-side logic and PPU-side logic, thus improving cache performance.

If save-game files could include emulator *logic* in a portable format, then they could be processed by any emulator that can interpret the bytecode language.

User avatar
Zepper
Formerly Fx3
Posts: 3190
Joined: Fri Nov 12, 2004 4:59 pm
Location: Brazil
Contact:

Re: Defining a cross-emulator save state format

Post by Zepper » Fri May 31, 2019 2:59 pm

About the savestate, is there a problem of setting APU (and possibly) PPU timings to zero instead of saving cycle counters? Good about saving on scanline -1. I believe it's no so hard to create a standard savestate format with tagged blocks.

supercat
Posts: 161
Joined: Thu Apr 18, 2019 9:13 am

Re: Defining a cross-emulator save state format

Post by supercat » Fri May 31, 2019 9:12 pm

Zepper wrote:About the savestate, is there a problem of setting APU (and possibly) PPU timings to zero instead of saving cycle counters? Good about saving on scanline -1. I believe it's no so hard to create a standard savestate format with tagged blocks.
The DMC timer can be used to generate raster-synchronized interrupts without requiring a special mapper, but losing the APU state would totally break things.

User avatar
rainwarrior
Posts: 7639
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Defining a cross-emulator save state format

Post by rainwarrior » Fri May 31, 2019 10:14 pm

In addition to the DPCM IRQ...

The APU frame counter can also be polled or used as an interrupt.

The APU status bits will change e.g. when a DPCM sample reaches its end.

So there's a few APU things that a game might rely on. Probably a lot of games wouldn't have any discernable effect of losing APU state at a save, but I'm sure there are notable ones that would be affected.


There are some expansion audio devices that are effectively "write only" though. Those at least might potentially have no non-audio consequences. ...and of course if you care about audio consequences, this whole idea isn't even on the table anyway.

Post Reply