It is currently Tue Jun 25, 2019 12:37 am

All times are UTC - 7 hours





Post new topic Reply to topic  [ 25 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Mon Dec 19, 2005 1:59 am 
Offline
User avatar

Joined: Fri Nov 19, 2004 7:35 pm
Posts: 4182
Are there any standardized savestate formats to use, or does each emulator need to create its own custom incompatible proprietary format?

_________________
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 19, 2005 2:48 am 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
There was some discussion of one earlier this year ("RSNS Savestate Proposal") but it didn't continue for very long. I've also done some further work on it in my emulator (including a movie format with one or more key frames for quick seeking). I'm interested in working on a common format, but only if it's done carefully and based on experimentation with alternative designs before committing to one.

If you're interested in doing some design work, it might be good to refine some ideas in e-mail and then present them here.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 19, 2005 4:26 pm 
Offline
User avatar

Joined: Sun Sep 19, 2004 10:59 pm
Posts: 1465
SNSS was the only real attempt at a standardized savestate format. It did not work, for several reasons:

1. It was difficult to use - multibyte values were stored in BIG ENDIAN, requiring nearly all emulators to use byte-swapping code (considering a vast majority of NES emulators run on LITTLE ENDIAN hardware). The reason given was to make it more "human readable", which is completely ridiculous as it is a BINARY file format.

2. It didn't divide information into separate blocks properly. The state for the CPU and PPU were crammed together into a single "base registers" block.

3. It didn't store enough detail. The CPU state had nothing for interrupts or open bus, the PPU state had nothing for the VRAM address latch (or the value that lingers on the PPU I/O bus and which is readable from any write-only register), and the APU state block held nothing but the last values written to $4000-$4013 and $4015, which is horribly insufficient for proper emulation.

4. Some state information was stored in the wrong place. The PPU block stored the current mirroring state, and the mapper block directly stored the current 8KB PRG ROM and 1KB CHR ROM banks mapped at $8000-$FFFF and $0000-$1FFF, respectively. Information like this should be encoded within the "custom mapper state" to take up an optimum amount of space, as well as allow additional detail to be stored (such as RAM banks mapped, along with write protection). The "SRAM" block also contained a "write enable" bit (which wasn't even documented properly, labeled as "0=no, >0=yes"), which properly belongs within the mapper state.

5. The Mapper state block was too restrictive. It allocated exactly 128 bytes for mappers to store their state information. For more complex mappers, such as the FDS, the Namco 106, and VRC7, this was not enough to store the complete state of their sound logic. For simpler mappers (such as UNROM, CNROM, or even NROM) it was a complete waste of space, since these mappers could store their complete state in a single byte (or in no space at all, in the case of NROM).

6. Some state data wasted space where not necessary - as mentioned above, the "custom mapper state" was fixed at 128 bytes even if the mapper itself stored only a single byte within it. Controller data was fixed to 4 bytes per frame including a "repeat count", and always stored data for both controllers (even if a 2nd controller was not even connected). PRG RAM (SRAM) was always stored in 4KB multiples (along with a "write enable" byte which doesn't even belong there - see #4), and the same was true for CHR RAM (VRAM) except for the "write enable" bit.

7. Some types of data were not stored at all, such as applied Game Genie codes or updated disk data on FDS games.

8. The "mapper state" blocks were ridiculously over-simplified, containing only the information which was not implied by the other bits of mapper-related state scattered throughout the rest of the file (again, see #4). The MMC3 state consisted of only four bytes - the IRQ counter, the IRQ latch, the IRQ counter enable, and the last value written to $8000 (by comparison, the MMC3 state in my emulator is 16 bytes long).

9. It pandered to obsolete file systems by restricting itself to the extensions .SS0 through .SS9 and supporting only 10 savestates, even though it was used almost exclusively on Windows 9x/NT, which has no such limitation.


If anyone is to attemp to devise a new "standard savestate format", it would be highly advised to avoid making these same mistakes again.

The main problem is that not all emulators are created equally - some are more precise than others. If you implement a savestate format according to a less precise emulator (like Nesticle), more accurate emulators will be unable to use it (as it will not properly preserve the system state they are maintaining). If you implement it according to a more precise emulator (like Nintendulator), then less precise emulators won't be able to fill in all of the necessary state information and, as a result, savestates created by them will not work correctly with more precise emulators. Even if you pick a "middle ground", you'll still run into the same problem, though with fewer casualties.

_________________
Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 19, 2005 6:35 pm 
Offline

Joined: Wed Nov 16, 2005 6:38 pm
Posts: 47
A standardize format would be great. I would be willing to help. Send a pm on the board to exchange email.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 19, 2005 9:22 pm 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
Hmmm, your last point dwarfs everything else. The main functionality that would be nice is basic transfer of saved games. That's one nice thing about battery RAM saves; the format can be standard for all emulators. But of course implementing similar for save states might require full accuracy when restoring the state.

Quietust, are you interested in exploring the issue of whether it's even possible to design a format that captures all information, but doesn't require Nintendulator's level of emulation to properly fill out? At the core, it's an issue of finding the smallest "state bottleneck" to save state at. If it's an impossibility, I'd like to see clearly why.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 8:46 am 
Offline
User avatar

Joined: Thu Nov 11, 2004 5:30 am
Posts: 121
Location: San Francisco, CA
Agreed, SNSS is a pile. Most of its failings, save the lack of maintenance, have been described by Quietust above. Let's never mention it again.

I'm wary of another format. Basically, if a new piece of information is discovered, say blargg's APU clock jitter, does that invalidate all previous savestates? Really, if emulators don't have similar implementations of some systems, it will be hard to make the library work for everything.

It might be easier just to make an extremely accurate NES emulation library available. ;)

_________________
...patience...


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 3:23 pm 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
Quote:
Basically, if a new piece of information is discovered, say blargg's APU clock jitter, does that invalidate all previous savestates?


This argument also applies to save states from older versions of the same emulator. This seems like a good miniature version of the problem to work on. The only reasonable approach seems to be filling in missing information and hoping for the best. There could still be code which doesn't work in this hybrid case, but does work with both the old version and new version. A contrived example would be code which determined which version it was running on, then kept verifying that it was running on the same version (i.e. checks for APU jitter, then keeps checking whether it magically started being emulated in the middle of the run, broken by a save state and restore in the new emulator).

The issue of discovery of new details is problematic. What comes to mind for me is the NMI suppression behavior I recently mapped out. One approach regarding a common file format is to deal with each new discovery individually, perhaps adding new fields that override the old ones and defining what a newer emulator should put into the old fields. If a few of us can spend some time considering actual scenarios, without rushing to something, I think progress can be made on the question of whether it's possible, and if so, how to achieve it. While we're on the subject, it's extremely time-intensive to reliably map out the details of these things. There are so many fallible links in the reverse-engineering chain for a given test that it requires hours of careful design to do tests on a given aspect.

So far it's clear that the best compatibility of save states between emulators wouldn't likely exceed that of the same emulator with older versions of itself. To take a more black-box empirical approach, how about we do some actual tests of moving save states between our emulators for common games to see how well it works in practice? I think that would give most of the benefits of perfect compatibility at a fraction of the cost.

I just got an image of the kinds of processes we're dealing with at an information level. There are some which amplify differences (open-loop) and others which suppress them (closed-loop). For example a random number seed loop amplifies small timing differences, while an infinite loop that waits for the next NMI suppresses them. What matters is that any differences in system state when restoring in a different emulator are not picked up by amplifying processes. The NMI suppression is a great example of a hardware-level amplifier: reading from $2002 at one particular PPU clock out of the almost 90000 per frame can suppress the NMI and thus cause very different behavior for that frame. Fortunately most games are written to suppress long-term effects of this, even though it might cause a doubled frame in the short-term.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 4:59 pm 
Offline

Joined: Wed Nov 16, 2005 6:38 pm
Posts: 47
Ideas:

Wouldn't it be less complicated to limit the place where you can save the state, such as only between frames and instructions. (immediately before VBLANK) Based on my board readings emulators render Video VBLANK by VBLANK and cpu instruction by instruction.

Using standardized descriptions we could also have a set of imprecise ones that would not be too difficult to solve into a more precise state. Less precise emu should be able to interpret the states of the more precise emu always (as long as that emu support the game), but not the other way around. Using the imprecise definitions the more precise emulators could make an educated guess based on the ROM or user input.

I know a binary format will save space compared to this but using xml with utf-8 encoding would allow easy manual editing, parsing, and be more platform independent. Short descriptive words and phrases or hex values could be used. Since this is a single byte text format (ie not utf-16 or 32) endianness would not be a problem, also mutiple save states could be contained into a single file. This website has a basic tutorial of what xml is and how it works http://www.w3schools.com/xml/default.asp

Hope I was not as confusing as any of my previous posts.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Dec 22, 2005 4:08 am 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
Quote:
Wouldn't it be less complicated to limit the place where you can save the state, such as only between frames and instructions?


Certainly. That was what my "find the state bottleneck" was about. And actually, the bottleneck is smaller just after the NMI has been vectored (due to the NMI suppression behavior I mentioned above).

Quote:
Less precise emu should be able to interpret the states of the more precise emu always (as long as that emu support the game), but not the other way around.


It needs to be the other way around, because it's the more precise emulators that will get even moreso with the discovery of new information. A (impractical) brute-force approach to this would be to store multiple versions of state so that less-precise emulators would use the older subset, with a new version spawned for each new set of discoveries. But this kind of intricate bloat would likely be rejected, so the approach to this issue will have to be more streamlined.

Quote:
I know a binary format will save space compared to this but using xml with utf-8 encoding would allow easy manual editing, parsing, and be more platform independent.


I'm quite sure that XML would be rejected by most emulator authors, not just due to space, but many issues. It has to be straightforward to parse with only a page or so of ANSI C code. Using a simple binary variable-sized block format with nesting would be more than sufficient.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 23, 2005 3:28 pm 
Offline

Joined: Wed Nov 16, 2005 6:38 pm
Posts: 47
blargg wrote:
Quote:
Less precise emu should be able to interpret the states of the more precise emu always (as long as that emu support the game), but not the other way around.


It needs to be the other way around, because it's the more precise emulators that will get even moreso with the discovery of new information. A (impractical) brute-force approach to this would be to store multiple versions of state so that less-precise emulators would use the older subset, with a new version spawned for each new set of discoveries. But this kind of intricate bloat would likely be rejected, so the approach to this issue will have to be more streamlined.


Coping with variation in version is something xml can handle, unknown tags are easily skipped. Since there are emulators that can handle just about every game I doubt there will be any new discoveries that are so fundamental to require a major update. Instead of taking a "what each thing does" approach a "what is the value of this thing" may be a better way to look at things. The values (or configuration) of these things in the hardware are the basis for modelling of all emu's. (Registers, blocks of ram, mirroring bits, etc).

This scenario is what i imagine happening with emu's of different precision use the same save state:
Consider an emu that does not support sound save states generated from emu would not work well in another that does support sound. If the situtation were revered the save state would work well. The best way I see of coping with this situtation would be to implement the APU registers as 0 initalized ram and write any updates to the section of the file. There may be enough data for an a precise emu to cope with but eventually, by change of the stage for example, it would be able to continue with sound emulation.

Having a minimum level of information must be a part of the standard.

blargg wrote:
Quote:
I know a binary format will save space compared to this but using xml with utf-8 encoding would allow easy manual editing, parsing, and be more platform independent.


I'm quite sure that XML would be rejected by most emulator authors, not just due to space, but many issues. It has to be straightforward to parse with only a page or so of ANSI C code. Using a simple binary variable-sized block format with nesting would be more than sufficient.


I understand that authors would want something very easy to code, however 1 page seems ambitious to me. Unless there was a library that was doing all the leg work. If there were a common library, I imagine calling something like getUchar ("/savestate/nes/cpu/register$value="x\'") to poll for information. I know xml will lend itself to that formatting. Xml already has several libraries availible for it too.

Another varition is a single file with an xml portion and binary block section. It would prevent editing with some text editors but a hybrid file seems easy enough to do.

Whether or not we can decide how we save the data, we need to talk about what to save.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 23, 2005 5:17 pm 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
The APU state is a good example. If an emulator isn't handling it at all, any games which use the APU interrupt or poll the status register won't work, thus we can assume that no save states lacking APU state will be using the APU. The default state should have everything silent and the frame interrupt disabled ($4017=$40).

As far as file format, about the simplest I can come up with that is extensible is the AIFF chunk style, where the file is a series of data blocks, each having a header with a fixed-length type tag and a fixed-length size, followed by size data bytes. This is easy to write and read without any extra state information. Any additional complexity in reading and writing has to give some useful benefit.

Here's something I just threw together as an example. The use of an end marker block allows nested groups of blocks, for example when you have multiple save states at the key frames in a movie file. Note how it also allows expansion of a given block, with older emulators reading a subset of the data, and newer emulators only reading as much data as the file provides.

Code:
void write_block( long type, long size, const void* in, FILE* out )
{
    unsigned char b [8] = { type, type>>8, type>>16, type>>24,
        size, size>>8, size>>16, size>>24 };
    fwrite( b, 8, 1, out );
    fwrite( in, size, 1, out );
}

void write_state( FILE* file )
{
    write_block( 'NAPU', sizeof (apu_state), &apu_state, file );
    write_block( 'NPPU', sizeof (ppu_state), &ppu_state, file );
    /* etc... */
    write_block( 'endb', 0, "", file );
}

void read_data( long size, void* out, FILE* in )
{
    unsigned char b [4];
    if ( fread( b, 4, 1, in ) )
    {
        long actual = b[3]<<24 | b[2]<<16 | b[1]<<8 | b[0];
        fread( out, (size < actual ? size : actual), 1, in );
        if ( actual > size )
            fseek( in, actual - size, SEEK_CUR );
    }
}

void read_state( FILE* file )
{
    unsigned char b [4];
    while ( fread( b, 4, 1, file ) > 0 )
    {
        switch ( b[3]<<24 | b[2]<<16 | b[1]<<8 | b[0] )
        {
        case 'NAPU':
            read_data( sizeof apu_state, &apu_state, file );
            break;
       
        case 'NPPU':
            read_data( sizeof ppu_state, &ppu_state, file );
            break;
       
        /* etc... */
       
        case 'endb':
            fseek( file, 4, SEEK_CUR ); // skip size
            return;
        }
    }
}


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 23, 2005 6:37 pm 
Offline

Joined: Wed Nov 16, 2005 6:38 pm
Posts: 47
Wow that code is a lot cleaner that I thought it would be for a binary file.

Your format has maximum space saving. The only real limitation I can see at this point are block names having a limited format and endianness.

I know it sounds like i won't let my xml idea die, but I have to say this. The hybrid xml file idea would allow binary chunks without limiting the space for tags and endianness.

xml Hybrid example
Code:
<savestate>
<nes>
<state>
<description>Motherbrain with 255 Missles and max health</description>
<cpu>
<register>0x0 0x5</register>
<ram>0x6 0x800</ram>
</cpu>
...
</state>
...
</savestate>
<binary data here>


The hex numbers above show the start offset and length from the first byte of <binary data here>. I can see the resistance to xml is because of the formatting involved but it might be worth the extra effort allow for updates and variation.

Edit: Changed first word of paragraph 2 from "This" to "Your" for the sake of clarity.


Top
 Profile  
 
 Post subject:
PostPosted: Sat Dec 24, 2005 8:23 am 
Offline
User avatar

Joined: Thu Mar 24, 2005 3:17 pm
Posts: 355
I'm all for a standardized savestate format. Though it's a very hard task, to design a format everyone would be happy with. You'll have to think, not what you'd prefer yourself, but what the majority of emulator authors would prefer.

In the case of XML, I don't think the majority would prefer that..


merry christmas :D


Top
 Profile  
 
PostPosted: Sat Dec 24, 2005 9:39 am 
Offline

Joined: Thu Sep 15, 2005 9:23 am
Posts: 1236
Location: Berlin, Germany
Dwedit wrote:
Are there any standardized savestate formats to use, or does each emulator need to create its own custom incompatible proprietary format?


I think that it would be easier for us all to stick to the our own savestate format for two reasons. First of all we can't even get a 16-byte ROM header right and secondly each emulator has it's own accuracy. If one were to be created then we would need full knowledge of every bit and byte register that the NES (specifically the PPU) has. Really, just like fixing the iNES header, it's too late.


Top
 Profile  
 
 Post subject:
PostPosted: Sat Dec 24, 2005 2:52 pm 
Offline
User avatar

Joined: Mon Sep 27, 2004 8:33 am
Posts: 3715
Location: Central Texas, USA
Quote:
The only real limitation I can see at this point are block names having a limited format and endianness.


What practical problems do these pose? The main limitations are the layout of the actual state structures, specifically preventing compiler-inserted padding and putting multi-byte values into little-endian format. Both are fairly trivial. I've been able to lay out structures for PPU and APU state in a logical way that is entirely usable as the in-memory format during emulation, the only step necessary before saving is to swap the multi-byte values to little-endian format (I develop on a big-endian Mac).

Quote:
I'm all for a standardized savestate format. Though it's a very hard task, to design a format everyone would be happy with. You'll have to think, not what you'd prefer yourself, but what the majority of emulator authors would prefer.


What if a few of us organize to crate a save state format that initially only aims to move games between emulators (as you can already do with battery RAM saves), perhaps with slight inaccuracies? This is a more modest goal, and would give experience on whether full accuracy were possible and worth the effort.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 25 posts ]  Go to page 1, 2  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group