.fds format: Can checksums be heuristically detected?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems.

Moderator: Moderators

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Wed Dec 09, 2020 6:19 pm

anikom15 wrote:
Wed Dec 09, 2020 5:07 pm
Who thought stripping checksum data was a good idea?

Stupid decisions of the past don’t preclude us from making better decisions in the present.
There's nothing stupid about it. Other standard floppy disk images do not include checksum either. Its primary function is to preserve integrity when reading the disk, which is entirely unnecessary when we're no longer using the unreliable floppy disk medium to preserve it. Your hard disk has its own CRC already.

Is there some game that you want to emulate that does something meaningful with a bad checksum? It would be nice to know an example of this so that we're not just speaking hypothetically.

Similar question for gaps. We can complain that gaps are missing from the format, but what problem does putting them in solve? A naive dump will just fill the gap with whatever bytes it reads. A real gap can be full of garbage not representable that way. If there was theoretically some copy protection scheme that relied on tricks with gap encoding, I'm certain it wouldn't be solved by just copying bytes read from the gap verbatim. That'd be an incredibly weak attempt at such a thing. More likely we'd get things like strange encodings intended to confuse the reader/copier, or something that relies on specific data timings, which would need a lot more than just a naive gap byte dump to encode. Floppy disk formats I know of that encode information that's useful for copy protection emulation tend to be insanely complicated... a series of strange additions, each one to address a new creative method of protection that was found. You can't can't future proof your format for this stuff, it just has to be extensible so you can keep adding to it.

So what's stupid about a straightforward format that correctly encodes almost (?) every FDS game with minimal fuss? What would a CRC in there have bought you? Which game needs it?
Last edited by rainwarrior on Wed Dec 09, 2020 7:16 pm, edited 2 times in total.

anikom15
Posts: 22
Joined: Mon Nov 30, 2020 2:41 am

Re: .fds format: Can checksums be heuristically detected?

Post by anikom15 » Wed Dec 09, 2020 6:45 pm

If your goal is to preserve information than omitting information is stupid.

Emulators and tools are free to completely ignore the CRC. Its existence doesn’t create any problem whatsoever. If you want to hack SMB2 to be easier because you suck at games you can update the CRC, you can delete the CRC, you can do whatever you want. It’s a corrupt image so whatever value it has doesn’t mean anything anyway.

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Wed Dec 09, 2020 7:01 pm

anikom15 wrote:
Wed Dec 09, 2020 6:45 pm
If your goal is to preserve information than omitting information is stupid.
No information is omitted. The value of the CRC is implicit in what you're given.

The only thing that is omitted is the possibility of encoding sectors with bad CRC. ...something which you've not offered any justification for. You just wanna call people stupid for not thinking it was important to include a thing only needed to encode bad dumps? There are many other ways a disk dump can be bad. They don't all need encodings.

Under normal circumstances, if you're trying to dump a FDS disk and you get a bad CRC, you would reread it because you have a bad dump. If the CRC was wrong, the data was corrupt, and you really shouldn't be preserving that unless you're desperate. Maybe you want a format to be able to encode these desperate cases, but I don't think it's stupid at all to leave those out of it.

Again: which game is this needed for? If there are FDS games that need a custom disk encoding scheme, I highly doubt a CRC alone could make any difference at all.

anikom15
Posts: 22
Joined: Mon Nov 30, 2020 2:41 am

Re: .fds format: Can checksums be heuristically detected?

Post by anikom15 » Wed Dec 09, 2020 7:28 pm

Do we know if every FDS disk’s CRC is correct for its data?

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Wed Dec 09, 2020 8:56 pm

anikom15 wrote:
Wed Dec 09, 2020 7:28 pm
Do we know if every FDS disk’s CRC is correct for its data?
Check back to page 1 for loopy's answer. I think few people can speak as authoritatively on the FDS library as he can.
loopy wrote:
Tue May 02, 2017 2:30 pm
Show me one game that does this (Hint: there aren't any). I've seen plenty of arguments like this. Foggy memories, claims that some game might/could have done this. Never seen one instance where it was actually an issue.

For emulators: CRC is not useful. The BIOS doesn't look at it, emulators just need to set the "CRC GOOD" status flag. Maybe insert dummy bytes for padding. Emulators will always need to do a little "magic" anyway to deal with GAP data etc.
For games: CRC could -theoretically- be used for copy protection or something. But it isn't. And won't be, ever.
For homebrew: Actually an annoyance, requiring an extra step in the compilation process.
For "preservation" (the No-Intro crowd): CRC is redundant. Games would fail to load without a correct CRC so nothing is being lost by not including it.
A mismatched CRC would be found while dumping a game, as it needs to be checked to verify that you got a good dump in the first place. That is really the whole purpose of this CRC: making sure you read the data from the disk correctly.

I don't think anyone disagrees that it could have been used for something else that could have been relevant to emulation and need to be preserved... but it wasn't. Nobody has found an actual example of this. Every single game we know of used the CRCs as intended. They're entirely redundant.

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Wed Dec 09, 2020 8:57 pm

anikom15 wrote:Do we know if every FDS disk’s CRC is correct for its data?
Excluding corrupt disks and the hacked .QD files from the Virtual Console, we do.

LocalH
Posts: 180
Joined: Thu Mar 02, 2006 12:30 pm

Re: .fds format: Can checksums be heuristically detected?

Post by LocalH » Wed Dec 09, 2020 10:44 pm

I understand the justification for no-CRC dumps, although the preservationist in me doesn't really like anything less than bit-perfect dumps of the entire disk contents, whether or not we "need" them. Not sure if flux transition dumps would be necessary but that's kinda the ultimate concept for "perfect" preservation of exactly what is stored on the disks, if so.

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Wed Dec 09, 2020 11:02 pm

The more low-level your image format is, the more entropy you are preserving, entropy that takes you away from the actual preservation-worthy data. Entropy coming from the alignment of the drive used for dumping, its speed variations, manufacturing tolerances ...

Initial gap length and in-gap data will be prone to varying degrees of randomness on the binary or bitcell level, and on the flux level, you might as well fingerprint the imaging drive with it.

So I reject the popular idea that proper preservation automatically means capturing more data and especially more low-level data. The idea comes from the Software Preservation Society who needed donors and collectors without any knowledge about Amiga copy protection methods to capture as much data as might be possibly needed for dump submissions until somebody knowledgable from the Society had the time to review the image data. The actual preservation-grade images (.IPF) that came out of this manual review process are not particularly low-level except for the fraction of the disk that was determined to require such.

I suppose the existence of a low-level image can be useful for independent verification that the higher-level image indeed is unmodified and contains all the necessary information. But I would not want to deal with the low-level image outside that narrow application. The idea of loading a 50 MiB set of Kryoflux stream files just to run a game off a 360 KiB disk image revolts me.
Last edited by NewRisingSun on Wed Dec 09, 2020 11:18 pm, edited 2 times in total.

anikom15
Posts: 22
Joined: Mon Nov 30, 2020 2:41 am

Re: .fds format: Can checksums be heuristically detected?

Post by anikom15 » Wed Dec 09, 2020 11:06 pm

NewRisingSun wrote:
Wed Dec 09, 2020 8:57 pm
anikom15 wrote:Do we know if every FDS disk’s CRC is correct for its data?
Excluding corrupt disks and the hacked .QD files from the Virtual Console, we do.
Call me a skeptic then.

I suppose using FQD for ‘good’ disks with no information in the CRC and QD for disks where the CRC doesn’t match the data. Otherwise, if you want a universal format it will need to be able to accommodate the VC versions. Digital or not, they are releases! Unless you’re arguing that the CRCs in corrupt disks should be thrown away. I don’t agree. Even if the information is worthless, it still tells us something and needs to be preserved.

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Wed Dec 09, 2020 11:09 pm

Yes, if you absolutely want to preserve disks with bad CRCs, there already are formats available for that. And the utility that I posted this week will let you convert between eleven different formats of various low-levelness, back-and-forth, so why is not everybody happy now? :lol:

LocalH
Posts: 180
Joined: Thu Mar 02, 2006 12:30 pm

Re: .fds format: Can checksums be heuristically detected?

Post by LocalH » Wed Dec 09, 2020 11:28 pm

NewRisingSun wrote:
Wed Dec 09, 2020 11:02 pm
I suppose the existence of a low-level image can be useful for independent verification that the higher-level image indeed is unmodified and contains all the necessary information. But I would not want to deal with the low-level image outside that narrow application. The idea of loading a 50 MiB set of Kryoflux stream files just to run a game off a 360 KiB disk image revolts me.
I mean, I understand why it's not desirable to have the file format that represents an exact preservation of the contents of the media be the daily-use file format, don't get me wrong. I also understood from the beginning why flux-transition dumps probably wouldn't be necessary for FDS preservation. But, it's kind of like making a file copy of an MS-DOS disk and saying that it's suitable for preservation, because the data stored in the unallocated space is not "necessary". Still, the data *is* stored on the disk, whatever the contents, so a preservation-oriented dump should absolutely include that data, as written on the original media. It's kinda the same idea behind the way LaserDisc preservation is being handled, by recording the raw RF waveform (when for most usage, a proper D1-resolution capture would be sufficient and more practical).

anikom15
Posts: 22
Joined: Mon Nov 30, 2020 2:41 am

Re: .fds format: Can checksums be heuristically detected?

Post by anikom15 » Wed Dec 09, 2020 11:47 pm

NewRisingSun wrote:
Wed Dec 09, 2020 11:09 pm
Yes, if you absolutely want to preserve disks with bad CRCs, there already are formats available for that. And the utility that I posted this week will let you convert between eleven different formats of various low-levelness, back-and-forth, so why is not everybody happy now? :lol:
Probably because the popular format is unable to accommodate them. Whether there be one format or two is not something I really care about, but I can understand how people would want a single file format that can handle everything.

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Wed Dec 09, 2020 11:59 pm

I went away from the idea of a "kitchen sink" format for one reason: I cannot stand that the same disk may be represented in more than one way within a single format.

lidnariq
Posts: 10277
Joined: Sun Apr 13, 2008 11:12 am
Location: Seattle

Re: .fds format: Can checksums be heuristically detected?

Post by lidnariq » Thu Dec 10, 2020 12:07 am

anikom15 wrote:
Wed Dec 09, 2020 11:06 pm
it will need to be able to accommodate the VC versions.
Keep in mind that the hacked VC images require an emulator that is inaccurate, namely requiring not enforcing said CRCs. If you take those images and put them on a real disk on a real FDS, the bad CRCs will error instead of working: viewtopic.php?p=254256#p254256

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Thu Dec 10, 2020 12:14 am

Yes, if the FDS format theoretically had CRC data in it, a bad dump also won't work on hardware or emulators anyway, so you'd be wasting everybody's time by distributing it.

If you need to preserve a bad dump that requires restoration work, use a specialized format that facilitates that. There are very few people that can make use of stuff like this. If you have a bad dump on your hands, your dumping tool should already let you know this. The only way a bad dump gets into an FDS file should be by someone deliberately ignoring that.

We don't need the popular format to permit stuff like bad CRCs that are not just useless but immensely frustrating for the general public who expects an FDS file to contain something worth running.
Last edited by rainwarrior on Thu Dec 10, 2020 12:21 am, edited 1 time in total.

Post Reply