.fds format: Can checksums be heuristically detected?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems.

Moderator: Moderators

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Tue May 02, 2017 3:05 pm

hex_usr wrote:
loopy wrote:Show me one game that does this.
Absence of evidence is not evidence of absence.
To avoid getting into meta-argument about what you can prove...

Loopy's point is there's no demonstrated utility in including the CRC.

If you want, you could make a working disk image to prove that you could make something that relies on a bad CRC. Such an example doesn't demonstrate utility though, just that it could be done. It's obvious that it could be done, nobody here is doubting that.


I don't think there's a harm in trying to emulate/record the CRCs in some way, but arguing that it's theoretically possible isn't a very convincing way to make anyone interested in it. If you could actually discover a cool new game that it was meaningfully relevant to, maybe people would care, but otherwise it's an extremely academic desire.

...and I still think trying to do it with heuristics on the disk image is underperforming for this academic goal. There's not much point in adding only a CRC, add the whole enchilada and be done with it! If you're trying to cram it into the .FDS format instead of a new format, do something that doesn't affect the integrity of the disk image you're trying to test! That's why I suggested using the header and changing the FOURCC.

Though, actually I think Loopy's suggestion to just use a new file extension is a much better idea than reusing .FDS and changing the header. Simply changing the extension to, say ".FDC" would solve the whole issue with no effort, and no "heuristic" workarounds.

...or if you just want CRCs use the .QD format since it already has them?
Last edited by rainwarrior on Tue May 02, 2017 3:23 pm, edited 1 time in total.

hex_usr
Posts: 92
Joined: Sat May 09, 2015 7:21 pm

Re: .fds format: Can checksums be heuristically detected?

Post by hex_usr » Tue May 02, 2017 3:22 pm

"fds" stands for Famicom Disk System, though. You should know that I am against the continued use of copier extensions such as "smc", "swc", "fig", "gd3", "gd7", and all that, because they do not match the console's initials and do not clearly indicate which console the ROM is for. Especially "bin". Is this .bin file a Mega Drive ROM, a PlayStation disk image with a cue sheet, or something else entirely?

"qd" is similarly non-indicative. It just stands for "QuickDisk" and says nothing about which console or computer the disk image is for.

...but I'm just going to drop it. This discussion showed me why I usually don't bother trying to inspire positive changes. And I'm sensitive and don't take harsh words well. I was called "racist" once, and I near-permanently quit the board where that happened.

byuu was right. It's not worth it.
bsnes-mcfly: the bsnes v073 and bsnes-classic killer (GitLab repository)

User avatar
rainwarrior
Posts: 8006
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by rainwarrior » Tue May 02, 2017 3:25 pm

So really your whole problem is that the only possible solution you can accept has to be named ".FDS"?

I would suggest that you're putting some aesthetic needs about 3 letters above the practical needs of a file format, and you might have an easier time of this if you adjusted that priority.

(And I'm not trying to insult you by suggesting this. I'm just trying to give you my perspective about a way to dissolve this problem.)

lidnariq
Posts: 10273
Joined: Sun Apr 13, 2008 11:12 am
Location: Seattle

Re: .fds format: Can checksums be heuristically detected?

Post by lidnariq » Tue May 02, 2017 3:56 pm

hex_usr wrote:This discussion showed me why I usually don't bother trying to inspire positive changes.
This reminds me of that time where this rando came to the forum whining about how it's impossible to store Galaxian in .nes not as an overdump and that therefore we should clearly allocate a new mapper for only Galaxian that would have its own completely incompatible definition of what the size bytes mean in the header.

tepples
Posts: 22288
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: .fds format: Can checksums be heuristically detected?

Post by tepples » Tue May 02, 2017 5:08 pm

hex_usr wrote:"qd" is similarly non-indicative. It just stands for "QuickDisk" and says nothing about which console or computer the disk image is for.
Nor does .zip tell what system board MAME is supposed to emulate. In fact, there are a bunch of containers that are renamed zipfiles, such as .smzip (StepMania), .wsz (Winamp), .jar (Java Standard Edition), .apk (Android), .odt (LibreOffice), and .docx (Microsoft Word).
hex_usr wrote:And I'm sensitive and don't take harsh words well. I was called "racist" once, and I near-permanently quit the board where that happened.
Sometimes I take a couple weeks off when I feel I've become less than wanted.

User avatar
Bregalad
Posts: 8008
Joined: Fri Nov 12, 2004 2:49 pm
Location: Chexbres, VD, Switzerland

Re: .fds format: Can checksums be heuristically detected?

Post by Bregalad » Tue May 02, 2017 11:26 pm

The .FDS format is awful - it "saves" bytes by removing the CRC and '0' gaps between files, but requires to fill with "0"s until a size of 65500 is reached - cancelling all the file size "savings".

I think that this is unfortunate but we have to deal with it. Inventing new "formats" for FDS would only make things even worse. Also the way you (the OP) react when we point out that your idea isn't that great really shows a lack of maturity. It just happens to everyone - you get an idea and it happens that after all it wasn't a good idea. It's just normal and you need to admit it.

I'm not familiar with the .QD format (never seen it) but if it includes checksums it's great. The different file extension makes it clear it's different and no confusion is possible. Then use the .QD format. It being 65536 bytes instead of 65500 is meaningless, especially since actual disks are smaller than both of those sizes.

User avatar
aquasnake
Posts: 207
Joined: Fri Sep 13, 2019 11:22 pm

Re: .fds format: Can checksums be heuristically detected?

Post by aquasnake » Fri Dec 04, 2020 6:56 pm

I'm sorry to let this issue restart.

The continuous zero gap between files is meaningful: For real quick disk hardware, because it is not a track-sector level coordinate format, there is no so-called "format". It stores data like a tape, and the head recognizes the end of a file by reading a certain length of consecutive zeros. It's just like the end of a string '\ 0'. Because it is not polar coordinate format, the file gap is meaningless for software, and the dump is inaccurate. After the head loads a file, the motor stops running and so on. At this time, the head does not return to 0, and then it is suspended above the gap data. However, the length of the gap data read each time by forced dump is different. Maybe the dumper thought that these data could not be exactly dumped and deleted.

Because of the unique disk reading mechanism of quick disk system, these problems are left over by history. It must scan the entire disk surface(~ 64KB data) before the head returns to zero, instead of randomly accessing the smallest disk cluster by looking up the formatted partition table. Therefore, the file gap must be "removed" and then the *.fds image format is generated.

As for the CRC byte, no game relies on this value to achieve authentication against piracy. Maybe this function is reserved, but if you need to write to the disk file during the game, in principle, you need to recalculate the CRC and modify this value. Also, because of the unique disk reading mechanism of quick disk, it is necessary to scan the whole file for CRC verification of a file, and scan the entire disk for any file. Maybe game developers can't tolerate multiple disk scanning times (the initial scan load can be tolerated), instead of checking the CRC, they directly return true.

For 65500 bytes single-sided disk size, I think it is better to define it as 65536. Because 65536 is 64K aligned. What are the benefits? If a game contains more than one disk side, if it is not 64K alignment, we must do a linear transformation of the address, which is converted by base address + offset address, but cannot be accumulated directly. Non 64K alignment requires hardware to map a0-a14 again, which wastes 15 IO pins. However, it can also be read out with registers in the way of loopy, which requires modifying the BIOS.

User avatar
loopy
Posts: 399
Joined: Sun Sep 19, 2004 10:52 pm
Location: UT

Re: .fds format: Can checksums be heuristically detected?

Post by loopy » Fri Dec 04, 2020 7:31 pm

I think the gap issues, and mechanism behind it is pretty well understood. There are games larger than 64k per side, so a 64k restriction doesn't make sense to me.

NewRisingSun proposed a "kitchen sink" format that seemed reasonable to me but it was pulled. Seems like something like that is necessary to put this to rest, as everyone wants something different.

Pokun
Posts: 1760
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: .fds format: Can checksums be heuristically detected?

Post by Pokun » Sun Dec 06, 2020 6:29 am

I don't fully understand this stuff, so don't take me too seriously. :P
If gaps are arbitrary in length, predictable for emulators and FDSStick etc to reproduce, and thus not part of the digital data in a disk (unlike the CRC), I don't see how the official QD format is any different from what Aquasnake describes. It's not a format made up by Nintendo only for the virtual console, but used universally for other systems using quickdisks, or am I wrong? At least Nintendo must have stored the QD data in some kind of digital format on their Disk Writer ROM cartridges long before the Doubutsu no Mori Famicom emulator and the Wii VC.

I don't really see the point of having a file extension that describes the system the game is for, if it's for a media that's used by other systems as well anyway (though that could be said about raw ROM dumps as well). You don't have a specific name for MSX 3.5" floppy disk images (which are often DOS-compatible and contains some x86 machine code) or cassette tapes, nor for IBM-PC. They are just DSK, FDD, CAS, TAP etc.
Besides, FDS isn't the official abbreviation for the Famicom Disk System, "FCD" is. Famicom is already universally abbreviated as "FC" and Super Famicom is "SFC", so "FCD" is only natural for a 3-chara abbreviation (though product codes uses HVC, SHVC, FMC, FSC, CGB, NUS, GCN etc).

My foolish guess is that the 64 kB size is Mitsumi's official specs as the recommended maximum size of a quickdisk or something, and that Game Doctor is simply out of spec. 65500 byte is an odd value (in binary speak) but on the other hand the FDS format doesn't include the CRCs. The only adequate solution that would allow for Game Doctor disks that I see, is a format without a maximum size like Loopy suggested.

And about CRC, isn't it useful for checking that the disk data is still intact and not tampered with?

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Sun Dec 06, 2020 8:37 am

Pokun wrote:It's not a format made up by Nintendo only for the virtual console, but used universally for other systems using quickdisks, or am I wrong?
I have never seen .QD-style disk images for any other system. The structure of a .QD file is not parsable without intricate knowledge on how Nintendo stores data. And the exact 64 KiB per side still make me think that these are actually dumps of the disk writer cartridges, which would want to bankswitch between sides in powers-of-two amounts.
Pokun wrote:The only adequate solution that would allow for Game Doctor disks that I see, is a format without a maximum size like Loopy suggested.
Game Doctor and Magic Card disks are already supported by FDSConv and NintendulatorNRS and are used by interested parties, using the .FQD format. FQD (in its finalized version) only indicates the particular device (Game Doctor/Game Master/Magic Card) it is for and if necessary, its configuration, and explicitly denotes the lengths both of every side and of every block, the latter to account for protection types that demonstrably need it. It does not denote gap lengths and CRC field values, as no disk has ever been spotted that requires explicit inclusion, but could be easily added if the need were to arise. Game Doctor and Magic Card disks have the same gap lengths as any licensed disks; at least all the ones whose raw data I have seen do, and the Game Doctor/Magic Card BIOSes require correct CRC fields computed in the same way as Nintendo's FDS BIOS does. So to answer the question in the thread topic: yes, you can easily infer the gaps and CRC field values from the data you do have in FDS images.

As for file extensions, Windows operating system users will want file extensions that are specific enough to allow associating the file type with the correct emulator to allow double-clicking on the file to "run" it. Doing so does indeed necessitate platform-specific file extensions.
Pokun wrote:And about CRC, isn't it useful for checking that the disk data is still intact and not tampered with?
Indeed, the presence of the CRC field in Nintendo's .QD images reveals that almost all the Virtual Console ones have been tampered with. ;) NintendulatorNRS will correctly signal CRC error to the FDS BIOS when using such disk images.

Pokun
Posts: 1760
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: .fds format: Can checksums be heuristically detected?

Post by Pokun » Sun Dec 06, 2020 9:56 am

Thanks for clarifying this!

So QD is probably ROM dumps of the Disk Writer ROM cartridges. I thought I had seen the format somewhere else, but I can't find it. None of the MSX emulators emulates quickdisk drives yet (as quickdisk software for MSX is very obscure and none of it is unique for the format AFAIK), and I don't know what Mame uses.

OK so the FQD format wasn't dropped after all, but it isn't a kitchen-sink format, but rather a specific format for specific purposes.

Yeah well since neither QD, FDS nor FQD are universal quickdisk formats, only used with emulators and some very specific tools, it makes sense to use a platform-specific extension.

I looked a bit more into the QD format, and it appears it's based on the "Q.D. RAM Disk Format" used by Intelligent System's Disk System development hardware "FDT-ICE". The only difference between the QD format and the Q.D. RAM Disk Format is that all sides are merged into a single file in the QD format while FDT-ICE required each disk side to be in a separate file (and probably 64 kB large).


I tried to search for FDSConv but I can't find anything. What is this and where can it be found?

NewRisingSun
Posts: 1287
Joined: Thu May 19, 2005 11:30 am

Re: .fds format: Can checksums be heuristically detected?

Post by NewRisingSun » Sun Dec 06, 2020 10:45 am

FDSConv is my command-line utility to convert between Famicom Disk System disk image file formats.

To convert from the common headered .FDS format to a binary format that includes CRC fields and gaps, use the command line:

Code: Select all

fdsconv --outfmt fdsstick-binary outputName inputName.fds
Note that "outputName" must be specified without filename extension, as the program will append the output-format-specific file extension on its own. "inputName.fds" on the other hand must be specified with filename extension. The file format of any input file is detected automatically. Conversely, to convert several flux-level ("raw") files into one multi-side .FDS format file, use the command line:

Code: Select all

fdsconv --outfmt fds multiSidedFile sidea.raw sideb.raw
Refer to the command-line help for further information, including a list of supported disk image file formats. You cannot batch-convert several input files into several output files; attempting to do so will only produce one output file with many, many sides. Use the Windows "FOR" syntax to perform multi-file conversions instead.

Source code is included, wherein you may learn how to reconstruct the gaps and CRC fields from input file formats that do not contain that information. I hope you like C++, if not, I don't want to hear it.
Attachments
FDSConv-v1.04-2020-07-26.7z
(248.17 KiB) Downloaded 64 times

Pokun
Posts: 1760
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: .fds format: Can checksums be heuristically detected?

Post by Pokun » Sun Dec 06, 2020 1:50 pm

Thank you for this tool. I do prefer C++ over other high-level languages.

anikom15
Posts: 22
Joined: Mon Nov 30, 2020 2:41 am

Re: .fds format: Can checksums be heuristically detected?

Post by anikom15 » Wed Dec 09, 2020 5:07 pm

Who thought stripping checksum data was a good idea?

Stupid decisions of the past don’t preclude us from making better decisions in the present.

Pokun
Posts: 1760
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: .fds format: Can checksums be heuristically detected?

Post by Pokun » Wed Dec 09, 2020 5:55 pm

Getting rid of it does make development and hacking easier as you don't need an extra tool to calculate the checksums every time you change something as Loopy pointed out earlier.
Now when the lot check is exposed to the public, the original factory data of all (I think?) released and possibly some unreleased games should be known.

Post Reply