It is currently Mon Dec 18, 2017 4:03 am

All times are UTC - 7 hours



Forum rules


Related:



Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Bad game dumps
PostPosted: Sat Oct 08, 2016 12:29 am 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1339
Hope it's okay to post this as its own topic here. If not, mods feel free to delete this.

I've been redumping the SNES game library, and every so often I find that my dump hash doesn't match the ones in GoodSNES/NSRT/No-Intro. For instance, recently AWJ did an amazing job tracking down what some changes to Spider-Man & Venom actually impacted. So I'd like to use this thread to publicly share such findings, in the hope AWJ or someone else might be interested in taking a look at what's going on. Not required of course, worst case this is just documentation.

To start the thread off, here's another bad dump I just found:

Code:
Fatal Fury (PAL)
Serial: SPAL-GN-0

[Correct]
SHA256: 450df78c9b7c92e9f8ce5c2ee0e1dbf939031c1e4f9e10c52c8d8f874364d1d6
0x122764: 0x00
0x179fd4: 0x0f

[Hacked/Corrupt]
SHA256: 16009eb74966d642a01a3eabd3e5364a006faff36e1ebaa306ff4e93ea8fda63
0x122764: 0x08
0x179fd4: 0x07


Note how the sum of the bytes add up to a correct internal ROM header checksum either way.

Given this one's only two bytes, and doesn't appear to affect programming code, I'm suspecting accidental corruption.

I have to say though, finding two bad dumps already ... it seems that the PAL set was verified a whole lot less than the US set has been. And I imagine the JP set will be full of even more surprises than the PAL one.


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sat Oct 08, 2016 5:52 am 
Offline

Joined: Mon Nov 10, 2008 3:09 pm
Posts: 431
byuu wrote:
[Correct]
SHA256: 450df78c9b7c92e9f8ce5c2ee0e1dbf939031c1e4f9e10c52c8d8f874364d1d6
0x122764: 0x00
0x179fd4: 0x0f

[Hacked/Corrupt]
SHA256: 16009eb74966d642a01a3eabd3e5364a006faff36e1ebaa306ff4e93ea8fda63
0x122764: 0x08
0x179fd4: 0x07[/code]


The first difference shows up as a very obvious stray pixel in an uncompressed sprite (between the character's legs on the right):

Image

The second difference looks like it's in some compressed graphics. I'm too busy right now to debug the game to figure out where it's loaded and what it is, or to find the correct palette for the bad uncompressed sprite.

Oh, and both bytes in your redump match the US ROM (which is much more similar to the Euro one than Maximum Carnage is)

Can you provide the PCB type and ROM labels (should be 2 ROMs based on the size) for this game?


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sat Oct 08, 2016 1:58 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1339
> The first difference shows up as a very obvious stray pixel in an uncompressed sprite

Ah, nice. Thanks for tracking this down so quickly!

> The second difference looks like it's in some compressed graphics.

That's certain to have a much bigger impact. As you and I know, bad data in a compressed stream can corrupt anything that comes after it.

> Oh, and both bytes in your redump match the US ROM

Ah, nice! I hate to say it, but my anxiety goes right to 11 every time I find these. I dumped this cart four times and cleaned all the contacts twice, making sure it wasn't a bad dump on my part. Even still, there's a lot of pressure because if I screw up even once in 2800+ game dumps, I'll never hear the end of it.

Although usually what I find is one of the address lines not registering so you get half the data duplicated and the resulting image won't even boot. Easy to spot/fix those cases.

> Can you provide the PCB type and ROM labels (should be 2 ROMs based on the size) for this game?

Sure thing! Here you go:

Image

(click for "holy shit why would anyone want a picture that huge?!" size. Proper 600DPI scans will follow in the near future.)

SHVC-2A0N-10 and SPAL-GN-0

This information for all of my dumps is also available here:

https://preservation.byuu.org/


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sat Oct 08, 2016 3:52 pm 
Offline

Joined: Mon Nov 10, 2008 3:09 pm
Posts: 431
byuu wrote:
That's certain to have a much bigger impact. As you and I know, bad data in a compressed stream can corrupt anything that comes after it.


Well, I set a read breakpoint on that address ($2F:9FD4) and played the game until it got hit, and it looks like the bad bit is in the background graphics for Duck King's stageone of Duck King's compressed sprites, but it's in a sequence of bytes that gets copied literally to the output stream, so it just translates to one miscolored pixel somewhere in the backgroundsprite, rather than throwing the decompression entirely out of whack.

ETA: whoops, I assumed that since there are uncompressed sprites in the ROM that all compressed graphics would be BG tiles, but actually the character sprites are a mixture of compressed and uncompressed. It looks like the compressed sprites are decompressed into WRAM at the start of a match and DMAed to VRAM on the fly from there, but "excess" sprites that don't fit in WRAM are stored uncompressed and DMAed directly from ROM to VRAM. So characters with a lot of moves (like the main heroes) have a larger portion of their sprites uncompressed than characters with fewer moves or simpler animations. That explains why some characters don't seem to have nearly enough sprites when you look through the ROM in a tile editor.


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sat Oct 08, 2016 9:35 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1339
I see. As always, I thank you kindly for your help here!

It's one thing to say, "there's a bad bit in this ROM file", and nobody really cares. But it's a lot more useful to say, "this graphic is corrupted due to this."

Here's hoping I don't find any more.


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 8:26 am 
Offline

Joined: Mon Nov 10, 2008 3:09 pm
Posts: 431
And, for the record, here's what the other bad bit looks like after decompression. Left is the bad PAL ROM, right is the US ROM which byuu's redump matches. I edited the palette's transparent color to make the bad pixel show up better:

Image


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 8:41 am 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19355
Location: NE Indiana, USA (NTSC)
And in case you already got a headache playing "spot the difference" in the comics in this week's Sunday newspaper:


Attachments:
findthediff.png
findthediff.png [ 1.3 KiB | Viewed 2688 times ]
Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 11:01 am 
Offline

Joined: Fri Oct 24, 2014 1:56 am
Posts: 75
Anyone want to test another "bad" dump. I've been redumping the Nintendo Power (SF Memory) carts and came across an interesting dump of Metal Max 2.

The normal Metal Max 2 has CRC 0x9516F39. I dumped a cart with Metal Max 2 and it contained the same revision but with a bad CRC 0xB542C0B6. I compared the ROMs and there was a 1-byte difference at 0x74569. The normal (good) Metal Max is 0x23 while the bad Metal Max is 0xA3. I assumed that my cart had somehow gotten corrupted over the years so I made note of the difference but basically ignored it. Several months later, another dumper shared their dump of Metal Max 2 and it contained the exact same bad byte!

The corrupted byte at 0x74569:
Code:
Good:  0x74569:  23
Bad:   0x74569:  A3


The two carts with the bad Metal Max 2 were programmed at two different Lawson stores at the end of 1998/beginning of 1999. I have a cart with the good Metal Max 2 that was programmed in September 1998. So sometime near the end of 1998, the data that Nintendo used for their programming kiosks got corrupted.

Code:
Good:  09/08/1998
Bad:   12/17/1998
Bad:   01/09/1999


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 11:31 am 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1339
AWJ, as I keep saying, this is a really bad idea:

https://github.com/mamedev/mame/commit/ ... 03a5b831ae

Code:
<dataarea name="rom" size="1572864">
  <rom name="spal-gn-0 p0.u1" size="1048576" crc="b48cfd3d" sha1="fc4910aa8dc8945f3d2402b737eb9e999424908f" offset="0x000000" />
  <rom name="spal-gn-0 p1.u2" size="524288"  crc="9df54331" sha1="36c5081a05353cc0b200b26b0d19f33057cb2ae7" offset="0x100000" />
</dataarea>


Here's why. Let's take the case of Street Fighter II:

NTSC 1: http://www.snescentral.com/pcb.php?id=0 ... side=front
NTSC 2: http://www.snescentral.com/pcb.php?id=0 ... side=front

PAL 1: http://www.snescentral.com/pcb.php?id=0 ... side=front
PAL 2: http://www.snescentral.com/pcb.php?id=0 ... side=front

This is really extremely common. About 70% of the time I have the same cart twice and open it up, it's a different PCB type. About 30% of those times, it's a 2 vs 1 ROM chip situation.

Unless you want hundreds of identical games (possibly more than 1000) in your database, or to omit a large part of history, this approach is no good.

I know why you do it for arcades. But home game cartridges are not arcade PCBs. A one size fits all approach results in hammering square pegs into round holes.

Also, it's 2016. It's probably time to drop CRC32 and SHA1 in favor of at least SHA2. Browsers have been trying to do that for years. SHA1 has very little life left in it. And due to the way hashes work, you want to update them *before* they get broken, not after.

> And, for the record, here's what the other bad bit looks like after decompression.

Damn you're good :D

Well, I guess we can see how this one went unnoticed, eh?

> Anyone want to test another "bad" dump. I've been redumping the Nintendo Power (SF Memory) carts and came across an interesting dump of Metal Max 2.

Compute the internal header checksum to see which one has a good checksum. That one is most likely the correct game image.

As a side note, I can't endorse cartridge dumps from flash memory. Not only is the data 20+ years old and highly vulnerable to bit rot, but nocash and others helped provide the documentation necessary to reprogram these carts.

It's possible if you buy a cart off eBay that it's been reprogrammed with a bad dump off the internet. So verifications of reprogrammable content are now worthless. The same goes for BS-X Satellaview, Nintendo Super System, and any other flash or EEPROM-based medium.

And as much as I hate to say it, it applies to my two very expensive EEPROM-based Super Famicom prize cartridges. But at least one is a unique gold cartridge with custom label, and the other has a custom label and box.


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 12:21 pm 
Offline

Joined: Fri Oct 24, 2014 1:56 am
Posts: 75
The normal Metal Max 2 with CRC 0x9516F39 is the good checksum. Two carts dumped independently with the exact same bad byte from a previously unknown dump is not bit rot or the result of someone recently reprogramming carts.

About half of my carts were purchased prior to our release of the reprogramming info. Suspected reprogrammed carts have only recently started to appear for sale on Yahoo! Japan. Along the lines of what you mentioned in the Nintendo Power thread, the suspect carts are multiples of the high value games from the same seller.

Take Care!


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 12:43 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19355
Location: NE Indiana, USA (NTSC)
byuu wrote:
Also, it's 2016. It's probably time to drop CRC32 and SHA1 in favor of at least SHA2. Browsers have been trying to do that for years. SHA1 has very little life left in it. And due to the way hashes work, you want to update them *before* they get broken, not after.

A "break" for a hash varies depending on the purpose for which the hash is used. Some attacks need only a collision (existence of two different dumps with the same hash), while others need an actual preimage (ability to create a bad dump with the same hash as a given good dump).


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 12:57 pm 
Offline

Joined: Mon Mar 27, 2006 5:23 pm
Posts: 1339
> Suspected reprogrammed carts have only recently started to appear for sale on Yahoo! Japan. Along the lines of what you mentioned in the Nintendo Power thread, the suspect carts are multiples of the high value games from the same seller.

Yeah. Zero surprise. We also have confirmed Nintendo Super System fakes now, including games that were never released on the system.

Thankfully most NP and BSX packs were already dumped, and the remaining ones are going to be never-released games so a forgery would be a hell of a lot more work. But unfortunately, the public audit logs (scans, documentation of who dumped what) is not available for them, and it's too late to do that now.

But, reprogramming was always possible, it was just harder without a nice blueprint in the form of fullsnes telling you how to do it. Worst case, you could desolder chips, reprogram, and put them back on the board.

> A "break" for a hash varies depending on the purpose for which the hash is used.

Of course. It's one thing to make a collision, and another to break CRC32 as badly as we have (hack the game all you want, then change four bytes to fix the sum.)

CRC32 is still fine for personal integrity, when you're not worried about collision attacks. I use that in formats like beat. One in four billion chance of corruption going undetected. Those odds are good enough for me.

But the reason to use SHA1 is security, and SHA1 is no longer suitable for that. If you care about that, use SHA2. If you don't, then lose the SHA1/MD5 sums that other sets use.


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 4:29 pm 
Offline

Joined: Sat Apr 25, 2015 1:47 pm
Posts: 337
Location: FL
skaman wrote:
Anyone want to test another "bad" dump. I've been redumping the Nintendo Power (SF Memory) carts and came across an interesting dump of Metal Max 2.


That's another difference in compressed data. Look at the space between the "E" and "T" in "metal", near the bottom of the letters:

ImageImage


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Sun Oct 09, 2016 6:53 pm 
Offline

Joined: Fri Oct 24, 2014 1:56 am
Posts: 75
That should make it easy to distinguish between the good and bad versions of Metal Max 2.

Let's see if I can find more of the bad versions without having to buy the carts.

Thanks Revenant!


Top
 Profile  
 
 Post subject: Re: Bad game dumps
PostPosted: Wed Oct 12, 2016 6:49 pm 
Offline
User avatar

Joined: Thu Jan 03, 2008 1:48 pm
Posts: 544
Just so these ROM images are preserved somewhere, are any of you open to being a facilitator for an archive.org (or similar) section?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group