NES Cart DB down again

You can talk about almost anything that you want to on this board.

Moderator: Moderators

tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES Cart DB down again

Post by tepples »

What happens if you boot from Xubuntu on a USB flash drive and then try to mount the problematic drive read-only?
BootGod
Posts: 359
Joined: Wed Jul 13, 2005 3:14 pm

Re: NES Cart DB down again

Post by BootGod »

I haven't tried any linux based approaches, and I have very little experience with linux based OS's in general. If there is a ready to go image file that I can just drop on a USB drive or burn onto a disc, I guess it's worth a shot.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: NES Cart DB down again

Post by koitsu »

Purely some educational responses -- you've got your game plan (sending the primary to the $300 recovery place), stick with it. The tools a data recovery company use won't be "attach the drive to a Windows PC and let it go" (it may be "attach a drive to a Windows PC using PC-3000 and its controller and then begin seeing what the behaviour is". PC-3000 lets you control everything down to the SATA signalling (yes, the signalling!)).

1. Windows does a lot of bullshit I/O to a drive without giving the user any indication it's doing so. Linux, FreeBSD, etc. don't generally do this; they tend to issue ATA IDENTIFY to get drive parameters and characteristics, followed by reading LBA 0 (and deciding to read MBR or GPT), followed by what's called "tasting" (looking for partitions and partition identification), and that's mostly it. You can turn off "tasting" in a lot of situations (at least on FreeBSD). Rephrased: when dealing with misbehaving or questionable disks, you want as *little* I/O done to the drive by the OS as possible. With actual data recovery suites like PC-3000, basically no I/O is done to the drive until the operator tells it to. There's a reason for that. :-)

In short, Windows tends to give the user absolutely no indication (or way to find out) what it's *actually doing* under the hood. It sucks for troubleshooting. Furthermore, Windows also wants to do a lot of things "automatically" -- a common one is to prompt a user for drive reinitialisation (read: repartitioning + formatting) even sometimes in I/O timeout scenarios (varies per SATA chipset). It's very dangerous. Automatic CHKDSK or journal recovery is another one that can cause all sorts of problems once I/O to a drive starts working -- it's very hard to get your data back after that (this is one of several reasons why not to work on source/original material! -- No data recovery company does that).

2. The drive being put into an eSATA enclosure (and continuing to act questionably) is a good test, but usually this isn't necessary. In the recovery industry, they actually do the opposite -- take the drive out of such enclosures and work on it via whatever native I/O interface it has. Enclosures contain SATA/USB bridges most of the time (some of the SATA/eSATA ICs used are considered this because they actually have USB support, regardless if the interface is wired up or not), and those bridges tend to do very questionable things (they literally intercept every ATA command and payload sent to the drive, often filtering out some and/or rewriting several -- they also commonly lie about drive capacity (LBA count)). The worst of the bunch are ones which support RAID.

Regardless, you doing this as an additional test (to rule out interface problems) is understandable, so don't feel like your troubleshooting efforts for were naught.

3. Hot-swapping is another worthwhile technique, as long as you know the interface you're hooking it up to has proper hot-swap support (rephrased: just because something has a SATA port doesn't mean hot-swap will function correctly). I've actually done this on machines which don't have proper hot-swap backplanes and caused physical sparks when hooking up the SATA power connector (there are capacitors and other things involved in actual hot-swap backplanes that keep this from happening) followed by the machine abruptly powering off. Scary.

But with a proper hot-swap drive bay or backplane, yes, it's sometimes worth trying. Occasionally (read: rarely) I have gotten drives which wouldn't allow I/O on a cold boot to actually work this way (i.e. attach drive + SATA power cable, but no SATA data cable; wait 30-60 seconds, then attach SATA data cable).

4. The HDD activity light staying on just means the SATA chipset (either in the enclosure, or the one natively attached to your motherboard, depending on how you have the drive hooked up) hasn't been told to shut the LED off. In other words: an ATA command or series of commands have been issued to the controller, the controller has issued them to the drive, but the controller is still waiting for response for the CDB. The question then becomes "what CDB did the OS send to induce that?" For example, maybe reading LBA 0 works, but reading LBAs that make up the primary GPT area cause the drive to go catatonic -- in which case, you need to be able to tell the OS to skip reading the primary GPT and skip immediately to the secondary (at the end of the drive). It's just one of hundreds of possibilities of course.

Figuring out if a specific LBA is what causes a drive to go catatonic/lock up or not, or if it happens even on something like ATA IDENTIFY, is important. I've done recovery (see those links I posted before) where a drive would go catatonic on *any I/O* past about 5-6 seconds of being powered on, and I was able to find a very clever workaround which let me work around the problem and allow for 100% data recovery. I actually streamed it live too (it was several weeks ago so it's no longer on Twitch, sorry to say), so some friends/viewers found it interesting.

5. "SMART checks" inside of system BIOSes do not do actual verification of drive functionality. What they do is issue an ATA CDB to the drive that asks the drive itself for the "SMART health status" and the result is either "OK" or "BAD". I'm keeping it simple here. The drive itself decides this based on SMART attribute thresholds being crossed. It is NOT a drive test, nor is it the same as a SMART short/long/conveyance/selective test. All the BIOS setting does is inform you whether or not the "overall SMART health status" is OK or not, and this normally takes about, oh I don't know, a few milliseconds. SMART actually has tons of very useful features for testing and analysis, but they aren't something a BIOS makes use of. I have *extensive* familiarity with SMART, so I could talk about this for days. (I'm kind of the "go-to guy" on dslreports.com for disk and storage issues)

The good thing you should know: if the SMART health check took 2 minutes, then what that means SMART RETURN STATUS either a) took 2 full minutes to return successfully (which means it's VERY slow and that indicates something anomalous, but that I/O to the drive can/does work in some circumstances!) or b) took 2 full minutes to return and internally timed out and the BIOS didn't report such (very possible -- I can't tell you how many BIOS bugs I've found, heh...).

Really, a review of SMART attributes on the drive would be most helpful, but the attributes are -- ready? -- stored on the platters in a reserved area, so if one of those has gone bad, and the drive has problems with LBA reallocation within reserved areas, then this could actually block the drive from returning anything (until power-cycled).

Anyway, I think that covers that part of the "SMART check". I could go into more details but eh, it doesn't help much at this point (if I have the drive it'd be more definitive).

I will say this much, though: it's very likely your drive has a fully function/working firmware (guess where that's stored too, on most drives (but not all)? You guessed it: platters!), but that something like the reallocation table has become corrupted or gone bad in some way (this is what I dealt with on the earlier drive I mentioned where I got 100% recovery), or that there's a specific LBA/sector that the drive locks up when attempting to access (this is incredibly common).

Like I said, this post was purely for educational/learning purposes, it doesn't help get you any closer to getting the data back. The data recovery company, on the other hand, should take care of that for you. PLEASE LET US KNOW HOW IT GOES! I always want to know what data recovery services work for people and don't -- I like being able to refer people to one or more, especially for things I can't do recovery on (i.e. physical/mechanical problems).
User avatar
Fisher
Posts: 1249
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: NES Cart DB down again

Post by Fisher »

Pretty nice explanation koitsu!!
Could you suggest some readings about this subject? It's very interesting and I would like to know more.

Well, as a suggestion, I second the one that told you to try to boot Ubuntu and mount the drive that seems fine. It happened many times with me (Windows won't mount but Linux did) that's one of the reasons a left Windows a while ago. I also suggest you to try Ubuntu Mate, since it has a lighter graphical interface. Ubuntu has some problems with older hardware, if that's the case, but it's really easy to use!!

Other thing that I second is the use of Testdisk! Man, this program is excellent!! It saved me and some friends just many times :-) and there's a Windows version too, just make sure you get yours from http://www.cgsecurity.org, to be safe.

Another suggestion is to host Bootgod's site together with NesDev. Could this be possible? I think it would be nice to have bootgod.nesdev.com!! So Bootgod would be part of the staff (if he is not already).
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: NES Cart DB down again

Post by koitsu »

Fisher wrote:Pretty nice explanation koitsu!!
Could you suggest some readings about this subject? It's very interesting and I would like to know more.
Sadly there isn't much information of this sort "available to the general masses" for two reasons: 1) it's fairly low level (bordering on obscure), and 2) data recovery companies really don't like it when people give out anything even remotely close to their "secret sauce". I've been tempted to blog about the latter, because as I found out doing recovery myself, there are actual people in the data recovery industry who intentionally will try to sabotage your efforts (apparently including, in one case, a guy who sent his drive to a person who worked for a data recovery company to help him with the recovery, who proceeded to intentionally brick the drive so that the fellow *had* to pay for data recovery services), or bare minimum insult you to the point where you lose faith. The HDDGuru forum is filled with these people; it's why a couple of the more helpful and intelligent users created the HDDOracle site.

I got my knowledge via a couple decades of dealing with bad ATA and SCSI disks, examining behaviour of ATA and SCSI drivers (in FreeBSD mainly), talking to driver developers, and actually reading as much documentation as I could find (including T10 and T13 specifications for both protocols). I do not work in the data recovery industry. I'm just a UNIX SA who knows a *lot* about "everything storage".

Well, as a suggestion, I second the one that told you to try to boot Ubuntu and mount the drive that seems fine. It happened many times with me (Windows won't mount but Linux did) that's one of the reasons a left Windows a while ago. I also suggest you to try Ubuntu Mate, since it has a lighter graphical interface. Ubuntu has some problems with older hardware, if that's the case, but it's really easy to use!!
Fisher wrote:... Other thing that I second is the use of Testdisk! Man, this program is excellent!! It saved me and some friends just many times :-) and there's a Windows version too, just make sure you get yours from http://www.cgsecurity.org, to be safe.
I warned about this kind of (extremely bad) advice already in my earlier posts:
koitsu wrote: ... One thing *not* to do is run any "recovery software" on the origin/source drive, no matter what anyone online tells you -- trust me on this. It will very likely make the situation worse if you need to rely on a data recovery company.

... (this is one of several reasons why not to work on source/original material! -- No data recovery company does that).
TestDisk from CGSecurity will make modifications to a drive in attempt to do recovery. DO NOT USE THIS PROGRAM ON THE DRIVE WHICH IS MALFUNCTIONING, especially since the drive is going to a data recovery company! I cannot stress this enough! You **never** want to issue any writes (NOT EVEN ONE!) to a drive which is malfunctioning. All of these programs (examples: TestDisk, SpinRite, Recuva, ZAR, GetDataBack) will issue writes to the drive when trying to "recover" data (SpinRite is the worst of the bunch).

The only time you want to run these kinds of programs on a malfunctioning drive is if you have NO PLANS to use a data recovery company, have no problems with the strong possibility of things being made worse, or just want to do it for educational purposes. That's it. (Data point: I am a paid customer of ZAR and GetDataBack, and I do use these utilities from time to time when recovering NTFS/FAT data from drives which have already undergone being duplicated/copied).
Fisher wrote:Another suggestion is to host Bootgod's site together with NesDev. Could this be possible? I think it would be nice to have bootgod.nesdev.com!! So Bootgod would be part of the staff (if he is not already).
That could be made possible already by simply adding a DNS CNAME record (bootgod.nesdev.com IN CNAME bootgod.dyndns.org.). You'd still have to visit it via http://bootgod.nesdev.com:7777/ though because he runs the website on an alternate port. He also might have to modify his Apache configuration (depending on how he set it up to begin with) to allow Host: headers matching bootgod.nesdev.com.

But really, think about it for a moment: none of that really matters. People won't care if the site is at bootgod.nesdev.com or bootgodssupercooldb.supersnakesonaplane.net or anywhere else. My point is that sticking it under the nesdev.com domain just makes it "vanity", which is something that should've died in the early 2000s (people wanting hosting or hostnames under a specific domain because "it looks cool"). :P But yes, it's completely possible to do.

Furthermore, if he was to be hosted by 8bitalley (who hosts nesdev/etc.), that's putting even more eggs in one basket (it's all hosted on one machine, etc.). If this situation has taught us (as a group) anything, it's: 1) check and test your backups regularly (I recommend doing a full bare-metal restore once a year), and 2) have some form of redundancy in place -- or if not, at least have a "contingency plan" for situations like this (i.e. an alternate hosting provider to whom you can rsync a backup over to and get things back up and running in a day). I'd honestly recommend just doing hosting at a VPS provider and paying US$5-10/month for service, then doing backups by rsync'ing, and/or SSH+mysqldump'ing the data off the VPS (and storing it at home, and/or somewhere else) on a daily basis, or maybe even twice a day. That's what I do with my VPS, and have done for years (and yes I've had to recover a few times).
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES Cart DB down again

Post by tepples »

One reason that I make the offline HTML snapshots of wiki.nesdev.com is to test my own backup of the wiki. After this situation, I'd end up doing the same for a revived NesCartDB once XML dumps are made available regularly.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: NES Cart DB down again

Post by lidnariq »

I suppose it's worth asking if BootGod will be willing to share, or at least make distributed backups of, the sql database that was underneath.
User avatar
Fisher
Posts: 1249
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: NES Cart DB down again

Post by Fisher »

Oh yeah, living and learning Koitsu. :D
I've always suspected of many misleading information on these HDD forums... You confirmed my theory!
I thought that testdisk didn't save anything on the "sick" hard drive unless you tell it to do so. I'll take extra care from now since doing this is like trying to quench your thirst with french fries. :-)
Thanks for showing me all this!!

About Bootgod's destiny, I gave a suggestion and it seems possible to be done, I've seem the wiki defaced a couple of times and moments later it was ok again. This shows that Tepples' backup routines are working fine. In the end, it's up to Bootgod to decide what's better and what to do.
The name, as Koitsu said, doesn't matter that much, it's really the content that does!!
User avatar
ultrageranium
Posts: 16
Joined: Thu Aug 20, 2015 2:02 am

Re: NES Cart DB down again

Post by ultrageranium »

I was actually trying to access the NES Cart DB right now, as part of my slow learning about mappers, NES/FC PCBs and cart making. After searching a bit I ended up back on this forum, reading the bad news.

It seems to be a very valuable resource for the NesDev community, and I'm happy to donate a bit if this recovery service works out.

As for the rest, and to paraphrase others, redundancy of such material should be something to think about once everything is back online. Web scraping is good, but ideally if that's possible, the source files and database dump could be made accessible via http for others to pull, maybe considering something like git. There are many other options (rsync server would indeed make it simpler than a DVCS repos and optimise bandwidth usage).

Regarding hosting, OVH offers a barebones service called kimsufi that offers dedicated servers starting 5€. Pros: 500GB disk, 5TB traffic, excellent uptime. Cons: you're on your own, the 5€ box is not very powerful (but FreeBSD runs wonderfully on it).

Good luck!
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: NES Cart DB down again

Post by Myask »

Careful use of Google's cache can get you at some of the data, just not the images, if you know exactly what to look for. But, it's very inconvenient. I eagerly await it coming back online.
math
Posts: 1
Joined: Tue Jun 14, 2016 2:10 pm

Re: NES Cart DB down again

Post by math »

Hi bootgod,

I have sent you a PM. I can offer you a free VPS hosted in the Netherlands. Let me know if you are interested.

Regards,

Mathijs
BootGod
Posts: 359
Joined: Wed Jul 13, 2005 3:14 pm

Re: NES Cart DB down again

Post by BootGod »

Just wanted to post an update, I didn't manage to ship out drive yet before I left for a long overdue vacation. I just got back tonight, I will get it shipped out for sure in the next couple days. I've gotten floods of support for both hosting and for helping out with recovery costs. I really appreciate all of the community support both here and from other sites.

Rest assured, once this gets straightened out, I will make sure others have a way to get backups of the database so if something like this was to happen again, multiple people would be in a position to recover it.

People have often requested an offline version of the site but I've never quite known how exactly to go about that. Of course it would be incredibly easy to make static versions of the profile pages, but how do you tie it all together? I feel like the most useful aspect of the site is the extensive search capabilities and results formatting and all of this stuff is done server-side.

Anyways, once I hear back from recovery service I will post an update!

Thank you everyone for your patience!
BootGod
Posts: 359
Joined: Wed Jul 13, 2005 3:14 pm

Re: NES Cart DB down again

Post by BootGod »

Excellent news today! I sent in the drive earlier this week, they received it yesterday, and already today they are telling me they can recover 100% of what I need and sent me a file manifest of the contents of the drive and it looks like everything is there! So long as the transfer goes smooth, I would expect to have the data back by mid next week!
User avatar
Dwedit
Posts: 4924
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: NES Cart DB down again

Post by Dwedit »

Client-side or static page stuff is possible with javascript, it's just that a big database file would need to be sent to the client to filter it or search it. It might be possible to reduce the data transfer needed.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!
plgDavid
Posts: 27
Joined: Tue Aug 07, 2012 11:01 am

Re: NES Cart DB down again

Post by plgDavid »

BootGod wrote:Excellent news today! I sent in the drive earlier this week, they received it yesterday, and already today they are telling me they can recover 100% of what I need and sent me a file manifest of the contents of the drive and it looks like everything is there! So long as the transfer goes smooth, I would expect to have the data back by mid next week!
Hello BootGod.

Is there a way to donate some funds? Paypal preferably? I've been depending on your database for years now! Its one of the most well made and thorough tech resources I know of of. Again hats off to you!.
Locked