NES Cart DB down again

You can talk about almost anything that you want to on this board.

Moderator: Moderators

8bitGuru
Posts: 1
Joined: Sat Jul 09, 2016 7:06 pm

Re: NES Cart DB down again

Post by 8bitGuru »

Hi, I'm a long time lurker that just learned of this due to searching for answers as to why the database was down. I registered in order to post this actually. I have used your site for some time and would also love to help donate some funds ( Do you have a paypal registered email? Would that work for you?). I'm very sorry to hear about these unfortunate events, please do keep up your amazing work! :beer:
BootGod
Posts: 359
Joined: Wed Jul 13, 2005 3:14 pm

Re: NES Cart DB down again

Post by BootGod »

I got all the data back, everything was recovered thankfully. It's so close to being back online but I'm running into one snag and it's really got me stumped. Something is going seriously wrong with PHP. Very shortly after scripts start to execute, it will crash, restarting apache as well. In the apache error log I get either "zend_mm_heap corrupted" or apache will just terminate with some random error code. It has nothing to do with the script running, it will just crash at some random point unless the script is very short.

The most puzzling part is i'm using the exact same versions of everything that I was before, with the exact same configs from before so I'm at a loss as to why this is happening. Any ideas anyone?

The final damage was $390 for the recovery + new HD. My paypal address is prlacey_6942 AT hotmail DOT com if anyone wants to help out, I would definitely appreciate it!
dubcl
Posts: 2
Joined: Thu Jun 30, 2016 2:11 pm

Re: NES Cart DB down again

Post by dubcl »

Hi BootGod,

The error apparently is because a bug in php [1], reading more here [2] talk about set "USE_ZEND_ALLOC=0" on php config. Maybe you can try that, or try upgrade the *AMP install, obviously in a separate server to keep the original untouched.

Cheers.

[1] http://drupal.stackexchange.com/questio ... mmatically
[2] https://developers.oxwall.com/forum/topic/41865
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES Cart DB down again

Post by tepples »

I second the suggestion to try newer PHP (especially PHP 7) on a different server.
plgDavid
Posts: 27
Joined: Tue Aug 07, 2012 11:01 am

Re: NES Cart DB down again

Post by plgDavid »

BootGod wrote:The final damage was $390 for the recovery + new HD. My paypal address is prlacey_6942 AT hotmail DOT com if anyone wants to help out, I would definitely appreciate it!
Done!
mimeoNES
Posts: 6
Joined: Tue Jul 12, 2016 4:57 pm

Re: NES Cart DB down again

Post by mimeoNES »

Long time lurker here. Great news Bootgod!!! Can't wait for the site to be back up.

I was also able to put together a mini version of NESCartDB based off one of your XML files. It's def not as comprehensive as your site but I streamlined it for anyone looking for a quick donor list. Also it's mobile friendly.

http://mimeones.com/nescartdb

Hope you guys find it useful and suggestions are welcome!
BootGod
Posts: 359
Joined: Wed Jul 13, 2005 3:14 pm

Re: NES Cart DB down again

Post by BootGod »

dubcl wrote: The error apparently is because a bug in php [1], reading more here [2] talk about set "USE_ZEND_ALLOC=0" on php config. Maybe you can try that, or try upgrade the *AMP install, obviously in a separate server to keep the original untouched.
I did try that actually, but the server will still crash none-the-less, just without that error message :/
tepples wrote:I second the suggestion to try newer PHP (especially PHP 7) on a different server.
Easier said than done unfortunately. The code for the public version of the site is ancient and heavily makes use of the old magic quotes "feature" (I didn't know any better at the time). Currently site is using 5.2.9-2, magic quotes is still available I believe in 5.3.x but was removed after that. I did try popping in a version of 5.3 and with that, apache wouldn't even start up (and gave absolutely no indication as to why).

I had been working on a new version of the site that's much cleaner code-wise and uses a newer PHP and no magic quotes nonsense. The database structure for the new site is completely different than the old one and the data is way out of sync. Not looking forward when the day comes to re-sync them either! In any case, the new version is not ready to be made live.

I'm sure I'll figure out something to get old one back up and running though.

Also thanks everyone for donations so far! I'll be sure to give credit on the site once its back up and running.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES Cart DB down again

Post by tepples »

mimeoNES wrote:http://mimeones.com/nescartdb

Hope you guys find it useful and suggestions are welcome!
Permalinks please. Could you make result pages have URLs, either through fragment manipulation or through history manipulation? I wanted to link "list of carts you could use to make Wheel of Fortune" in this post, but the URL in the location bar still points to a blank form.
mr-volk
Posts: 18
Joined: Mon Oct 21, 2013 11:32 pm

Re: NES Cart DB down again

Post by mr-volk »

I can organize a mirror at my home server.
Do you need it?
zzo38
Posts: 1096
Joined: Mon Feb 07, 2011 12:46 pm

Re: NES Cart DB down again

Post by zzo38 »

Can the NES Cart data be made available as a SQLite database?
(Free Hero Mesh - FOSS puzzle game engine)
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES Cart DB down again

Post by tepples »

Or, if that's too much trouble, at least as a MySQL dump so that one of us can do the conversion as needed.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: NES Cart DB down again

Post by koitsu »

zzo38 wrote:Can the NES Cart data be made available as a SQLite database?
Please god no. The internal DB format tends to change and be non-backwards-compatible quite often. We deal with this mayhem in FreeBSD regularly (once a year usually), because the package tools rely on SQLite 3.x as a database.

Let me educate people: if you want to export a database into something that people can use themselves, what needs to be provided is a text-based format. I don't care if it's mysqldump with certain options disabled, XML with or without a DTD, or even a bloody CSV. UTF8 output would be great, but even ASCII (with \xXX for chars outside ASCII range) would be fine.

Also, people putting up "mirroring sites" of this: if you think you're just going to run some kind of recursive wget --mirror then I will remind you as the guy who ran Parodius that this is EXTREMELY RUDE AND UNRELIABLE. Don't do this. Instead, work with the site owner/admin (BootGod) and get something that doesn't scrape an entire site. There are TONS of ways to do this that don't destroy resources like a "website mirror" would.

I wish people would let BootGod get the thing back up and working first. He just spent US$390 out of his own pocket on data recovery. One step at a time, and please have some patience!
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: NES Cart DB down again

Post by Sik »

koitsu wrote:
zzo38 wrote:Can the NES Cart data be made available as a SQLite database?
Please god no. The internal DB format tends to change and be non-backwards-compatible quite often. We deal with this mayhem in FreeBSD regularly (once a year usually), because the package tools rely on SQLite 3.x as a database.
Ironically, they say that using SQLite's file format as your own is a good idea (going as far as stating "Content stored in an SQLite database is more likely to be recoverable decades in the future"). Either something is wrong on your end, or something is wrong on theirs. (・_・)

EDIT: also before I forget, text based formats (at least when they have newlines) have the advantage of being easy to compare with a diff, and even to merge two files that were updated separately. Doing that with a binary format is a pain in the ass.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: NES Cart DB down again

Post by koitsu »

Sik wrote:Ironically, they say that using SQLite's file format as your own is a good idea (going as far as stating "Content stored in an SQLite database is more likely to be recoverable decades in the future"). Either something is wrong on your end, or something is wrong on theirs. (・_・)
It's definitely not my end, not when I see commits coming across that mention such matters. The documentation you linked makes me chuckle, especially when you consider that to use said database, you need to use something that links in to their API -- which is regularly changing in incompatible ways. (Yes, that's not the file format, but it's equally as important)

The two "biggest" projects I know of which use SQLite exclusively are pkg and ircd-ratbox, and I have already seen several reports of "weird SQLite database corruption" with pkg (these PRs have gone completely unanswered/ignored, which never sits well with me in an open-source projkect) -- but I will say that bugs in pkg are certainly possible. It's also possible that the problems I read about were related to the filesystem used as the backing store; SQLite, unlike other databases, relies exclusively on filesystem-level locks (which on some filesystems, including NFS, are known to be somewhat problematic; don't get me started on lockf vs. flock vs. fcntl).

My point here is this: don't just copy around a binary database file, even with SQLite. Either use .dump or a CSV export.

And yeah, text is good for lots of reasons, and you covered one of several. :-)
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: NES Cart DB down again

Post by Sik »

koitsu wrote:especially when you consider that to use said database, you need to use something that links in to their API -- which is regularly changing in incompatible ways.
This seems to be true of all major libraries these days, sadly. One of the reasons I hate semantic versioning, people use it as an excuse to break old stuff that was otherwise working perfectly fine. When your library gets so cluttered that the only reasonable way to keep going forward is to break existing programs*, then maybe it just means the design is at its limit and you should consider starting a new library from scratch, with a new name, and leave the old one to be maintained by those who still rely on it.

I can see the idea behind semantic versioning but I'd like to get rid of its major version (effectively leaving just two numbers), which would leave no room for breaking backwards compatibility. People underestimate just how important it is to keep old things working.

*Ignoring those that were doing something that wasn't valid in the first place and hence were working just by pure luck =P
Locked