How can we keep our NES development resources around?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems.

Moderator: Moderators

cppchriscpp
Posts: 93
Joined: Fri Dec 27, 2013 4:28 pm

How can we keep our NES development resources around?

Post by cppchriscpp » Sat Sep 28, 2019 8:51 am

Recently in another topic some discussion started about preserving some of the learning resources people used to get started with NES development. This conversation stemmed from the links to some of the nerdy nights sound tutorials going offline.

This led me to question: what if the nerdy nights tutorials went offline? What if nintendoage had a catastrophic database failure, or someone missed paying a bill? Or what if the downloads for the tutorials were lost?

I ended up writing a quick-and-dirty mirror site for the nerdy nights tutorials (including sound) to help mitigate this risk. It's plain html/js, and I offer a zip download of the entire site (with tutorial zips, etc) for offline use and/or to bring the site back up if my host goes down.

http://nerdy-nights.nes.science

Nerdy Nights is probably the most commonly-cited resource, but I think most of us learned from a lot of other places too. (Such as nesdoug's tutorials, topics here, various texts on the wiki, and some other sites I'm a little afraid may have already gone offline)

So, I'm wondering: should we do something to try to preserve all of this? Assuming the answer is yes... what resources do we want to preserve, and how do we want to preserve them? The backup website approach could work, but the answer might be as simple as making sure everything is on archive.org too. Let's discuss!

-----

Update: It seems like there is some interest, especially with some other goings-on on the internet, so I've created http://archive.nes.science to serve this purpose. Feel free to post about more stuff that might fit here.
Last edited by cppchriscpp on Tue Oct 22, 2019 7:27 pm, edited 2 times in total.

Drag
Posts: 1327
Joined: Mon Sep 27, 2004 2:57 pm
Contact:

Re: How can we keep our NES development resources around?

Post by Drag » Sat Sep 28, 2019 9:50 am

The thing that terrifies me more is if NesDev as a whole were to go down, because there's over a decade's worth of information on this forum alone, and that's before you consider how indispensable the wiki is. :P

Really, redundancy is the way to go, whether it's in the form of backups or mirrors. If Nerdy Nights is such a handy tool for newcomers, why not host it here too?

There's also code repositories, such as Github or BitBucket, which can serve as a nice anchored place to host a project, Nerdy Nights could potentially live there too, which gains the benefit of discussion and multiple contributors.

tepples
Posts: 22161
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: How can we keep our NES development resources around?

Post by tepples » Sat Sep 28, 2019 10:09 am

I mirror the wiki but not (yet) the forums.

lidnariq
Posts: 9864
Joined: Sun Apr 13, 2008 11:12 am
Location: Seattle

Re: How can we keep our NES development resources around?

Post by lidnariq » Sat Sep 28, 2019 10:14 am

For 3rd party links I've cited on the wiki, I've started adding (commented out) links to backups of these 3rd party links using archive.is, and replacing the original link with the backup when I become aware that the original went away.

The Internet Archive has fairly good coverage of the wiki ("200000 URLs have been captured") and forums ("100000 URLs have been captured").

calima
Posts: 1246
Joined: Tue Oct 06, 2015 10:16 am

Re: How can we keep our NES development resources around?

Post by calima » Sat Sep 28, 2019 10:17 am

The archive's weakness being their retroactive robots.txt policy. A squatter buys the domain, says no crawling, and all the previous saved content is deleted by them.


cppchriscpp
Posts: 93
Joined: Fri Dec 27, 2013 4:28 pm

Re: How can we keep our NES development resources around?

Post by cppchriscpp » Sat Sep 28, 2019 8:48 pm

I should probably note that the site I made is on github, but I have it private while I do a little bit of cleanup. (The scripts I used to grab the content are also there and they're a bit ugly/unclear right now)

I fully plan to release it though!

User avatar
Memblers
Site Admin
Posts: 3890
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Re: How can we keep our NES development resources around?

Post by Memblers » Sat Sep 28, 2019 11:24 pm

I do make backups of the forum database, but maybe I should do it on a more regular schedule. If the worst were to ever happen, we would lose months, rather than decades of info. I probably don't have any backup at all of the old forum though. Originally this website, and my NES projects, were backed up to an LS-120 SuperDisk.

I actually did lose my earliest NES stuff. What's funny is that I had posted one of the ROMs I made on Usenet. Sometime later I went back there and asked "anybody have testr.nes?". And yep, someone replied with it attached.

User avatar
Bregalad
Posts: 7993
Joined: Fri Nov 12, 2004 2:49 pm
Location: Chexbres, VD, Switzerland

Re: How can we keep our NES development resources around?

Post by Bregalad » Sun Sep 29, 2019 11:55 pm

Drag wrote:The thing that terrifies me more is if NesDev as a whole were to go down, because there's over a decade's worth of information on this forum alone, and that's before you consider how indispensable the wiki is. :P
I was going to say the same, except it's closer to two decades than one, if you include the WWWthreads boards before 2004. Also the NESdev main page is basically an archive, while everything covered by it is covered in some other form by the wiki, it'll never be the same.

Considering the major issues with the server and the inability to contact WhoMan, this is really worrysome.

tepples
Posts: 22161
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: How can we keep our NES development resources around?

Post by tepples » Mon Sep 30, 2019 6:23 am

I've been trying to collect my own old one-off test ROMs in a GitHub repository called little things.

LocalH
Posts: 173
Joined: Thu Mar 02, 2006 12:30 pm

Re: How can we keep our NES development resources around?

Post by LocalH » Fri Oct 04, 2019 9:47 am

lidnariq wrote:For 3rd party links I've cited on the wiki, I've started adding (commented out) links to backups of these 3rd party links using archive.is
Slightly relevant, if a user happens to be using Cloudflare's 1.1.1.1 DNS resolver, these archived links will be unreachable (something to do with Cloudflare not supporting EDNS and archive.is treating such requests as invalid). I actually switched away from using 1.1.1.1 once I learned that.

8bitMicroGuy
Posts: 314
Joined: Sun Mar 08, 2015 12:23 pm
Location: Croatia

Re: How can we keep our NES development resources around?

Post by 8bitMicroGuy » Sun Oct 13, 2019 7:35 pm

I know some MySQL. A system administrator of forums.nesdev.com could do a full extraction of the topics, posts, forums, categories, public user profile data, avatars, uploads, etc., and export all of that into some sort of an XML/JSON file for each "page" (category view, (sub)forum view, topic view, profile view, etc.) and there would be a HTML+CSS+JS frontend for navigation.

I don't know about programming the frontend part XD I suck at CSS. But I could make an Windows EXE as a reader of those XML/JSON files. This could make it super easy for stuff like searching. Searching by multiple criteria such as must-have-exact-phrase or avoid-this-exact-phrase and such; or regular expressions or whatever. I think that that would be seriously cool as there wouldn't be that need of wasting a server's CPU resources just to find what you need.

Due to this being very retro, I say we stick to some Windows XP-times HTML/JS/CSS standards + browser security considered. Better yet, a Windows XP-compatible EXE application (MIT LICENSED!) for browsing that stuff (I'll have to code it so that it stays MIT licensed :3). Windows 10 isn't very sustainable in terms of privacy and such. Windows XP is the latest Microsoft OS that I deem good regarding retro. We could make it even more retro by making a CD/DVD with an Autorun running that EXE Viewer.

So if you have some old computer that's off-grid and you wanna snuggle up inside the comfort zone of offline life, you will be able to do so. This is how my school had these computer magazines with new software and hardware and games and fashion and there was always that CD/DVD which had freeware, demo, trial, shareware and other software as well as some other good resources.

And in these days of great data compression, compressing the plain text could be relatively easy! The zlib library could compress the data for the CD/DVD and the EXE could decompress it. Alternatively, if one doesn't have a Windows machine to run the EXE but Linux or BSD, the solution is to use wxwidgets to make the Viewer because wxwidgets can cross-compile to multiple platforms.

What do you think?

BUT! Before that, I wish to revisit my old posts to delete some things that I wish I hadn't posted XD

User avatar
dougeff
Posts: 2775
Joined: Fri May 08, 2015 7:17 pm
Location: DIGDUG
Contact:

Re: How can we keep our NES development resources around?

Post by dougeff » Fri Oct 18, 2019 6:56 am

cppchriscpp,

your mirror doesn't have week 8 and week 9.

http://nintendoage.com/forum/messagevie ... eadid=7155
nesdoug.com -- blog/tutorial on programming for the NES

tepples
Posts: 22161
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: How can we keep our NES development resources around?

Post by tepples » Fri Oct 18, 2019 6:07 pm

Yahoo! Groups is going away. This means the nesdev mailing list is going away, as it was hosted on ONElist, which was bought by eGroups, which was bought by Yahoo!.

NESdev BBS has gone through what I consider four generations:

BB0: Yahoo! Group.
BB1: wwwThreads
BB2: phpBB2 that existed prior to mid-2012
BB3: This one, which was seeded with a copy of BB2

As for their archive status:

BB0 is archived up to March 1999, but message history from then until February 2006 has not. Yahoo! will delete all message history in a few weeks.
BB1 and BB2 I may have archived in 2012 in preparation for the sunset of Parodius Networking.
BB3 I have completed a proof of concept archiving script (without attachments), and jsr2a03.com is hosting a mirror. I plan to add attachments before the end of October.

cppchriscpp
Posts: 93
Joined: Fri Dec 27, 2013 4:28 pm

Re: How can we keep our NES development resources around?

Post by cppchriscpp » Sat Oct 19, 2019 1:59 pm

Couple updates...

First off, the mirror now has chapters 8 and 9, as well as advanced chapter 4. NintendoAge has two directories of the topics, and one was a little out of date. Please take a look and let me know if anything else is missing. (May have to do a hard refresh/clear cache to get the latest version) Thanks for the tip!


Second, the yahoo groups thing is pretty interesting. Looks like there are some archiving utilities out there for it, such as this one: https://github.com/Frankkkkk/yahoo-group-archiver -- it might be worth running one of those and sticking it somewhere. (I might try to do this soon). It also might be nice to stick BB2 somewhere, though it seems like that might already be lost?


The mirror site for the forums is AWESOME and makes me feel way better. I'll probably throw a mirror on nes.science and try to set up some automatic updating.




Next, something rather scary happened today. I'm going to share a posting from the ex-NintendoAge discord (which has now renamed itself) Emphasis and bracketed text are mine

- snipping out some of the announcement, as this is very long -

[2:56 PM] Gloves: 4) NINTENDOAGE IS NOW CHANGING

I do not want to incite panic or anything like that, but we felt it important to let you all know. Despite what I said under #3 above, GoCollect's hands have been forced. As I warned them - NA as it is today is a ticking time bomb, waiting to explode. That explosion is now apparently imminent. GoCollect received a message from IBM that the server that it's [NintendoAge and companion sites] sitting on is going to be decommissioned due to it's age. They HAVE to move it. Their intention is to back up and move everything to a completely new forum software (again, as I have proposed to them before and was told "no").

I would suggest that, just in case, you go and back up anything important to you on NA. Any threads or info or whatever - go save them to your computer somehow. The general method would be to right-click a page and "Save as HTML" at least.
[2:56 PM] Gloves: 5) So are there now going to be two new sites?

Yes. One will be owned by GoCollect and one will be managed by members of the community. GoCollect continues to own the "NA" brand, while Video Game Sage will be comprised of former NA members.

- more content removed here -

7) What can I do?

Seriously, back up anything important to you off of NA.

Right now we need at the least a temporary home for our userbase. For now that is this Discord. We know that NA as it exists today is set to disappear at the END OF THIS MONTH. OCTOBER 30TH. The users are the most important part of the community, and so we encourage you to bring your friends from NA onto this Discord so we can continue the community. I hate the thought of the lights suddenly going out and people losing their internet home, but this is the situation we're most likely facing, so we better not take any chance if we want to preserve and resume our community the best we can.
So, direct and to the point, what information on NintendoAge, AtariAge, etc do we want to back up? I'm very willing to tweak my backup tools to take care of anything major we care about.

Post Reply