C http library (Now works during regular gameplay!)

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

C http library (Now works during regular gameplay!)

Post by cppchriscpp »

Ever since I saw the ConnectedNES project, I've wanted to see if I could take that further. I finally got something working almost-reliably, so I thought I'd share. Announcing...

NESNet Library

Current Version: 0.0.4

Binaries: https://gh.nes.science/nesnet/releases (Bleeding edge CI images available from the README file in GitHub)
Source Code: https://gh.nes.science/nesnet

Description: NESNet is a neslib-compatible C library for the NES. Using the ConnectedNES hardware (Particle Photon) and a new firmware, it is able to make http requests to the internet.

It comes with a demo forum browser app, which is able to browse these forums from the NES itself. (There are also a few simpler demos)

There's much more information available on GitHub. Here's a code sample using it:

Code: Select all

int resCode;
char[16] theMessage;
// This triggers the get request, which will run in the background
http_get("http://yoursite.com/time.php", theMessage, 16);
// Now, we have to wait a while. You could do all of this in your game's logic loop instead.
while (!http_request_complete()) {
  // This method should be run once per frame. It does all of the web request stuff in the background.
  nesnet_do_cycle();
  ppu_wait_nmi();
}
if (http_response_code() == 200)
  // theMessage will now contain the first 16 chars of output from time.php on yoursite.com.

Current features:
- http GET/POST/PUT/DELETE requests
- Asynchronous requests while the game logic runs
- Usually doesn't respond with random errors. Usually.
- Multiple demo roms, including a NESDev forum browser


Demo video of the forum browser: https://youtu.be/wIXFcDRR34U
Demo video of asynchronous stuff: https://youtu.be/661rmHYVC0g
ezgif-1-59e2378fd0.gif
ezgif-2-724a26224a.gif
I realize the documentation is kinda ugly right now, but everything you need is in there. If you've got other questions, ask away!

Not actually an April fools joke. Probably was a bad idea to release this at 9pm the day before April fools, huh?
Last edited by cppchriscpp on Fri Jan 26, 2024 10:14 pm, edited 10 times in total.
lidnariq
Posts: 11429
Joined: Sun Apr 13, 2008 11:12 am

Re: NES http library for C

Post by lidnariq »

Video transcript:
Hello, my name is Chris, and today I would like to demonstrate for you a forum browser for the NES.

It is written in C, and uses two libraries, one known as neslib, and the other is relatively new and known as nesnet.

The first thing you might realize when I say that is that the NES was not built with any modem or any sort of internet-enabled hardware. You might question how that would be possible.

A while ago there was a project called ConnectedNES that was released by a woman known as Rachel, better known by her internet handle ? ? HXLNT. This device, combined with a special ROM would allow you to display tweets on your NES. As soon as I saw this, I knew that had to do something with this myself.

What you saw in the corner of the screen—I've blown it up a little bit now—is an original NES. It has been A/V modded, but is otherwise untouched, as well as a PowerPak, a screen, and the hardware from the ConnectedNES. It has a new firmware on it, but is otherwise identical to the original project.

On the NES, you'll notice that I just have this home screen for the forum browser. I'm going to go ahead and play the background music, which is After the Rain by Shiru, because I want some background music for this video.

If I press A, I will go to the forum list. Sometimes it's a little bit buggy. This is very timing based, so on occasion you'll still have problems. I'm hoping to iron out these issues eventually.

And I go through any forum I like, and I'm able to look at recent posts, and click into them. And I can see the most recent posts. Go ahead and view a couple of them. Pretty basic.

If I go back to the index, I can actually prove that this is completely live. I'll go ahead down here to the test forum where I can just throw a random post.

I'll go back to my browser for a bit. So there's our list of current posts. Let me bring up the forum itself.

Let me go ahead to that same forum over here. I'm going to create a new topic. Just make a nice quick one. "Testing NesNet. Hello from the internet!" Let's go ahead and just post this thing.

Going back to the NES itself, let me just refresh this page. Look: "Hello from the internet!" Get it open... and hey look, it's my post. If I want I can even go and reply to it. Should actually bring up my browser. I'm just going to post another one that says "Hi there!". Sent. And back to the NES itself. If we refresh ... we get a 404.

There's my second post that says "Hi there". If I press the down button, I'm able to get my original post back.

This is all done using a C library that is hopefully pretty easy to use. It's a little bare-bones right now, but it is completely available. Here's the C code that is behind this app, and there right here on the highlighted line is all that is needed to GET a web request.

I'm hoping that this could allow for some pretty interesting projects on the NES, so have fun, and I hope this is something that people want to use.
User avatar
Fisher
Posts: 1249
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: C http library (Browse the forums on the NES!)

Post by Fisher »

Would a YouTube viewer be possible by using this + libcaca?
Maybe it would need an external helper to live convert the videos and just throw ASCII data to the NES.
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

Probably not, unfortunately. The library is only going to support http get/post/put/delete - all of which are kind of a point-in-time sort of thing. For something like youtube you'd need something more designed for a continuous stream of data. You might be able to simulate it by getting 1 frame per get request, but with the speed of the library it would probably be pretty slow/ugly. You can definitely give it a shot; would be interested to see what happens.

If you're so inclined you could use the same hardware as this/ConnectedNES and write a new firmware to do something like that. This path would likely lead to better results.

Semi-related, If you haven't seen it yet, I think you'll like what the TASers at AGDQ 2017 did with a NES/SNES. I won't spoil it, but it's directly related to your interests. http://tasvideos.org/GamesDoneQuick/AGDQ2017.html (ps: If the first game doesn't really interest you, skip to about 17:00 with SMB2)
User avatar
Fisher
Posts: 1249
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: C http library (Browse the forums on the NES!)

Post by Fisher »

WOW!! :shock:
That's so crazy and awesome at the same time that's kind of scary!! :D
I really need to wath more TAS videos!

Maybe a cartridge can be made with RaspberryPi or an Ardruino and do the heavy job, just throwing the ASCII screen to the NES. The sound also will need even to go straight to the audio output or somehow be compressed to abuse the PCM channel.
Unfortunatelly, I won't have the time and resources to try this. :cry:
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: C http library (Browse the forums on the NES!)

Post by Myask »

Not to burst your bubble, but ACE TASes, particularly odd video-payload ones, are very much out of the ordinary.

At first, there was a slow, music+still image payload (recommend skipping load sequence, it takes a while): http://tasvideos.org/2187M.html
Then, there was a tiny character-animation payload, with song: http://tasvideos.org/2341M.html
The latest, however, is about up to the AGDQ TAS level. http://tasvideos.org/3358M.html
User avatar
pubby
Posts: 583
Joined: Thu Mar 31, 2016 11:15 am

Re: C http library (Browse the forums on the NES!)

Post by pubby »

Can this act as a server?
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

So no, this can't act as a server. I don't currently plan to add that functionality, though it's definitely possible with the hardware.

Can I ask what you would want server functionality for? I'm wondering if there might be a sane way to do it with the library now. (Or future iterations that support PUT/POST)
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

New version v0.0.2 out! Adds support for pretty much all commonly-used http methods. (get/post/put/delete) I have only tested put/post lightly - they seem to work reliably with small payloads.

Haven't gotten to the bottom of the URL sometimes corrupting, thus getting a bogus response. I suspect this will also affect put/post data, so be aware there's a chance post/put data could be corrupted until I figure it out.

Next steps will be stabilizing it a bit more, and probably cleaning up some of the documentation. I need to document the limits on various lengths better. (Eg urls cannot be more than 255 chars, post data can't be more than 1000 bytes, responses can't be over 500 bytes) I also probably need to tweak those - the photon has a lot of RAM; these limitations are all artificial. I also probably need to make another demo rom that shows off the new request types.
habstinat
Posts: 3
Joined: Wed Jul 31, 2013 10:45 am

Re: C http library (Browse the forums on the NES!)

Post by habstinat »

Hey cppchriscpp,

I bought a Photon and set it up with NESnet and it works! Tested both of your test ROMs in my PowerPak and was able to browse the forums and get random words.

I agree that I think it would be amazing to have either a webserver, or some sort of telnet server. I'll look into that eventually, but for now I'll definitely be playing around with the HTTP requests and make some sort of app of my own!
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

Hey- glad to hear it. I'll be really curious to see what you do with this, so please let me know!

Also, feel free to reach out if you run into issues. There's still a bug where the URL going to the photon sometimes gets corrupted - I don't quite understand why yet though. (Also if you end up looking at the photon source, I'm sorry. I'll clean it up some day...)

I'm participating in ludum dare this weekend, so I'll be somewhat scarce, but I'll have some time for this during the week.

Happy hacking!
User avatar
Bregalad
Posts: 8055
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Re: C http library (Browse the forums on the NES!)

Post by Bregalad »

Because, you mean, this wasn't an april fool ?! :shock:
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

Nope, completely serious project with a very poorly chosen release date!

Update: I don't have "real" news so I'll hijack my last post. I played around this a lot this weekend, and found a few things:
1) A lot of the delays I built in are completely unnecessary, and can be removed. This results in a huge speed increase.
2) This library is slooow. It's not reasonabe to expect to miss a couple frames and have data returned.
3) I can decrease the number of mistaken calls by breaking apart the logic with more waits for interrupts. I suspect I was trying to do too much per frame before.
4) I need to take over input handling to do it reliably. This means providing replacements for neslib's input routines so I don't trigger unexpected latches.
5) If I add a way to run code every frame and break code up well, I can make web requests while accepting input from p1 and updating sprites/etc with virtually no frame drops.

That said, I've realized I need to refactor this entire library a lot. I may release what I did this past weekend, but it's really not intuitive to use. I'm thinking of a new interface roughly like this:
- get/put/post/delete methods that initiate a request, and return immediately (Same input params as current)
- a method to call 1x per nmi (should be usable via straight assembly, for minimal overhead - as well as C for code simplicity) that will run whatever steps need to be taken for the current request, or just return if no request is in progress
- a method to call to check the status of the current request - has the last request completed yet?
- A method to get the status code of the last request (might merge with the status of the request - just say 0 = request in progress, for example.)
- One or more methods to get the state of the control pad w/o interrupting nesnet.



Not gonna lie - this is going to take a while to do. I'd probably need a free weekend to have a chance of getting this done. (And I don't have one of those for a little while... plus my attention's split between this and some potential improvements to Missing Lands) However - this rewrite should allow for games that make web requests in the background during regular play. The thing I did this weekend allowed me to semi-reliably fetch 8 bytes roughly 1x/second to put a sprite on the screen while also moving my own player around with the controller. I'm excited about the possibilities here.

I'll update once I get there, and if anyone really wants the half-done junk let me know. Worst case I can commit it to a branch for now.
habstinat
Posts: 3
Joined: Wed Jul 31, 2013 10:45 am

Re: C http library (Browse the forums on the NES!)

Post by habstinat »

Just saw the update -- that sounds fantastic. Would love to see the new changes committed to a branch if you have the time, even if they're incomplete for now.

I was trying to get nesnet.asm working in ca65 yesterday (as opposed to including nesnet.h in cc65), but I ran into some trouble. Granted I'm very new to nesdev, but after making some changes I'm getting compilation errors about "pushax" and "popax" being undefined, which I think depend on the C stack anyways.

Thanks for your work on this, it's very impressive.
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

Hey!

So my changes were actually made as part of a separate demo, and live in another git repo (That only lives on my pc for the moment) so it's not quite as simple as making a branch. NESNet is linked in, so I'd need to make a separate branch for that and setup would be a pain. That said... I don't mind sharing.

I grabbed the current source in the repo including a heavily doctored nesnet, and dumped it into the attached zip file. It's extremely rough, and doesn't line up with what I wrote above. There's a method you can give to nesnet to run every frame. (That's a very weird pattern, and the reason I don't want to release it just yet) All source that you care about is in hello_world.c as well as lib/nesnet_config.asm (NOT lib/nesnet/nesnet_config.asm)

It expects a server to be running at http://192.168.1.201:3000 with the nodejs code in the server directory. The url is defined at the top of hello_world.c - change it to the local ip of your pc, or run the nodejs app on a server. (Another thing I would want to polish!) All the server does is set a variable for the sprite's x position, then move it 8 pixels every time a request is made. If it's working, you'll see a character slowly move across your screen. Each position change is a completed http request!

Server setup should be as simple as installing node (> 4.x), doing an npm install, then doing an npm start.

I did also include a compiled version of the code as-is in hello_world.nes, just in case that's useful. The photon firmware doesn't need to be updated for this.

Hopefully this is enough to get you started if you want to play with it!
nesnet-world.zip
Tech demo!
(78.85 KiB) Downloaded 119 times
-----

About using the library from assembly, that's definitely something I haven't spent any time on yet. The pushax and popax stuff is to get the parameters being passed into the function. You could probably create a macro to get these to read from the 6502 stack (ie using the `pha` and `pla` instructions) - or just replacing that stuff with setting URL and URL+1 to the low and high bytes of the memory address of your url.

For a little more explanation on that, check out Shiru's documentation, here: https://shiru.untergrund.net/articles/p ... s_in_c.htm (Scroll down to " Writing functions in assembly code" to see what this is supposed to do.)

Last note - I'm hoping to find some time this weekend to play with this a little. Maybe I can get a new version out with some of these tweaks! No promises, but I'll see what I can do.

-----

Sneaky update #2:

I've been working on the newer version of the library that does asynchronous stuff better and so far I've been successful. I've ported http get requests to a more sane asynchronous format.

You call http_get as before (now a void) but it returns immediately. You call nesnet_do_cycle 1x/frame to trigger the library to do it's thing and you then poll http_request_complete() until it returns true. At that point you can get the status code with http_response_code(). (And your response data will be in the provided variable)

There's also a new nesnet_pad_poll method which you must use for all player input while web requests are running. Without it you risk corrupting the data the NES gets back. Taking this over seems to have helped get successful responses much more frequently.

Finally, there's a new demo in demos/character-movement that showcases the new asynchronous request types by having a character moved by a nodejs-based server. (The same demo I shared above, ported)

The existing photon firmware will still work - I don't actually anticipate having to change that much.

The documentation is still out of date, and post/put/delete requests don't work. You can check it out here: https://gh.nes.science/nesnet/tree/async - I'm hoping to round out some of the missing stuff and get this thing officially released in the next few days. Stay tuned! (I promise I'll actually make a new post when that happens, too ;))

5/26: ... and then life happened. I've been making incremental improvements but this weekend is completely shot for progress. Docs are mostly up to date, but I still need to fix the connection test. There is now a web-based demo too that lets you move a character around the screen on your pc, and see it on the NES. I want to apply real sprites to it soon. Sorry guys.
Last edited by cppchriscpp on Fri Jan 26, 2024 10:15 pm, edited 1 time in total.
Post Reply