C http library (Now works during regular gameplay!)

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Browse the forums on the NES!)

Post by cppchriscpp »

Finally got a new version of this out!

v0.0.3 is a breaking change - all interfaces to use the library changed slightly. This is because all requests are now done asynchronously - meaning you can make web requests during gameplay! First post and github updated with new example code and docs.

The get/put/post/delete methods now trigger a request and return immediately. After that you run nesnet_do_cycle() once per frame. Finally, you check if the request is complete with nesnet_request_complete(), and once it is, you use http_request_code() to get the status code. (200 OK, 404 Not Found, etc)

There is also a requirement that you don't poll inputs directly or using neslib. You should use neslib_pad_poll() instead. The reason for this is neslib takes over all input handling, so it can buffer network requests at safe times. If you don't do this, most web requests through the library will fail.

I think I've updated all relevant documentation and such, but if you see anything out of date or run into issues, please let me know.

I also created a new video demonstrating the asynchronous capabilities: https://www.youtube.com/watch?v=661rmHYVC0g

Enjoy!

Update: I snuck out a version v0.0.4 this weekend too - it functions mostly the same, but requires you to include a new configuration file (nesnet_config.asm) in your startup script.

This file lets you choose how many bytes to fetch per frame. You can bump this up if you're making something that requires getting large http responses. (I use this in the forum browser demo).

It should also be marginally faster than v0.0.3, as I doubled how fast we fetch data in general since that release. (1 byte/frame to 2)
User avatar
Memblers
Site Admin
Posts: 4044
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Re: C http library (Now works during regular gameplay!)

Post by Memblers »

chris, despite the fact that I haven't replied in this thread yet, I've actually been super-excited about the possibilities of this. When I saw that post in the test forum, I immediately did a google search and found the code repo before you even posted about it. :D

I've done some work on something with a similar goal, but it's extremely primitive. Though I'd LOVE to do a wireless peripheral, and I have played with the HC06 and ESP8266, the real killer for me is that to legally distribute it, I'd have to go through the FCC unintentional radiator testing process (and for Canada and EU, I have no idea..). While it's in the realm of possibility, it creates a massive pricing problem for something that will only sell in small numbers. It's almost possible to make it a dev-board type of thing where users buy their own module to plug in to it, but even that seems to be a legal minefield (like requiring soldering is OK, just a connector is not, advertising the wireless features not OK, as a breakout/devboard is OK, those are the impressions that that I got). So yeah for those reasons, I've been more focused on a wired solution. One that would be as cheap as possible, for that reason I think the low-end model could simply be a USB adapter, connecting the NES controller port to a PC. And I already build those, a crappy one for now but I do have plans to make a much nicer one eventually.

So I've been thinking about diving into your code to see if it would be appropriate for me to, at best, merge my project with yours, or at the minimum, aim for some degree of compatibility. I bought the Photon board a few weeks ago, but I haven't sat down with it to try it out. I've been kind of in the middle of a project-frenzy at the moment, trying to get several hardware projects done (all of them NES-related, of course) while battling feature-itis on the larger one.

Note that my coding background is only assembly and C (virtually nothing outside of stdlib and SDL), so that's why everything is kinda shoehorned into that. I'm fine with writing low-level stuff from scratch, but when it comes to networking, sockets, etc. it's all new to me. Tons of stuff I don't know. So it's likely that I'm useless to your project, but I don't know yet.

The goals I have now are the following:
1: FTP access. I do have this working, though I haven't written an actual file browser and the way I do it seems kind of lame. The PC-side program has a template for a session script. You add in the filename, write it to disk as a text file, then use system() to call ftp.exe and give it that session script. Then the PC uploads the file to the NES. I'm guessing that probably not portable to another OS (Win7 here), though maybe there's a similar way to do this with a cross-platform FTP client.
2: User account system. From a little bit of looking into this, it looks like going down a rabbit hole with security problems, etc. I'm pretty clueless. Google authentication seems like a decent option, they provide the API for Java. So I suppose then, the main program would be Java for that, and the bulk of the program (in C) be a DLL file. Seems kinda yucky.. I really should learn to use Java, it doesn't look bad.
3: SQL database. I need multiple NES's to be able to communicate and store data. I suppose a server program would be written, the NES games would communicate with that and it somehow arbitrates communication between NES's and also the database? Or alternately, a peer-to-peer type thing where the program on the local PC accesses the database and somehow communicates through that? I'm thinking the server is needed, this ideally wouldn't be the type of thing where you just drop off files and disconnect (if it was just that, I suppose FTP would be enough). Ideally, the players should to be able to interact with each other in (near) real-time. Once again, I'm pretty clueless. One game is already written for this, just missing the networking parts, so this definitely needs to happen somehow. :)

I would appreciate any input from anyone based on those 3 features I have in mind, or anything I overlooked or was unclear about. Am I on the right track, or not? I suppose it's possible on some level to replace that photon board with a PC, but is this different enough that I'm barking up the wrong tree? Should I instead just make my own thing and later think about some kind of compatibility layer/wrapper thing for the NES side?
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: C http library (Now works during regular gameplay!)

Post by cppchriscpp »

Memblers wrote:chris, despite the fact that I haven't replied in this thread yet, I've actually been super-excited about the possibilities of this. When I saw that post in the test forum, I immediately did a google search and found the code repo before you even posted about it. :D

I've done some work on something with a similar goal, but it's extremely primitive. Though I'd LOVE to do a wireless peripheral, and I have played with the HC06 and ESP8266, the real killer for me is that to legally distribute it, I'd have to go through the FCC unintentional radiator testing process (and for Canada and EU, I have no idea..). While it's in the realm of possibility, it creates a massive pricing problem for something that will only sell in small numbers. It's almost possible to make it a dev-board type of thing where users buy their own module to plug in to it, but even that seems to be a legal minefield (like requiring soldering is OK, just a connector is not, advertising the wireless features not OK, as a breakout/devboard is OK, those are the impressions that that I got). So yeah for those reasons, I've been more focused on a wired solution. One that would be as cheap as possible, for that reason I think the low-end model could simply be a USB adapter, connecting the NES controller port to a PC. And I already build those, a crappy one for now but I do have plans to make a much nicer one eventually.

So I've been thinking about diving into your code to see if it would be appropriate for me to, at best, merge my project with yours, or at the minimum, aim for some degree of compatibility. I bought the Photon board a few weeks ago, but I haven't sat down with it to try it out. I've been kind of in the middle of a project-frenzy at the moment, trying to get several hardware projects done (all of them NES-related, of course) while battling feature-itis on the larger one.

Note that my coding background is only assembly and C (virtually nothing outside of stdlib and SDL), so that's why everything is kinda shoehorned into that. I'm fine with writing low-level stuff from scratch, but when it comes to networking, sockets, etc. it's all new to me. Tons of stuff I don't know. So it's likely that I'm useless to your project, but I don't know yet.

The goals I have now are the following:
1: FTP access. I do have this working, though I haven't written an actual file browser and the way I do it seems kind of lame. The PC-side program has a template for a session script. You add in the filename, write it to disk as a text file, then use system() to call ftp.exe and give it that session script. Then the PC uploads the file to the NES. I'm guessing that probably not portable to another OS (Win7 here), though maybe there's a similar way to do this with a cross-platform FTP client.
2: User account system. From a little bit of looking into this, it looks like going down a rabbit hole with security problems, etc. I'm pretty clueless. Google authentication seems like a decent option, they provide the API for Java. So I suppose then, the main program would be Java for that, and the bulk of the program (in C) be a DLL file. Seems kinda yucky.. I really should learn to use Java, it doesn't look bad.
3: SQL database. I need multiple NES's to be able to communicate and store data. I suppose a server program would be written, the NES games would communicate with that and it somehow arbitrates communication between NES's and also the database? Or alternately, a peer-to-peer type thing where the program on the local PC accesses the database and somehow communicates through that? I'm thinking the server is needed, this ideally wouldn't be the type of thing where you just drop off files and disconnect (if it was just that, I suppose FTP would be enough). Ideally, the players should to be able to interact with each other in (near) real-time. Once again, I'm pretty clueless. One game is already written for this, just missing the networking parts, so this definitely needs to happen somehow. :)

I would appreciate any input from anyone based on those 3 features I have in mind, or anything I overlooked or was unclear about. Am I on the right track, or not? I suppose it's possible on some level to replace that photon board with a PC, but is this different enough that I'm barking up the wrong tree? Should I instead just make my own thing and later think about some kind of compatibility layer/wrapper thing for the NES side?
Thank you very much for the reply! I'm excited to see something interesting with networking done on the NES. (Using neslib or something else)

I will warn you that while I've been getting into lower-level code I'm still pretty green - you will see some pretty dumb choices in my assembly, and worse in my C++. (The photon library is basically one big TODO comment.) It also still isn't completely stable - requests still fail sometimes, and data gets corrupted. (I could really use some checksums or something..) That said, have at; I've released everything I have. (Or if something's missing I'll be happy to share) Also, I hear you on the project frenzy... always too much to do, and far too little time.


So.. onto your projects. I would definitely suggest you play with the photon, even if you ignore my library. The biggest thing I like about the photon is the support/community around it - you mentioned raw sockets and stuff like that, but in reality I didn't have to do any of that. There's a pretty solid http library I'm using here: https://github.com/nmattisson/HttpClient, and if you don't mind the GPL you can also do https really easily: https://github.com/glowfishAPI/httpsclient-particle. I haven't tried it, but there's an ftp library too: https://github.com/jychuah/ParticleFtpClient. All I really had to do was send bytes to/from the controller port bit-by-bit, then reassemble them on the other side.

Here's the parts that do networking from the photon, a whole 20 lines of code: https://gh.nes.science/nesnet/blob/mast ... t.ino#L210 -- it really is quite fantastic how simple the http library makes it.

For your goals,
1. You've got this one working, so it's probably worth sticking with something that works. I believe there's an ftp binary installed by default on most linux distros that behaves in a similar way to windows. (Might be true for mac os too) Porting that might be more feasible than you think. (And even if not, most people can get access to a Windows machine/at least a VM) You could figure out how to set up an ftp server on the photon (probably writing it from scratch), or set up something where you ftp the file to a server and set up an http server and have the NES download it, but unless there's something you want to fix by doing that, it doesn't seem worth it.
2. Yeah, security can be a nightmare. I think my biggest challenge is that using http like I am now, everything goes over the internet in plain text... which isn't good when you're sending a username/password. I want to do something with https, but I also don't want to relicense nesnet as GPL. (Maybe I can split out the photon firmware and put that under GPL though...) For users specifically, google auth may be a good option. I have been experimenting with auth0 myself, and they seem to have a few decent libraries. I have some half-formed ideas around giving each device an ID, and using public key cryptography to help some of this too, but i haven't figured out how to make that work.
3. I don't understand peer-to-peer networking at anything past a concept level, so I tend to opt for server-based solutions. I think for the case you described, a single server probably makes a lot more sense. I see two ways to go about it - an efficient way, and a slower/easier way.

The slower/easier way would be to set up a server, and have the clients make regular http requests containing that player's coordinates/other data, and the server responds with the coordinates of the other players. This is how I did that demo with 1 character on the NES and one on the browser. (Note: you might not need a database at all, depending on the language you use. There's no database involved with that demo, just a few variables in the server script.)

The major limitation to that approach is data size - nesnet isn't very efficient when you start sending more than a few bytes per request. You might be able to make this better by hacking the library to hard-code the URL on the photon, so you can immediately send coordinate data from the NES. (The requested URL is sent bit-by-bit through the controller port for every request - this is actually one of the slowest parts of the library.)


The more efficient option would be to build a custom firmware for the photon that accepts data in exactly the format you expect, then use TCP sockets (or make http requests to a known server, for that matter) from the photon to talk to a server that expects data in the same format. You could probably reuse some pieces of nesnet - particularly the 6502 assembly for sending bytes through the controller port, and the logic for transforming those back into meaningful strings. This avoids most/all of the overhead nesnet introduces.


Anyway... sorry if that was way too much text, but hopefully some of it was helpful. I'm really excited to see someone else playing with nes networking. (particularly someone who has contributed a lot to the nesdev scene) I really think there's potential for something interesting. Happy to be a sounding board for ideas going forward, too.
Post Reply