NES to USB Keyboard interface?

Discuss hardware-related topics, such as development cartridges, CopyNES, PowerPak, EPROMs, or whatever.

Moderator: Moderators

emerson
Posts: 34
Joined: Fri Nov 18, 2016 7:29 am

Re: NES to USB Keyboard interface?

Post by emerson »

As promised, here is my approach to interfacing a ps/2 keyboard (and ps/2 mouse) to an NES. A dip switch is used to configure the adapter to either emulate the Family Basic keyboard, or provide full enahnced keyboard and mouse support. More details are in the ReadMe. I have a few more ideas I would like to implement in this design, so if I continue with this project Ill start a new development thread.

I currently can't test the Family Basic keyboard emulation with anything but the test rom I wrote, so if someone else builds this, could you please let me know if it works or not?

A brief description taken from the ReadMe:
-Support for all 3 keyboard scan codes
-Automatic scroll wheel detection for mouse
-Power led error blink codes
-Serial debug data available
-Adapter is automatically reset when console reset is pressed
-Save/Load functionality with PC compatibility
Attachments
nes_keyboard_adapter.7z
(321.28 KiB) Downloaded 379 times
User avatar
bazza
Posts: 94
Joined: Fri Nov 24, 2017 1:36 pm
Location: Argentina
Contact:

Re: NES to USB Keyboard interface?

Post by bazza »

emerson wrote:As promised, here is my approach to interfacing a ps/2 keyboard (and ps/2 mouse) to an NES. A dip switch is used to configure the adapter to either emulate the Family Basic keyboard, or provide full enahnced keyboard and mouse support. More details are in the ReadMe. I have a few more ideas I would like to implement in this design, so if I continue with this project Ill start a new development thread.

I currently can't test the Family Basic keyboard emulation with anything but the test rom I wrote, so if someone else builds this, could you please let me know if it works or not?

A brief description taken from the ReadMe:
-Support for all 3 keyboard scan codes
-Automatic scroll wheel detection for mouse
-Power led error blink codes
-Serial debug data available
-Adapter is automatically reset when console reset is pressed
-Save/Load functionality with PC compatibility
Is posible run in arduino?
emerson
Posts: 34
Joined: Fri Nov 18, 2016 7:29 am

Re: NES to USB Keyboard interface?

Post by emerson »

Unfortunately it's not arduino compatible. You would have to start from scratch to make it work.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

I don't have the hardware to try it out but that's pretty cool! So it's a PIC based PS/2 to Family Basic converter? + some extra protocol to pass through some mouse data and the original PS/2 key stream?
emerson
Posts: 34
Joined: Fri Nov 18, 2016 7:29 am

Re: NES to USB Keyboard interface?

Post by emerson »

rainwarrior wrote:I don't have the hardware to try it out but that's pretty cool! So it's a PIC based PS/2 to Family Basic converter? + some extra protocol to pass through some mouse data and the original PS/2 key stream?
Basically yes... It has two modes of operation. Mode 1 is strictly a Family Basic converter. Mode 2 passes through the ps/2 keyboard scan codes along with the ps/2 mouse values.

Somehow I forgot to include some crucial information in the ReadMe about how the "Mode 2" function actually works. Sorry about that... Off the top of my head, you should know that when in "Mode 2", the PIC will convert all scan codes, regardless of config switch settings, to scan code 3 before sending the key values to the NES. This is because of its simple byte structure. If keyboard is forced to scan code 3 via the config switches, typematic repeat is disabled. If the mouse is enabled via the config switch, the mouse is set to report mode, and responds with the changes in xy position relative to its last read.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

So, here's a tentative protocol/spec that I've had on paper for a while. Haven't yet implemented it in hardware, and am still in very early experiments, but maybe worth sharing just to get feedback or see what others are up to.

Edit 1: extended report from 3 to 4 bytes (assuming 32-bit shift register is just as easy as 24-bit to build). Added scroll wheel and relative coordinates as an option. Added note about variability of hardware implementation. Edit 3: dropped scroll wheel to just 4 bits.

Connections, intended to plug into the second controller port (NES) or through the expansion port (Famicom), via $4017:
  • D0 - standard controller pass through
  • D3 - keyboard report (4 bytes)
  • D4 - mouse/status report (4 bytes)
The protocol is that you strobe, then read 4 bytes from D3+D4. Plugging in a standard controller to D0 is optional, as are either keyboard or mouse (indicated in status report).

D3 is just a buffer of scancode events, exactly what you'd get from a PS/2 interface, so the host can just collect these and deliver them up to 4 at a time. If there are less than 4 bytes to send, put $00 in the unused ones.

D4 would have one status/ID byte, and then X/Y position. A fourth byte for the middle wheel, if you've got one:

Code: Select all

7 bits 0 (MSB first) - 1st byte status
--------
LRMKS110
.....110 - Device ID
....S... - 0 for absolute coordinates, 1 for signed mickeys
..MK.... - Mouse or Keyboard present
LR...... - Mouse buttons

7 bits 0 - 4th byte wheel
--------
3SWWW...
3....... - Third button
.SWWW... - 4-bit signed mickey for scroll wheel (two's complement)
.....xxx - Reserved/unused.
Following this is X then Y position. Each is one byte, absolute pixel position. This makes it very simple to get a screen position for the mouse, and obviates the problems of dealing with relative position / mickeys, speed differences of timing (polling too much, too little), gives the host device option to implement sensitivity / scaling. (Y can report >= 240 if desired, but the NES application can decide how to handle that.)

Absolute positioning makes it very easy/unambiguous for an emulator to translate existing mouse coordinates into this, and also eliminates the need to capture the mouse and could allow better multi-tasking with the emulator, i.e. similar to how a lot of VMs have host integration, you could seamlessly move the mouse in and out of the emulator window and interact with it while doing other things.

Hardware:

I would expect to implement this with something like an Arduino or RPi acting as a USB host, and 2 x 4-byte shift register buffers.

After strobe, the host should wait for some reasonable delay for the NES to unload the shift registers (1-2ms?), then refill the buffers automatically and wait for the next strobe. That way, the microcontroller/host has time to prepare between polls, and can fill the shift registers serially. (Experience with the SNES mouse's "speed limit" kinda clued me into this need.)

With the keyboard, the host will queue up keyboard event scancodes as received and just dump them into the shift registers if there are any pending after a poll. Extended key reports might split across more than one poll, but on the NES side we should just be processing this queue one byte at a time: if we're halfway through an extended report, it can wait for the next frame for the next byte. After the first $00 byte in a keyboard report, all further bytes should be $00 (ignored) as well. (Multi-byte reports must be in-order but could be delivered in 1 byte per poll, interspersed with $00s this way, so even a device with only an 8-bit register could still serve compatibly if needed.)

The intent is to poll once per frame. Not sure how long to spec the refill period after the poll, which depends on the implementation, but 10+ms remaining in a frame after a poll is probably an eternity here. Presumably it could be polled more often, but I don't think anyone needs this.

There would be a 1 poll latency on the keyboard and mouse, but I think this is negligible/reasonable for this purpose. (If we need absolute lowest latency for gaming purpose, the controller is still there, of course.)


Edit 1: Implementations aren't required to implement everything. The 4th keyboard byte could be all zeroes for a simpler mouse with no button 3 or scroll wheel. Relative mode (or absolute mode) could be omitted. A device can provide mouse and not keyboard, or vice versa. If device configuration is needed (sensitivity, reset, etc.) mechanisms for that should be built into the controller device in some way.

Keyboard scancodes would suffer from the usual regionality problem, however, since there is a host device in between the keyboard and the NES, an implementation could remap them. If it's cheaper, the hardware implementer could ignore this problem and just expect the user to have the right region's keyboard. ...or the NES program could keep multiple scancode maps. This old problem left intact, because it keeps 10 pages of key mappings out of the spec, and leaves the door open for some more lightweight/cheap implementations.


Edit 2: Memblers published a very good adaptation of the USB scancode set on the next page: NES keyboard DRAFT. I'd suggest standardizing on that.
Last edited by rainwarrior on Wed Apr 08, 2020 12:00 pm, edited 13 times in total.
calima
Posts: 1745
Joined: Tue Oct 06, 2015 10:16 am

Re: NES to USB Keyboard interface?

Post by calima »

Absolute controls need much external interfacing, you need a way to reset, calibrate... Otherwise it'd be like VMs with a mismatched mouse integration. You may also want to include a 4th mouse byte for the wheel and third button.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

calima wrote: Mon Mar 30, 2020 2:07 amAbsolute controls need much external interfacing, you need a way to reset, calibrate... Otherwise it'd be like VMs with a mismatched mouse integration. You may also want to include a 4th mouse byte for the wheel and third button.
Well, yes I think any calibration like that is best kept external (part of your USB host device) and not something the NES program should have to deal with. If you need a reset button or a speed control or other specific-hardware-dependent controls, put it on that box. (Or... maybe most practically, it can reserve special keys on the keyboard for these functions.)

(Similar in the way the Hyperkin's on-mouse speed button is much more useful to a user than SNES mouse's software protocol.)

The comparison to VMs is weird to me though. If we're talking about an NES emulator, it already has an operating system to deal with calibrating the hardware, so this is a complete non-issue for those? It's only a problem for the hardware to NES interface, and that isn't VM-like at all? The mouse would be dedicated in that case.

On the part of keyboards, for regional scancode mismatch at least the host device has an advantage of being able to remap them if desired by the implementation. (...and cheaper implementations can leave you to find a regional matching keyboard instead. There's a variety of options here.)


As for putting a 4th byte for a wheel / 3rd button, as long as all 0s means no press/wheel (i.e. applications that don't need it can ignore it), that's probably fine. It seems pretty germane as an optional extra report.

So... 1st bit reports button 3, remaining 7 bits is a signed number of wheel mickeys? 7 bits feels like overkill for that, but anything else feels pretty arbitrary.


Actually, I suppose a 32-bit serial shift register is more normal than a 24-bit one isn't it? I haven't really looked at parts for this. Maybe extend both the keyboard and mouse reports to be 32-bit if that's the case. 4 bytes each?
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES to USB Keyboard interface?

Post by tepples »

If you need a reset button or a speed control or other specific-hardware-dependent controls, put it on that box.
What sort of controls would an interface box need? I can think of at least buttons to increase and decrease sensitivity.
(Or... maybe most practically, it can reserve special keys on the keyboard for these functions.)
Which would confuse the player when the game asks the player to press a particular key, but a particular interface adapter has reserved that key for sending configuration commands to the interface adapter. Or if the user is using a game that uses the mouse and does not use the keyboard, such as someone using a paint program or a shooting game, should the user have to turn off the NES Control Deck, find a keyboard, and connect it just to be able to configure the mouse?
The comparison to VMs is weird to me though. If we're talking about an NES emulator, it already has an operating system to deal with calibrating the hardware, so this is a complete non-issue for those? It's only a problem for the hardware to NES interface, and that isn't VM-like at all? The mouse would be dedicated in that case.
Are you familiar with the difference in behavior of a VM when it it set to capture the mouse compared to when it is set not to capture the mouse?

Using the mouse for something other than pointing, especially to turn a game object to the left or right, doesn't work so well if the mouse is already at the far left or far right of the range. For example, think of the use of the Super NES Mouse in the games On the Ball, Jurassic Park, and Doom, which rotate the player character. Or think of use of the mouse in Marble Madness, where the trackball imparts acceleration to the player character. In these cases, if the user were to move the cursor to the left when the adapter thinks the cursor is at x=0, to the right when the adapter thinks the cursor is at x=255, up when the adapter thinks the cursor is at y=0, or down when the adapter thinks the cursor is at y=255, the game will not be able to tell that the mouse was moved. Or would the interface box need to provide a button to let the user enable or disable a mode where the position horizontally or vertically wraps around?

Even using the mouse to control a cursor on the screen doesn't help when the cursor's range isn't the full screen. Consider the title screen of Thwaite. The boundary in which the mouse can move is not 256x256 pixels; it's much less tall. Even during gameplay, the cursor can't enter the top overscan or approach within a certain distance of the ground to protect the houses. Or think of the use of the mouse in the cannon placement phase or build phase of Rampart, where each player's cursor cannot cross the river to the opponent's territory. The behavior that the user expects when moving the cursor toward and away from the boundary is that movement into the boundary is clamped, but movement away from the boundary takes effect immediately. It's possible to simulate this behavior from an absolute mouse interface, much as Arkanoid does for its controller, but it's not any simpler. A game would have to convert the absolute readings to relative displacements, add the relative displacements to the cursor position, and then clamp the cursor position.

Your objective of resiliency to rereading without dropping mickeys would be served well by just accepting the absolute-to-relative-to-absolute conversion model and allowing the (x, y) coordinates to wrap within [0, 255) range.
calima
Posts: 1745
Joined: Tue Oct 06, 2015 10:16 am

Re: NES to USB Keyboard interface?

Post by calima »

The VM comparison was what first came to mind for absolute vs relative mouse issues. Surely everyone who's used VMs more than a few times has seen it.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

tepples wrote: Mon Mar 30, 2020 10:07 am
(Or... maybe most practically, it can reserve special keys on the keyboard for these functions.)
Which would confuse the player when the game asks the player to press a particular key, but a particular interface adapter has reserved that key for sending configuration commands to the interface adapter. Or...
So if you need the "perfect" one you can build an expensive box with like some dedicated configuration console just so you can be guaranteed to have every single key on your keyboard available. I dunno. I don't really care. Having absolutely every possible key combo available is a pointless goal. Non-issue.

It's just a cheap suggested way to squeeze out a few extra controls that has practically zero added cost. Just pick an unlikely key combo (e.g. Ctrl + Win + Up). Lots of keyboards on the market actually have special controls like this that you'd never notice without reading the manual (...or sometimes not even in the manual). If you don't like this solution, build the box differently. The NES side doesn't need to know about this. (...please don't let your next comment be "but what about supporting keyboards that don't have a Win key?")
tepples wrote: Mon Mar 30, 2020 10:07 amShould the user have to turn off the NES Control Deck, find a keyboard, and connect it just to be able to configure the mouse?
Why would you have to turn off the NES to configure the input host box? This is the same reason I think that stuff shouldn't be on the NES-side at all. The NES program doesn't need to know about device configuration, and shouldn't. If you don't like my suggestion that a keyboard could cheaply configure a mouse in this situation, then build something else for it. Can the USB host not handle hotplugging a keyboard? Doesn't matter to the NES.
tepples wrote: Mon Mar 30, 2020 10:07 am(Captured mouse vs. non-captured mouse.)
This isn't intended to do every possible thing under the sun. My primary aim is to make GUI interfaces easy and practical, as well as ease of integration with emulators. This is about being able to point and click and multitask. Dropping the possibilities of relative motion helps this, IMO. More or less every GUI API does screen or window relative coordinates already, for very good reason.

I want a point and click, a touch screen, not a dedicated capture. I want an emulator to be able to respond to the mouse like just another window on my desktop.

Another example is Pico 8, which made a similar choice with its mouse controls. It works great, and plays extremely well embedded in a webpage in a multitasking environment. There's tons of examples of this.

Yeah it won't work like a Marble Madness trackball. It's not a marble madness trackball. Yeah, it's not a drop-in replacement for SNES' Jurassic Park controls... but even both of these game types have very practical control alternatives.

Thwaite's cut off title screen was designed around relative controls. There's nothing essential about it having to be cut off, and that title screen would have worked perfectly well with other designs. There's plenty practical game applications for absolute coordinates.

However, compromise:

Put a switch on your box to change the absolute pixels to signed mickey bytes. Done. You could even take one of the status bits to indicate it if you like. Basically becomes two different device IDs I guess... but detectable at runtime.

If you're cheap and only care about one mode or the other, just build a device that only does one of the two. :P

Has the limitation that an NES program must really be designed for one or the other mode, and not be able to switch by itself... but that's the way it should be. In an emulated host environment we have to go from uncaptured to captured for that, or just always be captured... much better to make that an explicit user choice.

For playing on a real NES, yes we're essentially always captured to begin with, but I want something suitable for both kinds of environments, and that's absolute coordinates.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: NES to USB Keyboard interface?

Post by tepples »

rainwarrior wrote: Mon Mar 30, 2020 12:15 pm
tepples wrote: Mon Mar 30, 2020 10:07 amShould the user have to turn off the NES Control Deck, find a keyboard, and connect it just to be able to configure the mouse?
Why would you have to turn off the NES to configure the input host box?
I had in mind a host box that didn't fully support hot swapping. I must have had my own negative experience with the consequences of hot swapping PS/2 peripherals on the brain.
rainwarrior wrote: Mon Mar 30, 2020 12:15 pm I want a point and click, a touch screen, not a dedicated capture. I want an emulator to be able to respond to the mouse like just another window on my desktop.
Now I'm starting to understand your goal here. You're not trying to reflect a mouse's functionality as much as a tablet's. It's an NES counterpart to the existing Oeka Kids tablet protocol, just not quite as similar as the Famicom and NES versions of the Arkanoid controller are to each other.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

tepples wrote: Mon Mar 30, 2020 6:00 pmNow I'm starting to understand your goal here. You're not trying to reflect a mouse's functionality as much as a tablet's.
I have no actual desire to use a tablet for NES here, the primary goal with absolute coordinates is multitasking and ease of GUI implementation. However, games designed for tablet are a good source of ideas for how to use this type of control. (...as are many games designed for web browser, or windowed interfaces.)

How does the relative mode proposal suit you though? IMO it doesn't add any burden to the spec to repurpose one of the device ID bits for that. (Probably wouldn't be hard to include in the hardware implementation too, even though it's not the mode I want to use.)
tepples wrote: Mon Mar 30, 2020 6:00 pmI had in mind a host box that didn't fully support hot swapping. I must have had my own negative experience with the consequences of hot swapping PS/2 peripherals on the brain.
I suppose that this concept could be implemented via parallel port with an old PC if you'd like. ;P
User avatar
Memblers
Site Admin
Posts: 4044
Joined: Mon Sep 20, 2004 6:04 am
Location: Indianapolis
Contact:

Re: NES to USB Keyboard interface?

Post by Memblers »

IBM defined 3 sets of scancodes, which one should be used? There is also the USB set, defined by Microsoft. Which standard to use was a question I've also had, and I haven't started on any of the keyboard code yet.

My plan is to have a host program work on windows, linux, mac, (maybe try android later) and I was hoping to use SDL for keyboard/mouse. So you have a PC, or an RPi. You can connect to it through a "dumb" adapter (asynchronous) with an off-the-shelf part, or a "smart" adapter (synchronous) with an MCU to the controller port or expansion port. Then you'd use USB or Bluetooth keyboard and mouse. I'll release the first version soon, source will be free under MIT or similar license. The only working service right now is running code remotely, sending APU register updates, the app is an NSF player.

I'm using packets for everything, COBS encoded, and that allows for a control character in the 8-bit data, without the overhead of an escape character. Zero is idle state and packet terminator (it's $7D in my code now, but I'll be changing it to zero). The first byte after an idle character defines the type of packet, each type can have it's own special handler and format etc. for everything beyond that first byte. I'm having my packet types also include a CRC16, a packet number, followed by the data. The NES program sends an ACK with the last good packet number received, and that also serves as a request to send the next packet. I was thinking I might add a channel number in there, it might make more sense to let each channel have its own packet number sequence. Or maybe a simpler packet number per each packet type, allowing only one connection for each packet type at a time.

But all that's about moving the data, the interesting part is using it. This project seems to be putting me in orbit of writing-my-own OS territory. I've been thinking about how hardware drivers could be written on NES. It seems like keyboard and mouse support could benefit from this, as well as network and file access. What I as thinking is something along these lines:
-ROM distributed as a normal .NES file with the developer's preferred driver pre-loaded
-there's signature somewhere (fixed location, or anywhere?) in ROM where the developer maps out the load address, RAM address, ZP RAM address available to drivers (and maybe maximum stack depth?)
-the end user selects a driver suit their hardware, this is saved by the OS
-when the OS loads the program, it uses the address assignments to either recompile the driver, or do address fix-ups on the driver binary. The driver is then patched into the ROM.
edit:
-the actual code the user program JSRs to would in internal NES RAM, to keep it independent from any mapper (making it easier for the driver to be in a different PRG bank from the code calling it)
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: NES to USB Keyboard interface?

Post by rainwarrior »

Memblers wrote: Fri Apr 03, 2020 10:31 amIBM defined 3 sets of scancodes, which one should be used? There is also the USB set, defined by Microsoft. Which standard to use was a question I've also had, and I haven't started on any of the keyboard code yet.
Honestly, I don't think this is as much a problem as it seems. Just use the USB set.

For a host device, it should be almost trivial to remap scancodes if you really want to use an old non-USB keyboard.

(As an aside, with the old "3 sets" thing, keyboards were generally backward compatible and could be set at runtime to the desired scancode set. IIRC DOS normally puts a keyboard in set 1 during startup?)
Memblers wrote: Fri Apr 03, 2020 10:31 amI'm using packets for everything, COBS encoded, and that allows for a control character in the 8-bit data, without the overhead of an escape character.
For keyboard data, 0 is already the "no event" value so COBS isn't necessary, but I guess you're also transmitting other kinds of data with whatever it is you're doing.
Last edited by rainwarrior on Fri Apr 03, 2020 3:12 pm, edited 1 time in total.
Post Reply