Question about Gameplay speedrun on real hardware

A place where you can keep others updated about your NES-related projects through screenshots, videos or information in general.

Moderator: Moderators

User avatar
jpx72
Posts: 178
Joined: Tue Sep 28, 2010 3:27 am
Location: Slovakia
Contact:

Question about Gameplay speedrun on real hardware

Post by jpx72 »

Hello
I hope this topic wasn't discussed here before, but are there any projects that implemented speed-runs / or any other runs to an actual NES ROM? I mean like changing the demo run in SMB to run somebody's actual gameplay (there's the game genie code for enabling the sound in the SMB demo run AOSYLZEI + AUNPAAEY =thanks nesondubois) or, that really interrest me, to implement a Rockman run to actual Rockman.nes - you see my point...
I know there are sync/timing problems with real hardware as discussed here:
http://tasvideos.org/forum/viewtopic.php?t=4288
but nevertheless I would be really interrested in this, even if the succes rate of that run would be only like 50%.
But as a non-programmer I would only be interrested in some already made compiler, just add salt and water and it works -> that's adding rom(SMB and Rockman1 are my real interrest) + run(something like emulator "video") = final product.

EDIT: I know about the extra Famicom hardware (can't remember the name now) that can repeat button presses/ and/or using a PC to do the same thing, but that's not the question...
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Post by thefox »

You mean adding the playback stuff in the ROM itself? You won't find a easy solution for that, every game is different. It's possible but making an automated tool isn't an easy task...

I thought about doing a replayer like this for, guess what, PowerPak. :) I think it should be possible by allocating part of the PRG-ROM for controller states, stealing a couple of cycles off NMI (to read the controller data from PRG-ROM and pass it to FPGA) and using the FPGA to watch for controller reading instructions and substituting (like my save state mappers' turbo feature already does). That is if the game actually keeps NMI enabled at all times...
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

Technically, if there's enough free PRG-ROM space, you could easily store the button log (i.e. emulator movie, but compressed to save more space) there and modify the game's controller reading routine to read from that log instead. You'll also need a couple of free RAM bytes, to keep track of the button data.

Since games use vastly different mappers (which means that the way in which the button logs will be stored will vary due to the way PRG-ROM is bankswitched), have their controller reading routines in different places, and use RAM very differently, you can't automate this process. Each game is a particular case.

A tool can even be made for this, but it will only support games that have been explicitly added by the person making the program, it will not automatically support "every ROM ever".
Nessie
Posts: 133
Joined: Mon Sep 20, 2004 11:13 am
Location: Sweden
Contact:

Post by Nessie »

It does not address your question, but this guy hooked up an Arduino to the controller port, then fed an FCEUX movie to it so it played through a speed run on a real NES.

Really impressive!
User avatar
jpx72
Posts: 178
Joined: Tue Sep 28, 2010 3:27 am
Location: Slovakia
Contact:

Post by jpx72 »

I like the button log idea, I had something like that in my mind. Keeping it simple. Updating the finished compilation software due to the differences between games seems normal... But developing something like that would be probably easier for the powerpak first (although I would like the idea better if I had one).

I don't understand the mechanics of storing the emulator videos in current emlators, it isn't just simple time(when) + button(what) log right? So there would have to be also a compiler for those runs. The "special" TAS runs are out, many of those use routines which aren't even supported by the real hardware (AFAIK). But there are many "normal" runs and speed-runs that can be reproduced by a human (understand "real hardware").

The Famicom hardware button-log storage unit was called "Hori Game Repeater". Some info here:
http://www.famicomworld.com/forum/index.php?topic=4847
Pictures here:
http://www.forum.emunes.pl/index.php?topic=2927.0

EDIT: Thanks for the Arduino link, haven't seen that before, nice!

EDIT2: I just found a relevant discussion here:
http://nesdev.com/bbs/viewtopic.php?t=6162
User avatar
Kasumi
Posts: 1293
Joined: Wed Apr 02, 2008 2:09 pm

Post by Kasumi »

TAS' generally work from a cleared save file, from the start of a game, and use no cheats or anything. They're nothing but pure button input. This being true, anything a TAS does should absolutely be possible in the game, on the actual console even if it's not HUMANLY possible. Unless "special" TAS refers to something I haven't seen.

TAS button input may not work on a console, but the cause of this would be that the TAS was made on an inaccurate emulator, and not really the TAS' input itself. In fact this may even break "normal" speed runs done on an emulator. For instance, emulators tend to work from cleared RAM, but NES games might rely on the "random" RAM at startup to seed their random number generators.

In fact there are even several verified on console runs, and now that methods like the Super Mario Bros. video exist, more runs will probably be verified.
Last edited by Kasumi on Fri Mar 25, 2011 1:25 am, edited 1 time in total.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Post by thefox »

tokumaru wrote:Technically, if there's enough free PRG-ROM space, you could easily store the button log (i.e. emulator movie, but compressed to save more space)
Any ideas for a compression/decompression algorithm (I remember you being active in the CHR compression thread)? Trying to do this on PowerPak tickled my interest again. :)
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
mic_
Posts: 922
Joined: Thu Oct 05, 2006 6:29 am

Post by mic_ »

Any ideas for a compression/decompression algorithm
Wouldn't you be fairly likely to have some data sequences repeated fairly often in a button log from a speed run? So a LZ variant should perform fairly well. aPLib maybe? Or PuCrunch.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Post by thefox »

mic_ wrote:
Any ideas for a compression/decompression algorithm
Wouldn't you be fairly likely to have some data sequences repeated fairly often in a button log from a speed run? So a LZ variant should perform fairly well. aPLib maybe? Or PuCrunch.
Hmm, I was thinking about something that could be decompressed easily. If it could be done on the FPGA that would be even better. :) I have to say I'm (almost) completely clueless about compression methods.
mic_
Posts: 922
Joined: Thu Oct 05, 2006 6:29 am

Post by mic_ »

Here's an aPLib decoder I ported to the 6502: http://jiggawatt.org/badc0de/decrunch/a ... h_6502.asm
Note that it isn't fully optimized.

I think the PuCrunch archive includes a decoder for the 6510 that could be used pretty much as-is.
User avatar
qbradq
Posts: 972
Joined: Wed Oct 15, 2008 11:50 am

Post by qbradq »

I worked on an AVR-based solution to play back a TAS over the controller port (never finished :( ). Here's the compression scheme I used for this project. I've converted it to C for readability.

Code: Select all

typedef struct
{
	byte num_frames;	// The number of frames to hold this input byte.
	byte input;			// The litteral bits to load into the latch register
} input_record;

struct input_dictionary
{
	input_record records[128]; // The dictionary of input records
}

typedef union
{
	byte dict_entry;	// If bit 7 is set, bits 0-6 are an index into input_dictionary.
	input_record record; // If bit 7 is not set, this is a litteral input record.
} input;
That way you have your 128 most frequent input records compressed down from two bytes to one. And you can still represent holding down the same inputs for over two seconds without using another input record.

I did write a program to convert FCEU format input movies into this format, but I can't seem to find it.

So with all that said I was working with a very limited platform. The input movie would have been burned into the ROM of the AVR, then read back using slow program reads. I only had 32 bytes of RAM to work with (not counting a return stack), so I never looked into fancy compression and the like.

Ya'll are way beyond me with the PowerPak stuff. I can't even read VHDL or whatever it is you all use :D

Wow, this really makes me want to finish up that AVR input playback device. I'll have to go work on that over the weekend :D
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru »

thefox wrote:Any ideas for a compression/decompression algorithm (I remember you being active in the CHR compression thread)? Trying to do this on PowerPak tickled my interest again. :)
In my game I just used RLE (buttons + frame count), because the demos are fairly short. I have a recording routine that outputs to SRAM, so I can later grab the data from the save file and incbin it in the ROM for playback.

How much memory do you have for this on the PowerPak? Keeping track of several minutes (possibly more than an hour) of button logs might be a challenge.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Post by thefox »

mic, I think those algos are too complex for this purpose. Thanks anyways.
tokumaru wrote:
thefox wrote:Any ideas for a compression/decompression algorithm (I remember you being active in the CHR compression thread)? Trying to do this on PowerPak tickled my interest again. :)
In my game I just used RLE (buttons + frame count), because the demos are fairly short. I have a recording routine that outputs to SRAM, so I can later grab the data from the save file and incbin it in the ROM for playback.

How much memory do you have for this on the PowerPak? Keeping track of several minutes (possibly more than an hour) of button logs might be a challenge.
Well, the idea was to store the states in the the upper half of the PRG-ROM (512K on PowerPak, so game can still use 256K max), hook NMI to a memory area where FPGA dynamically generates code to do BIT $xxxx JMP original_nmi (4+3 cycles). FPGA then grabs the read value from the data bus and returns it the next time game reads $4016/$4017.

So 256K would allow ~72 minutes worth of uncompressed controller states (single player). I don't know how long the longest speedruns are but I'd like the limit to be bit bigger than that.
qbradq wrote:I did write a program to convert FCEU format input movies into this format, but I can't seem to find it.
Let me know if you find it, I would be especially interested in the compression rate.
User avatar
qbradq
Posts: 972
Joined: Wed Oct 15, 2008 11:50 am

Post by qbradq »

Good news! I got bored so I whipped up a Python program to compress FCEUX FM2 input movies into the format I documented above. I tested this with a ~18 minute run of SMB1 and a ~6 minute run of SMB1. Both came out to about 6% of the uncompressed (one byte per frame) size.

Download Link (GPL V3, Requres Python 2.5.4 or Higher)

Edit: Oh yea, I forgot to mention. This program only supports first player, only with a Game Pad, and there are *no* sanity checks. I may add that stuff later, or anyone else is free to.

I am very excited to work on my AVR-based playback thing-a-majig now. Only problem is I only have *very* small AVRs with 2K Flash space. I'll have to order bigger ones to see full speed runs :oops:

Here's the Stats! Yay!

Code: Select all

Title:			NES Super Mario Bros (JPN/USA PRG0) "warpless" in 18:41.7 by Lee (HappyLee)
Download URL:	http://tasvideos.org/movies/fm2/happylee3-smbwarpless.zip
Total Frames:   67413
Output Length:  4391 Bytes
Compressed %:   6.513580%

Title:			NES Super Mario Bros (JPN/USA) "walkathon" in 06:47.2 by Lee (HappyLee).
Download URL:	http://tasvideos.org/movies/fm2/happylee4-smb-sidestroller.zip
Total Frames:   24472
Output Length:  1498 Bytes
Compressed %:   6.121281%
Python Program (WARNING: Ugly Code :D )

Code: Select all

'''fm2bin.py

Convert an FCEUX .FM2 Emulator Movie File into a binary form for playback on
NES hardware.

Copyright (C) 2011 Norman B. Lancaster

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

'''

import os
import sys
import re
import struct


input_bytes = []
rle_words = []
word_count = dict()
dict_words = dict()
output_bytes = []


def main():
	if len(sys.argv) != 3:
		print '''Usage: fm2bin.py input output
input   *MUST* be a .fm2 version 3 file. This is created by FCEUX. Only port 0
        input will be processed. Port 0 must have been a game pad. Save states
		are not supported either.
output  The output binary file. See "fileformat.txt" for details.
'''
		sys.exit(1)
	
	# Load the input file
	input = open(sys.argv[1], "r")
	while True:
		line = input.readline()
		if line == "":
			break
		line = line.strip()
		
		# Input record line
		if line.startswith('|'):
			byte = 0
			match = re.search('\|([^\|]*)\|([^\|]*)\|', line)
			token = match.group(2)
			for char in token:
				byte = byte << 1
				if char != " " and \
				char != ".":
					byte += 1
			input_bytes.append(byte)
	input.close()
	
	# REL-compress the data
	last_byte = -1
	count = 0
	for byte in input_bytes:
		if ( byte != last_byte and \
		count > 0 ) or \
		count >= 128:
			rle_words.append(((count - 1) << 8) | (last_byte & 0xff))
			count = 0
		last_byte = byte
		count += 1
	rle_words.append(((count - 1) << 8) | (last_byte & 0xff))
	
	# Build the word dictionary
	for word in rle_words:
		if not word in word_count:
			word_count[word] = 0
		word_count[word] += 1
	
	# Sort the word dictionary
	tupples = []
	for key in word_count:
		tupples.append((word_count[key], int(key)))
	tupples.sort()
	tupples.reverse()
	
	# Take the top 128 elements as the compression dictionary
	tupples = tupples[0:128]
	for i in range(0, len(tupples)):
		dict_words[tupples[i][1]] = i
		tupples[i] = tupples[i][1]
	
	# Output the dictionary
	# NOTE: tupples is now an array of INTs, should have thought that one through :D
	output_bytes.append(len(tupples))
	for word in tupples:
		# Just like in the record stream, count first, then byte
			output_bytes.append(word >> 8)
			output_bytes.append(word & 0xff)
	
	# Output the input record stream
	for word in rle_words:
		# Compress-able record
		if word in dict_words:
			# Bit 7 is the "this is a dictionary word" flag
			output_bytes.append(dict_words[word] | 0x80)
		# Non-compress-able record
		else:
			# Count first, then byte
			output_bytes.append(word >> 8)
			output_bytes.append(word & 0xff)
	
	# Output the file
	# Note: there may be a much faster way to do this.
	output = open(sys.argv[2], "wb")
	for i in range (0, len(output_bytes)):
		output.write(struct.pack("B", output_bytes[i]))
	output.close()
	
	print "Total Frames:\t%d" % len(input_bytes)
	print "Output Length:\t%d Bytes" % len(output_bytes)
	print "Compressed %%:\t%f%%" % float((float(len(output_bytes)) / float(len(input_bytes))) * float(100))


if __name__ == "__main__":
	main()
File Format Documentation:

Code: Select all


File format description for the binary output file created by fm2bin.py

by Norman B. Lancaster



Offset	Length	Desscription
0		1		(N) Length of the compression dictionary in entries
1		2*N		Compression entries.
				Byte	Meaning
				0		The number of frames to hold this data in the output latch for controller 1
				1		The data that should be loaded into the output latch for controller 1
2*N+1	XXX		Input record entries.
				Byte	Meaning
				0		The number of frames to hold this data in the output latch for controller 1
						If bit 7 is set this is an index into the compression entries table (expressed
						as an offset in compression entires, NOT bytes) to load.
				1		The data that should be loaded into the output latch for controller 1
						If bit 7 of byte one is set, skip this byte.

Results for Reference FM2 Files:

Title:			NES Super Mario Bros (JPN/USA PRG0) "warpless" in 18:41.7 by Lee (HappyLee)
Download URL:	http://tasvideos.org/movies/fm2/happylee3-smbwarpless.zip
Total Frames:   67413
Output Length:  4391 Bytes
Compressed %:   6.513580%

Title:			NES Super Mario Bros (JPN/USA) "walkathon" in 06:47.2 by Lee (HappyLee).
Download URL:	http://tasvideos.org/movies/fm2/happylee4-smb-sidestroller.zip
Total Frames:   24472
Output Length:  1498 Bytes
Compressed %:   6.121281%
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Post by thefox »

qbradq wrote:Good news! I got bored so I whipped up a Python program to compress FCEUX FM2 input movies into the format I documented above. I tested this with a ~18 minute run of SMB1 and a ~6 minute run of SMB1. Both came out to about 6% of the uncompressed (one byte per frame) size.
Wow, thanks, that's a much better ratio than I expected. I guess this means that I could actually use the 32K PRG-RAM for my PowerPak replayer instead of PRG-ROM. Makes some things easier, because recordings could be loaded using the standard PowerPak menus.

EDIT: FYI, I checked tasvideos.org and the longest "speedrun" seems to be DW4 at a bit over 2 hours. I should try couple of more videos with your script to see how well they compress in general.
Post Reply