It is currently Tue Oct 17, 2017 3:36 pm

All times are UTC - 7 hours





Post new topic Reply to topic  [ 52 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
PostPosted: Wed Jan 18, 2017 10:46 pm 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 5710
Location: Canada
tokumaru wrote:
I'm not opposed to clearing the RAM as an extra safety measure on your release build, but during development, it may give you the false sense of security that your modules are robust when they're actually not, if they neglect to initialize something that just happens to work if it starts out as 0, but fails with other values and you won't know it until you happen to re-run that module after that variable was left in a "bad" state.


Thanks, that clarifies the kind of problem you're thinking about. I think the primary difficulty of this argument is that I am viewing this as a yes/no proposition about whether to include a RAM wipe in the startup. It's not really an all-or-nothing issue, just it seemed initially phrased as one to me, and that reading felt dangerous to me. I'd like to clarify my priorities here:


Overall what I am adamantly opposed to is not setting RAM to a consistent state at startup on deployed code.

Whether or not you think it's safe to assume something is still 0 from the RAM wipe, that's a different issue. I think it's perfectly practical to do this carefully, but I can see how that can be a problem if you code in way that makes that error prone. If you want to initialize to $FF or $55 or change it infrequently during development to try and probe for accidental assumptions, I think that's a perfectly fine practice. Any consistent RAM state is fine for deployment, it doesn't have to be 0. I just happen to find 0 useful, but it's not what's important to me here at all.

I don't think randomizing it all the time while developing is the best way to approach that problem, though, because I think you should spend as much development time as you can testing with something as close to the deployment version as you can manage. So, from the premise that you must deploy this way, I think you should normally test this way too.

(Occasional smoke test sessions where you temporarily turn some part of the engine upside down and see if it still works are great to do, though.)



If you'll permit me to have a "koitsu" moment (please forgive me koitsu) I'd like to offer that the reason I have these priorities comes from my own personal experiences.

What I'm worried about is the "works on my machine" problem. If you don't initialize all of RAM, you'll now have a different startup environment on different emulators, on different flash cartridges, even on the same type of flash cartridge with different directory structures, etc. Deploying such a project to the world is inviting a nightmare of unreproducible errors. When I release a program and send it out to hundreds or thousands or people, difference-of-environment problems are the most frequent and most difficult thing I find myself dealing with, so they're at the very top of my list in terms of priority. It's really my number one deal, and it's why I say I feel strongly about this.

Here's one recent example of such a failure in deployment, but I can think of several others off the top of my head just on the NES alone. I've spent hundreds or maybe thousands of hours on these kinds of problems when releasing software (as both professional and amateur, on every single platform I've release software on), and they still fill me with fear.

On the other hand, I wipe RAM in all my NES programs and I can't think of a single time I've accidentally assumed something would be 0 in the wrong place (or if I have, it's still latent and nobody's noticed yet). I don't think I tend to have a lot of code that isn't clearly part of startup or not. Even when coding something like a main menu, I often use JMP $FFFC to return to the title screen if I can, instead of trying to rely on any assumptions about state since reset.

So... that's where my perspective is coming from. For me the "assumed 0 by accident" has been a non-issue, and the other one dominates the worst parts of my professional life, so if I'm weighing one against the other, there's no question to me which is the important one.


Outside of the NES, uninitialized variables is a big problem, especially in a language like C or C++ where guarantees about the environment are quite outside the language. There are some good tools for this, though-- modern C++ compilers can do static analysis and give you warnings about a lot of this kind of mistake. For some tougher ones there's great tools like Valgrind. In general I've learned the habit of initializing at declaration whenever it's not trivial (and leave a good comment if I really do want to leave it initialize). I'd love to see an NES debugger with some valgrindesque capabilities, though you can do a lot already with FCEUX + lua.


Top
 Profile  
 
PostPosted: Wed Jan 18, 2017 11:34 pm 
Online
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10046
Location: Rio de Janeiro - Brazil
rainwarrior wrote:
What I'm worried about is the "works on my machine" problem. If you don't initialize all of RAM, you'll now have a different startup environment on different emulators, on different flash cartridges, even on the same type of flash cartridge with different directory structures, etc.

OK, so we can finally agree that the main argument for putting all of the RAM in a consistent state is to normalize the environment, so that from that point on, the chances of the code running the same on any machine (bugs and all!) increases. I'm fine with that.

What I'm really getting at now are the bugs. What can one do to expose bugs caused by lack of memory (or even register) initialization? Neither clearing nor leaving the RAM alone will help much with detecting bugs. Clearing might help if you intercalate modules in the same play session and run the same module more than once (if it only works well the first time, something's fishy), but that sounds like more trouble than starting out with the garbage already in RAM and going straight to the module you want to test.

A full robustness test would require all RAM combinations to be tested, as well as every possible execution path the code can take, but that's just mathematically impossible. For now I'm satisfied with giving a few turns of testing on top of (consistently) "dirty" RAM every once in a while in hopes of exposing initialization issues.


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 4:31 am 
Offline

Joined: Tue May 28, 2013 5:49 am
Posts: 797
Location: Sweden
Garth wrote:
Regarding the macro allowing varying numbers of input parameters to set or clear flags:
Pokun wrote:
Once that list of flags grows enough, you loose the space you save by not clearing the RAM in a loop.

That's if it's practical to keep them together, for loop indexing. You might want to keep them separate though, with the parts of the program that use them. The macro could still be made to see if there are enough to justify a loop, and see if they're at contiguous addresses, and if so, assemble the loop, rather than straightlining them.

I have variables that needs to be initialized to 0 all over the place, for example RAM page 2 and 3 is for OAM and VRAM buffers and page 4 is reserved for the sound routine, so looping isn't easy. But I'm with Rainwarrior on this one, I'm not convinced by the advantages to not clear RAM so I'm going to continue doing so for reasons Rainwarrior explained. I want all of my buggy code to behave the same on all emulators, flashcarts and repro carts if possible.


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 6:29 am 
Online
Formerly WheelInventor

Joined: Thu Apr 14, 2016 2:55 am
Posts: 895
Location: Gothenburg, Sweden
First of all, thanks everybody, especially tokumaru and rainwarrior for continuing the topic on different methods concerning RAM wipe/randomization. Discussions like these are helpful when orienting; and in a way wikis, manuals, tutorials and technical writeups are less likely to capture. It helps me, and hopefully others, to set up methodology based on conscious choices.

tokumaru wrote:
A full robustness test would require all RAM combinations to be tested, as well as every possible execution path the code can take, but that's just mathematically impossible. For now I'm satisfied with giving a few turns of testing on top of (consistently) "dirty" RAM every once in a while in hopes of exposing initialization issues.
Would one cover potential bugs more efficiently by specifically testing RAM preset edge cases, ie. setting RAM to $00, $01, $FF, plus/minus flag edges, at pre-initialization? It wouldn't cover everything, but maybe go straight for common culprits.

Additionally, every literal used for condition checking could maybe be defined in the beginning (CONST1 = some value) rather than being literal (#$some value). That way, it'd be a more convenient procedure to set up a macro that can both check for edge cases, user defined edges, and, in the end, be used as a guarantee that RAM looks the same on start regardless hardware/emulator variant.

Code:
.macro fill_RAM value
LDA value
(your RAM 'wipe'/fill code here)
.endm

Changing value to any common edge case or CONSTNAME between diagnostic builds is just as easy as changing a seed for a pseudo random function. Maybe an offset parameter could be versatile, too, in the cases of using a CONST as parameter.

Note that this is not intended as a full replacement for a randomized/garbage memory test, just a complement.

If you still want to check random cases a few passes, adding a mode toggle parameter and if clause to the macro can be done in most assemblers (Ophis currently can't but i've recieved word conditional assembly is on the roadmap), and in CA65, it could be as simple as identifying if there's a second parameter or a specific parameter is used (ie .macro set_RAM value, rndseed), in which .ifnblank rndseed will branch to the right subfunction of the macro.


Some sort of batch process that outputs a series of identical binaries, excepting the instruction on what to fill RAM with, would be more time efficient. I'm not sure how i'd approach that, though.

_________________
http://www.frankengraphics.com - personal NES blog


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 7:37 am 
Online
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10046
Location: Rio de Janeiro - Brazil
WheelInventor, I like your ideas. Comparing execution logs from the different binaries running the same input could be an interesting way to look for initialization bugs.


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 8:08 am 
Online
Formerly WheelInventor

Joined: Thu Apr 14, 2016 2:55 am
Posts: 895
Location: Gothenburg, Sweden
Hm.. Let's say an emulator has the ability to load the program, filling RAM ($0), run, write a log, test next fill value, up until every possible fill has been tried, then compare logs, and sort out every unique log, and attach what range each one of those logs count for?

Running the diagnostic would then only be a matter of temporarily commenting out any wipe code in your source (most quickly done if the wipe is in a macro or include), assemble, and run the emulators' hypothetical test.

The trouble is how to automate/substitute human interaction with the program, if the diagnostic runs $0-$FF, (edit: or even more extense, every possible combination cell for cell in RAM. :shock:). Some sort of script that gives user input specified in seconds + frames?

If it only runs a few preset edge cases or, say, 10 randomized ones, you can perhaps get away with manual user input and a 'next test' command.

_________________
http://www.frankengraphics.com - personal NES blog


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 8:46 am 
Offline
User avatar

Joined: Mon Jan 03, 2005 10:36 am
Posts: 2961
Location: Tampere, Finland
I think you'd get the same benefit from an emulator which can log accesses to uninitialized RAM? (Like NDX.) For an emulator which supports this, it's important to not clear all memory on initialization during development, otherwise stray reads can't be detected. Stray writes can't of course be detected in any case (although maybe this would be possible in some cases with some help from the assembler to annotate the used memory areas; definitely it would be possible to trap writes to unexpected areas like $2008..$3FFF).

(I do think all memory should be cleared in the release build.)

_________________
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: kkfos.aspekt.fi


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 8:53 am 
Offline
User avatar

Joined: Sat Jul 12, 2014 3:04 pm
Posts: 936
Wheelinventor wrote:
every possible combination cell for cell in RAM.
very no.
2^(8*2048=16384) ~ 10^4932 >> universe_time_existed.

Current processors do things at roughly 2^30/s (~1GHz). The age of the universe is about 2^34 years. Years:seconds conversion rate is roughly 2^25.
Running one cycle of a program, with all possible RAM initvalues, thus would take a modern processor about 2^16300 (10^4906) times as long as the universe has existed.

(phew, almost ran that without remembering that bytes ≠ bits!)


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 8:59 am 
Offline
User avatar

Joined: Sat Jul 12, 2014 3:04 pm
Posts: 936
Ah, one of the holy-war questions: wipe or not. Aaand I see you've both covered most-all of the bases.

The point of wipe is indeed "faster/smaller initializing of RAM".

The point of not-wipe is that that might not be the right time to initialize variables, and thus makes finding variables not-properly-reset after certain segments harder, because it'll work the first time.
And, for advanced emulators, you can trap a truly-uninitialized read.
But, this doesn't actually catch the improper reset.


rainwarrior wrote:
What I mean is I feel strongly enough about it that I disagree with "to each his own". I think leaving unallocated parts of RAM up to chance is bad form and feel a necessity to advise against it. (Summarizing again: you should not intentionally create conditions which aren't reliable from system to system.)
As usual, I'll invoke one traditional counterpoint: Final Fantasy 1 uses uninitialized RAM ($F5-6) for where to start on encounter tables, causing encounters to vary across a RESET-load but not [usually] across a POWER-load. Contrariwise, the in-battle RNG is saved in the save file, and not erased on NEW GAME…but it has been noticed that this variable is eventually zeroed by the code that writes "THE END" in fancy letters, and speedrunners use it to try to achieve a consistent start-up state to get their first encounter escapable as-desired.

The initial state of RAM has been found to vary somewhat per-system, but is an annoyingly analog effect, and can take variable amounts of time to return to from what I've seen of feasel's runs.

Pokun wrote:
I have variables that needs to be initialized to 0 all over the place, for example RAM page 2 and 3 is for OAM and VRAM buffers and page 4 is reserved for the sound routine, so looping isn't easy.
Initializing OAM to FF is more useful than 00.

[hr]

Unfortunately, the typical way a loader is run on PowerPak is to run it out of $0400-07ff, which means a naïve RAM-initializer loop would not suffice. Hmm…but that's for another topic, I think.


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 9:06 am 
Online
Formerly WheelInventor

Joined: Thu Apr 14, 2016 2:55 am
Posts: 895
Location: Gothenburg, Sweden
Quote:
very no.

Haha, i assumed the numbers to be well over high enough to be a practical no-no - but didn't think them to be physically impossible.

_________________
http://www.frankengraphics.com - personal NES blog


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 9:07 am 
Online
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10046
Location: Rio de Janeiro - Brazil
thefox wrote:
I think you'd get the same benefit from an emulator which can log accesses to uninitialized RAM? (Like NDX.)

I guess you are right. Initializing to different values might be good when a human is eyeballing any differences, but if the computer is doing an objective analysis it can detect the problem right at its root instead of looking for side effects. Alternated use of memory by different modules is still a problem though, if for any reason you can't switch the order in which the program modules run (i.e. one might depend on the processing done by another).


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 10:48 am 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 5710
Location: Canada
WheelInventor wrote:
The trouble is how to automate/substitute human interaction with the program, if the diagnostic runs $0-$FF, (edit: or even more extense, every possible combination cell for cell in RAM. :shock:). Some sort of script that gives user input specified in seconds + frames?

Have you heard of a Tool Assisted Speedrun? ^_^ FCEUX actually has this feature built in, and it's quite useful for repetitive testing. Several emulators have a "record movie" feature where you can record and playback an input stream, but the TAS editor lets you edit it frame by frame, if you need to. I mostly just use the simple record/playback, but TAS editing made a few tests possible for me that I couldn't do with live input.

Myask wrote:
rainwarrior wrote:
I think leaving unallocated parts of RAM up to chance is bad form and feel a necessity to advise against it. (Summarizing again: you should not intentionally create conditions which aren't reliable from system to system.)
As usual, I'll invoke one traditional counterpoint: (Final Fantasy 1's PRNG)

The key word I was trying to emphasize, before I got very explicit about my point, was unallocated. PRNG of course is supposed to be different from run to run, but it's under deliberate control. Two or three bytes of an unknown PRNG seed is a well managed and mitigated problem. Leaving hundreds of bytes in an unknown state for long periods of execution is not.

Myask wrote:
The initial state of RAM has been found to vary somewhat per-system, but is an annoyingly analog effect, and can take variable amounts of time to return to from what I've seen of feasel's runs.

It doesn't just vary per-system, it can vary tremendously from power-on to power-on on the same system. I think some SRAMs do often start in all/most $FF or $00 in some conditions, but both my NES and Famicom basically don't have any reliable startup bits. I wrote a ROM you can use to test it yourself, though, if curious: thread with test ROM


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 5:02 pm 
Offline

Joined: Tue May 28, 2013 5:49 am
Posts: 797
Location: Sweden
Myask wrote:
Pokun wrote:
I have variables that needs to be initialized to 0 all over the place, for example RAM page 2 and 3 is for OAM and VRAM buffers and page 4 is reserved for the sound routine, so looping isn't easy.
Initializing OAM to FF is more useful than 00.

Of course I don't zero out the OAM buffer. I just mean my RAM map is organized in a number of groups like OAM, VRAM, sound, score etc. So looping only the flags and other variables that needs to be zeroed out won't work.


Top
 Profile  
 
PostPosted: Thu Jan 19, 2017 7:14 pm 
Offline
User avatar

Joined: Sat Jul 12, 2014 3:04 pm
Posts: 936
rainwarrior wrote:
Have you heard of a Tool Assisted Speedrun?

It feels to me like very, very few [other] people are simultaneously involved in speedrunning, TASing, and NESdev.

Myask wrote:
rainwarrior wrote:
I think leaving unallocated parts of RAM up to chance is bad form and feel a necessity to advise against it. (Summarizing again: you should not intentionally create conditions which aren't reliable from system to system.)
As usual, I'll invoke one traditional counterpoint: (Final Fantasy 1's PRNG)

The key word[…]was unallocated.[/quote]Whoops. Missed that word.
rainwarrior wrote:
Myask wrote:
The initial state of RAM has been found to vary somewhat per-system, but is an annoyingly analog effect, and can take variable amounts of time to return to from what I've seen of feasel's runs.

It doesn't just vary per-system, it can vary tremendously from power-on to power-on on the same system. I think some SRAMs do often start in all/most $FF or $00 in some conditions, but both my NES and Famicom basically don't have any reliable startup bits. I wrote a ROM you can use to test it yourself, though, if curious: thread with test ROM

Was kind of trying to say "may vary between power-cycles" with the " an annoyingly analog effect, and can take variable amounts of time to return to [the 'cold' state]".

That said, the system they were using at AGDQ'17 must have some very consistent cells or their technique has been improved since I'd watched Feasel; they were power-cycling for a quarter-second tops rather than quarter-half hours.


Top
 Profile  
 
PostPosted: Fri Jan 20, 2017 3:13 am 
Online
Formerly WheelInventor

Joined: Thu Apr 14, 2016 2:55 am
Posts: 895
Location: Gothenburg, Sweden
tokumaru wrote:
that dreadful STA $2003 ; set the low byte (00) of the RAM address comment that comes straight from the Nerdy Nights tutorials


Has this been mentioned in previous threads? I haven't informed myself on this.

myask wrote:
rainwarrior wrote:
Have you heard of a Tool Assisted Speedrun?
It feels to me like very, very few [other] people are simultaneously involved in speedrunning, TASing, and NESdev.
I saw my first TAS livestream this week, and only accidentaly so, as my flatmate's boyfriend came over and put it on the screen.

rainwarrior wrote:
Several emulators have a "record movie" feature where you can record and playback an input stream, but the TAS editor lets you edit it frame by frame, if you need to. I mostly just use the simple record/playback, but TAS editing made a few tests possible for me that I couldn't do with live input.

That's really useful! Must TAS scripts be played @ real-time emulation speed?

_________________
http://www.frankengraphics.com - personal NES blog


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 52 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: Bing [Bot] and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group