Graphical glitches on the Everdrive
Moderator: Moderators
Graphical glitches on the Everdrive
Need some help. Working on a game for the nesdev competition and we're seeing a glitch. Basically, when a level starts, we disable sprites, setup the level, then enable sprites and off we go. This works fine on emulator (FCEUX) and powerpak, but we're seeing a weird glitch on everdrive. Basically, for a brief moment (presumably one frame), the sprites show up all wacky. I assume its the sprites because the background has already been loaded and looks fine. Its also weird because the first 2 sprites (the top of the frog) are always in the correct place and some of the health is in the right place, and there are some consistencies in each levels glitched state... like some health seems to be changed to r's and some of them are shifted to the same spot halfway across the screen... but which ones aren't always consistent. There are a couple other consistent ones, but I'm not sure how much that matters.
The difficult part is I can't debug on my everdrive. I did try to recreate the "screen garbage" on emulator by never disabling the sprites to test if it was correct sprite states but being displayed too early on everdrive, but when I did, the sprites never looked out of place. I wasn't sure what to try after this. I sort of assume its either something sketchy the everdrive does (although I couldn't find any meaningful differences between everdrive and powerpak) or we're interacting innappropriately with the PPU and some way the Everdrive works exposes it... but I couldn't figure anything out by walking through the code and the PPU is confusing . We're not doing anything with mappers, if that helps.
Our code is here: https://github.com/mitch3b/ChickenOfThe ... 1583-L1701 It's a lot to look at, but I've linked to the section that we use to load the level so it might be useful.
Here is the video: https://clips.twitch.tv/AgileGracefulYakinikuPrimeMe that shows the first 2 levels. The first level the glitch happens in real time and the second level I pause buffered so it loads into the first frame and stays that way. I've included screen shots including what the second level looks like when loaded, although its a little dark (sorry about that, I just plucked an image from the paused twitch video).
Any ideas to recreate or theories would be helpful.
Thanks!
The difficult part is I can't debug on my everdrive. I did try to recreate the "screen garbage" on emulator by never disabling the sprites to test if it was correct sprite states but being displayed too early on everdrive, but when I did, the sprites never looked out of place. I wasn't sure what to try after this. I sort of assume its either something sketchy the everdrive does (although I couldn't find any meaningful differences between everdrive and powerpak) or we're interacting innappropriately with the PPU and some way the Everdrive works exposes it... but I couldn't figure anything out by walking through the code and the PPU is confusing . We're not doing anything with mappers, if that helps.
Our code is here: https://github.com/mitch3b/ChickenOfThe ... 1583-L1701 It's a lot to look at, but I've linked to the section that we use to load the level so it might be useful.
Here is the video: https://clips.twitch.tv/AgileGracefulYakinikuPrimeMe that shows the first 2 levels. The first level the glitch happens in real time and the second level I pause buffered so it loads into the first frame and stays that way. I've included screen shots including what the second level looks like when loaded, although its a little dark (sorry about that, I just plucked an image from the paused twitch video).
Any ideas to recreate or theories would be helpful.
Thanks!
Re: Graphical glitches on the Everdrive
Did you try it on mesen?
https://twitter.com/bitinkstudios <- Follow me on twitter! Thanks!
https://www.patreon.com/bitinkstudios <- Support me on Patreon!
https://www.patreon.com/bitinkstudios <- Support me on Patreon!
Re: Graphical glitches on the Everdrive
No. The only emulator I've tried is FCEUX although the person I'm developing with (link_7777) I THINK uses EmuHawk and didn't see the issue either. Is there a reason why Mesen might show something different than FCEUX or does it have a feature that might help? (BTW, i'll try Mesen anyway bc I assume it'd be a better emulator to be using when I dev on my mac)
Re: Graphical glitches on the Everdrive
Mesen is more accurate overall. For example, it doesn't start RAM with all zeroes, which is more accurate.
Also it presents some more debugging features.
Also it presents some more debugging features.
https://twitter.com/bitinkstudios <- Follow me on twitter! Thanks!
https://www.patreon.com/bitinkstudios <- Support me on Patreon!
https://www.patreon.com/bitinkstudios <- Support me on Patreon!
Re: Graphical glitches on the Everdrive
As a developer, you should not rely on a single emulator, specially if accuracy isn't one of that emulator's strongest suits. FCEUX is a great debugging tool, but a little lacking in accuracy.mitch3a wrote:The only emulator I've tried is FCEUX
Unfortunately I can't help much with C code, but if a ROM is included I will try to do a little debugging when I'm on my PC.
Re: Graphical glitches on the Everdrive
Attached the latest version. Just please don't go overboard on playing too far/gameplay feedback Saving that for the competition and still hoping to make some significant changes before submission.
- Attachments
-
- ChickenOfTheFarm_02.nes
- (40.02 KiB) Downloaded 500 times
Re: Graphical glitches on the Everdrive
I had a similar bug before and it was because the OAM was not initialized properly and you would see some residual data at at startup. It could be the cause here.
Re: Graphical glitches on the Everdrive
It could be an issue with missing a frames worth of the DMA to the OAM (which we're doing), but in this case, because sprites are set and can't move, assuming nothing is corrupting it then the OAM should remain unchanged. The question would be (if that is the issue), why would it be getting corrupted on Everdrive and not powerpak/emu.
Re: Graphical glitches on the Everdrive
Different menu software using RAM differently, I guess. That and OAM DMA appears to use slightly different memory access timings compared to CPU code or data fetches, and early PowerPak revisions had to be modded with resistors to account for that.
Re: Graphical glitches on the Everdrive
Seems like we're starting to narrow down the issue. There are a handful of examples here that can corrupt the OAM so I'll probably step through one by one to verify if any are happening:
https://wiki.nesdev.com/w/index.php/Err ... nd_Sprites
I'm also going to try Mesen to see if I can recreate the issue in an emulator. and take a harder look at PPU/OAM DMA writes... unfortunately the code isn't organized the best way to do this so it'll probably be a little error prone and take a chunk of time, but it definitely feels like it could expose my issue.
https://wiki.nesdev.com/w/index.php/Err ... nd_Sprites
I'm also going to try Mesen to see if I can recreate the issue in an emulator. and take a harder look at PPU/OAM DMA writes... unfortunately the code isn't organized the best way to do this so it'll probably be a little error prone and take a chunk of time, but it definitely feels like it could expose my issue.
Re: Graphical glitches on the Everdrive
Actually, it does pre-initialise RAM with all zeros (that's the default). You can change/select this behaviour, however, but it's up to the user to change that: Options -> Emulation -> Advanced -> Default power on state for RAM: All 0s (Default), All 1s, or Random Values are the choices. I tend to recommend folks try all of them when testing a game they've worked on.nesrocks wrote:Mesen is more accurate overall. For example, it doesn't start RAM with all zeroes, which is more accurate.
The subject of default RAM values on power-on has been discussed heavily in the distant and recent past, and there is no universal default that works for every game. Some games depend on certain values otherwise they act weird/different than on hardware, while others rely on the exact opposite/different values than the other games.
Anyway, let's not let this subject distract from the thread unless it turns out to be the root cause.
In general, testing on actual hardware still has to be done on real carts (read: not Everdrive, not PowerPak, etc.) for reasons demonstrated here. The implementation of those flash carts using native 6502 code and thus the NES's RAM can cause unexpected behaviour.
It's probably going to turn out to be some piece of code that is relying on values not initialised (incl. OAM), or possibly some PPU quirk ($2000/2005/2006), or timing thing. It almost always is on the NES.
Re: Graphical glitches on the Everdrive
Someone I know used to have an issue with Everdrive where sprites would mess up during lag frames in Zelda (IIRC, sprite RAM bits would get set). I believe this turned out to be a power supply issue, since the cart draws more power than is typical. During lag frames in Zelda, OAM DMA isn't performed, so maybe inadequate power was impacting OAM decay in some way. I notice here that your glitching is setting bits; for example, tiles from the health bar have been moved what looks like #$80 pixels to the right. So, seems similar. [Edit: This OAM issue is actually caused by the cart and is apparently fixed by placing 100 ohm resistors on the cart's CPU data lines. It has been seen in flash carts and on the FDS RAM adapter. See my investigation and test ROMs later in this thread.]
The behavior of your code is that you do OAM DMA while rendering is off, and then enable rendering almost 30,000 cycles later. This should be plenty of time for decay. In fact, in Mesen, if you turn on decay emulation (Options -> Emulation -> Advanced -> Enable OAM RAM decay), you can see by viewing Sprite / OAM RAM in the Memory Tools that by the time you've enabled rendering, OAM has decayed. Mesen currently emulates this by setting decayed RAM to #$10 (and you get a frog head on the top left of the screen). You need to not wait a long time between OAM DMA and enabling rendering.
@sour: Speaking of which, could you make Mesen use random values on OAM decay? IIRC, the manual says they get set to random, I remember seeing a Mesen code comment saying they get set to a specific value, and then the code sets it to a different value. With everything at #$10, it's hard for me to see the glitching here because it goes under your pause icon.
The behavior of your code is that you do OAM DMA while rendering is off, and then enable rendering almost 30,000 cycles later. This should be plenty of time for decay. In fact, in Mesen, if you turn on decay emulation (Options -> Emulation -> Advanced -> Enable OAM RAM decay), you can see by viewing Sprite / OAM RAM in the Memory Tools that by the time you've enabled rendering, OAM has decayed. Mesen currently emulates this by setting decayed RAM to #$10 (and you get a frog head on the top left of the screen). You need to not wait a long time between OAM DMA and enabling rendering.
@sour: Speaking of which, could you make Mesen use random values on OAM decay? IIRC, the manual says they get set to random, I remember seeing a Mesen code comment saying they get set to a specific value, and then the code sets it to a different value. With everything at #$10, it's hard for me to see the glitching here because it goes under your pause icon.
Last edited by Fiskbit on Wed Aug 05, 2020 1:19 am, edited 1 time in total.
Re: Graphical glitches on the Everdrive
As expected, the OAM was being corrupted on Everdrive so an extra DMA OAM was required (code fix here: https://github.com/mitch3b/ChickenOfThe ... arm.c#L916). Didn't really investigate any more than giving it a try, but seemed to work and competition deadline approaching so I'll take it and move on. Thanks for the help. Always impressed by how quick/helpful you folks are!
Re: Graphical glitches on the Everdrive
Not sure why I wrote that they "randomly change" in the documentation.Fiskbit wrote:@sour: Speaking of which, could you make Mesen use random values on OAM decay? IIRC, the manual says they get set to random, I remember seeing a Mesen code comment saying they get set to a specific value, and then the code sets it to a different value. With everything at #$10, it's hard for me to see the glitching here because it goes under your pause icon.
This used to return $FF, which would hide all the sprites (that's what the comment says), but it was changed to $10 to make the sprites visible (and because it was a value that some people said their OAM bytes tended to decay to). Looks like I didn't change the comment to match.
I could change this to something even more obvious, though. E.g $05 multiplied by the sprite index, which in this case would result in this: Pretty hard to miss, at the very least.
Also, there is a "Break on decayed OAM read" option in the debugger that will cause execution to break whenever a decayed byte is read by the PPU (which is the safest way to find these issues really). The only downside is that this requires keeping the debugger window opened.
Re: Graphical glitches on the Everdrive
That's certainly a lot more visible! I'm actually planning to do some OAM decay testing in the near future; I'd like to get an idea of how long it takes before decay occurs and how it tends to decay. Beyond just understanding it better, I'm curious if it could have utility (could it reasonably be used as a source for randomness?). For now, I'll be testing on Everdrive, which I can apparently expect to influence my results.
As far as emulation goes, for development, having realistic decay timings seems pretty important. High visibility might be helpful, but as you mentioned with the debugger feature, it's not necessary (though the debugger breaks on decayed reads even when sprite rendering disabled, which is of questionable utility and makes the feature pretty useless for this Chicken of the Farm ROM because of so many false positives). If flash carts do actually impact decay timings and/or behavior even on healthy power supplies, having an option to run with these more strict timings would also be helpful (though in the case of decay possibly being faster than vblank as I described with Zelda, I'm not sure what the right solution is. OAM DMA during lag frames is a bad idea because it can draw sprites that aren't fully written yet. This is especially noticeable in Metroid with the HUD).
For casual play, I like accuracy and would prefer standard realistic timings and realistic decay, especially given that it's possible a game could require this at some point if decay is sufficiently random.
(All of this depends on getting good data about how this tends to work across different consoles, of course.)
As far as emulation goes, for development, having realistic decay timings seems pretty important. High visibility might be helpful, but as you mentioned with the debugger feature, it's not necessary (though the debugger breaks on decayed reads even when sprite rendering disabled, which is of questionable utility and makes the feature pretty useless for this Chicken of the Farm ROM because of so many false positives). If flash carts do actually impact decay timings and/or behavior even on healthy power supplies, having an option to run with these more strict timings would also be helpful (though in the case of decay possibly being faster than vblank as I described with Zelda, I'm not sure what the right solution is. OAM DMA during lag frames is a bad idea because it can draw sprites that aren't fully written yet. This is especially noticeable in Metroid with the HUD).
For casual play, I like accuracy and would prefer standard realistic timings and realistic decay, especially given that it's possible a game could require this at some point if decay is sufficiently random.
(All of this depends on getting good data about how this tends to work across different consoles, of course.)