A/V synchronization

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

A/V synchronization

Post by James »

In this thread, gdkchan asked about A/V synchronization in his emulator. Here, I'll discuss the method I developed for nemulator.

Most emulators sync to the audio playback rate and adjust the video rate to accommodate. This results in tearing (no vsync) or stuttering when scrolling (vsync). nemulator syncs to the video frame rate, and continuously adjusts the audio rate to match. The result is smooth video and glitch-free audio. Here's a block diagram of how it's done.

Code: Select all

                                           audio callback -> sound card
                                                     ^
                                                     |
                                                     |
input -> low pass -> resampler -> high pass -> output buffer
                         ^                           |  
                         |                           |
                         +---------feedback----------+

Input is audio at its native rate (1.79MHz in the case of NES). The input is low-pass filtered and then passed to the arbitrary rate resampler. The output from the resampler is high pass filtered, then stored in an output buffer. When the soundcard needs data, the callback function is invoked, pulling data from the buffer.

Each frame, a frame's worth of audio data is processed. When it is placed in the circular output buffer, the distance between the write and play cursors is measured. The goal is to keep this distance consistent between frames so that the audio buffer doesn't underflow, resulting in audio glitches, and so that no blocking occurs while waiting for the buffer to drain, resulting in delayed frames/video glitches.

The feedback block:
- Low pass filters the distance measurement
- Adds that measurement to a circular buffer containing the last 60 measurements (1 second)
- Computes the slope of those measurements
- Adjusts the resampler output rate based on how far the distance is from the target and whether it's converging on or diverging from that target

I've created a sample SDL app that implements this system. Source code is available on github, or you can download a Windows version here (requires sdl2.dll). I've only tested this on Windows, but it should work on other platforms. A few notes:
- nemulator uses DirectSound without callbacks, which permits for more accurate measurement of the audio buffer's cursors and, thus, better synchronization
- Accuracy in this app is limited by the callback frequency (every 512 samples), so some tweaking may be required
- Pre- and post- resampler filtering is not implemented
- Input to the resampler should be at least 8x oversampled
- The green line represents the distance measurement, the red line represents the resampler output frequency
- If your display isn't running at 60Hz, this won't work properly

Image

James
get nemulator
http://nemulator.com
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: A/V synchronization

Post by lidnariq »

Just for reference ... is the audio supposed to be a clean square wave?
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: A/V synchronization

Post by rainwarrior »

Hmm, do you find that the aliasing/distortion resulting from the jittering pitch is a worthwhile tradeoff though? Like, you're offering a choice between occasional skipped frames, tearing, or audio distortion, right? Not sure which is the best of these options. (Maybe ideally you can choose!)

One thing I might suggest, though, if you're using a sync-to-audio solution, you could slightly adjust the clock rate so that 1 frame worth of cycles = reported vsync rate. Doing this would greatly reduce the frequency of dropped frames, while detuning the output pitch just the slightest amount (at least for a 60hz monitor).
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A/V synchronization

Post by James »

lidnariq wrote:Just for reference ... is the audio supposed to be a clean square wave?
Yes, a 240Hz square wave.
rainwarrior wrote:Hmm, do you find that the aliasing/distortion resulting from the jittering pitch is a worthwhile tradeoff though?
Absolutely. I find visual artifacts, like stuttering while scrolling, to be intolerable. That said, the audio pitch changes made by this method are inaudble to me. On my system, the frequency in the SDL version varies over about a 30Hz range (.06%). The DirectSound version does even better, with a variance of only a few Hz. Both are below the threshold of audibility (https://en.wikipedia.org/wiki/Pitch_(mu ... difference).
rainwarrior wrote:you could slightly adjust the clock rate so that 1 frame worth of cycles = reported vsync rate
It may not have been clear, but this does rely on vsync at 60Hz*. So this is, in effect, exactly what I'm doing.

* Or, e.g., FreeSync and a 50Hz clock source.
Last edited by James on Fri Jan 20, 2017 12:53 pm, edited 1 time in total.
get nemulator
http://nemulator.com
User avatar
Disch
Posts: 1848
Joined: Wed Nov 10, 2004 6:47 pm

Re: A/V synchronization

Post by Disch »

In my emulator I allow the option. If you choose to sync to video frame rate, it will produce more or less audio to match.... which can be accomplished by running audio channels for more/fewer cycles as needed without actually changing the clock rate.

This is complicated by the DMC, which has CPU-visible side-effects (IRQs, stolen cycles). To solve that, I run 2 separate DMC units side-by-side. The "DMCPU" which is not audible but does the DMA and generates IRQs. And a separate audible DMC which is what you actually hear.

This results in smooth video + smooth audio playback with no pitch changes, regardless of the framerate (so it even works for something like Fast-Forward). Only exception being raw, timed $4011 writes, which will crackle a bit due to no longer being spaced properly. But games which use those are pretty rare.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: A/V synchronization

Post by lidnariq »

Whatever's going on here is mixing really badly with linux ALSA dmix, continuously emitting the "sample underrun" sound even without warning about it in the terminal.


Also, somehow, I managed to get it to start oscillating, which really doesn't sound good. (The green line rises at a constant slow rate, gets to about 2/3 up the window, then decays in a rapid exponential, then restarts)
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: A/V synchronization

Post by tepples »

Disch wrote:Only exception being raw, timed $4011 writes, which will crackle a bit due to no longer being spaced properly. But games which use those are pretty rare.
Such as Battletoads and most of the Wheel of Fortune games.

Here's an idea for an emulator feature: Count $4011 writes over the past several frames, and if it exceeds (say) 40 writes in 4 frames, log that. Then we can make a list of licensed games using raw PCM and cross-reference it with the developer/porter list from NesCartDB to improve my pun.
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A/V synchronization

Post by James »

lidnariq wrote:Whatever's going on here is mixing really badly with linux ALSA dmix, continuously emitting the "sample underrun" sound even without warning about it in the terminal.
Is vsync working/is it running at relatively consistent 60fps?

You may need to reduce the callback frequency (try changing wanted.samples to 1024) and/or increase BUF_LEN.
get nemulator
http://nemulator.com
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: A/V synchronization

Post by lidnariq »

James wrote:Is vsync working/is it running at relatively consistent 60fps?
Yup. Seems to eventually converge on 60.00Hz vsync and 47972Hz sample rate.
You may need to reduce the callback frequency (try changing wanted.samples to 1024) and/or increase BUF_LEN.
With SDL_AUDIODRIVER=alsa (and whatever default routing it's picking), the clicking frequency happens once every BUF_LEN samples.
With SDL_AUDIODRIVER=dsp (i.e. OSS emulation), or with AUDIODEV=plughw:0,0, it seems to work as intended
With AUDIODEV=dmix, it emits a pile of fuzz with minimal tonal content

I've seen the aforementioned oscillation show up all of the above audio methods.
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A/V synchronization

Post by James »

lidnariq wrote:I've seen the aforementioned oscillation show up all of the above audio methods.
In nemulator, there's some code to look for this condition and reset, but I didn't include that here. I'll update it.

No idea re: clicking issues with alsa. I'll see if I can reproduce it.
get nemulator
http://nemulator.com
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: A/V synchronization

Post by rainwarrior »

James wrote:
rainwarrior wrote:you could slightly adjust the clock rate so that 1 frame worth of cycles = reported vsync rate
It may not have been clear, but this does rely on vsync at 60Hz*. So this is, in effect, exactly what I'm doing.
What I meant was a fixed rate. You're adjusting the rate dynamically. I meant adjust the fixed rate (once at startup) so it matches your vsync and stick with it, so you don't get audio distortion from constantly changing it to try and match. You'd end up with no audio distortion and skipped visual frames almost never.

Also you misquoted this; I said it, not lidnariq.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: A/V synchronization

Post by tepples »

But how long would it take the emulator to accurately measure the actual vsync rate and the actual sample rate when it starts? A rate like the "47972Hz" that lidnariq reported isn't related in the expected 735:1 or 800:1 ratio. In addition, the oscillators in the video card and sound card may drift over time.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: A/V synchronization

Post by rainwarrior »

tepples wrote:But how long would it take the emulator to accurately measure the actual vsync rate and the actual sample rate when it starts? In addition, the oscillators in the video card and sound card may drift over time.
What?? Just query it from the hardware! Edit: Koitsu points out that APIs for this are inadequate. Maybe just measure it and let user override manually?

I don't believe drift over time is significant here. Have you really encountered this with modern monitors or sound cards? (e.g. I've never found a computer sound device to be measurably out of tune, at least not in any of the many times I've tuned a musical device to one.)

However. I did say this was part of the "sync video to audio" solution where occasionally you have to display a frame twice (or skip display of a frame) to resync. If there's a small amount of mismatch for some reason this will occasionally do that to correct. I said "almost never" because I'd expect it to be so infrequent as to be essentially nonexistant.

If your monitor isn't running close to the game's target refresh rate (i.e. 50 or 60hz) then you're going to be living with doubled/skipped frames anyway. Some people want NES accurate timing, too, so they also must live with that for its nonstandard 60.1hz rate. That tends to be quite noticeable though because it's a frame skipped every 10 seconds.


Edit: On the subject of drift over time this website claims to be a vsync test: https://www.vsynctester.com/index.html

On this laptop it thinks the real refresh rate is 60.002 Hz, so we're talking about one frame doubled every 8 minutes. Is that not insignificant? I think that's pretty good?
Last edited by rainwarrior on Fri Jan 20, 2017 9:09 pm, edited 1 time in total.
User avatar
James
Posts: 431
Joined: Sat Jan 22, 2005 8:51 am
Location: Chicago, IL
Contact:

Re: A/V synchronization

Post by James »

rainwarrior wrote:Also you misquoted this; I said it, not lidnariq.
Sorry about that -- copy/paste error. Fixed.
rainwarrior wrote:so you don't get audio distortion from constantly changing it to try and match
If the audio rate change is sufficiently small, there is no audible distortion.
rainwarrior wrote:I meant adjust the fixed rate (once at startup) so it matches your vsync and stick with it
higan used this method at some point, requiring the user to go through a process to measure video and audio rates. Those settings are no longer there, but byuu can probably comment on how well this method works.
get nemulator
http://nemulator.com
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: A/V synchronization

Post by rainwarrior »

James wrote:
rainwarrior wrote:so you don't get audio distortion from constantly changing it to try and match
If the audio rate change is sufficiently small, there is no audible distortion.
In the test program you posted, I find the displayed rate (once it settles, which takes a bit of time) is in a constant state of oscillation, and even minute adjustments are clearly audible through the varying timbre it produces. It never sounds stable to me.

Here, I made a recording. Does this sound significantly different than it does in your own tests?

Maybe another suggestion: would it be useful to smoothe the rate of change, maybe averaging over many frames and adjusting rate only once a second or something instead?
Attachments
audio_sync.flac.zip
(1.52 MiB) Downloaded 337 times
Post Reply