Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

USB headsets to digital audio workstation software...
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sat Sep 11, 2021 6:08 pm

Many folks, including myself, have been plagued by resampler instability. Either at program startup, or whenever VAC is restarted (like on any change in VAC settings), or just randomly, over and under-flows will start accumulating, sometimes at a very high rate.

Recently, Richie exposed a couple of controls, accessible via the "Advanced" tab. Of the two, really only the Feedback Gain setting matters with respect to the aforementioned stability.

For a couple of days now I've been messing with the Feedback Gain settings, unfortunately with little improvement.

The standard, default setting for Feedback Gain is 4x10^-6, aka 0.000004, or 4 millionths of whatever the unit is. In the interests of brevity I'm just going to call this "4", and other settings will be relative to that.

I've tried settings between 5 and .1, relative to the default setting of 4. 4 definitely seems to be the largest setting that gives good performance. Lower settings seem marginally more stable. But there's really no difference in gross stability at any setting between 4 and .1. I can be sitting in front of the machine, and everything might be fine for 20 minutes or 200 minutes. Then, for no apparent reason, over or underflows just start rapidly accumulating. As usual, to stop this I will cycle one of the Ringbuffer manual checkboxes to get VAC to restart.

I'm now trying the absolute minimum setting possible, which is .01 relative to 4 (or 0.00000001 if you prefer). More to follow with those results.
w9mdb
Posts: 446
Joined: Sun Apr 09, 2017 5:53 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w9mdb » Sat Sep 11, 2021 9:50 pm

Where does one get this version? I don't seem to see any update to the git projects.

Mike W9MDB
Mike W9MDB
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sat Sep 11, 2021 9:52 pm

Mike,

It's -21h. I was just trying to show that it was anything that revision or newer.

Scott
w9mdb
Posts: 446
Joined: Sun Apr 09, 2017 5:53 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w9mdb » Sat Sep 11, 2021 10:05 pm

Mike W9MDB
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sat Sep 11, 2021 10:49 pm

I guess you have not been keeping up with the forum. See here:

viewtopic.php?f=9&t=3817
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby W4WMT » Sun Sep 12, 2021 10:28 am

[quote="w-u-2-o"]
For a couple of days now I've been messing with the Feedback Gain settings, unfortunately with little improvement.
/quote]

This, I believe, is the same setting that Walter found to be the panacea for all of his resampler woes. It's frustrating that results from one user to the next are so utterly different!
User avatar
DL2XY
Posts: 116
Joined: Sun Jul 30, 2017 9:47 am

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby DL2XY » Sun Sep 12, 2021 12:33 pm

W4WMT wrote:
w-u-2-o wrote:For a couple of days now I've been messing with the Feedback Gain settings, unfortunately with little improvement.
/quote]

This, I believe, is the same setting that Walter found to be the panacea for all of his resampler woes. It's frustrating that results from one user to the next are so utterly different!


I too can't replicate stability with the values used in my patched 2.8.11 release?!
In fact changing values has little to no effect at all.
Maybe the differences between 2.8.11 rel. and 2.8.11.h2 are too big, there a quite a few additional [threads] functionalitys running wich are even more garbling the buffer count sampling times?

To analyse the problem in depth i would need access to the actual source, maybe Richi can provide on NDA basis?

73 Walter
Last edited by DL2XY on Sun Sep 12, 2021 2:44 pm, edited 1 time in total.
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby W4WMT » Sun Sep 12, 2021 1:12 pm

More threads you say, Walter?
Now that's interesting...
w9mdb
Posts: 446
Joined: Sun Apr 09, 2017 5:53 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w9mdb » Sun Sep 12, 2021 1:13 pm

Using 21h and 2.1.18 firmware on 7000DLE MKII.

The wideband behavior and panscope look odd in this video.
I see base level variations in the wb that are worrisome when average is turned off.
And the way an FT8 transmission looks is odd. It starts off with the noise level in the panscope high and the wb display shows jittering noise level. Then the panscope level decreases and the wb stabilizes after about 4 seconds of transmit both at the same time.
This has been going on for a long time...P1 and prior versions so not new behavior.

https://www.dropbox.com/s/4idhmw7socq93 ... .webm?dl=0

Another thing I noticed while playing with wb and panscope is the 0Hz line -- the level does not seem to depend on a signal being present but does oscillate when an antenna is unplugged/plugged. I assume this is a bogus frequency resulting from the FFT 0th element.
Untitled.png
Untitled.png (233.91 KiB) Viewed 48551 times
Mike W9MDB
User avatar
DL2XY
Posts: 116
Joined: Sun Jul 30, 2017 9:47 am

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby DL2XY » Sun Sep 12, 2021 3:16 pm

W4WMT wrote:More threads you say, Walter?
Now that's interesting...


Sorry, my fault. In fact the release has more (72) threads running than h2 (68).
So the extensions ( gradient filling, blobs,active peak hold and many others) must be implemented in already existing thread extending their slicetime.

Comparing processor load charts of both versions shows sligtly more positive fluctuations in h2, but i dont realy belief this small difference is the problem. It' more the high fluctuation at all that make it difficult to get a stable ringbuffer consumption measurement for the regulation loop (have a look at the unaveraged % values in the new VAC Monitor tab!) .

But i'm stabbing in the dark, i can only speak for shure on the release, wich is still running a quite stable SR with my patch (only about +/-5ppm pp).

Walter
G3ZQH
Posts: 111
Joined: Mon Apr 10, 2017 5:21 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby G3ZQH » Sun Sep 12, 2021 4:38 pm

I also have instability problems with my VAC and the re-sampler which have been present for all versions of Thetis/firmware over the years. Of interest, I also use SDR Console software with my 200D. Although this lacks many of the ANAN refinements present in Thetis, one of the big advantages for me is that the virtual audio (using Voicemeeter Potato) is completely rock solid. I do not see any of the dropouts, re-sampler problems, instabilities etc that occur with Thetis. With SDR Console the audio just works flawlessly in the background. For digimodes I use SDR Console in preference to Thetis. For CW and SSB I find the extra features in Thetis have advantages (filters, noise reduction, PureSignal, +others).

So possibly there is something to learn from how the audio is processed in SDR Console if this is accessible?

73 Dave G3ZQH
User avatar
DL2XY
Posts: 116
Joined: Sun Jul 30, 2017 9:47 am

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby DL2XY » Sun Sep 12, 2021 5:18 pm

I got a stable solution on Thetis 2.8.11H2:

H2-Stable-1.png
H2-Stable-1.png (16.95 KiB) Viewed 48519 times

H2-Stable-2.png
H2-Stable-2.png (10.73 KiB) Viewed 48519 times


Some Hints:

We want buffering to be done mostly in the ringbuffer, so set PortAudio buffer to manual 1. Do not use "0", which is a "magic number" telling windows to use an automatic approximation (30 to 60ms), it would be the same as manual off. Edit: It had been turn out this is not true for PortAudio: 0 or 1 should make no difference -Walter

Set ringbuffer size to high values, more means higher resolution in rate measurement and less fluctuations. On the downside you will get more latency.

Set VAC buffersize to lowest possible value. Lower values mean more (smaller) buffer transfers per second, reducing fluctuations in time.

Disable cpu power managment. A variable CPU-Clock would be catatrophic to the regulation loop.

Run Thetis at high priority. Everthing running with higher priority on the PC will defer timesclice assignment to thetis causing more fluctuations.

I had to modify FB-Gain and Slew-Time a small amount to adopt to 2.8.11H2 thread timing behavior. Note there is a settling time of 2-3 minutes before ratio gets stable within +/-2 ppm.

73 Walter
Last edited by DL2XY on Tue Sep 14, 2021 11:58 am, edited 1 time in total.
User avatar
DL2XY
Posts: 116
Joined: Sun Jul 30, 2017 9:47 am

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby DL2XY » Sun Sep 12, 2021 5:55 pm

G3ZQH wrote:I also have instability problems with my VAC and the re-sampler which have been present for all versions of Thetis/firmware over the years. Of interest, I also use SDR Console software with my 200D. Although this lacks many of the ANAN refinements present in Thetis, one of the big advantages for me is that the virtual audio (using Voicemeeter Potato) is completely rock solid. I do not see any of the dropouts, re-sampler problems, instabilities etc that occur with Thetis. With SDR Console the audio just works flawlessly in the background. For digimodes I use SDR Console in preference to Thetis. For CW and SSB I find the extra features in Thetis have advantages (filters, noise reduction, PureSignal, +others).

So possibly there is something to learn from how the audio is processed in SDR Console if this is accessible?

73 Dave G3ZQH


Yes, there is.

For a pure digital signal chain, for example [Thetis > VAC > WSJTX > VAC > Thetis] the adative samplerate converter is not usable!
In this scenario there is no second clock oscillator involved, every part of the chain is only processing buffers as fast as it can. This is not even done in realtime, it only must be in time to prevent buffer underrun. If using the Converter it gets an input buffer back every time it outputs a buffer, so the regulation-loop is chaising its own tail. Thats the reason for oscillating samplerate with the well known "serpentine waterfall".

The samplerate converter is (only) usefull if there is another (hardware) clock involved, for example by using an audio-interface or soundcard.

Walter
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sun Sep 12, 2021 9:02 pm

Edited to preface the following with a note of clarification: the stability problems I'm seeing have to do with the resampler occasionally running wild. Not the short or even long term stability problems that plague some folks who are running such modes as FT8. Where that sort of stability is concerned I have no problems at all. My FT8 and other digital signals are rock solid on receive and transmit.

But every once in a while over and underflow counters start to accumulate at a rapid clip, and things will stay that way for many minutes if I do nothing, then it will all go back to normal for a period, and then it happens again. There is no rhyme or reason, no obvious change in PC state, no regularity to it at all.

DL2XY wrote:We want buffering to be done mostly in the ringbuffer, so set PortAudio buffer to manual 1. Do not use "0", which is a "magic number" telling windows to use an automatic approximation (30 to 60ms), it would be the same as manual off.

This is an excellent, and important, detail. While I had found this to be true empirically, I never knew why "0" was a poor choice. Thank you, Walter!

Note: I've been running with PortAudio set to "1" for many, many months now, so this is not a factor for me.

Set ringbuffer size to high values, more means higher resolution in rate measurement and less fluctuations. On the downside you will get more latency.

This is not something I'm willing to do. It is indicative of a problem in the code, or algorithm, and also generates far too much latency for my taste. And it was a solution that worked without exposing the Feedback Gain setting or requiring any adjustments to that setting.

Currently my Ringbuffer sizes are both "0".

Set VAC buffersize to lowest possible value. Lower values mean more (smaller) buffer transfers per second, reducing fluctuations in time.

Not something I can do with the hardware and drivers I have. 256 is as short as I can get it without clicks/pops. This is a function of the interface. I have tested this outside of Thetis. And, again, it should have no bearing on resampler stability, it is merely a band-aid.

Disable cpu power managment. A variable CPU-Clock would be catatrophic to the regulation loop.

Not sure this is true, or even if it matters. For while Thetis runs, why would the CPU go into any sort of power management state?

Run Thetis at high priority. Everthing running with higher priority on the PC will defer timesclice assignment to thetis causing more fluctuations.
I have tried this. It has had no effect on stability here. It does have an effect on clicks/pops in terms of certain threads being blocked and not completing in time. I normally run with it set to "Real Time".
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sun Sep 12, 2021 9:06 pm

DL2XY wrote:
The samplerate converter is (only) usefull if there is another (hardware) clock involved, for example by using an audio-interface or soundcard.

Walter

That doesn't make sense to me, Walter. There is a clock in the ANAN hardware that has set the sample rate of the IF data, and the audio data sample rate is derived from that. Meanwhile that data is dumped into the PC audio hardware which is running at a slightly different sample rate.

So there definitely are two different clock domains that need to be reconciled, with different hardware in each one of them.
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sun Sep 12, 2021 9:15 pm

G3ZQH wrote:So possibly there is something to learn from how the audio is processed in SDR Console if this is accessible?


The answer to that question is an unqualified "Yes!"

However, SDR Console is not "accessible". It is proprietary code, and Simon Brown is not going to share it.

However, however :) , there is plenty of great audio processing code on the internet free for inspection and possibly even wholesale use. Code that successfully solves the stable and glitch-free transfer of audio data across multiple clock domains in a very complex environment, and with low latency.

Perhaps the most extreme example of this is the open source digital audio workstation (DAW) software, Ardour. I have used Ardour and it works brilliantly. Personally I prefer Reaper because its UI and workflow better match my requirements, but Ardour is a top shelf, the equal of any other DAW out there. And, again, Ardour is open source.

I've suggested to various dev's that Ardour be inspected as a means of achieving better performance in Thetis, but it would be a TON of work for someone to do that and then make wholesale changes in Thetis. It would be a major effort.
User avatar
ramdor
Posts: 1468
Joined: Wed Jul 03, 2019 3:07 pm
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby ramdor » Sun Sep 12, 2021 10:59 pm

I just quickly set up some values from Walters examples above (interesting to know about the 0/1 setting for PA), and let it sit for a few minutes.

Initially var ratio was swinging either side of 1.0, but eventually settled, perhaps took 3-4 minutes. I am not sure if it will switch into an unstable state at some point however. Defaults used for feedback/slew. I must add that I tend to ignore all this stuff, sick everything on forced etc. and just work datamodes ;)

So the issue is that it will become 'settled', and then for some reason just becomes unstable? I wonder if something is going on with usb or audio sub system, perhaps power management.

As I type this, it has been 1.000000 for 5+ mins but I have just noticed that it is starting to swing around again on the 'TO VAC' and has reached 1.000000 again and has settled there. This was followed by 'FROM VAC' doing the same thing, but has settled again.

R.

resampler1.jpg
resampler1.jpg (107.81 KiB) Viewed 48479 times
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Sun Sep 12, 2021 11:07 pm

It can take hours, sometimes, Richie. I normally leave everything running 24/7. If I leave for a few hours and come back everything will be fine but the under and overflow counters will show many thousands of counts accrued.
K1LSB
Posts: 639
Joined: Wed Feb 05, 2020 5:25 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby K1LSB » Sun Sep 12, 2021 11:21 pm

DL2XY wrote:We want buffering to be done mostly in the ringbuffer, so set PortAudio buffer to manual 1. Do not use "0", which is a "magic number" telling windows to use an automatic approximation (30 to 60ms), it would be the same as manual off.

73 Walter

Where did you get that information, please?

TIA,
Mark
User avatar
ramdor
Posts: 1468
Joined: Wed Jul 03, 2019 3:07 pm
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby ramdor » Mon Sep 13, 2021 12:40 am

w-u-2-o wrote:It can take hours, sometimes, Richie. I normally leave everything running 24/7. If I leave for a few hours and come back everything will be fine but the under and overflow counters will show many thousands of counts accrued.


i'll leave it running (actually it has been for about an hour now whilst playing Deep Rock Galactic with some friends... lol). I wonder if may be perhaps being upset by screensaver/monitor standby which depending on monitor may have audio over hdmi etc. I'll report back tomorrow.

R.
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby W4WMT » Mon Sep 13, 2021 1:47 pm

w-u-2-o wrote:I've suggested to various dev's that Ardour be inspected as a means of achieving better performance in Thetis, but it would be a TON of work for someone to do that and then make wholesale changes in Thetis. It would be a major effort.


Just for the record: Ardour uses the popular "Zita" resampler, which was indeed considered early-on for Thetis. Warren's evaluation of the Zita was that its spur output rejection would be insufficient for Thetis. Here, he was not thinking of phone or cw, but the digi modes. Our modern digi modes can operate at such fantastic (s+n)/n ratios that the spur output from Zita would become the limiting factor, rather than the digi protocol itself. So he designed a super-low spur adaptive variable resampler from scratch. So that's how we wound up with what we have now.

The resampler is very "compute intensive", but that was known from the beginning. And by that I don't mean average CPU usage on time-scales of seconds, but rather peak usage over time-scales on the order of microseconds.

73 to all,

Bryan W4WMT
User avatar
DL2XY
Posts: 116
Joined: Sun Jul 30, 2017 9:47 am

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby DL2XY » Mon Sep 13, 2021 4:06 pm

w-u-2-o wrote:
DL2XY wrote:
The samplerate converter is (only) usefull if there is another (hardware) clock involved, for example by using an audio-interface or soundcard.

Walter

That doesn't make sense to me, Walter. There is a clock in the ANAN hardware that has set the sample rate of the IF data, and the audio data sample rate is derived from that. Meanwhile that data is dumped into the PC audio hardware which is running at a slightly different sample rate.

So there definitely are two different clock domains that need to be reconciled, with different hardware in each one of them.


Thats right, if you are using additional audio hardware in your signal chain there are 2 clock domains present and the resampler may be usefull.

But there also is a complete different scenario, the typical DIGI-MODE setup:
Thetis connected via virtual audio drivers (thats software) to one of the many digimode applictions like WSJT-X for example. In this case everything in the outer chain is software, there is no second clock domain, audio buffers are exchanged in a loose timing manner, just fast enough to support the data rate. Here the resampler is not needed, in fact it has problems to determine the incomming buffer rate because it is measuring a delayed version of it's own outgoing rate, scrambled by windows multitasking timeslicing. You definitly want to set the resampler to fixed 1.0 in this case.

But carefull, there may be traps: For example the virtual audio interfaces of Voicemeter are quite good for digimodes, as long as the DAW is not started.
If the DAW is running it redirects the audio streams trough itself syncronizing them to its main output interface and there is the second clock domain again!

Walter
User avatar
ramdor
Posts: 1468
Joined: Wed Jul 03, 2019 3:07 pm
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby ramdor » Mon Sep 13, 2021 4:14 pm

I ran it all night, around 15hrs, power saving for screens disabled. No buffer under/overruns reported. During the first 2-3hrs I was actually multiplayer gaming with some friends using the same PC without any impact on Thetis it seemed.

I did however discover some interesting behaviour, which can be seen in the following vid. Something interesting and odd happened around 8mins into the vid.

https://youtu.be/i1WDMTF5Yao

Richie.

resampler2.jpg
resampler2.jpg (111.72 KiB) Viewed 48390 times
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Mon Sep 13, 2021 4:23 pm

Bryan & Walter: great info from both of you.

Richie: I think you are not seeing the problem because you are using a very high Ringbuffer value. I typically use a value of "0".

I've increased mine to 32 just to see if that seems to correct the problem. If so, then a multi-day witch hunt will have to start in order to find the lowest value that is unconditionally stable over a 24 hour period. Perhaps that will provide some valuable data.
User avatar
ramdor
Posts: 1468
Joined: Wed Jul 03, 2019 3:07 pm
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby ramdor » Mon Sep 13, 2021 4:36 pm

w-u-2-o wrote:Bryan & Walter: great info from both of you.

Richie: I think you are not seeing the problem because you are using a very high Ringbuffer value. I typically use a value of "0".

I've increased mine to 32 just to see if that seems to correct the problem. If so, then a multi-day witch hunt will have to start in order to find the lowest value that is unconditionally stable over a 24 hour period. Perhaps that will provide some valuable data.


Running test again with 8/8 for the RB's

R.
User avatar
Tony EI7BMB
Posts: 651
Joined: Sun Apr 09, 2017 2:31 pm
Location: Dublin
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby Tony EI7BMB » Mon Sep 13, 2021 5:12 pm

I notice my VAR ratios never really settle down, I thought this was the normal behaviour for this but perhaps not. here's my set up.


audio.png
audio.png (654.54 KiB) Viewed 48368 times
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby W4WMT » Mon Sep 13, 2021 5:35 pm

Tony EI7BMB wrote:I notice my VAR ratios never really settle down, I thought this was the normal behaviour for this but perhaps not. here's my set up.


Hi Tony,

What you show there is perfect! It's completely normal for the var ratios to "hunt" slowly in the last 1 or 2 digits. The resampler is doing its job in a textbook fashion. Your overflows & underflows are zero. And the loop is steering the sample rates in order to keep the ringbuffer exactly half full.

Perfect in every way!

73!
User avatar
Tony EI7BMB
Posts: 651
Joined: Sun Apr 09, 2017 2:31 pm
Location: Dublin
Contact:

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby Tony EI7BMB » Mon Sep 13, 2021 6:16 pm

Thanks, that's reassuring
User avatar
w-u-2-o
Posts: 5539
Joined: Fri Mar 10, 2017 1:47 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby w-u-2-o » Mon Sep 13, 2021 7:48 pm

Still randomly unstable at RB 32. Not worth going bigger, that's already a ridiculously large value.
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Thetis 2.8.11-21+++ VAC advanced controls: results and discussion

Postby W4WMT » Mon Sep 13, 2021 8:34 pm

w-u-2-o wrote:Still randomly unstable at RB 32. Not worth going bigger, that's already a ridiculously large value.


Do the random instabilities vanish when you Force=1 ?

Return to “Digital ("Virtual") Audio”