LatencyMon Benchmarks

Post your awesome projects here for everyone to marvel at!
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Sat Jul 22, 2017 4:37 pm

There are no benchmarks the way you want them, Gary. What people have posted here are "informal" benchmarks. That is to say, people have posted what they are getting and how it works for them.

73,

Scott
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Sat Jul 22, 2017 4:44 pm

and sure enough - go back and let LatencyMon run and within a few minutes it starts to rise ... it now says it is not suitable for audio and the values are above 3mS for the Highest Reported DPC and ISR.

I guess I'll need to try to hunt down what is causing it - assuming it is one thing. It says the 'culprit' is NDIS.sys (network driver), suggesting I turn off the wifi - its not on ... and wdf01000.sys or Kernel Framework Driver Routine for the DPC (says to check cpu settings).

I'll do a google search on these - if anyone knows up front (wouldn't that be fortunately) what to look for I'd (and others I'm sure) would appreciate it.

Gary

still can't say that I've ever heard any anomalies re the audio though.
User avatar
WA0VY
Posts: 70
Joined: Sun Apr 09, 2017 3:41 pm

Re: LatencyMon Benchmarks

Postby WA0VY » Sat Jul 22, 2017 9:51 pm

Gary, I'm not aware that there any published "benchmarks" showing what others are seeing in LatencyMon (LM). There are published benchmarks associated with Passmark, but that is measuring overall and system sub-component speed performance, which is not the same as latency. As far as where you are good to go in terms of latency, obviously, the lower, the better. But, if LM is indicating that you are capable of running real-time audio, and you are not seeing audio drops and pops, you're there. I am not presently seeing any drops or pops even though LM indicates occasionally spikes associated with the NVIDIA driver. I am currently using Windows Performance Analyzer (WPA) and related Recorder to try to track that down. Its a very complex program but it is giving some useful, and surprising, data. More on that later.
73 Brent WA0VY
User avatar
WA0VY
Posts: 70
Joined: Sun Apr 09, 2017 3:41 pm

Re: LatencyMon Benchmarks

Postby WA0VY » Sun Jul 23, 2017 12:56 am

I did not realize that this thread had gone into page 2 when I posted my last response so I did not see Scott’s or your last response before I posted. Sorry to hear that you have the dreaded latency issue. However, if it’s not causing drops or pops, resolution is more a matter of seeking perfection than necessity. But I have latency OCD so I’m not one to suggest that any one shouldn’t chase this thing down and put a stake through it. I think you will be disappointed with the info you will get from Google. Other than updating your drivers, most of the information is so generic and unspecific to your build and software environment as to be basically worthless.

As mentioned, I am currently using Windows Performance Analyzer (WPA) to trace this. It’s a time-consuming and complex process. You can’t run the recorder too long as the files become huge and unmanageable. So, I had to keep running the recorder while all potential problem software was running as well as latencyMon (LM) and dump the data if I didn’t see an interesting event within one minute. After about 10 minutes in on 7/19, DPC latency jumped to early red and I captured it in about 31 seconds of data. I processed it and will show you in some screen shots what I found. Again, every different build and software environment will probably show something different, but if you download and run WPA, you can get a pretty good idea what your specific issues are. The first screenshot is of the entire 31 second recording. I arranged the data to show GPU and CPU usage in detail. I could see from the other graphs that nothing interesting was happening in storage, power or memory. The event occurred at about 18.5 seconds in and is pretty obvious in the graph.

ScrnShot 2.jpg
ScrnShot 2.jpg (445.2 KiB) Viewed 1934 times


The second shot shows the mouse hovering in the center of the event generating a horizontal line through both graphs which indicates that (a) Firefox was the process involved and (b) it did not cause a spike in CPU usage, only in GPU activity under rendering. That wasn’t a huge surprise since I am using a Xeon E5 2699 v4 with 22 cores, so it takes a serious anomaly to get its attention.

ScrnShot 3.jpg
ScrnShot 3.jpg (448.93 KiB) Viewed 1934 times


The next shot shows (in the blue spikes) an “unknown” process is generating some GPU activity/latency due to “memory transfers.” I have no idea exactly what that means but I note that there is no unusual system RAM data so I assume that is in the graphics GDDR.

ScrnShot 4.jpg
ScrnShot 4.jpg (442.63 KiB) Viewed 1934 times


The next shot shows the fairly large amount of time that was involved in rendering in Firefox at the inception of the event. It is about 60% of all the rendering time used by Firefox in the entire 31 seconds of the recording, so A LOT! Hence, the DPC spike.

The last shot shows the comparative highest amount of time involved in the mystery “memory transfer” issue. It suggests that is not the problem, whatever process is causing it.

So, what do I know now that I didn’t before? Microsoft Edge will become the only browser up when I’m running HPSDR.
73 Brent WA0VY
User avatar
WA0VY
Posts: 70
Joined: Sun Apr 09, 2017 3:41 pm

Re: LatencyMon Benchmarks

Postby WA0VY » Sun Jul 23, 2017 12:59 am

For whatever reason, this blog would not allow me to post the last two shots in the previous post, so here they are:

ScrnShot 5.jpg
ScrnShot 5.jpg (496.98 KiB) Viewed 1934 times


ScrnShot 6.jpg
ScrnShot 6.jpg (495.24 KiB) Viewed 1934 times
73 Brent WA0VY
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Sun Jul 23, 2017 1:18 am

That's some badass analysis, Brent!

73!

Scott
User avatar
W1JA
Posts: 13
Joined: Sun Apr 09, 2017 3:19 pm

Re: LatencyMon Benchmarks

Postby W1JA » Mon Jul 31, 2017 8:13 pm

I'm not having any issues, but I was intrigued by this LatencyMon thread. Here are my results, taken recently while tuning around, having a QSO, running PowerSDR, DXlab, Chrome, VMB, VSPE, Pidgin.
Image

I built this PC in February. It's
Gigabyte GA-Z270XP SLI motherboard
i7-7700T
nVidia GTX 750Ti
Samsung 960 EVO m.2 SSD
16GB DDR4 2400 RAM

73,
John, W1JA
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Mon Jul 31, 2017 8:56 pm

Nice!
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: New PC Build: now UPDATED with overclocking performance

Postby K9RX » Mon Jul 30, 2018 3:46 pm

Scott,

I did a search and didn't find any dedicated thread on using and interpreting Latency Mon ... I know it is mentioned in this thread ... please let me know if i need to move it ...

so I have a new computer - it is in the 12,500 range for passmark for the CPU, 96%, and 91% for memory but is way behind in graphics (using the onboard Intel 630 for now, card not installed yet) and in disk (no SSD yet, researching whether or not it will do an NVMe or needs just a normal SSD) ...

Latency mon is better than my currently used i5-4460 .... but it still says unacceptable typically in an hour or so of running - always tcpip.sys. So how does one go about finding out whether or not I have the latest version (and then how do you upgrade it if you don't)? I have upgraded everything on the computer from the "brand new out of the box" state using HP's facilities ... that included BIOS, Graphics, Audio... nothing else is being reported as needing an upgrade... my suspicion is, since Windows was the first thing it upgraded (to the latest) it has the latest tcpip.sys ... so I think this is a red herring and i'll not be able to do a great deal (read: anything) about it ... it is currently using wifi as its in the dining room and not connected to ethernet as it will be when in the shack.

Gary
K9RX
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: New PC Build: now UPDATED with overclocking performance

Postby K9RX » Mon Jul 30, 2018 4:04 pm

actually the passmark on the CPU is 14842!

Gary
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Mon Jul 30, 2018 5:23 pm

Gary,

This is the main LatencyMon topic. It is the only topic with the title "LatencyMon...", which a quick search easily reveals. I have moved your posts here.

I am by no means an expert on how to solve LatencyMon reported problems. I have had good luck updating drivers, and finally built a new, very fast, computer. I use a combination of Windows Update, PC manufacturer update utilities, and specific, individual component (e.g. NIC card or NIC chip) update utilities.

One very good idea is to disable devices and/or services that are giving you problems to confirm they are the source, such as Wi-Fi NICs, etc. After that it tends to be a lot of googling to see if anyone else has lucked onto the solution. Mostly you find the best info on gamer and DAW websites, since those are really the only other demographics that are badly affected by these problems.

I don't think I'm writing anything you don't already know. Unfortunately it is not always an easy problem nor an exact science to make LatencyMon happy.

73,

Scott
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Mon Jul 30, 2018 5:30 pm

Thanks Scott... unfortunately a search is only as good as knowing what to search for - I searched for Latency Monitor, not LatencyMon! So now they're both linked to the same thread.

Ok - I'll keep trying to find out where I get access to tcpip.sys... I looked under Device Manager and wasn't able to locate it under many categories ....I'll keep searching.

Gary
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Mon Jul 30, 2018 6:48 pm

I searched on "latencymon tcpip.sys" and found this search result interesting:

https://communities.intel.com/thread/54594
User avatar
WA0VY
Posts: 70
Joined: Sun Apr 09, 2017 3:41 pm

Re: LatencyMon Benchmarks

Postby WA0VY » Tue Jul 31, 2018 12:40 am

w-u-2-o wrote:I searched on "latencymon tcpip.sys" and found this search result interesting:

https://communities.intel.com/thread/54594


Well, hmmmh. Soooo, let me see if I got it: 20 some pages later, MS is still working on it ("we feel your pain"), the thread finally gets into the 21st century, a few posted highly meaningful, good-looking LM results for exactly 6 seconds (exaggeration, kinda), traffic on the issue died out (most likely out of frustration) and still no apparent generic solution. Distilling what seemed pertinent, I decided to experiment with some NIC power settings although I am no longer experiencing any audio glitches of any kind. I have long since found and disabled every possible power saving feature in the BIOS and Win 10. However, I found and disabled the "Energy Efficient Ethernet" option on the Advanced menu of the dedicated NIC in Properties and did notice that when I powered HPSDR back up, I had "0" under/over flows and "0" OOOPS from the git-go. I have never seen that before. Usually, there is some initial instability and then it becomes highly stable. Time will tell if that is meaningful or another rabbit hole. Stay tuned.

73 Brent WA0VY
73 Brent WA0VY
User avatar
WA0VY
Posts: 70
Joined: Sun Apr 09, 2017 3:41 pm

Re: LatencyMon Benchmarks

Postby WA0VY » Tue Jul 31, 2018 1:21 pm

Time will tell if that is meaningful or another rabbit hole. Stay tuned.


Rabbit hole. Never mind.
73 Brent WA0VY
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Wed Aug 01, 2018 12:30 pm

Re: LatencyMon, a setting change and a bit of improvement

So I ran my current shack computer, an i5-4460 on LatencyMon again - ran it for over an hour - never fell out of "suitable"! This new super fast i7 within 30 minutes would fall out ... frustrating. Then I found a setting I hadn't seen before. Under CONTROL PANEL | SYSTEM | Advanced System Settings | Advanced then the top line "Performance, settings" selection ... it was defaulted to "let the computer decide", changed to max performance and restarted the computer. Now the i7 seems better - it has run for as much as 90 minutes without falling out ... but eventually it will. Now more so with the second line "highest measured interrupt to process latency" ...and there is no indicated 'chief offender' listed for this one ... searching for this one I've not found, surprise surprise, a solution or even a good method to track it down.

I've got a few days before I run out of time to return this computer if I decide to do so ... so far other than passmark its not been an overwhelming standout. Maybe just having a very fast processor just isn't enough.

Gary
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Wed Aug 01, 2018 6:48 pm

so apparently I lucked out and hit some of the items that need to be (re)set. I added another one after that last one - I don't remember where it was ... the new computer has now run for over 6 hours and still is "acceptable" ... although I suspect JUST so as the one category that it previously failed at is sitting at 860uS and I suspect 1mS is the trip point ... which got me thinking - I assume that no one knows the relevance of the numbers - the actual worst case numbers that are posted and their impact on real world results re clicks/pops and tics ... ? The numbers are just very slightly better than the i5 system ... but would the pure processing power of the new system (passmarks 200% one compared to the other) make other potential 'issues' better? For example I cannot run above 192K on the i5 ... it really acts up in strange ways ... now for the most part I (almost) never use above 92K which is perfectly suited to my needs (except when working split on 40 when I need to go to 192K) ... but I'll be curious if the i7 works in this area ... also hopefully, the reason why I bought the i7 was to future proof - hopefully if and when we have an extra VAC (and/or one dedicated to I/Q for CW Skimmer) and I use Skimmer it runs fine ...

I ask about the values because thinking about it - just simplistically, a 1mS latency in response back from an interrupt would mean a huge loss of data for a 92Khz stream ... or at least it seems. Yet that isn't the case - so I wonder how the numbers relate overall.

Gary
User avatar
w-u-2-o
Posts: 1172
Joined: Fri Mar 10, 2017 1:47 pm

Re: LatencyMon Benchmarks

Postby w-u-2-o » Wed Aug 01, 2018 8:55 pm

Gary,

It sounds like you are making progress.

As for where LatencyMon and DPC Latency Checker set their pass/fail limits, I have no idea.

I'd suggest, however, that anything that stalls the primary DSP processing thread in PowerSDR (or Thetis) for longer than either a half or certainly a full DSP buffer time period is likely to generate bad results. I'm guessing quite a bit, but if we use the DSP buffer size in Setup > DSP > Options as a guide, then consider that 64 IQ samples at 192KHz sample rate = 333uS.

I'd further suggest that the real problem is in time spent away from the primary DSP thread is the real issue, not problems with VAC. The audio runs at a sample rate of 48KHz. With a buffer size of 256 samples that's 5.3mS, which is a long, long time.

73,

Scott
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Fri Aug 03, 2018 10:11 pm

Just as an aside, for me an important one, I had changed that setting:

CONTROL PANEL | SYSTEM | Advanced System Settings | Advanced then the top line "Performance, settings" selection ... it was defaulted to "let the computer decide"

and this improved or reduced the latency issues on the new computer .... with this setting alone it went for 10 hours without 'tripping' .... I did the same setting on my current radio computer, the i5 ... and after having shut it down and back up again noticed that the fonts were woefully dim and thin and ill-defined... it was this setting! So the trade off is performance for poor visual display .... a real shame.

Gary
Joe-W4WT
Posts: 52
Joined: Sun Apr 09, 2017 5:27 pm
Location: Cumming, GA

Re: LatencyMon Benchmarks

Postby Joe-W4WT » Sun Aug 05, 2018 7:22 pm

Gary,

I've recently been able to reduce my settings to the absolute minimum with rock solid performance. My Ring Buffers are 512, 512. I am running with Primary audio setting at 64 and VAC at 128. I have Buffer Latency at zero. I run HOURS with no uptick in overflows/underflows or OOOPs now. I typically see only the last digit of the "Var Ratio" changing while running with the next to last sometimes changing. Here is how I managed this...

1) I went to the advanced settings on my network card that attaches to my Anan 8000 board and set virtually everything OFF/Disabled. I increased both receive and transmit buffers as high as they would go, 2048. I did leave "Interrupt moderation" enabled and the moderation rate is set to "Adaptive". I also have "Packet Priority" enabled.

2) I then found that under DSP Options, SSB/AM TX buffers MUST be higher than 64. I tried 128 then settled on 256 and that seemed best for me. I set all the other TX buffers at 128 for CW, FM, and Digital.

3) I use VoiceMeeter Banana and have the Behringer UMC202HD. My buffer settings on the UMC202HD are 128 samples. Safe mode unchecked. In VoiceMeeter I have "Buffering ASIO" set to default. All my sound settings are as Scott suggested; 48khz. I use 24 bit. I'm sure 16 bit is probably fine too.

All of the above had to be done to get to the levels I'm now running.

My CPU is an I7-3770. Not super fast. It runs a Passmark of 4674, CPU Mark of 9552. I typically run Firefox with multiple screens, including radar tracking, Dxmaps, my local security cameras displayed, all the DxLab apps running, JTDX, and JTAlert, and email (Thunderbird). So lots of apps running.

I don't know if any of the above settings might help you but thought I would provide what I had done here just in case.

73,

Joe W4WT
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Mon Aug 06, 2018 1:10 pm

Joe,

Thanks for the post - unfortunately some of this doesn't relate directly to my setup where I am using, what is it WDS or something like that under VAC ... I'm using VAC-M (Eugene's) ... I'm not sure how I'd edit the network 'card' ... for me its going through a 1Ghz switch then to the main computer which has MB networking. So don't know if that switch has a bearing on it or not or even where I'd edit the internal (computer) network connection if that is what you mean. Sorry for my ignorance on these issues...

I can try setting the buffers - although I'd have thought I'd be setting the DIGITAL one and not SSB. Also the CW/FM have no bearing on this I'd guess and they're working fine so don't want to mess with them. Possibly you use the VAC connection for all and thus it might be germane for your setup.

I notice that WSJT doesn't decode all that well sometimes... signals are clearly there and nothing decodes ... and wonder if this has something to do with the radio settings. I do know that on TX I see these burps and bubbles and splatter (as a general term of what it is) that is at times only 35 db down from the peak signal (using DUP, not there without DUP)... Scott wants me to post a picture of it - I'll need to do a video as its only there for the briefest of moments and there is no "freeze" function for the display (its on my list of "wants" turned in to the development team, probably very low priority) ... one of these days I'll get it done. I've been busy working new ones on 6 ... started the season with around 56 countries - now at 94!

Thanks for the info though - I'll see what I can get out of it on my setup. Have a great day.

Gary
K9RX
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Mon Aug 06, 2018 3:05 pm

I'm curious ... even though I'm an embedded designer both HW and SW I'm not familiar with the details of how a PC handles interrupts. So I'm throwing this out there...

I have an old program that is used to download firmware to the product that I sell ... and for it to work reliably I have to set the affinity to one processor. I was wondering if anyone has played with this either regarding 'shunting' any offending drivers/exe's and/or redirecting PowerSDR. So if a core handles interrupts, a big unknown for me - and you could focus an offending program to not use say core 6 and set up PSDR to only use core 6 it might eliminate the effects of interrupt latency at least for that (most) offending program. I don't know that there is any way to test this using Latency Mon as it by default is looking at everything ... but if there were someone that had issues with pops/clicks for example and one could use Latency Mon to find the worst case offenders ... if they can figure out how to set the affinity for each of these they could turn off say 2 of the 6 cores that are used (if the processor for example had 6 total) ... and then set up PSDR to only use those 2 cores....

just a thought.

Gary
K9RX
Joe-W4WT
Posts: 52
Joined: Sun Apr 09, 2017 5:27 pm
Location: Cumming, GA

Re: LatencyMon Benchmarks

Postby Joe-W4WT » Mon Aug 06, 2018 7:09 pm

Hi Gary,

I supplied the info for all the modes the radio operates but certainly digital is your focus so I would try setting that to 128 buffers on the filter page, if it is at 64 and see if that makes any difference. I also run digital modes at 48K rather than 192K where I run all other modes. This seems to work much better for that mode and certainly anything above 48k is unnecessary on digital mode.

I noticed this year you guys to the East of us here in the Atlanta area seemed to get more of the "E" cloud than we did. Congrats on the big pickup in 6m countries! I picked up quite a few but nothing like you did.

If I were you, I would try connecting my NIC directly to the Anan rather than going through a switch/router just as a test. Obviously there is more latency going through a switch/router path than a direct connection to a NIC from the radio. For this test I would also turn off the different advanced options on the NIC as I discussed above before assuming it makes no difference initially.

I tried playing with Affinity some time in the past but discovered it didn't "stick" and you would have to reset it every time you started the App and some Apps changed it even while they were running so I had no luck with this setting. It sounded like a good idea as I noticed that most apps don't use all cores; most use just the first one or maybe the first two. My thinking was I could assign HPSDR to cores 7 & 8 to get it away from most other programs but it just didn't work out for me to solve the latency problems I had then.

Also, as Scott has stressed many times, everything in the audio chain MUST be set to identical levels; 48K and either 16 or 24 bit. If not, problems are sure to occur as I discovered when I missed one. Missing one is not hard to do as there are so many in different areas to keep up with. Such a pain particularly when Microsoft deems it necessary to undo what you've done frequently with updates. They seem to love to screw with the audio settings with almost every update.

Your mentioning of the "burps and bubbles" on transmit reminds me of what I saw on SSB (and have seen on FT8) with PureSignal OFF and running a two tone test and seeing what I thought looked like a "irregular shaped cloud" popping up from below on the waterfall frequently. This happened even with VAC OFF. The problem causing this turned out to be the TX buffer size settings on the "DSP Options" page, as I mentioned earlier. I had it set to 64 and found it had to be at least 128 to mostly get rid of this problem. Even at 128 I would occasionally see a burp so I settled on 256 and it seemed to completely eliminate it. This was why I mentioned I set all other modes to 128. Some were set to 64 previously. Those may need to go to 256 also but I haven't tested them all yet. This "cloud" looked very similar to the display anomaly caused when running PureSignal with the noise blanker on in previous versions of HPSDR which made me think it was just a display issue but when I ran PureSignal with the buffers at 64, I saw a glitch in the "Amp View" display of PureSignal every time I saw the "cloud" popped up so my thought was it is a real problem and not just a display anomaly.

73,

Joe W4WT
Joe-W4WT
Posts: 52
Joined: Sun Apr 09, 2017 5:27 pm
Location: Cumming, GA

Re: LatencyMon Benchmarks

Postby Joe-W4WT » Mon Aug 06, 2018 8:26 pm

Hi Gary,

I thought I would update you on my fooling around with Affinity today. I decided to play with it to see what I could discover. I assigned PowerSDR.exe to cores 5,6, & 7. I assigned all the firefox instances, JTDX, and JTAlert to cores 0 thru 4 as these were the major users of CPU cycles. I then opened resource monitor and just watched the different cores and I did notice a distinct difference in cores 5,6, & 7 compared to cores 0 thru 4. There was indeed less total activity. This didn't really help me as I don't have issues anymore but it did clearly show that most other traffic was off of cores 5,6,&7. Here is the picture of the cores and as you can see the majority of usage is on cores 0 thru 4 now.

73,

Joe W4WT

Image
K9RX
Posts: 209
Joined: Fri Apr 14, 2017 3:47 pm

Re: LatencyMon Benchmarks

Postby K9RX » Mon Aug 06, 2018 11:07 pm

and I tried changing PSDR to just one core - core 6 on my i5 (6 cores total) ... there was no change that was obvious. The CPU usage was the same as I'd expect it to be ... note my thought on this is that if one were to do this you'd want to have the programs you're concerned with, for example WSJT, set to the same core .... the purpose here being to reduce the chance that it will be on a core that has something that has long latency and as a result be held up by it ... of course without knowing what that program is (although LatencyMon tells us the worst case one(s)) we can't eliminate it but we might reduce the frequency.

Gary
K9RX

Return to “PC's & Homebrew Hardware”