GTN650, declining frame rate (resolved)

[posixPilot 0]

  Hi,

I'm having an issue where my frame rate decreases substantially after a period of using the GTN650 on X-Plane.  First, let me start off by saying that I don't have a GPU, and I know that I need one.  As anyone that has shopped for a GPU recently can tell you, they are virtually impossible to find.  Since I've been unable to purchase one, I've been running with very constrained resources.

Back to the issue; I'm able to start up and take off with the GTN650 running and achieve about 22-24FPS.  This frame rate holds up while I practice my first approach and missed procedure.  I usually setup for a second approach and go missed again, with the frame rate holding steady above 20 FPS.  However, usually when I begin my 3rd approach, I begin to see the frame rate drop into the 16-18 range, and as I continue the approach, it drops to 10-12.  By the time I'm at the missed approach point, the frame rate sometimes drops as low as 5-6FPS.  At this point, I open the plugin manager and stop the GTN plugin.  The frame rate immediately springs back into 20+ FPS range.

The issue with frame rates below 20 in X-Plane is it causes the simulation to lose time.  Approaches take longer, control input is delayed.  Generally below about 15 FPS  just isn't usable.

As for troubleshooting, I've monitored my CPU load and temperatures.  Temps are well below 60 C at all times, and cpu usage is very low, with peaks around 50% on a single core (16 core Intel 11900).  I know my integrated graphics are pegged, but oddly enough, when I see low frame rates, the GPU usage drops substantially.

I've also systematically removed each plugin in X-Plane while having the FPS issue; this is how I determined the GTN plugin was causing the issue.  My graphics settings have all of the X-Plane sliders set fully to the left (lowest settings).  The texture quality is so bad, I can't even read the numbers on the runway.  I've tried setting the GTN CpuAffinity to auto and all (-2).  I've also set the GTN display refresh to half.  I'm flying the Laminar C172 modified with the RSG G5 instrument.  Vulkan is disabled as it is incompatible with my graphics.

Any other ideas how I can troubleshoot or prevent the GTN plugin from affecting my frame rate?

 

  Thanks

  Mike

[RXP 2,338]

Hi,

First of all, the solution is really not meant to run without a discrete GPU and the RXP GTN User's Manual explains how to configure Windows in order to make sure the trainer is running on a discrete GPU (NVidia or AMD) instead of the integrated one.

What I'd suggest you do though, which might help maybe finding whether there is something which you can do or that we can do, is the following:

1) right before you fly, open the Windows Task Manager (CTRL+SHIFT+ESC) and display the "Performance" tab.

2) during your flight look up the total GPU memory used and see whether it keeps increasing,

3) compare the GPU memory figure with the one reported inside X-Plane Graphics settings (it displays next to the texture resolution slider)

If both are increasing it is most likely XP11 itself which is using more and more GPU memory, if both are growing and shrinking about in the same proportions, it is your CPU RAM which is struggling most likely.

The above is not a precise methodology but it might help seeing better what could be the potential reason(s).

-

[posixPilot 0]

  Hi Jean-Luc,

Thanks for taking a look; I ran the test as you suggested.  I setup a few way points in a flight plan and let the plane fly for a while.  Before starting X-plane, I had 0.3G of GPU memory used.  Immediately after starting X-Plane, I had 1.6G of GPU memory and 8.0G of system memory used.  X-Plane reported 208MB of textures loaded.

During the flight I watched the GPU memory steady increase from 1.6G to 2.8G, while the system memory increased from 8.0 to 9.5G.  However, the X-Plane textures went from 208MB to 207MB.  It took about 40 minutes of flying to reach the "point of impairment" where FPS dropped from the low twenties to 15 FPS or less.

Once the FPS dropped below 10, I stopped the GTN plugin.  The FPS returned to the low twenties, GPU ram went to 1.6G and system ram to 8.2G.  X-Plane textures remained at 207MB.

Let me know if this is helpful.  Overall, it doesn't seem like a lot of CPU/GPU ram is being consumed, but I think restarting the plugin shows the memory growth from the GTN.

I've also had a few X-Plane crashes recently; I don't know if they are related, but it seems to happen in low FPS situations.  I might let the flight continue past the point of impairment to see if we eventually get a crash.

 

  Thanks

  Mike

[RXP 2,338]

Thank you for the additional details! 

If I understand correctly, 2.8GB VRAM and 9.5GB RAM were the peaks your system did reach when fps where starting to drop? 

I find the 40min figure is interesting and this is giving me a probable hint, but I'd need to be sure about the behaviour of RAM/VRAM for a longer period after the fps did drop. If they where the highest you've noticed but if this was before fps dropping, did they further grow when you were running with < 20 fps afterward, and did they still continue to grow afterward?

If you look in the Windows Task Manager, what is the reported VRAM available (both dedicated and total) and how much RAM do you actually have too?

[posixPilot 0]

  Hi Jean-Luc,

Quote

If I understand correctly, 2.8GB VRAM and 9.5GB RAM were the peaks your system did reach when fps where starting to drop? 

Correct, on yesterday's test I stopped the plugin after about 50 minutes of run time.  I conducted a longer test today to answer the question about peak memory usage, and, yes VRAM usage continues to grow.  See the table below.

 

Time            RAM of          VRAM of         Textures        Frame/Time      FPS
                31.8G           15.9G Shared
Boot            4.2G            0.2G            --              --              --
Start           7.7G            1.4G            208M            0.041           24
@20min          8.2G            1.8G            208M            0.045           22
@30min          8.6G            2.0G            208M            0.042           23
@35min          8.7G            2.2G            208M            0.049           20
@40min          8.8G            2.3G            208M            0.044           22
@43min          8.9G            2.3G            208M            0.048           20
@45min          9.0G            2.4G            207M            0.052           19
@46min          9.0G            2.5G            207M            0.054           18
@48min          9.1G            2.6G            207M            0.057           17
@50min          9.3G            2.8G            207M            0.070           14
@55min          9.8G            3.3G            207M            0.112           8
@60min          10.3G           3.8G            207M            0.165           6
@70min          11.2G           4.7G            208M            0.181           5
@80min          12.3G           5.6G            208M            0.201           4
@90min          13.2G           6.5G            208M            0.240           4
Stop GTN        8.1G            1.6G            208M            0.037           26

A couple of points on the data; I started the timer when X-Plane dropped the plane on the runway.  My previous timing was an estimate, where I started the timer after take off.  Still, it is clear something starts to change about 40 minutes into the flight.  I did my best to interpolate the FPS and frame/time column.  The actual rate was highly variable, so these are averages.

Regarding VRAM, since I'm using integrated graphics, Taskmanager reports the VRAM as shared.  I'm pretty certain it is just using CPU RAM as you can see the RAM grow and shrink at the same rate as VRAM.  I have 32GB of physical RAM on this system.

  Thanks

   Mike

 

[RXP 2,338]

It looks like the GTN trainer was consuming about 5GB of "memory" here (VRAM+RAM - both the same in this case), which I find suspicious. Even at about 46 mins it was not consuming this much, yet fps started to go down.  As for the significant drop at about 50/55 min, this sounds to me like the process is hitting the available memory address space (the trainer is a 32 bits program limited to 2GB RAM).

It is also possible the problem lies in XP11 SDK as well. For example there might be a bug where the more you run and redraw the plugin, the more resources XP11 allocates and don't release, and this bug might be only exhibiting because you're using an integrated GPU instead of a discrete one. I'd suggest you try using this tool now: https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer

It will allow you to monitor the actual working set, and total memory (both RAM and VRAM) that the trainer process itself is using, as opposed to the entire system memory consumption. This might help to pin point the root cause and maybe a solution.

If this problem would be solely GTN related, I'd suspect there would be more widespread reports in the forum. This is why I'd first suspect probably a bug with the Integrated GPU Drivers remains the most probable cause (are you Intel or AMD?). You might want to also cross check any update for your integrated GPU and eventually whether there are other reports with other games of increasing memory consumption over time for example.

[posixPilot 0]

  Hi Jean-Luc,

I did a bit more testing, by removing the GTN plugin.  After boot, I had 4.3G of RAM and 0.3G of VRAM used.  I started up X-plane, and started a timer once the simulation started.  Initial memory usage was 7.6G RAM and 1.3G VRAM in X-plane.  I stopped the GTN plugin and then took off.  The plane flew for 90 minutes like the last test and all FPS samples were above 20 (usually 24-26FPS).  After 90 minutes the memory usage was 8.1G RAM, 1.5G of VRAM.

I saw your note about process explorer while I was running this test, so I left the flight going and downloaded it.  X-Plane.exe showed 5.7G of private memory and 3.7G of working set.  I decided to enable the GTN plugin and immediately the Trainer process started.  The Trainer remained at 0.5G of private memory and never grew.  However, X-Plane.exe grew to 6.2G private and 4.2G working set over a 10 minute interval.

It doesn't look like I can isolate the memory usage inside of X-Plane.exe any further, but the process returned to 5.7G private/3.7G working when I disabled the GTN plugin.

As for the integrated graphics, I have Intel (11900K cpu).  While my graphics drivers are only about a month old, I did find an update and applied that.  I'm running the test now, but I'm seeing about the same behavior with memory consumption through process explorer.

Let me know if you think of any other troubleshooting that I can do.  I can't think of a way to isolate the integrated graphic drivers, without having a discrete GPU card.

  Thanks

  Mike

[posixPilot 0]

  Hi Jean-Luc,

I've done a good bit of testing on this issue to try and isolate the problem.  Here's what I've found:

  - The issue happens only when the GTN is displayed in the panel AND in a pop-out window.

  - This affects both 650 and the 750.

I think the first item is relevant as it likely explains why we haven't seen other reports of the issue.  I'm overlaying the 650/750 in the 3d panel and I'm using a pop-out window at the same time.  If I remove the overlay or close the pop-out the issue does not occur.  If I use the overlay and the floating window the issue does not occur.  I'm assuming that most other users don't have both.

I can probably do without the panel overlay, so removing it is my current work around.  I'm using the overlay to basically "cover up" the X-Plane GNS 530 in the default 172.  If I could blank the screen, that would work just as well.

Since I'm following the slow GNS530/Win11 issue, I also tried changing my HyperThreading BIOS setting.  When disabling HT on my BIOS, I notice that this issue takes longer to manifest.  With HT enabled, I see FPS reduction in 40-45 minutes, with HT disabled it takes over an hour.

I've posted my INI file below so you can see exactly how I'm using the 650/750.  Both the 650 and 750 configurations in the file produce the problem on my system.  Also, I've tested my system with other 3d games per your request.  Running Portal1, Portal2 and HalfLife 2 for about 30 minutes doesn't cause significant memory growth or FPS drop in any of those games.

Thanks for taking a look at this issue.

  Cheers

  Mike

...

[GTN]
; comma separated list of gauges
gauges = GTN_650_1

[GTN_650_1]
; comma separated list of render targets.
drawto = WINDOW, PANELS
; is the master device if true.
MasterDevice = true 
; aviation db: 0: nav_db2_grm, 1: nav_db2, 2: nav_heli_db2_grmn, 3: nav_heli_db2
NavDbType = 1
; connects to: 'PFC_GTN' or any '#PID' (#D067 for PFC_GTN)
; no value connects to first found, 'OFF' disables connection.
HardwareDevice = 
; hardware selector number or -1 to disable.
HardwareIdx = -1
AutoCdiSrc = true 
UseSimGpsCmds = true 
LinkObs = true 
LinkCrs = true 
CpuAffinity = -2
LinkVor = true 
ReduceRate = true 
PowerSource = AVIONICS_BUS

[GTN_650_1.WINDOW]
; display window if true.
visible = true 
; show screen only gauge if true.
nobezel = false
; display type: WINDOW,PANEL2D,PANEL3D,PANELS
frame.type = WINDOW
; display position and dimension (left,top,width,height)
frame.rect = 61,798,1280,577
; screen only border size (pixels).
border.size = 0
; screen only border color (#RGB or #RGBA).
border.rgba = #000000
; enable mouse clickspots if true, disable if false (use SHIFT to override)
usemouse = true 
; left mouse button on the screen toggles popup window if true (requires SHIFT+left mouse button).
popleft = false
; right mouse button on the screen toggles popup window if true (requires XP11.1).
popright = true 
; VR mode position and dimension (left,top,width,height) (requires XP11.20)
vrmode.rect = 0,175,415,175
; display as popout window if true (requires XP11.1).
undocked = true 
; display mouse tooltips if true.
tooltips = false
; freeze display position if true.
locked = false
; auto hide in external view if true.
autohide = false

[GTN_650_1.PANELS]
; display window if true.
visible = true 
; show screen only gauge if true.
nobezel = true 
; display type: WINDOW,PANEL2D,PANEL3D,PANELS
frame.type = PANELS
; display position and dimension (left,top,width,height)
frame.rect = 0,1024,517,389
; screen only border size (pixels).
border.size = 0
; screen only border color (#RGB or #RGBA).
border.rgba = #000000
; enable mouse clickspots if true, disable if false (use SHIFT to override)
usemouse = true 
; left mouse button on the screen toggles popup window if true (requires SHIFT+left mouse button).
popleft = false
; right mouse button on the screen toggles popup window if true (requires XP11.1).
popright = true 

[GTN_750_1]
; comma separated list of render targets.
drawto = WINDOW, PANELS
; is the master device if true.
MasterDevice = true
; aviation db: 0: nav_db2_grm, 1: nav_db2, 2: nav_heli_db2_grmn, 3: nav_heli_db2
NavDbType = 1
; connects to: 'PFC_GTN' or any '#PID' (#D067 for PFC_GTN)
; no value connects to first found, 'OFF' disables connection.
HardwareDevice = 
; hardware selector number or -1 to disable.
HardwareIdx = -1
PowerSource = AVIONICS_BUS

[GTN_750_1.WINDOW]
; display window if true.
visible = true
; show screen only gauge if true.
nobezel = false
; display type: WINDOW,PANEL2D,PANEL3D,PANELS
frame.type = WINDOW
; display position and dimension (left,top,width,height)
frame.rect = 324,942,830,790
; screen only border size (pixels).
border.size = 0
; screen only border color (#RGB or #RGBA).
border.rgba = #000000
; enable mouse clickspots if true, disable if false (use SHIFT to override)
usemouse = true
; left mouse button on the screen toggles popup window if true (requires SHIFT+left mouse button).
popleft = false
; right mouse button on the screen toggles popup window if true (requires XP11.1).
popright = true
; VR mode position and dimension (left,top,width,height) (requires XP11.20)
vrmode.rect = 0,395,415,395
; display as popout window if true (requires XP11.1).
undocked = true
; display mouse tooltips if true.
tooltips = false
; freeze display position if true.
locked = false
; auto hide in external view if true.
autohide = false

[GTN_750_1.PANELS]
; display window if true.
visible = true
; show screen only gauge if true.
nobezel = true
; display type: WINDOW,PANEL2D,PANEL3D,PANELS
frame.type = PANELS
; display position and dimension (left,top,width,height)
frame.rect = 0,1024,517,389
; screen only border size (pixels).
border.size = 0
; screen only border color (#RGB or #RGBA).
border.rgba = #000000
; enable mouse clickspots if true, disable if false (use SHIFT to override)
usemouse = true
; left mouse button on the screen toggles popup window if true (requires SHIFT+left mouse button).
popleft = false
; right mouse button on the screen toggles popup window if true (requires XP11.1).
popright = true

...

Edited January 20, 2022 by RXP

[RXP 2,338]

Hi Mike,

I'm sorry for the delay, and I thank you for your patience with this issue and your help to solve this.

If I understand the test cases:

- render to 3D panel -> no issue
- render to popup window -> no issue
- render to popped out window -> no issue
- render to 3D panel + popup window -> no issue
- render to 3D panel + popped out window -> memory leak/growth over time.

I'll review if there could be anything "leaking" a resource in our code but I doubt it in this case: each of the render targets (the 3D panel "overlay" and the popped out window) are not allocating any resources "per frame". In other words, once created, they are pretty much not allocating anything. Furthermore, the code running behind the popup or the popped out window is the very same. The only difference is an X-Plane flag telling whether the window is popup or popped out.

Quite confusing but I'd think it is something related to X-Plane popped out windows internally, or maybe something system wide which is hooking on any window and which is causing this memory leak/growth over time?

Something you might want to try: running in the configuration 3D+Popped-out, when you start experiencing the issue, go to the RXP menu and change the window from popped-out to popup, then observe whether you restore perf/memory, then change it back from popup to popped-out, and observe if memory grows again.

With this information I shall be able to file a report to LR.

[posixPilot 0]

  Hi Jean-Luc,

Sorry I haven't responded, I wasn't getting notifications from this thread.

The test cases you listed above are correct, aside from the ones with "pop-up".  I didn't test any pop-up test cases, just 3D and "pop-out".

I finally managed to find and purchase a GPU card in late January and as you suspected, the frame rate issue went away immediately.  I've been able to maintain high frame rates using 3D + pop-out for over 2 hours.

I'm not sure if you still want to report something to LR.  I don't think I can test the pop-up scenario without removing the GPU from my PC, which I'm not terribly keen to do.

 

  Thanks

  Mike

[RXP 2,338]

Thank you for the follow up!

In all cases, X-Plane is not really meant to run on an embedded GPU like the Intel and actually, all simulators are usually better running on a discrete GPU anyhow. I believe at this stage it is a resolved issue then!