DXGI_ERROR_DEVICE_HUNG error: Is there a solution?

[joepoway 241]

1 hour ago, Pete Dowson said:

Only recently (since P3D4.3, and still with P3D4.4) I started to get these DXGI device has hung messages. Originally, when they started, they only occurred at dusk or at night. But I've just tried re-enabling my FTX England scenery to see what it looks like in version 4.4.

Now the crash occurs in the daytime, 100% consistently, within a couple of minutes of the scenario starting up (my default, at UK2000's EGCC).

The GPUs (I have 3 x 1080Ti's in SLI formation) are never loaded to more than 45% (according to Task Manager), so I don't understand what is going on. Is this a P3D problem, or an nVidia driver problem? The error report appears to be from P3D. I have tried various nVidia drivers, Currently I'm on 416.94.

My installation is pretty well loaded up with addons as well as Orbx, but I tried with the FTC OpenLC Europe disabled, and with all my UTX layers disabled too, to no avail. With no FTX Europe regions enabled all is well -- for daytime flying. I can't use it at night because of the same errors.

I'll ask on the P3D forum too. I need a solution, or my Orbx purchases are useless (I have all of the FTX Europe country regions and was looking forward to trying them, but I can't get passed the first couple of minutes after loading).

Whilst awaiting answers I'll continue experimenting. Things to try:

1. Disabling SLI
2. Changing drivers yet again (go for the latest first ...
3. ... er: there is no 3 ... yet.

Thanks,
Pete

 

Pete

I know of 3 friends with 1080ti cards that all had this problem and we tried everything, in all cases the cards went bad and after changing out cards with new cards the problem disappeared. We first swapped out the cards with other cards that didn’t have the issue and after that remedied the problem they returned their cards under warranty for new ones.

I’m not saying this is the problem for everyone but I’ve seen it be the cause 3 times. If it’s possible for you to swap a card at a time that may identify if you have a card going bad.

As a note there seems to be more quality issues with the 1080ti cards then the previous generations from what I’ve seen. Additionally a friend of mine just had a 2080ti completely fail after 2 weeks. 

Joe

 

[Nyxx 6,993]

42 minutes ago, Pete Dowson said:

Thanks. Another thing to try. I usually do the nVidia "clean install", but i assume DDU does a better job.

Pete

 

Yes, it does you have to start PC in safe mode before you use it.

[pracines 823]

29 minutes ago, Pete Dowson said:

I set unlimited in P3D and enable VSync.

I think there is something amiss with Vsync - several gamers and simmers have turned off Vsync altogether and DXGI never came back.

If you have not tried this possible solution, its worth a shot to just leave Vsync off and lock FPS to 30 within P3D.

Edited December 4, 2018 by pracines

[simbol 5,819]

44 minutes ago, Pete Dowson said:

That may be so. That will be my next test, whether by disabling one of the GPUs or removing its SLI link. Or, thinking on how they are linked, sort of circularly, I may have to find that 2-way link I think I had.

One result so far from my testing. Now that DXGI crashes don't occur I've been testing further. First with the time set for night. And it works fine -- but with abysmal frames rates (14-17 at EGLL), and then in the day also at EGLL -- the same! So I think Orbx FTX England (and the others I'd hoped to enjoy) are not viable with my setup. 😞

I think without the FTX stuff I should be okay at night though, without fear of DXGI crashes at last!

Pete

 

Pete,

Rob M. just replied to your post on the LM forums, he said 3 way SLI is not native supported.. so I think 2 way SLI will be the way for you.

Regards,
S.

[Pete Dowson 558]

10 minutes ago, simbol said:

Rob M. just replied to your post on the LM forums, he said 3 way SLI is not native supported.. so I think 2 way SLI will be the way for you.

I see. Ok. Thanks. I think that probably accounts for the fact that one of the GPUs was hardly used, but maybe not for the DXGI crashes. Because they started occurring so consistently after I added FTX Europe I feel it's more to do with P3D loading.

Pete

 

Edited December 4, 2018 by Pete Dowson

[redchiefnr1 3]

Hi Pete,

I also had the same problem with my nvidia 1080Ti and the newer drivers since 4/2018 until I used this fix:

The problem: a thing called TDR checker monitors the computer’s hardware, particularly the GPU. When the GPU stops responding with the computer for 2 seconds TDR checker restarts the driver. The TDR checker was good on paper but did not work in the real world as a GPU can stop responding when it is working hard.

Here is the solution that worked for me, it requires an extra sub key entry into the Windows registry, so please only implement if you feel OK with editing your registry!

- Exit all Windows based programs,
- Click on the Windows Start button, type regedit in the Search box,double-click regedit.exe from the results above.
If you are prompted for an administrator password or confirmation, type the password or provide confirmation,
- Browse to and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers
- On the Edit menu, click New, and select the QWORD (64-bit) value from the drop-down menu,
- Type ‘TdrLevel’ as the Name and click Enter,
- Double-click TdrLevel and set the value as 0 (it is set to that by default, but double check) and click OK,
- Close the registry editor and restart your computer for the changes to take effect!

I saw this fix mentioned earlier in this thread, but make sure you create a Qword (64bit) key for Windows 64 bit OS and NOT a Bword (32 bit) key because that wil not give the right result.

If you encounter a DXGI hung/removed error again please make sure that this key still exists in your registry!
It happened to me once that the key vanished after a Windows Update, resulting in the dreadful DXGI hung error once again.

 

after using this fix I never had an "dgxi error device removed" again

greetings

Dirk

[Pete Dowson 558]

39 minutes ago, redchiefnr1 said:

I saw this fix mentioned earlier in this thread, but make sure you create a Qword (64bit) key for Windows 64 bit OS and NOT a Bword (32 bit) key because that wil not give the right result.

Ah, thanks! I had already tried that, but it was set as a DWORD! I've changed it now, and will re-test tomorrow with SLI re-enabled.

None of the 3 GPU's are heavily loaded (45% max in Task Manager), but i'm wondering if the problem is the lack of any real work done by one of them. Only two are used.

L-M say 3-way SLI isn't supported, so that may explain this. But if so, I'll end up removing a card and going 2-way after all.

BTW, with only one card in use I'm seeing it loaded up to 90% in heavy situations. With the "3-way" SLI they wre max 45, 8 and 45. So I think i need some sort of SLI to give me some headroom! But 2-way would obviously suffice.

Thanks again,
Pete

 

Edited December 4, 2018 by Pete Dowson

[cj-ibbotson 77]

4 hours ago, JoeFackel said:

I saw posts where someone clocked down (!) his 1080ti and got rid of the device_hung errors. At least the max. power consumption to 80% if i recall correctly.

Yes Joe that was myself, was the ONLY single thing that helped.  Absolutely nothing else and I mean nothing else worked at all. Tried every single driver, tried brand new OS installs of current W10, an older 1703 Build, old to new drivers (at least 18 of them now), even went back to W8.1 and still got the crashes.  Tried running a bare sim with just Orbx scenery, no addons like Active Sky or shader mods, tried the TdrLevel and TdrDelay edits in the registry, tried the ShowDeviceLostWarning=0 entry in the P3D config file, tried restoring my cpu overclock to default, tested my ram, like you Pete I do not overclock my GPU, its factory overclocked so I downclocked the core and memory speeds back the standard nvidia speeds.  After a final format of my system I reinstalled W10, fully updated it and installed a default P3D 4.3 (4.2 client also failed on a test install) and it still gave a Device_hung crash, default settings, default sim, no addons, default system clock speeds etc.  The latest thing to try which LM advised I think is to change PANELS_ALWAYS_ON_TOP=1 to 0.  I had hoped this had worked but on a test today after 3.5 hours I return home to see it had crashed.  The only thing that helps in my system is to set the Power Limit of my 1080ti to 75-80%, this value may give varied results with others.  Joe Young insists I have a faulty card but so many of us cannot have faulty cards, my card passes every benchmark and stress test like Furmark no problem.

I really had hoped you would have been the savior to these crashes Pete, I remember when I first got Prepar3D v1.3 or 1.4 back in 2012 and they were plaqued with D3D.dll crashes all the time which you fixed with FSCUIPC.  I believe there is something in the sim, or other games which might throw up this error, which does not like something an individual users graphics card is doing and gives us this error.

Chris

[Bob Scott 7,716]

1 hour ago, cj-ibbotson said:

Yes Joe that was myself, was the ONLY single thing that helped.  Absolutely nothing else and I mean nothing else worked at all. Tried every single driver, tried brand new OS installs of current W10, an older 1703 Build, old to new drivers (at least 18 of them now), even went back to W8.1 and still got the crashes.  Tried running a bare sim with just Orbx scenery, no addons like Active Sky or shader mods, tried the TdrLevel and TdrDelay edits in the registry, tried the ShowDeviceLostWarning=0 entry in the P3D config file, tried restoring my cpu overclock to default, tested my ram, like you Pete I do not overclock my GPU, its factory overclocked so I downclocked the core and memory speeds back the standard nvidia speeds.  After a final format of my system I reinstalled W10, fully updated it and installed a default P3D 4.3 (4.2 client also failed on a test install) and it still gave a Device_hung crash, default settings, default sim, no addons, default system clock speeds etc.  The latest thing to try which LM advised I think is to change PANELS_ALWAYS_ON_TOP=1 to 0.  I had hoped this had worked but on a test today after 3.5 hours I return home to see it had crashed.  The only thing that helps in my system is to set the Power Limit of my 1080ti to 75-80%, this value may give varied results with others.  Joe Young insists I have a faulty card but so many of us cannot have faulty cards, my card passes every benchmark and stress test like Furmark no problem.

Power supply to the GPU may be an issue.  I've seen, for example, where folks used a VGA power cable that has two connectors daisy-chained on the end of it to feed both aux power connectors on a high-end GPU.  That's a bad way to fly, as it is likely pulling the voltage on that single cable down enough to cause instability under load.  Another possibility would be a multi-rail power supply with an overloaded 12v rail connected to one or both power connectors on the GPU.  Or it could be as simple as a weak or underspecced PSU not keeping the voltage up.  Or a bad cable or bad cable connector.  The fact that reducing the power load limit stops the problem is suspicious.

The second possible culprit for errors under load is heat.  The GPU temp does not tell the story of on-board voltage regulator temps, memory temps, etc.  Could be the GPU core is operating at a temp that's comfortably between the lines, but another section of the GPU is overheating due to a poor cooling solution, limited airflow, or a malfunctioning fan or fans and/or a bad fan curve that doesn't ramp the cooling up enough.

Regards

[Guest]

4 hours ago, Pete Dowson said:

I see. Ok. Thanks. I think that probably accounts for the fact that one of the GPUs was hardly used

Sorry, I thought you were running 2 GPUs not 3?  3 GPUs on the X series MBs will work fine so long as you dedicate them to each display, but don't use SLI performance mode.  But in your case I think you have distortion issues that need to be corrected by external software/hardware for your projectors?  As you found out, per LM, only 2-way SLI "performance mode" is supported.  Even then, driver support of SLI can sometimes break various rendering aspects (usually FX or AG goes missing and/or pops in/out) ... best to find a driver that works and stick with it when it comes to SLI performance mode (even more so for dedicate sim setup).

I've only seen DXGI error once (and only once) and that was after installing FSL Spotlights many months ago.  I seem to recall that problem got resolved either with FSL Spotlight update or driver/DDU reset.

Per my testing, SLI adds CPU overheard, as much as 17%.  SLI is great IF (important IF) you aren't capping out your CPU.  If you're capping out your CPU then SLI is probably going to make matters worse, not better.  Like I've suggested in other threads, use SLI to get better AA and DL and weather performance, do NOT use it if you want high AI traffic or high road/boat traffic or high AG building density ... those are going to stress the CPU and SLI will just stress the CPU that much more and FPS will actually be worse.

If you look at my 9900K testing and 7900X testing scenario I selected that location and settings specifically so as to NOT overload the CPU.  If I overloaded the CPU then you'll see no gains from the GPU side at all, doesn't matter what GPU you use, if the CPU is at 100% any testing beyond that is not really valid.

As always, it's a balancing act and that's why no two systems will be the same and no single solution "for all".

Cheers, Rob.

 

[Guest]

56 minutes ago, w6kd said:

Power supply to the GPU may be an issue.  I've seen, for example, where folks used a VGA power cable that has two connectors daisy-chained on the end of it to feed both aux power connectors on a high-end GPU.  That's a bad way to fly,

Bob, agree with you ... in fact in my recent 9900K build I foolishly assumed that modular PSUs could use modular cables across different brands of PSU and/or the fancy braided cables available online.  I found out that I absolutely could NOT mix and match PSU modular cables from different vendors and/or use custom braided cables unless they were "extension" only cables going 1 to 1 on every wire.  That one had me frustrated for a while thinking I had band pumps or faulty MB only to discover it was the PSU "modular" cabling ... I thought there were standards to these "modular" cables/PSUs, apparently I thought wrong ... "modular" doesn't mean interchangeable between vendors.

Lesson for me was:

1.  Always use the PSU vendors cables that come with the PSU (no exceptions)
2.  Never buy online fancy colored braided cables unless they are pin to pin EXTENSION cables

Cheers, Rob.

[cj-ibbotson 77]

51 minutes ago, w6kd said:

Power supply to the GPU may be an issue.  I've seen, for example, where folks used a VGA power cable that has two connectors daisy-chained on the end of it to feed both aux power connectors on a high-end GPU.  That's a bad way to fly, as it is likely pulling the voltage on that single cable down enough to cause instability under load.  Another possibility would be a multi-rail power supply with an overloaded 12v rail connected to one or both power connectors on the GPU.  Or it could be as simple as a weak or underspecced PSU not keeping the voltage up.  Or a bad cable or bad cable connector.  The fact that reducing the power load limit stops the problem is suspicious.

The second possible culprit for errors under load is heat.  The GPU temp does not tell the story of on-board voltage regulator temps, memory temps, etc.  Could be the GPU core is operating at a temp that's comfortably between the lines, but another section of the GPU is overheating due to a poor cooling solution, limited airflow, or a malfunctioning fan or fans and/or a bad fan curve that doesn't ramp the cooling up enough.

Regards

Ive a good quality 1000w OCZ Gold with the correct cables to the card, no splitters or convertors.  Card needs a 6 and an 8 pin supply.  I did change the cables to the gpu a few months ago as I did once get an error on boot saying there was no power to the graphics card, only ever happened once.  Seriously dont want to risk trying another psu at a huge cost possibly for nothing and I do not know anyone with a pc, as we are a dying breed.  MSI Afterburner and HWMonitor tell me the gpu is currently at 66 degrees.  I have in past tests set the gpu fans manually at 100%, can see and hear they ramp up to full speed but it did not prevent crashes.  Ive also been working hard this week on my cpu temps changing the configuration of the AIO water cooler to different positions and fan arrangements.  Currently running at 4.5 and maxing at 61 degrees which sim running.  Current Min temp is less than 20 with one core showing it went as low as 14 degrees 🙂

[simbol 5,819]

3 minutes ago, Rob Ainscough said:

Never buy online fancy colored braided cables unless they are pin to pin EXTENSION cables

Not even nice pink ones? 🤔

[Snuffleupagus 327]

I'm having this same problem. I created the other thread topic referenced in here, so I'll just continue in here with you guys. I ran the DDU in safe mode, updated to the latest driver, and still the same crash. GPU (GTX 1080) usage was only 72%, GPU Memory 81%, and voltage 72%. Nothing anymore than it's been for all of my other time in the sim. I already have TdrDelay=8 too. I'm really stumped on this as I never had this before 4.4.

[Bob Scott 7,716]

18 minutes ago, pvupilot said:

I'm having this same problem. I created the other thread topic referenced in here, so I'll just continue in here with you guys. I ran the DDU in safe mode, updated to the latest driver, and still the same crash. GPU (GTX 1080) usage was only 72%, GPU Memory 81%, and voltage 72%. Nothing anymore than it's been for all of my other time in the sim. I already have TdrDelay=8 too. I'm really stumped on this as I never had this before 4.4.

There's another TDR setting you can try in addition to TdrDelay, which is TdrDdiDelay (a DWORD value created the same way and in the same registry location as TdrDelay).  The default is 5, I have been using 20 for years, along with 10 for TdrDelay.  The TdrDdiDelay sets the amount of time a thread can leave the driver.  Note that if you set TdrLevel to 0, none of these delay settings are used--they're meaningless--as the detection of driver errors by the TDR code is disabled.

Also, for those using factory-overclocked GPUs, a factory overclock IS still an overclock.  I don't know what sort of binning/testing process is used by the various OEMs, but several years ago I had a couple very high-end 980Ti water-cooled boards in an SLI configuration that would not run stable at their aggressive advertised factory overclock, but they worked like champs with a relatively modest downclock from the factory settings (still overclocked significantly above the factory settings).  So it would not be at all surprising to me that some folks might run into stability issues at the factory settings on some of the factory-overclocked boards.  And overclocked w/r/t GPUs usually means both the core clocks and the memory clock, which are set independently.  So if you're having trouble, I'd start with both core and memory clocks at nVidia stock settings and work progressively up from there.

Regards