Keep in mind that your 970 GTX has 4GB Vram but out of those 4GB only 3.5GB are actually usable without killing FPS.
Something with the last 500MB of Vram being adressed differently. Depending on your X-Plane settings and Plugins you might hit those last 500MBs of VRam which will drop FPS significantly. I know because I have a 970 GTX myself but just a regular 1920x1080 Monitor and occassionaly hit those last 500MBs of VRam while tuning X-Plane settings.
If you use maximum Resolution Textures without the Compression, try switching the Texture Compression on or switch to a lower Texture Resolution setting.
The Draw Shadows on Terrain costs a lot of Frames as frontendrob already mentioned.