How do you test? (And some personal conclusions)

[Guest]

I used to test various settings and tweaks by simply looking at the sim... I now found out that's a VERY bad way of testing things because you often only see what you expect or want to see. So I wonder, how do you guys test settings and tweaks?

 

Yesterday I spend my day testing various settings and tweaks, amongst others because of the fact that I wanted to give DX10 another try, but while I was at it I also tested some wellknown 'old' tweaks to see if they actually do something.

 

The best way to test things that I know of now is to start a flight, take off, put the plane on AP above heavy terrain and when the plane has settled, save the flight. To test a setting I now only have to load and run that flight without having to touch anything and I always take screenshots at specific momenst (when passing a certain building or when a specific texture has loaded). This way every testflight is exactly the same and not changed by my input. I also restarted the PC after every test!

 

Anyway, one of the things I noticed yesterday is that the famous TEXTURE_BANDWIDTH_MULT tweak makes NO difference at all on my PC. Well, at least... I always thought of that tweak as a way to influence when the ground textures would load. Different settings would result in later or earlier loading of textures. Well, I tested 40, 80 and 120 and did so by loading my testflight and pressing V as soon as a specific and very noticable texture had loaded. When I compared the screenshots, they were all exactly the same so the textures loaded at the exact same distance... It seems to do nothing at all! (I have always been using the default TBM setting and will remain to do so.)

 

Another thing I found out is how little influence the LOD_RADIUS setting has AND also that FSX simply ALWAYS loads textures rather late. I ran my testflight again and once more I clicked V when a certain texture had loaded in the distance, both at 4.500000 and 6.500000 radius. I don't know why but I always thought that 6.5 would make the circle around you in which textures are loaded a LOT bigger: I never really noticed that on my PC so I always thought there was something wrong with it. But my test showed that the difference between 4.5 and 6.5 is only a few hundred meters in the FSX world...! I always thought it would make a difference of kilometers but it doesn't. Another tweak that although it does work, at the expense of fps, it doesn't work as I have always thought. (I have always used 4.5 and will remain to do so because of the performance loss I get in certain situations with 6.5.)

 

Another 'funny' example... I wrote in another topic that DX10 seems to load textures later. Everytime if I would enable DX10 (with all the fixes) textures loaded later, no matter what I did. Until I tested this properly with the 'method' described above: when I compared various screenshots they were ALL the same so the idea that DX10 loaded textures later was all inside my head... However, I have more than one testflight and in another one DX10 was a little bit later with texture loading... (and it had some other sligght problems but I don't want to go off topic here, but I am sticking with DX9).

 

I also tested Affinity Mask this way. If it had any result, it would be that it made things more blurry on my PC. (I already stopped using this tweak some time ago.)

 

Anyway, it shows that only proper testing, in a way that's predictable and does NOT depend on simply watching the sim but actually comparing screenshots and results, can tell you if things work. And how things work. Funny thing is that after a day of testing I ended up with the exact same cfg's as I had when I started... (I only use ONE tweak, bufferpools at zero).

 

Question: I wonder how you (if ever) tested settings and tweaks, like for instance TEXTURE_BANDWIDTH_MULT. Did you simply play the game and looked at it...? Or do you have good way to test things and did you actually see any effect? If so, what kind of test do you use?

[JimmiG 244]

I don't bother with most of those "tweaks". I reduce FIBER_FRAME_TIME_FRACTION to 0.15 when not flying over photo scenery, because it increases FPS a tiny, tiny bit. I don't bother with the affinity mask or TEXTURE_BANDWIDTH_MULT (it's Mult, not Load, BTW).

 

Even the famous "miracle" BufferPools tweak only makes a small difference on my system.

 

The LOD_Radius is mostly noticeable when flying over photo scenery, but it's not a huge difference. I use 6.5 over BlueSky scenery but go back to 4.5 over OrbX, otherwise I would get OOM errors.

 

In general, the "tweaks" that have their own slider in the sim are the ones that make a difference. That's why the developers put in a slider for them.

[lodestar 186]

i test in saved situations, with and without weather, with default and payware planes, with fps locked(this is what i use)and unlocked to see fps difference etc... 

 

About tweaks, i agree with Jeroen except affinity mask which makes noticeable difference on my system. 

 

There are some tweaks which depends on system, settings and other applied tweaks

[Guest]

 

 

In general, the "tweaks" that have their own slider in the sim are the ones that make a difference. That's why the developers put in a slider for them.

 

LOL Very true. (And thanks about the LOAD > MULT : don't know wny I posted LOAD there... I corrected the OP. ^_^ ).

 

Yes, tweaks are very system dependant. On my PC only Bufferpools makes a noticable (and extremely HUGE) difference.

 

It's odd that it took me so many years to finally test a few things in a good way... I always tested tweaks and settings but never in a good way, always simply by watching the sim do its thing. Now at last I finally actually KNOW what works and what a few of those tweaks do. On the other hand it is funny to see that I ended up with the same cfg as I started with. So even though I now KNOW more, my old and not too good way of testing still led me to the best settings... ^_^

[Mork 34]

Hi Jeroen, as far as the TBM I have the exact same experience as you. I used to fly with the default 40 value, because it made no difference whatsoever. But I know other people swear by it, so I have set it at 80. At 120 I thought I saw some small irregularities, but like you I also acknowledge the 'it's all in your head' factor, lol.

Also the texturemaxload entry has an effect on this.

 

Did you also test FTFF? Because that one is suspicious too, because it should only work on single core CPU's I read in some old FSX documentation somewhere.

 

I do use higher LOD, but you are right, the circle of LOD doesn't grow as fast as you think. I have set it at 7.5 now I think, because with DX10 you don't get OOM's that quickly and it still performs well.

 

Affinitymask does seem to work here, but only a little bit. I have it set in my cfg. (2500K -> AM=14)

 

Bufferpools (=0) I have tested many times and does seem to work, however if I use bufferpools settings for high end PC's, I can almost see no difference. Tested this yesterday.

 

Internal or external FPS limiter: I have tested it many times, external ALWAYS worked better here, but yesterday I used the FSX limiter during testing and now I had about the same performance as external?! LOL (With Vsync active, I don't even know exactly if this matters at all)

 

The problem with all this is:

 

1) With DX10 the tweaks don't seem all necessary, and it is hard to compare tweak results with DX9.

 

2) It does seems to matter where you fly. Some tweaks only show their effect on certain scenery and when flying the NGX for example. The sim has different calculations going on, and I suspect that if you fly at detailed scenery, with bad weather at a detailed addon airport, you need different settings to get the best results then when you fly a GA in rural environments with FTX Global for example.

 

People all give their experiences, but you never EXACTLY know the combination of hardware (RAM timings, CPU-type + clock etc), Windows settings (and services running + what drivers and firmware installed), installed add-ons and their settings (Accufeel, UTX, AI-traffic) etc. so that makes it very hard to compare.

 

The best tweak is learning yourself to accept what you have I guess.

[stans 703]

My testing method is very simple,  Using a low wing aircraft, I go to a scenery intense area, take off, fly straight and level while watching the scenery pass beneath the leading edge of the wing.  I am looking for pauses and stutters.  Then i put the aircraft into a 45 degree bank and turn hard, watching the horizon and observe for pauses and stutters.  I use frame rate counts only as a matter of reference, what I am really seeking is a smooth display of the image.  Smooth and 15 fps is preferable to 40 fps with stutters and pauses.

[Mork 34]

My testing method is very simple,  Using a low wing aircraft, I go to a scenery intense area, take off, fly straight and level while watching the scenery pass beneath the leading edge of the wing.  I am looking for pauses and stutters.  Then i put the aircraft into a 45 degree bank and turn hard, watching the horizon and observe for pauses and stutters.  I use frame rate counts only as a matter of reference, what I am really seeking is a smooth display of the image.  Smooth and 15 fps is preferable to 40 fps with stutters and pauses.

 

But I can tell you that when you have it all optimized and then decide to fly the NGX on a detailed add-on airport, you will experience worse performance and need to start tweaking again to get the smoothness back. These airports also have more heavy AI traffic than when flying at small airports in non-heavy areas.

After that when you want to fly in VFR environment with a GA plane you know you are missing out on some details, because you now you have resources left compared to flying the NGX. If you really want the maximum eyecandy and performance, it heavily depends on what and where you are flying.

[mmann 72]

 

 

Then i put the aircraft into a 45 degree bank and turn hard, watching the horizon and observe for pauses and stutters.

 

This is what I find works best for testing.

[Guest]

Hi Jeroen, as far as the TBM I have the exact same experience as you. I used to fly with the default 40 value, because it made no difference whatsoever. But I know other people swear by it, so I have set it at 80. At 120 I thought I saw some small irregularities, but like you I also acknowledge the 'it's all in your head' factor, lol.

Also the texturemaxload entry has an effect on this.

 

Did you also test FTFF? Because that one is suspicious too, because it should only work on single core CPU's I read in some old FSX documentation somewhere.

 

I do use higher LOD, but you are right, the circle of LOD doesn't grow as fast as you think. I have set it at 7.5 now I think, because with DX10 you don't get OOM's that quickly and it still performs well.

 

Affinitymask does seem to work here, but only a little bit. I have it set in my cfg. (2500K -> AM=14)

 

Bufferpools (=0) I have tested many times and does seem to work, however if I use bufferpools settings for high end PC's, I can almost see no difference. Tested this yesterday.

 

Internal or external FPS limiter: I have tested it many times, external ALWAYS worked better here, but yesterday I used the FSX limiter during testing and now I had about the same performance as external?! LOL (With Vsync active, I don't even know exactly if this matters at all)

 

The problem with all this is:

 

1) With DX10 the tweaks don't seem all necessary, and it is hard to compare tweak results with DX9.

 

2) It does seems to matter where you fly. Some tweaks only show their effect on certain scenery and when flying the NGX for example. The sim has different calculations going on, and I suspect that if you fly at detailed scenery, with bad weather at a detailed addon airport, you need different settings to get the best results then when you fly a GA in rural environments with FTX Global for example.

 

People all give their experiences, but you never EXACTLY know the combination of hardware (RAM timings, CPU-type + clock etc), Windows settings (and services running + what drivers and firmware installed), installed add-ons and their settings (Accufeel, UTX, AI-traffic) etc. so that makes it very hard to compare.

 

The best tweak is learning yourself to accept what you have I guess.

 

Since I see no difference with TBM I like to keep it at the default 40. I don't like to change default settings if not necessary.

 

No, I didn't test FTFF because I already don't like how FSX loads textures and I really don't want FSX to do that even sloppier...

 

Bufferpools: you use DX10, right? In that case it indeed doesn't make such a big difference as it does with DX9. Which means that DX10 without tweaks runs better than DX9. However, with tweaks DX9 runs just as good as, if not better, than DX10 on my PC. After a LOT of testing I decided to stick with DX9 mainly because there is some screen tearing/wobbling in DX10 (no vysnc problem but something else). And just as with keeping TBM at default I like to use DX9 instead of DX10 with all kinds of fixes. The less tweaks and fixes the better imho.

 

I used to use an external limiter until I found out textures were loading slower in the distance compared to when I use the ingame limiter, so now I set 30 fps ingame and I use the 1/2 vsync Inspector option.

 

And yes, results are different in different locations. My heavy low and slow testflight in a valley with lots of trees showed different results when it came to comparing DX9 and DX10 than flying low and slow above London. The differences were in favor of DX9 there btw.

 

You're last sentence is very true indeed and the real 'magic bullet'! I have to add that I always had some problems accepting the FSX performance I had because I thought something was not working right on my PC. But now I actually found out by testing what for instance TBM and LOD_RADIUS do EXACTLY on my PC, I now know that the things I don't like about performance simply are limitations in the FSX engine and there isn't much you can do about it. My PC is working PERFECTLY fine and that made it a LOT easier to accept and enjoy what I've got!

[moria 0]

I have 3 test areas that I use and have set up three save flights for the same.

 

As I do most of my flying in the UK.. they are as follows....

 

1)  Open scenery roughly half way between Manchester and Edinburgh at 3500 feet..    used for ground scenery and flying straight and testing smoothness on banking.

2)  Coming in over the coast at Dover  (starts humming.,.  there'll be bluebirds over  etc)  at 5000 ft  as your starting to pick up some complex areas

3)  Approach to Heathrow (I have UK2000 and all the other UK London airports) at 2500 because of AI traffic and scenery density.

 

I then do all three with both a GA and a tubeliner and in each case I use clear skies and a set theme with lots of clouds.

 

So thats a total of 9 tests for each setting change.

 

Ultimately though, what it means is I have developed some 3 different settings files, depending on whether I am fluing GA, Tubeliner and where.

 

Again, I have no idea about FPS thesedays and dont even look at it,  to me it's

 

1) Is it smooth and no jaggies or artifacts.

2) Do all my controls get responded to immediately.

 

Regards

 

Graham

[Jim Young 2,356]

I like the testing conducted at this recent topic - http://forum.avsim.net/topic/414988-profiling-fsx-via-process-monitor/.  I have done something similar in the past and basically came up with the same findings as the topic OP at the link I provided. 

 

Best regards,

Jim