Archived

This topic is now archived and is closed to further replies.

Petraeus

The Petraeus Index

Recommended Posts

Shown below is version 1 of what has been termed the 'Petraeus Index', comparing the relative FPS performance of aircraft in FSX. My thanks to fellow-collaborator PontiusPilotus for helping me to put together this first set of figures.The figures are broken down for convenience into three sets - airline, GA and other (bizjet, military etc.)Full details of what it's all about can be found in this AVSIM Wiki entry. It's also the place where you can add in your own figures - there are full instructions there. You can either supply your own numbers for aircraft already shown, to give us a larger statistical sample, or add new numbers for aircraft we don't have details of yet. For example, we have no figures yet for Ariane 737's, or for the Leonardo MD80. So please pitch in and help this to become the definitive source of relative aircraft performance data. - the better the sample data, the better the overall reliability of the Index. The numbers are relative to the stock Microsoft CRJ700, which is given the value of 100. So something with a score of 33 will typically only give a third of the framerate of the CRJ on your system; something with a score of 150 will typically give 50% more frames per second.Enjoy! :( AirlineGAOther

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

Neat idea!I wonder how repeatable this measurement is across systems.. and ultimately in situations where the systemis taxed more heavily with scenery, weather etc and the aircraft has to compete with other computing demands.I also have noticed that the Carenado C182Q is very framerate friendly, so that part of the scale I can identify with. :(

Share this post


Link to post
Share on other sites

Thanks for the useful comparison, Petraus! If something like this can be consistent across platforms, it would be great to have a framerate rating system fo popular titles. I'm going to run some comparisons of my own, and I'm curious whether the fairest comparisons also necessitate fully restarting FSX for each test- Is there a difference in the buffer overheads between just switching planes, as opposed to completely restarting FSX?

Share this post


Link to post
Share on other sites

Great idea ! However, i am surprised to find f.e. the Twin Comanche so far down the list. It's a top performer in my hangar (a big hangar i might add), much better than f.e. the Saratoga. I'll check if i missed an update...

Share this post


Link to post
Share on other sites

Quite a cool idea, I like it. As Mango noted though, some aircraft will perform better or worse on some systems than others, based purely on system configs and the way the aircraft was modeled and textured. Generally a system with a lower end cpu but a very powerful video card is going to favour aircraft like the Carenado or RealAir planes which are perhaps more demanding than average on graphics power and less so on system or gauge computations.For example the RealAir Marchetti and Spitfire are both in your list, and both similarly constructed, but very much swapped in performance from what I experience. I adore them both but the Marchetti *vastly* outperforms the Spit on my machine, especially in multiplayer.Cool stuff though. :)

Share this post


Link to post
Share on other sites
I'm curious whether the fairest comparisons also necessitate fully restarting FSX for each test- Is there a difference in the buffer overheads between just switching planes, as opposed to completely restarting FSX?
There may be a difference, but if we all follow the present method, of just starting with a new plane, then at least we'll be 'consistently different'.
As Mango noted though, some aircraft will perform better or worse on some systems than others, based purely on system configs and the way the aircraft was modeled and textured. Generally a system with a lower end cpu but a very powerful video card is going to favour aircraft like the Carenado or RealAir planes which are perhaps more demanding than average on graphics power and less so on system or gauge computations.
That's why we're asking people to feed in their own numbers. Now we've taken out the major source of variability, the overall power of a system, we're left with lesser sources of variability. If we get enough data we can show just how much this residual variation actually is.

Share this post


Link to post
Share on other sites
There may be a difference, but if we all follow the present method, of just starting with a new plane, then at least we'll be 'consistently different'. That's why we're asking people to feed in their own numbers. Now we've taken out the major source of variability, the overall power of a system, we're left with lesser sources of variability. If we get enough data we can show just how much this residual variation actually is.
I think this is a great initiative and will be glad to contribute. Understanding the problem about variations in user hardware, perhaps even more sliders (not related to the user aircraft) could be turned down, eg. scenery complexity, terrain related etc. Even better, perhaps Petraeus could make a standard "saved settings" file available for download? While that would not eliminate every possible source of variation, it would perhaps help prevent wild fluctuations in results.RegardsSimon

Share this post


Link to post
Share on other sites

Thank you to all who contributed to this great idea. We have been in need of this type of info for a long time IMO. (FWIW the aircraft I own perform very close to the data on the graph.) This needs to be pinned in my opinion and more data added as time permits.Thanks again,Bill

Share this post


Link to post
Share on other sites

I like to add my findings as well. While running the tests, i saw that i have not all aircraft installed, but i hope that's enough anyways. The CRJ in mind, i adjusted the cfg until i reached exactly 100 FPS average with this model (autogen, scenery and traffic off). The aircraft settings/detail was at maximum. Interesting that some aircraft have very unstable FPS (at least on my system with FPS set to unlimited). However, those mentioned below perform all well with locked FPS - just the average result doesn't tell the truth... The F-16 is a surprise , i don't have problems to fly it fluid with my normal settings. The Bushhawk needs a patch if you ask me... Oh, and sorry about the Saratoga - the non glass model is of course not bad.Default DC-3 - 162Default Piper Cub - 161Default Lear - 144Default Robinson R22 - 142Default Mooney - 142Shockwave P-51D - 134Default King Air - 131Aircreation Trike - 129Carenado C182 RG - 125 (HD VC version, cfg set to 1024)Default DG 808 - 123Default C172 - 122Acceleration P-51D - 121Default Grumman Goose - 121Default C208B - 117Default Extra 300S - 117Default Baron - 116Default Maule - 115Shockwave B-17G - 115Default JetRanger - 112RealAir Decathlon - 111Default Mooney G1000 - 111Carenado C152 - 108Default Beaver - 103Default Boeing 737 - 102 Default CRJ700 - 100Default Airbus A321 - 100Default C172 G1000 - 100Carenado C172N - 99Eaglesoft Twin Comanche - 96Aerosoft Long-EZ - 96Eaglesoft CJ1 - 96RealAir SF-260 - 92Eaglesoft Liberty - 91FSD Saratoga - 91Flight1 Citation Mustang - 86CS C-130 - 86DA Cheyenne IIXL - 84Default Baron G1000 - 82Flight1 PC-12 - 78Acceleration EH101 - 77Aerosoft Beaver - 74Acceleration F-18 - 68FSD Saratoga Glass - 68Wilco Citation X - 60Eaglesoft Cirrus SR22 - 59Eaglesoft Columbia 400 - 55CS 757-200 - 48DA Do-27 - 46Aerosoft Bushhawk - 40Aerosoft F-16 - 37--------------------------------------------------------------------------------------------------Coolsky MD80 - 178 (very unstable between 25 and 310 FPS)Aerosoft DHC-6-300 - 166 (very unstable between 20 and 300 FPS)Shockwave Spitfire - 166 (very unstable between 34 and 345 FPS)Iris PC-9 - 148 (very unstable between 22 and 245 FPS)Shockwave Bf109E - 144 (very unstable between 32 and 340 FPS)Carenado Mooney - 118 (very unstable between 30 and 225 FPS)Default Boeing 747 - 114 (unstable between 80 and 155 FPS)Btw, this test on another system might show a completely different result.

Share this post


Link to post
Share on other sites

Does this index represent only the external model? Because posting a VC index would kill a lot of these values

Share this post


Link to post
Share on other sites
I like to add my findings as well. While running the tests, i saw that i have not all aircraft installed, but i hope that's enough anyways. The CRJ in mind, i adjusted the cfg until i reached exactly 100 FPS average with this model (autogen, scenery and traffic off). The aircraft settings/detail was at maximum. Interesting that some aircraft have very unstable FPS (at least on my system with FPS set to unlimited). However, those mentioned below perform all well with locked FPS - just the average result doesn't tell the truth... The F-16 is a surprise , i don't have problems to fly it fluid with my normal settings. The Bushhawk needs a patch if you ask me... Oh, and sorry about the Saratoga - the non glass model is of course not bad.Btw, this test on another system might show a completely different result.
Thanks Mango, we'll add these to the graph. Can you confirm that these are raw FPS readings (we can normalise them for you) and that the readings are for the VC view? You mentioned that the FPS was locked in some of these cases; I hope they in no way constrain any of the FPS measurements (causing them to be invalid).Thanks again for the contribution.Ponti

Share this post


Link to post
Share on other sites
Does this index represent only the external model? Because posting a VC index would kill a lot of these values
The wiki says measurements to be made inside the VC. Another chart of measures of the external model would be very interesting too!

Share this post


Link to post
Share on other sites

Very cool idea and initiative, hope to see it go further!I have a lower end rig , yet default FSX, does quite well with higher settings, so I tend to stay with aircraft that do perform closer to the defaults, such as those from Carenado.Really to bad we can't have a place like AVSIM or another large site that can get a review copy of every addon aircarft and then post /add to a chart like this. would help with purchases (but probably hurt developers that have resource intensive addons)Would also like to see other categories such as helicopters and vintage warbirds of which there is a big commercial market.

Share this post


Link to post
Share on other sites

What a great idea! Congratulations.What do you think about having a little application that would do the following things:-1) Backup the users FSX.CFG and Scenery.CFG files2) Set FSX.CFG and Scenery.CFG files known values (eg Graphics, Scenery, Traffic sliders etc)3) Launch FSX4) For each aircraft in the users system:- (a) Start a flight at your test airport with that aircraft loaded, and sit idle for say 3 mins(b) Write the average FPS and aircraft name to a .CSV log file5) Restore the FSX.CFG and Scenery.CFG from backup6) Upload the .CSV log file to your ftp/email for you to analyse/chart resultsThe advantages of such an app would be:-1) Consistent FSX test configuration2) Makes it easy for users to run test and so many more users might actually run the test, hence you will get a bigger sample3) You could easily test different scenerios (eg different FSX.CFG settings)4) Results are collated for you automatically

Share this post


Link to post
Share on other sites
What a great idea! Congratulations.What do you think about having a little application that would do the following things:-1) Backup the users FSX.CFG and Scenery.CFG files2) Set FSX.CFG and Scenery.CFG files known values (eg Graphics, Scenery, Traffic sliders etc)3) Launch FSX4) For each aircraft in the users system:- (a) Start a flight at your test airport with that aircraft loaded, and sit idle for say 3 mins(:( Write the average FPS and aircraft name to a .CSV log file5) Restore the FSX.CFG and Scenery.CFG from backup6) Upload the .CSV log file to your ftp/email for you to analyse/chart resultsThe advantages of such an app would be:-1) Consistent FSX test configuration2) Makes it easy for users to run test and so many more users might actually run the test, hence you will get a bigger sample3) You could easily test different scenerios (eg different FSX.CFG settings)4) Results are collated for you automatically
That's a great idea, Matthew. Always had something like that in mind. Are you volunteering? :( The more the merrier!Ponti

Share this post


Link to post
Share on other sites
Thanks Mango, we'll add these to the graph. Can you confirm that these are raw FPS readings (we can normalise them for you) and that the readings are for the VC view? You mentioned that the FPS was locked in some of these cases; I hope they in no way constrain any of the FPS measurements (causing them to be invalid).Thanks again for the contribution.Ponti
Hi Ponti,FPS were of course unlocked and all were measured in the VC at daytime (vsync off). I even took care of the different initial zoom levels and was setting it like i would fly the aircraft. Readings are average FPS rounded up or down. All systems were running, weather was set to clear sky. The CRJ had straight 100 FPS average.Like i said, some aircraft have very stable FPS at unlimited, some are all over the place... Wonder why.

Share this post


Link to post
Share on other sites
1) Backup the users FSX.CFG and Scenery.CFG files2) Set FSX.CFG and Scenery.CFG files known values (eg Graphics, Scenery, Traffic sliders etc)3) Launch FSX4) For each aircraft in the users system:- (a) Start a flight at your test airport with that aircraft loaded, and sit idle for say 3 mins(:( Write the average FPS and aircraft name to a .CSV log file5) Restore the FSX.CFG and Scenery.CFG from backup6) Upload the .CSV log file to your ftp/email for you to analyse/chart results
Well, no need to do anything with the scenery.cfg. Go to the FSX settings page - you will find four buttons (Save, Load, Customize and Reset Defaults). There you save your current settings as f.e. Standard and your FPS test settings as f.e. Test. Easy to switch. Set up your test situation (i was sitting at the Ayers Rock airport with the CRJ700) and save the flight as f.e. FPS Test. It can happen that some aircraft crash when you work through your hangar. Load the 'Test' cfg and the 'FPS Test' flight and you are back in no time.

Share this post


Link to post
Share on other sites

Hi all,I hate to sound like a party pooper, but it seems to me there are more variables than you can shake a stick at. Kindof like herding cats sort of. Setting up a test configuration environment is not an easy task.I hope you guys can figure it out as it may be somewhat valuable in the decision process to buy or not to buy. I must agree on the Carenado findings as all of their planes have done great on my system.RegardsBG

Hi Ponti,FPS were of course unlocked and all were measured in the VC at daytime (vsync off). I even took care of the different initial zoom levels and was setting it like i would fly the aircraft. Readings are average FPS rounded up or down. All systems were running, weather was set to clear sky. The CRJ had straight 100 FPS average.Like i said, some aircraft have very stable FPS at unlimited, some are all over the place... Wonder why.

Share this post


Link to post
Share on other sites
Hi all,I hate to sound like a party pooper, but it seems to me there are more variables than you can shake a stick at. Kindof like herding cats sort of. Setting up a test configuration environment is not an easy task.I hope you guys can figure it out as it may be somewhat valuable in the decision process to buy or not to buy. I must agree on the Carenado findings as all of their planes have done great on my system.RegardsBG
Agreed, but it's not about the test environment, it's a measure aganist airacraft, not the environment. Example, according to the two lists posted, the Aerosoft Bushhawk gave less than half of the FPS than they got with the default CRJ, so no matter what your environment setup, you should see a similar result between the two aircraft. So if you are getting 20 fps in the default CRJ, then I would expect to get around 10 FPS in the Aerosoft Bushhawk based on this test data. If enough people post their findings, then performance can be more precise based on the average.This is why I always ask users how an addon compares to the defaults since I get good performance with them.

Share this post


Link to post
Share on other sites

I always compare my addons to the default. And the the developers complain. Blah blah blah... It's best to compare them to default because everyone has default aircraft. I don't have nearly as many addons as a lot of you have but essentially say the C172 (non g1000) vc is 100 index:RealAir scout/decathlon 90Super 80 Pro 80LevelD 767 75SR22 G3 Turbo 60Mustang 50(all in the VC)

Share this post


Link to post
Share on other sites

Hmm, I guess your right. It really is relative when you look at it from your perspective. But I did see in another post that the F1 Mustang in one graph pretty much sat at the bottom but someone else posted and it was somewhere in the middle. So I am wondering.... Here is something to think about. You here so many talk about how increasing traffic (either car or ai) cripples their system. With my system, this is not the case. I can essentially turn up traffic on the ground and in the air and see very little change in my Frames. On the other hand, when I load up a glass cockpit, my frames tumble and I am running a 9650 processor. For me, autogen, traffic, clouds can be reasonably upped with out too much penalty, but sophistacated planes crash my system, even with the traffic and other sliders turned way down. This has always confused me. Anyway, good luck with the metrics. RegardsBG

Agreed, but it's not about the test environment, it's a measure aganist airacraft, not the environment. Example, according to the two lists posted, the Aerosoft Bushhawk gave less than half of the FPS than they got with the default CRJ, so no matter what your environment setup, you should see a similar result between the two aircraft. So if you are getting 20 fps in the default CRJ, then I would expect to get around 10 FPS in the Aerosoft Bushhawk based on this test data. If enough people post their findings, then performance can be more precise based on the average.This is why I always ask users how an addon compares to the defaults since I get good performance with them.

Share this post


Link to post
Share on other sites
That's a great idea, Matthew. Always had something like that in mind. Are you volunteering? :( The more the merrier!
If I had the time, yes I would, but unfortunately I don't, so I can't! :(

Share this post


Link to post
Share on other sites

This looks interesting. I applied the same procedure, but for FS9 (no FSX in my PC) and I got an index of 72 for the PMDG 747-400. Because there's no default CRJ in FS9, I used the default 747 (which has an index of 100, and if I'm not wrong, the FSX one is exactly the same as the FS9 one) as reference. Is this a good reference, or should I've taken another one? And has anybody else tried doing this on FS9?Also, the Wiki says AI and autogen should be turned down to zero. Do I have to do so with Scenery or that's not necessary?Best regards from ColombiaLuis Miguel

Share this post


Link to post
Share on other sites

Interesting!, For it to be considered an accurate gauge though, it would have to be shown these numbers are relatively consistent, by the same percentage of difference, across the wide range of different systems. Also looking at the charts on the Wiki Page, why are there business jets listed under other aircraft? Wouldn't they be considered GA? Same goes for the Wilco CRJ and ERJ-145, shouldn't they be under airliners? I'm going to run these tests myself and see what I come up with with my system.

Share this post


Link to post
Share on other sites