Archived

This topic is now archived and is closed to further replies.

Petraeus

A Benchmark Comparison

Recommended Posts

As much out of personal interest as anything, I decided to do a comparative benchmark of the FSX passenger planes in my hangar, to compare the FPS they would give under standardised conditions. I'm sharing the results because others may find them interesting.I've described my system configuration, and the FSX situation that I set up, at the end of this post. Obviously the absolute numbers for FPS that I achieved were very specific to this setup. However it is the relative performance of one aircraft versus another that is of most general interest, and should apply to many, if not all, FSX setups. I've summarised the result as a graph below.Perhaps not surprisingly, the four Microsoft stock aircraft perform the best. They don't have complex avionics such as an FMS, and one would expect MS to know how to build something that performs well in the environment it designed.However the two Flight 1 / Coolsky MD80's perform very well in comparison, especially as they both have flight management systems, albeit of different generations. The original MD80 is particularly impressive, given that it is ported across from FS9 rather than being a native FSX build.Then there comes "the pack" of reasonable performers from Level D, PMDG and feelThere / Wilco, although interestingly they are led by another port to FSX, the Level D 767."The pack" is followed by the feelThere / Wilco CRJ 700 and the iFly 747-400, both starting to struggle a bit, although bearing in mind that the iFly 747 is freeware.Finally we have the Captain Sim 757. This is actually the demo version with no avionics. It should be fairly obvious why I didn't go on to buy the full version, even with the recent deal at $9.99. For the sake of an open/close perspex bookcase cover in the cockpit and similar novelties, this thing just sucks the lifeblood out of my system like some software vampire. It needs 40+ FPS to feed its habit, leaving a mere 7.9 to drive scenery rendering, AI traffic, weather etc. Which is a great shame, because the 757 is my personal favourite real-life aircraft.I don't have figures for the PMDG 747 or Ariane 737, for the very simple reason that I just haven't bought them to date.It may be that there are components in my system that tend to favour one aircraft over another, so the precise ratios between the aircraft may differ on your system. However, I would be very surprised if the rankings or overall picture is vastly different.[System: Intel Core 2 Duo 8600; 4Gb RAM; GeForce GTX 280; nothing overclocked. Windows Vista 64.FSX Situation; FTX Australia scenery; Autogen dense; No AI (to remove a possible source of variability in readings); static weather (ditto); aircraft is lined up on YMML runway 27, all engines running and avionics active.Measurement: Situation and aircraft set up and left for 5 minutes, then average FPS reading taken.]

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

Nice work Petraeus. This will be very useful for others. All a/c are not created equal.Pierre

Share this post


Link to post
Share on other sites

Very cool!However, I find it odd that the 757 demo performs so poorly on your machine. With the demo I got default plane-like FPS. The full version gets FPS similar to the LDS-767. FSX is so odd...To make your graph completely accurate, I think you should comment on which planes are true FSX models. By default, they will give better FPS. Perhaps use the same colored bars or darker shades for FS9 portovers, and use lighter colors for the true FSX planes.This is a real neat idea though. It would be cool to see you do some graphs for other types of products too ;)

Share this post


Link to post
Share on other sites

Thanks for the project.Very informative.How about some GA prop planes.I love Premierair,but they seem to take a performance hit.Especially if I go to locked spot view often.I tend to get the dreaded OOM.Thanks,Ron

Share this post


Link to post
Share on other sites
This is a real neat idea though. It would be cool to see you do some graphs for other types of products too ;)
I'll do that when I have a bit of spare time. Although my hangar is a bit of a confused mixture... :(

Share this post


Link to post
Share on other sites

I've also wished reviewers would do the same thing when reviewing an aircraft. Reviewers telling me they have such and such a computer and that they get so many fps at some airport is pretty much useless. However, if they were to provide some comparison fps with some default aircraft and other addon aircraft I feel this would allow readers to better judge how the product would work on their own computer.Plus I don't have the CS 757 but I can't believe even CS would release such a poor performing aircraft so that me be an anomaly peculiar to your system. The rest of the numbers seem to back up what I have felt when using those aircraft that I have (eg the LevelD 767, Wilco CRJ and Wilco Airbus)

Share this post


Link to post
Share on other sites

Great comparison. Super80 performs excellent on my poor machine, and actually is only airliner addon that I can fly in FSX without problems. :(

However the two Flight 1 / Coolsky MD80's perform very well in comparison, especially as they both have flight management systems, albeit of different generations. The original MD80 is particularly impressive, given that it is ported across from FS9 rather than being a native FSX build.
Original Super80 FSX is not a port over. It was actually rebuild for FSX using FSX SDK and is 100% native FSX model. On the other side Wilco 737 is a cheap port over and it shows on frames.

Share this post


Link to post
Share on other sites

Great work,Totally agree with the results. I bought Captain Sim 757 and level D 767 on the same weekend. The 757 was removed from my system by the Monday, FPS was unplayable whereas 767 seemed as fast as any others (almost).

Share this post


Link to post
Share on other sites

I don't have a problem flying planes like the CS 757. The trick is to save different display settings in FSX and load them each time you fly to match the aircraft and environment you are going to fly in. With the CS 757, I load a file that has all the sliders turned down just enough to make it flyable around the biggest airports. You lose the eye candy, but I didn't pay for complex airliners to fly low and slow looking at scenery.

Share this post


Link to post
Share on other sites
Good job and very useful information.
Thanks a lot. Where the measurements taken inside the virtual cockpit or an exterior view? If you have the time and the inclination a chart comparing both for each aircraft would be very interesting. RegardsSimon

Share this post


Link to post
Share on other sites
Thanks a lot. Where the measurements taken inside the virtual cockpit or an exterior view? If you have the time and the inclination a chart comparing both for each aircraft would be very interesting.
In the VC, looking forward, zoom level 0.70.And yes, I will, if time permits.

Share this post


Link to post
Share on other sites

Good post!Wow I always knew Captain Sim 757 was a pile of crap (after I bought it, unfortumately), but this comparison really shows how incapable the Captain Sim team is. It is an insult to flight simmers to even release a product in this state. Everybody should be warned that this product is unusable.

Share this post


Link to post
Share on other sites

This kind of information is very useful indeed. I suggested to one of the after market folks that it would be nice if they would give the consumer a default ac reference number for their product. It didn't go over very well. But I still think it would be nice if those who offer add-on products for MSFS could all agree on some standard of measure regarding performance. For example, using pre-defined mid level settings in FSX (no traffic and no weather) and using the VC view in the default Baron at a pre-defined airport parking spot, product XYZ is yielding 60% of what the Baron does. Then we could get a pretty good idea of whether our systems would run their product by applying the same settings and observing the outcome. Granted it would not be fool proof due to all the variables involved in equipment. And there would be no guarantees. But at least we could get some idea regarding predicted performance with their product.Best,Bill

Share this post


Link to post
Share on other sites

Suggest:1) we sticky this2) we use normalised numbers, e.g. using the CRJ700 as a base reference of 1.0 (FPSnew/FPSref)That way we can add more benchmarks as we go along and still compare apples and apples. I do have some addons that I can add to the list but because my machine specs are different only if the figures are normalised would they be useful. If everyone chips in we'll have a bigger sample size as well. I suspect most GAs and twin props would have a rating of close to or above 1 for payware and default Microsoft. I hereby christen this the "Petraeus Rating". :(

Share this post


Link to post
Share on other sites
Suggest:1) we sticky this2) we use normalised numbers, e.g. using the CRJ700 as a base reference of 1.0 (FPSnew/FPSref)
I'm not in a position to be able to do 1, but 2 is an excellent idea. I don't have every plane in my hangar, but others will have the missing ones, so they can contribute.During the course of the next week I'll reissue the passenger plane numbers (including the PMDG 737-600/700 that I've just ported across to FSX), but normalised to the CRJ700 as you've suggested. Then hopefully others can chip in with Ariane, PMDG 747-400, others like Overland etc. After that I'll do my GA planes, so others can do likewise.I'll also suggest in the Review Forum that they do likewise.
I hereby christen this the "Petraeus Rating". :(
If that were my real name I'd decline on the grounds that it's too egocentric. :( But as it's only a pseudonym, why not? :(

Share this post


Link to post
Share on other sites
I'm not in a position to be able to do 1, but 2 is an excellent idea. I don't have every plane in my hangar, but others will have the missing ones, so they can contribute.During the course of the next week I'll reissue the passenger plane numbers (including the PMDG 737-600/700 that I've just ported across to FSX), but normalised to the CRJ700 as you've suggested. Then hopefully others can chip in with Ariane, PMDG 747-400, others like Overland etc. After that I'll do my GA planes, so others can do likewise. If that were my real name I'd decline on the grounds that it's too egocentric. :( But as it's only a pseudonym, why not? :(
Excellent Petraeus. I will send you some numbers over the course of next week and you can update that master Excel file (it would be a good idea to include sample size as well). If we can't sticky this let's figure out a way to post this somewhere on the Net (such as a blog, but we'll discuss offline). I still think it's a very useful exercise. :(

Share this post


Link to post
Share on other sites

May I suggest that the base Petraeus Rating be 100 instead of 1.0 (ie New FPS/CRJ700 FPS x 100)? Whole numbers are always easier to grasp than fractions.

Share this post


Link to post
Share on other sites