Benchmarking FSX and Prepar3D

Review By Doug Horton.

I often see claims in forums by flight simulator users who credit particular changes for framerate performance improvements, which when subjected to statistical testing of framerates, do not hold up. This article describes the framerate benchmarking procedure I’ve been using for several years to sort out performance claims for hardware and configuration options for running FSX and Prepar3D. Flightsim friends have sometimes asked me to comment on new systems builds, and often, I point out that some inexpensive components that will improve FSX performance, while some expensive components will not help FSX run any faster, and I prove these differences by benchmarking FSX framerate performance.

In some cases, users make hardware or settings changes, see a certain instantaneous framerate changes, and then optimistically draw conclusions to justify their hardware investments or settings changes. But the framerate while the sim is paused is a poor measure of performance in flight. And it’s important to standardize all settings, so results are repeatable. Where does the flight take place, what’s the track, and what sceneries are being used? What display options are set? What weather is selected? Which aircraft is being used? Is the 2D control panel or Virtual Cockpit displayed? What percentage AI aircraft traffic is set? This last setting strongly influences FSX performance, by the way. The list of variables is long, and in most cases, standardized settings and test methods are not used to justify the claims from “tweaking” time sim, so the results, in a statistical context, are neither repeatable(same user, same settings and equipment), nor reproducible (different user, same settings, different but identical equipment).

There is also a notion of “causality” in the study of statistics. Causality means that an observed effect or measureable result is due solely to one change in an independent variable, such as one hardware or settings change, with all other factors held constant. Many times, observed effects are coincident, not causal, such as, “It rained yesterday and I won the lottery!” Coincident yes, causal no!

It’s also important to check the data variation to ensure there are no data “outliers,” which might be caused by something happening in another application while benchmarking the sim. For example, if Windows begins to automatically download updates while FSX performance is being measured, results may be affected, and if result is significantly different than other results, the outlying data point could be ignored, particularly if a reason for the difference can be identified.

This is a very simple procedure, and once learned, in about ten minutes, you’ll never again wonder whether a hardware or settings change was responsible for a FSX or Prepar3D performance difference. The descriptions below cover installation and testing for FSX, with the installation for Prepar3D being the same, except for use of corresponding file locations. Also, it will be necessary to copy the Bombardier CRJ 700 files from FSX to Prepar3D.

Case Study

I began serious benchmarking exercises several years ago after I began receiving sample motherboards and processors from Intel. I was curious how a new processor performed, including cautious overclocking of the processor, with use of a more robust cooling fan than furnished with the processor. I’d previously learned that FSX is significantly “CPU-bound” and most hardware changes make no difference. Recognizing this fact can save readers from spending money on hardware upgrades that will provide no performance improvements.

One exception to this statement is investment in new processor, and as necessary, a motherboard and memory to support the new processor. As an example, I previously tested a relatively inexpensive Intel Core i5-2500K processor, which at the time could be purchased for about $200. This processor uses the socket LGA 1155 architecture, and I tested it on an Intel DP67BG motherboard, which at the time was selling for about $170 US. Though the i5-2500K is a four-core processor, it’s not capable of hyper-threading, so it presents only four cores to FSX, and because that’s the most FSX can utilize, it works fine.

Importantly, the i5-2500K processor also includes Intel’s Turbo Boost 2.0 feature, and this feature, with no user action, other than verifying in the BIOS that Turbo Boost is enabled, will boost the i5-2500K processor core frequency from its default value of 3.3 GHz, to as high as 3.7 GHz. Using Intel’s Extreme Tuning Utility, I incrementally overclocked the Turbo Boosted frequency to 4.4 GHz, and using the FSXMark07 benchmarking procedure, I verified linearly proportional increases in FSX framerate performance with increased processor speed, as illustrated in the accompanying graph. Note for statisticians: the graph also displays the spreadsheet-calculated linear regression equation and correlation coefficient, with a nearly perfect fit of the line to the data.

Graph of FSXMark07 framerate test results over range of i5-2500K processor speeds

What follows is a description of the proven method for benchmarking FSX framerate performance with standardized settings, to provide causal and repeatable results. Take the time to read the description, installation instructions, how to run tests, and how to view logged results. Installation can be completed in less than 10 minutes after downloading two files, and once installed, testing is as easy as starting FSX with the saved configuration file and benchmarking flight, confirming settings once per session, running about five, five-minute test flights, calculating the simple average of the five trials, and then reviewing results to ensure results make sense.

Benchmarking Procedure

The best means of accurately benchmarking FSX framerate performance is using the free FSXMark07 package, developed by flightsim user Gary Dunne. The file FSXMark07.zip, available in the AVSIM library, includes the necessary FSX files and simple instructions, which are summarized below. The free FRAPS utility is used for framerate tracking and logging. With FSX, standardized testing with FSXMark07 means:

• Only one hardware component or configuration setting is changed at a time, and a standard configuration for display settings is provided by loading a specified *.cfg file, such as the “FSXMark07 - Global High.cfg” file to standardize 38 parameters in the user’s “FSX.cfg file,” and then making a few other changes, as indicated below.
• If scenery, terrain, land class, or airport and facilities data (AFCAD files), or other files are added to FSX in the Scenery Library, they should be disabled for the test area, which with FSXMark07 is the greater Seattle, Washington, USA area.
• A selected default aircraft flies a set route, on autopilot, on auto-throttle, at a set altitude and airspeed, with other conditions (or “states”) set in the included FSXMark07.flt (flight), FSXMark97.fssave (states), FSXMark07*.pln (flight plan), and FSXMark07.wx (weather) files.
• FRAPS software is used to track and log framerate data for five minutes for each test run.

The FSXMark07 test flight begins paused, in the default Bombardier CRJ 700, north of downtown Seattle, and after un-pausing FSX, the flight continues toward the downtown area, turns to fly over Boeing Field and SeaTac airports, and it then heads toward the Auburn airport, with Mt. Rainier visible in the distance.

The flight provides a great variety of scenery, terrain, and clouds, with a modest amount of AI aircraft, auto, and boat traffic. The CRJ 700 flies at 1500 feet MSL and 250 knots, as FRAPS counts individual frames over the five-minute period. The end of the five minutes is signaled by the FRAPS framerate box reappearing on the screen. The test flight is then repeated about four more times, to ensure results are consistent.

After the recommended five test flights, the simple average of the FRAPS-recorded framerates is calculated. Pinning the file FSXMark07.flt to the Windows taskbar simplifies reloading the flight and resetting FRAPS for the additional trials.

Installation of FSXMark07 Files and FRAPS

Download FSXMark07.zip from AVSIM, and download FRAPS. Read the FSXMark07 instructions, which also cover FRAPS installation, as summarized below.

For the FSXMark07 procedure installation, extract the FSXMark07.flt, .fssave, .plan, .wx, and files to the C:\Users\(username)\Documents (or My Documents)\Flight Simulator X Files\ folder. Similarly, extract the FSXMark07 - Global High.cfg file to the same location. Note: though an FSXMark07 - Max.CFG file is provided, it is not used, because its settings are too aggressive.

FRAPS installation is from its executable file, and after installing, run FRAPS. On its setting interface, ensure the following settings are made:

• On the General tab, check “Start Fraps minimized,” “Fraps window always on top,” and “Minimize to system tray only.”

FRAPS interface – General tab settings for FSXMark07

• On the FRAPS “99 FPS” tab, designate a location for saving the benchmark results files, such as the Benchmarks folder under the FRAPS folder; set Benchmarking Hotkey to “P” (to coordinate with the FSX Pause keyboard command), check “MinMaxAvg” for file to save, and set “Stop benchmark automatically after...” to 300 seconds (5 minutes). Also, enter None for “Overlay Hotkey,” ensure the upper right corner is selected on the Overlay Corner image, “Hide overlay” is not checked, and “Only update overlay once a second” is not checked.

• Ignore the Movies and Screenshots tabs.

FRAPS interface – “99FPS” tab settings for FSXMark07

FSX Configuration

Start FSX and go to menu Options, Settings, Save, to name and save your current FSX configuration file and its included settings. This allows reloading your saved configuration after benchmarking is completed. Next, go to Options, Settings, and Load the “FSXMark07 - Global High.cfg” file, to standardize 38 parameters in your currently operative FSX.cfg file.

If a widescreen display is being used, it’s best to manually edit the FSX.cfg file to properly display the control panel and outside scenery. To do this, find and modify in the [Display] section, or enter if not there, the parameter line “WideViewAspect=” to “True”, and then save the FSX.cfg file. This change needs to be made when FSX is not running. When users compare results, they should indicate their respective display sizes and resolutions and whether the display is widescreen or not, along with a listing of other principal hardware components.

Experience confirms that adding one more configuration change provides less variation in test  results. On the FSX menu bar, select Options, Settings, Display, Graphics tab, and then check the box for “Preview DX 10.” A warning box will pop up to prompt your restarting FSX after making this setting. With “Preview DX 10” enabled, FSX runs from a “parallel code path” instead of its DX 9 code path. If this change is not selected, the results from each initial test flight after starting FSX will be lower than from subsequent test flights, likely because of a difference scenery caching. Note that you’ll need to restart FSX after selecting Preview DX 10.

Finally, after restarting FSX, ensure that the “Preview DirectX 10” box is still checked in Options, Settings, Display, Graphics tab, and for uniformity with other test results, select Anisotropic filtering and Anti-aliasing, as these latter two settings are not part of the FSXMark07 - Global High.cfg file modifications.

Benchmarking Procedure

After starting FSX, open the FRAPS interface, and check for the yellow framerate display in the upper right-hand corner of the screen, as seen in the image below. Next, click your mouse on the FSX screen to bring FSX into focus, wait about five seconds for the mouse cursor to disappear. If it doesn’t, framerate will be reduced slightly during the test flights. In this case, close and reopen the test flight, after which the mouse cursor should disappear.

To start the test flight, press the “P” key to simultaneously un-pause the benchmark flight and start FRAPS framerate logging. After this action, FRAPS will momentarily display a green box and the yellow framerate display will disappear within a few seconds. Do not touch the mouse or keyboard during the benchmark flight, as doing so will reduce the framerate.

View at beginning of benchmarking flight

Enjoy the scenery during the five-minute flight until you see the framerate display reappear with a momentary red box around the average framerate number. Not to worry if you miss seeing the red box, because FRAPS calculates and stores the framerate results files in the designated Benchmarks folder, the location of which was saved earlier.

View after five minutes of benchmarking flight, with FRAPS red “average” box in upper right

After each test flight is complete and the FRAPS framerate display has reappeared, there’s a quick means of starting another flight. Without hitting the “P” key, reload the flight from the FSX menu, or copy and use a shortcut on your desktop, preferably pinned to the taskbar, to C:\Users\(username)\Documents\Flight Simulator X Files\FSXMark07.flt. This action should reload the flight while keeping FSX and FRAPS coordinated, and the yellow FRAPS display should be visible after reloading.

Benchmarking Results

The accompanying image shows sample results displayed in a table, from a typical “FSX (date and time stamp) minmaxavg.csv” file, which can most simply be viewed in Windows Explorer. Open the Preview pane, and you can view the contents of the *.csv files one at a time without opening the files. If desired, you can copy sets of results to a spreadsheet and plot results in relation to computer and FSX settings. Again, running and averaging at least five trials is recommended for each hardware or settings change. If any test result is not within about 5% of the other results, run at least one more trial, and then take the average of all results.

Sample FRAPS results from one FSX benchmarking run

Cautionary Notes

Occasionally, FSX and FRAPS can become unsynchronized in relation to use of the P key for pausing and un-pausing the sim, and simultaneously starting and stopping FRAPS. This can lead to invalid FRAPS summary files, so if necessary after the flight is reloaded as paused , stop and restart FRAPS such that you see the yellow framerate number while the flight is still paused at the starting point. If you see FRAPS results files with less than 300 seconds shown (less than 300000 milliseconds in the *.csv file), these are from these unsynchronized pauses, and they should be deleted and not included.

Other computer programs and underlying services can affect framerate results significantly. It’s best to run after a fresh reboot of Windows, and with background processes disabled, such as anti-virus, file indexing, Windows updating, and other programs that may cause file reading and writing during benchmarking. Remember that FSX is “CPU-bound” and the objective is to provide full CPU resources for running FSX. Most important for comparison is changing only one hardware component or FSX setting at a time, to keep results “statistically significant.”

Author’s Note: there has been an FSXMark11 procedure floating around flightsim websites for the past two years, which copies the FSXMark07 procedure developed by Gary Dunne, without properly crediting Gary for his work. Operation of FSXMark11 is identical to that of FSXMark07, except for use of a different airplane, test flight location, and a configuration setting in FSMark11 that’s biased toward higher framerates by setting airline and general aviation AI traffic to zero, among other settings. For benchmarking consistency and proper author credit, I recommend staying with FSXMark07.