Review By Doug Horton.
I often see claims in forums by flight simulator users who credit particular changes for framerate performance improvements, which when subjected to statistical testing of framerates, do not hold up. This article describes the framerate benchmarking procedure I’ve been using for several years to sort out performance claims for hardware and configuration options for running FSX and Prepar3D. Flightsim friends have sometimes asked me to comment on new systems builds, and often, I point out that some inexpensive components that will improve FSX performance, while some expensive components will not help FSX run any faster, and I prove these differences by benchmarking FSX framerate performance.
In some cases, users make hardware or settings changes, see a certain instantaneous framerate changes, and then optimistically draw conclusions to justify their hardware investments or settings changes. But the framerate while the sim is paused is a poor measure of performance in flight. And it’s important to standardize all settings, so results are repeatable. Where does the flight take place, what’s the track, and what sceneries are being used? What display options are set? What weather is selected? Which aircraft is being used? Is the 2D control panel or Virtual Cockpit displayed? What percentage AI aircraft traffic is set? This last setting strongly influences FSX performance, by the way. The list of variables is long, and in most cases, standardized settings and test methods are not used to justify the claims from “tweaking” time sim, so the results, in a statistical context, are neither repeatable(same user, same settings and equipment), nor reproducible (different user, same settings, different but identical equipment).
There is also a notion of “causality” in the study of statistics. Causality means that an observed effect or measureable result is due solely to one change in an independent variable, such as one hardware or settings change, with all other factors held constant. Many times, observed effects are coincident, not causal, such as, “It rained yesterday and I won the lottery!” Coincident yes, causal no!
It’s also important to check the data variation to ensure there are no data “outliers,” which might be caused by something happening in another application while benchmarking the sim. For example, if Windows begins to automatically download updates while FSX performance is being measured, results may be affected, and if result is significantly different than other results, the outlying data point could be ignored, particularly if a reason for the difference can be identified.
This is a very simple procedure, and once learned, in about ten minutes, you’ll never again wonder whether a hardware or settings change was responsible for a FSX or Prepar3D performance difference. The descriptions below cover installation and testing for FSX, with the installation for Prepar3D being the same, except for use of corresponding file locations. Also, it will be necessary to copy the Bombardier CRJ 700 files from FSX to Prepar3D.
Case Study
I began serious benchmarking exercises several years ago after I began receiving sample motherboards and processors from Intel. I was curious how a new processor performed, including cautious overclocking of the processor, with use of a more robust cooling fan than furnished with the processor. I’d previously learned that FSX is significantly “CPU-bound” and most hardware changes make no difference. Recognizing this fact can save readers from spending money on hardware upgrades that will provide no performance improvements.
One exception to this statement is investment in new processor, and as necessary, a motherboard and memory to support the new processor. As an example, I previously tested a relatively inexpensive Intel Core i5-2500K processor, which at the time could be purchased for about $200. This processor uses the socket LGA 1155 architecture, and I tested it on an Intel DP67BG motherboard, which at the time was selling for about $170 US. Though the i5-2500K is a four-core processor, it’s not capable of hyper-threading, so it presents only four cores to FSX, and because that’s the most FSX can utilize, it works fine.
Importantly, the i5-2500K processor also includes Intel’s Turbo Boost 2.0 feature, and this feature, with no user action, other than verifying in the BIOS that Turbo Boost is enabled, will boost the i5-2500K processor core frequency from its default value of 3.3 GHz, to as high as 3.7 GHz. Using Intel’s Extreme Tuning Utility, I incrementally overclocked the Turbo Boosted frequency to 4.4 GHz, and using the FSXMark07 benchmarking procedure, I verified linearly proportional increases in FSX framerate performance with increased processor speed, as illustrated in the accompanying graph. Note for statisticians: the graph also displays the spreadsheet-calculated linear regression equation and correlation coefficient, with a nearly perfect fit of the line to the data.
Graph of FSXMark07 framerate test results over range of i5-2500K processor speeds
What follows is a description of the proven method for benchmarking FSX framerate performance with standardized settings, to provide causal and repeatable results. Take the time to read the description, installation instructions, how to run tests, and how to view logged results. Installation can be completed in less than 10 minutes after downloading two files, and once installed, testing is as easy as starting FSX with the saved configuration file and benchmarking flight, confirming settings once per session, running about five, five-minute test flights, calculating the simple average of the five trials, and then reviewing results to ensure results make sense.
Benchmarking Procedure
The best means of accurately benchmarking FSX framerate performance is using the free FSXMark07 package, developed by flightsim user Gary Dunne. The file FSXMark07.zip, available in the AVSIM library, includes the necessary FSX files and simple instructions, which are summarized below. The free FRAPS utility is used for framerate tracking and logging. With FSX, standardized testing with FSXMark07 means:
Only one hardware component or configuration setting is changed at a time, and a standard configuration for display settings is provided by loading a specified *.cfg file, such as the “FSXMark07 - Global High.cfg” file to standardize 38 parameters in the user’s “FSX.cfg file,” and then making a few other changes, as indicated below.
If scenery, terrain, land class, or airport and facilities data (AFCAD files), or other files are added to FSX in the Scenery Library, they should be disabled for the test area, which with FSXMark07 is the greater Seattle, Washington, USA area.
A selected default aircraft flies a set route, on autopilot, on auto-throttle, at a set altitude and airspeed, with other conditions (or “states”) set in the included FSXMark07.flt (flight), FSXMark97.fssave (states), FSXMark07*.pln (flight plan), and FSXMark07.wx (weather) files.
FRAPS software is used to track and log framerate data for five minutes for each test run.
The FSXMark07 test flight begins paused, in the default Bombardier CRJ 700, north of downtown Seattle, and after un-pausing FSX, the flight continues toward the downtown area, turns to fly over Boeing Field and SeaTac airports, and it then heads toward the Auburn airport, with Mt. Rainier visible in the distance.
The flight provides a great variety of scenery, terrain, and clouds, with a modest amount of AI aircraft, auto, and boat traffic. The CRJ 700 flies at 1500 feet MSL and 250 knots, as FRAPS counts individual frames over the five-minute period. The end of the five minutes is signaled by the FRAPS framerate box reappearing on the screen. The test flight is then repeated about four more times, to ensure results are consistent.
After the recommended five test flights, the simple average of the FRAPS-recorded framerates is calculated. Pinning the file FSXMark07.flt to the Windows taskbar simplifies reloading the flight and resetting FRAPS for the additional trials.
Installation of FSXMark07 Files and FRAPS
Download FSXMark07.zip from AVSIM, and download FRAPS. Read the FSXMark07 instructions, which also cover FRAPS installation, as summarized below.
For the FSXMark07 procedure installation, extract the FSXMark07.flt, .fssave, .plan, .wx, and files to the C:\Users\(username)\Documents (or My Documents)\Flight Simulator X Files\ folder. Similarly, extract the FSXMark07 - Global High.cfg file to the same location. Note: though an FSXMark07 - Max.CFG file is provided, it is not used, because its settings are too aggressive.
FRAPS installation is from its executable file, and after installing, run FRAPS. On its setting interface, ensure the following settings are made:
On the General tab, check “Start Fraps minimized,” “Fraps window always on top,” and “Minimize to system tray only.”
FRAPS interface – General tab settings for FSXMark07
On the FRAPS “99 FPS” tab, designate a location for saving the benchmark results files, such as the Benchmarks folder under the FRAPS folder; set Benchmarking Hotkey to “P” (to coordinate with the FSX Pause keyboard command), check “MinMaxAvg” for file to save, and set “Stop benchmark automatically after...” to 300 seconds (5 minutes). Also, enter None for “Overlay Hotkey,” ensure the upper right corner is selected on the Overlay Corner image, “Hide overlay” is not checked, and “Only update overlay once a second” is not checked.
Ignore the Movies and Screenshots tabs.
FRAPS interface – “99FPS” tab settings for FSXMark07
FSX Configuration
Start FSX and go to menu Options, Settings, Save, to name and save your current FSX configuration file and its included settings. This allows reloading your saved configuration after benchmarking is completed. Next, go to Options, Settings, and Load the “FSXMark07 - Global High.cfg” file, to standardize 38 parameters in your currently operative FSX.cfg file.
If a widescreen display is being used, it’s best to manually edit the FSX.cfg file to properly display the control panel and outside scenery. To do this, find and modify in the [Display] section, or enter if not there, the parameter line “WideViewAspect=” to “True”, and then save the FSX.cfg file. This change needs to be made when FSX is not running. When users compare results, they should indicate their respective display sizes and resolutions and whether the display is widescreen or not, along with a listing of other principal hardware components.
Experience confirms that adding one more configuration change provides less variation in test results. On the FSX menu bar, select Options, Settings, Display, Graphics tab, and then check the box for “Preview DX 10.” A warning box will pop up to prompt your restarting FSX after making this setting. With “Preview DX 10” enabled, FSX runs from a “parallel code path” instead of its DX 9 code path. If this change is not selected, the results from each initial test flight after starting FSX will be lower than from subsequent test flights, likely because of a difference scenery caching. Note that you’ll need to restart FSX after selecting Preview DX 10.
Finally, after restarting FSX, ensure that the “Preview DirectX 10” box is still checked in Options, Settings, Display, Graphics tab, and for uniformity with other test results, select Anisotropic filtering and Anti-aliasing, as these latter two settings are not part of the FSXMark07 - Global High.cfg file modifications.
Benchmarking Procedure
After starting FSX, open the FRAPS interface, and check for the yellow framerate display in the upper right-hand corner of the screen, as seen in the image below. Next, click your mouse on the FSX screen to bring FSX into focus, wait about five seconds for the mouse cursor to disappear. If it doesn’t, framerate will be reduced slightly during the test flights. In this case, close and reopen the test flight, after which the mouse cursor should disappear.
To start the test flight, press the “P” key to simultaneously un-pause the benchmark flight and start FRAPS framerate logging. After this action, FRAPS will momentarily display a green box and the yellow framerate display will disappear within a few seconds. Do not touch the mouse or keyboard during the benchmark flight, as doing so will reduce the framerate.
View at beginning of benchmarking flight
Enjoy the scenery during the five-minute flight until you see the framerate display reappear with a momentary red box around the average framerate number. Not to worry if you miss seeing the red box, because FRAPS calculates and stores the framerate results files in the designated Benchmarks folder, the location of which was saved earlier.
View after five minutes of benchmarking flight, with FRAPS red “average” box in upper right
After each test flight is complete and the FRAPS framerate display has reappeared, there’s a quick means of starting another flight. Without hitting the “P” key, reload the flight from the FSX menu, or copy and use a shortcut on your desktop, preferably pinned to the taskbar, to C:\Users\(username)\Documents\Flight Simulator X Files\FSXMark07.flt. This action should reload the flight while keeping FSX and FRAPS coordinated, and the yellow FRAPS display should be visible after reloading.
Benchmarking Results
The accompanying image shows sample results displayed in a table, from a typical “FSX (date and time stamp) minmaxavg.csv” file, which can most simply be viewed in Windows Explorer. Open the Preview pane, and you can view the contents of the *.csv files one at a time without opening the files. If desired, you can copy sets of results to a spreadsheet and plot results in relation to computer and FSX settings. Again, running and averaging at least five trials is recommended for each hardware or settings change. If any test result is not within about 5% of the other results, run at least one more trial, and then take the average of all results.
Sample FRAPS results from one FSX benchmarking run
Cautionary Notes
Occasionally, FSX and FRAPS can become unsynchronized in relation to use of the P key for pausing and un-pausing the sim, and simultaneously starting and stopping FRAPS. This can lead to invalid FRAPS summary files, so if necessary after the flight is reloaded as paused , stop and restart FRAPS such that you see the yellow framerate number while the flight is still paused at the starting point. If you see FRAPS results files with less than 300 seconds shown (less than 300000 milliseconds in the *.csv file), these are from these unsynchronized pauses, and they should be deleted and not included.
Other computer programs and underlying services can affect framerate results significantly. It’s best to run after a fresh reboot of Windows, and with background processes disabled, such as anti-virus, file indexing, Windows updating, and other programs that may cause file reading and writing during benchmarking. Remember that FSX is “CPU-bound” and the objective is to provide full CPU resources for running FSX. Most important for comparison is changing only one hardware component or FSX setting at a time, to keep results “statistically significant.”
Author’s Note: there has been an FSXMark11 procedure floating around flightsim websites for the past two years, which copies the FSXMark07 procedure developed by Gary Dunne, without properly crediting Gary for his work. Operation of FSXMark11 is identical to that of FSXMark07, except for use of a different airplane, test flight location, and a configuration setting in FSMark11 that’s biased toward higher framerates by setting airline and general aviation AI traffic to zero, among other settings. For benchmarking consistency and proper author credit, I recommend staying with FSXMark07.
Review by Doug Horton. Real Pilots wear headsets, to improve communications and reduce engine and other ambient noise, so why shouldn’t flight simulator users wear headsets? Corsair’s Vengeance 1500 USB gaming headset makes this a pleasure, with a professional quality headset that offers style, comfort, and most importantly, great sound reproduction. The product is also provided with software that offers many listening options, including an equalizer and synthetic 5.1 and 7.1 sound fields. According to Corsair marketing: “If you have on-board sound, the Vengeance 1500 provides easy one-cable USB connection and Dolby Headphone technology that lets you enjoy multichannel games the way they were meant to be played. Dolby Headphone is compatible with any game with 3D audio, and helps reduce ‘headphone fatigue’ while delivering the accurate 360-degree audio imaging you need if you're playing to win.” We’ll learn more about these features as we proceed, but first, look at the impressive design. Profile view of Corsair Vengeance 1500 USB gaming headset’s impressive design Headset Features Large Drivers Large 50mm drivers (speaker) are provided to greatly reduce the overloading and distortion that can occur with gaming headsets that typically use smaller 40mm drivers. This feature is also intended to achieve more accurate sound reproduction, particularly for the low-frequency effects that are more difficult to produce with small speakers, compared to a sub-woofer, for example. Corsair also suggests that use of large speaker drivers also provides a competitive advantage. Comfort The extra-wide, extra deep “circumaural” ear cups isolate sounds outside the headset, and they feel better than supra-aural (on-ear) headsets. The ear pads are constructed with memory foam, with microfiber covers, to provide “a comfortable and reliable fit that conforms to the shape of your head without binding or pinching,” according to Corsair. Microphone The Vengeance 1500 headset is furnished with a unidirectional, noise-cancelling microphone, which is tuned to capture the user’s voice loudly and clearly, while filtering out distracting background noises. Also, the microphone is radially hinged, so it can be moved upward or downward as desired. Headset Controls The Vengeance 1500 headset is provided with a customary in-line controller, which is a bit larger than my previous headset’s controller. Corsair suggests that it’s large so that it’s “easy to grab without taking your eyes off the action,” and I support that claim. The control includes a silver-colored elongated oval Mute button in the central area of the controller, and it has large minus and plus buttons for Volume, which are highlighted with blue lighted circles around the buttons. However, if the Mute button is pressed, the lighted circles around the two volume buttons turn from blue to red. Overall, the headset controller is just right – slightly larger than others, which makes it easier to use during game play, and it’s nicely designed and lighted, making it easy to see and use. Vengeance 1500 headset controller, with small Mute button and lighted minus and plus Volume buttons Installation When you first plug the Vengeance 1500 headset into a USB socket, Windows will search for a generic driver. With my up to date Windows 7 installation on two computers, Windows found a driver and properly identified the headset, so that it appeared in the Flight Simulator X menu for Options, Settings, Sound drop-down list of devices. Within five minutes of plugging in the headset, I was able to configure and verify operation of the headset in FSX to either play all sounds on the headset, or to play voice, such as ATC dialog, on the headset, while playing all other sounds on my speakers. FSX and Prepar3D Setup The most realistic use of a USB gaming headset of any name or brand, for flight simulation, is to set up outside sounds on speakers, and voice on the USB headset. This is easy to do, but sometimes the settings on separate Windows and flight simulator sound interfaces make this require an extra step. Regardless of the order in which the steps are taken the end point should result in coordinated settings as follows: The 1500 headset should be plugged into a USB connector and the headset must be turned on. In Windows, with Windows 7 Home Premium illustrated in this example: Headset speakers should be enabled as the “Default Communications Device.” System speakers, whether powered by motherboard sound module or add-in sound card, should be enabled as “Default Device.” Note that my Windows 7 installation also sees a digital sound option from the motherboard’s Realtek sound module, which is shown as Ready, as it’s not in use. Windows 7 sound settings to provide for voice from headset and other sounds from speakers powered by motherboard sound module or sound card Start FSX (or Prepar3D) On the main menu, click Options, Settings, Sound interface, and on the two drop-down lists, set Sounds to Speakers (Realtek High Definition, in example), and set Voice to Speakers (Corsair Vengeance 1500). Note slider for Voice, in case you need to adjust it in comparison to Engines, Cockpit, and Environmental sounds FSX sound settings interface, set to provide voice from headset and other sounds from speakers powered by motherboard sound module Prepar3DX sound settings interface, set to provide voice from headset and other sounds from speakers powered by motherboard sound module Documentation and Software The package included a small rolled up Warranty Guide, and at first glance, I couldn’t find a manual. At second glance, I found rolled up with the Warranty Guide a 5” square sheet of yellow paper, which contains brief instructions in six languages. Here is the English text: “Go to www.corsair.com/vengeance1500 to download the software. “ (Note: the download link has changed, and at time of writing, this is the link for downloading the driver and software.) PLEASE DISABLE YOUR ANTI-virus SOFTWARE Anti-virus software might incorrectly indicate that this software installation contains a virus or malware. To help ensure trouble-free installation, please temporarily disable your anti-virus software before installing the software included with this product. Remember to re-enable your anti-virus software after installation of the Corsair software is complete.” Well, that’s a short manual, but it was extremely good advice to download the software, because after downloading and running the driver file, I unplugged and re-plugged the headset’s USB connection, and I was pleasantly surprised to see the Corsair USB Headset Control Panel user interface displayed. Corsair USB Headset Control Panel, showing adjustable sliders for main and equalizer sounds on the left side, and settings for the Dolby Headphone sound environment on the right side The Headset Control Panel UI contains four sections: Upper left: three sliders for sound Volume, MIC Volume, MIC loopback, and Balance Lower left: equalizer profile selection, new profile addition option, and sliders for the 10-channel equalizer Upper right: selection buttons for Bypass, Dolby Headphone 5.1, and 7.1 Virtual Speaker Shifter Lower right: Environment Size icon, with virtual speaker adjustment box, and three buttons for selecting preset Studio, Cinema, and Hall sound environments Corsair USB Headset Control Panel, with settings for the 7.1 sound environment on the right side I was hopeful that pressing the “i”, “?”, or “globe” links in the far upper right corner of the UI would open a help document or instruction manual, but no luck with that wish. Fortunately, answers to my question were available in the Reviewers Guide I was furnished. For example, what do the three sound mode buttons do, according to the Reviewers Guide? Bypass: provides stereo mode with no HRF processing (I looked up HRF - Harmonic Reject Filter – suppresses electric network frequency, phone beeps, or any other drones) Dolby Headphone: provides 5.1-channel surround sound processing, and provides a more spacious sound that reduces ear fatigue (the feeling that the sound is “inside your head”) 7.1 Virtual Speaker Shifter: uses the on-board USB sound processor to provide virtual 7.1 surround sound, and the ability to ‘shift’ virtual speaker positions to fine-tune the sound. Pressing this button opens another UI, which allows the user to increase or decrease the level of each individual channel, rotate the sound field, and reposition any of the virtual speakers. The Virtual Speaker Shifter allows several individual adjustments. After writing this review, I recommended to Corsair that they re-title the Reviewers Guide as Users Guide, edit as appropriate, and provide a link on the 1500 headset product page for downloading this guide. Specifications Here are the specifications for the Vengeance 1500 USB Gaming Headset: Headphones Frequency Response: 20Hz to 20kHz Impedance: 32 Ohms @ 1kHz Dynamic Range: 95dB (A-weighted) Drivers: 50mm Cable Length: 3m USB power consumption: 250mW Connector: USB Type A Microphone Type: Unidirectional noise-cancelling condenser with adjustable, rotating boom Impedance: 2.2k Ohms Frequency Response: 100Hz to 10kHz Frequency Response: 20Hz to 20kHz Impedance: 32 Ohms @ 1kHz Dynamic Range: 95dB (A-weighted) Drivers: 50mm Cable Length: 3m USB power consumption: 250mW Summary There’s lot to like about the Corsair Vengeance 1500 USB headset product: professional quality; impressive design and construction; great fit and feel; versatile software interface; and quick adaptation to FSX (or Prepar3D) through Options, Settings, Sound menu. The fit and comfort are particularly nice, and I wish the expensive noise-cancelling headset I wear for real flying were as comfortable and provided with the elaborate sound adjustment software. The Vengeance 1500 headset is significantly better than my older on-ear USB headset from another manufacturer. The older headset was once a popular product, but it was of the “on-ear” design category, compared to the “over-ear” design of the 1500 headset. The older headset quickly went into “probably won’t ever use again” storage, where it will never squeeze my ears again! Corsair has clearly developed a premium headset, with great concern for comfort and reproducing sounds that are true to the source material. The headset and it easily adjustable software also help with the need for sound reproduction accuracy, detailed positioning, and sound awareness that lets users “play their best,” according to Corsair. The only shortcoming of this product is the lack of an included printed manual, or at least a downloadable manual. As I mentioned earlier, Corsair could rename the excellent Reviewers Guide to become the User’s Guide, and then setting it up for downloading from the headset’s product page of the Corsair website. Note: Corsair has now released a wireless USB headset, named the Vengeance® 2000 Wireless 7.1 Gaming Headset, with rechargeable battery. It sells for about 50% more than the wired 1500 model tested for this review. The Vengeance 2000 headset will be the subject of separate review.
Review By Doug Horton. I often see claims in forums by flight simulator users who credit particular changes for framerate performance improvements, which when subjected to statistical testing of framerates, do not hold up. This article describes the framerate benchmarking procedure I’ve been using for several years to sort out performance claims for hardware and configuration options for running FSX and Prepar3D. Flightsim friends have sometimes asked me to comment on new systems builds, and often, I point out that some inexpensive components that will improve FSX performance, while some expensive components will not help FSX run any faster, and I prove these differences by benchmarking FSX framerate performance. In some cases, users make hardware or settings changes, see a certain instantaneous framerate changes, and then optimistically draw conclusions to justify their hardware investments or settings changes. But the framerate while the sim is paused is a poor measure of performance in flight. And it’s important to standardize all settings, so results are repeatable. Where does the flight take place, what’s the track, and what sceneries are being used? What display options are set? What weather is selected? Which aircraft is being used? Is the 2D control panel or Virtual Cockpit displayed? What percentage AI aircraft traffic is set? This last setting strongly influences FSX performance, by the way. The list of variables is long, and in most cases, standardized settings and test methods are not used to justify the claims from “tweaking” time sim, so the results, in a statistical context, are neither repeatable(same user, same settings and equipment), nor reproducible (different user, same settings, different but identical equipment). There is also a notion of “causality” in the study of statistics. Causality means that an observed effect or measureable result is due solely to one change in an independent variable, such as one hardware or settings change, with all other factors held constant. Many times, observed effects are coincident, not causal, such as, “It rained yesterday and I won the lottery!” Coincident yes, causal no! It’s also important to check the data variation to ensure there are no data “outliers,” which might be caused by something happening in another application while benchmarking the sim. For example, if Windows begins to automatically download updates while FSX performance is being measured, results may be affected, and if result is significantly different than other results, the outlying data point could be ignored, particularly if a reason for the difference can be identified. This is a very simple procedure, and once learned, in about ten minutes, you’ll never again wonder whether a hardware or settings change was responsible for a FSX or Prepar3D performance difference. The descriptions below cover installation and testing for FSX, with the installation for Prepar3D being the same, except for use of corresponding file locations. Also, it will be necessary to copy the Bombardier CRJ 700 files from FSX to Prepar3D. Case Study I began serious benchmarking exercises several years ago after I began receiving sample motherboards and processors from Intel. I was curious how a new processor performed, including cautious overclocking of the processor, with use of a more robust cooling fan than furnished with the processor. I’d previously learned that FSX is significantly “CPU-bound” and most hardware changes make no difference. Recognizing this fact can save readers from spending money on hardware upgrades that will provide no performance improvements. One exception to this statement is investment in new processor, and as necessary, a motherboard and memory to support the new processor. As an example, I previously tested a relatively inexpensive Intel Core i5-2500K processor, which at the time could be purchased for about $200. This processor uses the socket LGA 1155 architecture, and I tested it on an Intel DP67BG motherboard, which at the time was selling for about $170 US. Though the i5-2500K is a four-core processor, it’s not capable of hyper-threading, so it presents only four cores to FSX, and because that’s the most FSX can utilize, it works fine. Importantly, the i5-2500K processor also includes Intel’s Turbo Boost 2.0 feature, and this feature, with no user action, other than verifying in the BIOS that Turbo Boost is enabled, will boost the i5-2500K processor core frequency from its default value of 3.3 GHz, to as high as 3.7 GHz. Using Intel’s Extreme Tuning Utility, I incrementally overclocked the Turbo Boosted frequency to 4.4 GHz, and using the FSXMark07 benchmarking procedure, I verified linearly proportional increases in FSX framerate performance with increased processor speed, as illustrated in the accompanying graph. Note for statisticians: the graph also displays the spreadsheet-calculated linear regression equation and correlation coefficient, with a nearly perfect fit of the line to the data. Graph of FSXMark07 framerate test results over range of i5-2500K processor speeds What follows is a description of the proven method for benchmarking FSX framerate performance with standardized settings, to provide causal and repeatable results. Take the time to read the description, installation instructions, how to run tests, and how to view logged results. Installation can be completed in less than 10 minutes after downloading two files, and once installed, testing is as easy as starting FSX with the saved configuration file and benchmarking flight, confirming settings once per session, running about five, five-minute test flights, calculating the simple average of the five trials, and then reviewing results to ensure results make sense. Benchmarking Procedure The best means of accurately benchmarking FSX framerate performance is using the free FSXMark07 package, developed by flightsim user Gary Dunne. The file FSXMark07.zip, available in the AVSIM library, includes the necessary FSX files and simple instructions, which are summarized below. The free FRAPS utility is used for framerate tracking and logging. With FSX, standardized testing with FSXMark07 means: Only one hardware component or configuration setting is changed at a time, and a standard configuration for display settings is provided by loading a specified *.cfg file, such as the “FSXMark07 - Global High.cfg” file to standardize 38 parameters in the user’s “FSX.cfg file,” and then making a few other changes, as indicated below. If scenery, terrain, land class, or airport and facilities data (AFCAD files), or other files are added to FSX in the Scenery Library, they should be disabled for the test area, which with FSXMark07 is the greater Seattle, Washington, USA area. A selected default aircraft flies a set route, on autopilot, on auto-throttle, at a set altitude and airspeed, with other conditions (or “states”) set in the included FSXMark07.flt (flight), FSXMark97.fssave (states), FSXMark07*.pln (flight plan), and FSXMark07.wx (weather) files. FRAPS software is used to track and log framerate data for five minutes for each test run. The FSXMark07 test flight begins paused, in the default Bombardier CRJ 700, north of downtown Seattle, and after un-pausing FSX, the flight continues toward the downtown area, turns to fly over Boeing Field and SeaTac airports, and it then heads toward the Auburn airport, with Mt. Rainier visible in the distance. The flight provides a great variety of scenery, terrain, and clouds, with a modest amount of AI aircraft, auto, and boat traffic. The CRJ 700 flies at 1500 feet MSL and 250 knots, as FRAPS counts individual frames over the five-minute period. The end of the five minutes is signaled by the FRAPS framerate box reappearing on the screen. The test flight is then repeated about four more times, to ensure results are consistent. After the recommended five test flights, the simple average of the FRAPS-recorded framerates is calculated. Pinning the file FSXMark07.flt to the Windows taskbar simplifies reloading the flight and resetting FRAPS for the additional trials. Installation of FSXMark07 Files and FRAPS Download FSXMark07.zip from AVSIM, and download FRAPS. Read the FSXMark07 instructions, which also cover FRAPS installation, as summarized below. For the FSXMark07 procedure installation, extract the FSXMark07.flt, .fssave, .plan, .wx, and files to the C:\Users\(username)\Documents (or My Documents)\Flight Simulator X Files\ folder. Similarly, extract the FSXMark07 - Global High.cfg file to the same location. Note: though an FSXMark07 - Max.CFG file is provided, it is not used, because its settings are too aggressive. FRAPS installation is from its executable file, and after installing, run FRAPS. On its setting interface, ensure the following settings are made: On the General tab, check “Start Fraps minimized,” “Fraps window always on top,” and “Minimize to system tray only.” FRAPS interface – General tab settings for FSXMark07 On the FRAPS “99 FPS” tab, designate a location for saving the benchmark results files, such as the Benchmarks folder under the FRAPS folder; set Benchmarking Hotkey to “P” (to coordinate with the FSX Pause keyboard command), check “MinMaxAvg” for file to save, and set “Stop benchmark automatically after...” to 300 seconds (5 minutes). Also, enter None for “Overlay Hotkey,” ensure the upper right corner is selected on the Overlay Corner image, “Hide overlay” is not checked, and “Only update overlay once a second” is not checked. Ignore the Movies and Screenshots tabs. FRAPS interface – “99FPS” tab settings for FSXMark07 FSX Configuration Start FSX and go to menu Options, Settings, Save, to name and save your current FSX configuration file and its included settings. This allows reloading your saved configuration after benchmarking is completed. Next, go to Options, Settings, and Load the “FSXMark07 - Global High.cfg” file, to standardize 38 parameters in your currently operative FSX.cfg file. If a widescreen display is being used, it’s best to manually edit the FSX.cfg file to properly display the control panel and outside scenery. To do this, find and modify in the [Display] section, or enter if not there, the parameter line “WideViewAspect=” to “True”, and then save the FSX.cfg file. This change needs to be made when FSX is not running. When users compare results, they should indicate their respective display sizes and resolutions and whether the display is widescreen or not, along with a listing of other principal hardware components. Experience confirms that adding one more configuration change provides less variation in test results. On the FSX menu bar, select Options, Settings, Display, Graphics tab, and then check the box for “Preview DX 10.” A warning box will pop up to prompt your restarting FSX after making this setting. With “Preview DX 10” enabled, FSX runs from a “parallel code path” instead of its DX 9 code path. If this change is not selected, the results from each initial test flight after starting FSX will be lower than from subsequent test flights, likely because of a difference scenery caching. Note that you’ll need to restart FSX after selecting Preview DX 10. Finally, after restarting FSX, ensure that the “Preview DirectX 10” box is still checked in Options, Settings, Display, Graphics tab, and for uniformity with other test results, select Anisotropic filtering and Anti-aliasing, as these latter two settings are not part of the FSXMark07 - Global High.cfg file modifications. Benchmarking Procedure After starting FSX, open the FRAPS interface, and check for the yellow framerate display in the upper right-hand corner of the screen, as seen in the image below. Next, click your mouse on the FSX screen to bring FSX into focus, wait about five seconds for the mouse cursor to disappear. If it doesn’t, framerate will be reduced slightly during the test flights. In this case, close and reopen the test flight, after which the mouse cursor should disappear. To start the test flight, press the “P” key to simultaneously un-pause the benchmark flight and start FRAPS framerate logging. After this action, FRAPS will momentarily display a green box and the yellow framerate display will disappear within a few seconds. Do not touch the mouse or keyboard during the benchmark flight, as doing so will reduce the framerate. View at beginning of benchmarking flight Enjoy the scenery during the five-minute flight until you see the framerate display reappear with a momentary red box around the average framerate number. Not to worry if you miss seeing the red box, because FRAPS calculates and stores the framerate results files in the designated Benchmarks folder, the location of which was saved earlier. View after five minutes of benchmarking flight, with FRAPS red “average” box in upper right After each test flight is complete and the FRAPS framerate display has reappeared, there’s a quick means of starting another flight. Without hitting the “P” key, reload the flight from the FSX menu, or copy and use a shortcut on your desktop, preferably pinned to the taskbar, to C:\Users\(username)\Documents\Flight Simulator X Files\FSXMark07.flt. This action should reload the flight while keeping FSX and FRAPS coordinated, and the yellow FRAPS display should be visible after reloading. Benchmarking Results The accompanying image shows sample results displayed in a table, from a typical “FSX (date and time stamp) minmaxavg.csv” file, which can most simply be viewed in Windows Explorer. Open the Preview pane, and you can view the contents of the *.csv files one at a time without opening the files. If desired, you can copy sets of results to a spreadsheet and plot results in relation to computer and FSX settings. Again, running and averaging at least five trials is recommended for each hardware or settings change. If any test result is not within about 5% of the other results, run at least one more trial, and then take the average of all results. Sample FRAPS results from one FSX benchmarking run Cautionary Notes Occasionally, FSX and FRAPS can become unsynchronized in relation to use of the P key for pausing and un-pausing the sim, and simultaneously starting and stopping FRAPS. This can lead to invalid FRAPS summary files, so if necessary after the flight is reloaded as paused , stop and restart FRAPS such that you see the yellow framerate number while the flight is still paused at the starting point. If you see FRAPS results files with less than 300 seconds shown (less than 300000 milliseconds in the *.csv file), these are from these unsynchronized pauses, and they should be deleted and not included. Other computer programs and underlying services can affect framerate results significantly. It’s best to run after a fresh reboot of Windows, and with background processes disabled, such as anti-virus, file indexing, Windows updating, and other programs that may cause file reading and writing during benchmarking. Remember that FSX is “CPU-bound” and the objective is to provide full CPU resources for running FSX. Most important for comparison is changing only one hardware component or FSX setting at a time, to keep results “statistically significant.” Author’s Note: there has been an FSXMark11 procedure floating around flightsim websites for the past two years, which copies the FSXMark07 procedure developed by Gary Dunne, without properly crediting Gary for his work. Operation of FSXMark11 is identical to that of FSXMark07, except for use of a different airplane, test flight location, and a configuration setting in FSMark11 that’s biased toward higher framerates by setting airline and general aviation AI traffic to zero, among other settings. For benchmarking consistency and proper author credit, I recommend staying with FSXMark07.
