Contemporary thoughts on the "out of memory" error

[Bob Scott 7,715]

Running FS9 with a recently-built cutting edge PC, I started seeing the dreaded Out of Memory error in FS9. My new PC has 2GB of RAM, and something like 85GB free on the drive FS runs on. How in the world could this be???Well, a thorough sweep of the forums revealed that I was not alone. The majority of recent complaints of this problem are from people running relatively strong PCs, but with complex add-ons like the PMDG or Level-D, lots of scenery, traffic etc. Note that this is not a sudden CTD, but an error window that says your application has run out of memory, advising you to try freeing up disk space and run it again, even though you still have plenty of RAM and HD space. The most common time to see this error is on approach.So I began the troubleshooting process. I found a scenario that produced an OOM virtually every time...a PMDG B744 flight from SimFlyers KDFW to SimFlyers KIAD, bad wx saved in ActiveSky v6.5, and Radar Contact controlling the way. Traffic was set at 34% from MyTraffic.I started by monitoring the RAM usage. I note from reading through the forums that many people do not understand what they are looking at w/r/t RAM usage. The numbers reported in task manager only reflect the working set, or the RAM actually loaded in memory at a given time. The best metric to use is the "private bytes" as reported by the WinXP performance tool...this shows how much memory has been allocated to the program, including memory swapped out to disk or no longer reachable due to a memory leak. My memory allocation was very high...the perf monitor reported nearly 1000 MB at the beginning of the flight, and on approach just before the OOM error, nearly 1400 MB. The allocation was stable, with normal ups/downs due to weather/traffic throughout the ~2 hr flight. But as I approached the airport, allocations climbed steadily, and at the point when the airport scenery and all the traffic moving around the airport started to load...BAM, OOM error.It's important to note that an OOM error occurs anytime a program attempts to allocate more than the 2GB max allowed for any Windows process. Doesn't matter what else is running on the PC...each process has its own 2GB virtual address space to stay within. It also doesn't matter how much actual RAM is in use...all that's needed is a request for more memory than WinXP can allocate. Seems to me that this sudden deluge of scenery and AI planes causes FS to ask WinXP for more RAM than it's allowed to have, causing the OOM error.So I tried again a few times, this time with the AI slider at 0. RAM usage went down...in fact I only saw one OOM in four tries configured this way. But memory allocation was still near 1000MB approaching the field.Something I saw in another post got my attention as well. Every .DLL file in the FS9 modules folder gets loaded with FS9. Every one. I ran WinDebug against FS9, and sure enough, every DLL in the folder was loaded and had an address range allocated to it. I had DLLs from the Flight1 Cessna 421 and Piper Meridian, Level-D, Lago's VimaCore, ActiveRadar etc, all loaded up and taking a bite out of a limited 2GB sandwich. If it's never called, it is paged almost immediately out and won't occupy RAM again...but it *is* still allocated and blocking available address space. Those watching the Task Manager will never see this, because once it's paged out, it's out of the working set reported by Task Manager--but not out of the picture.So I cleaned out my modules folder of everything I didn't need to run the PMDG, Radar Contact, or my PFC flight controls. My RAM usage at start was now under 800 MB...approaching the field RAM usage was 75-100MB less than before, and I have not had an OOM since. This tells me I need to manage my modules folder on a per-flight basis, and this weekend I'm writing some batch files to move the right files in for each add-on I fly. No reason to have the PMDGOptions.DLL loaded when flying the Level-D, or to ever have ActiveRadar.dll since I don't use it. There were 15 DLLs there that I didn't need during the first test flights.Last, I did a lot of experimenting with the /3GB option in WinXP Pro. After a lot of reading and experimenting, I'm still a bit puzzled...this option clearly results in some beneficial changes in disk caching behavior on my 2GB machine. Not sure why. But FS9 does *not* have the /LARGEADDRESSAWARE flag set in the executable's header, so even with /3GB specified, FS9 only gets the standard 2GB of address space from WinXP. I'm going to try making a change to the FS9 executable flag, and see if FS9, by some chance, is large address compliant, and can thus benefit from having an extra gig of allowable allocations. But that is complicated by the fact that every DLL loaded with FS9 will have to be compliant as well...I don't expect the results will be universally good.So that's it. My little odyssey, and maybe a cure for a few of those seeing this frustrating error on an otherwise powerful system.CheersBob ScottATP IMEL Gulfstream II-III-IV-VSantiago de Chile

[GHarrall 494]

Bob,I was going to tell you about a thread on the UT forum over at Flight 1 discussing the very same thing, but I see you are there already!Glenn

[Rottenlungs 0]

Very interesting information Bob. Thanks for making the effort to share these findings. I haven`t encountered the OOM error you mention but I am very keen to reduce unwanted load. Creating module 'profiles' for each flight scenario is a very good idea. Thank again James

[mgh 89]

A very interesting and informative post. It confirms what I've long thought that many, if not most, problems reported here (such as OOM etc) are caused by add-ons and not by FS9 itself, especially if there are lots of of add-ons.Maybe we'll see an end of posts blaming FS9 for these unwanted side-effects?

[Guest TomOOO]

Just a note on the 2Gb limit. This is because of the maximumaddressable size for a 32 bit OS architecture, ie all versionsof win except 64bit XP. There are complex ways about this, butFS will not be using this, and for all current programs thereis absolutely no way a 32bit Architecture OS can allow a singleprocess to obtain more than 2.147 bytes of memory. I therefore doubt you will get more memory.The problem you are getting with the addon may be the faultof FS2004. It may be down to a memory leak that has been foundwith the way the addon has been implemented. This will be the faultof the underlying program and not the addon. Of course, there mayjust be a too complex view - and this will be the fault of theaddon trying to do too much which requires too much memory.Just my 2pence worth.RegardsTom

[mgh 89]

>The problem you are getting with the addon may be the fault>of FS2004. It may be down to a memory leak that has been found>with the way the addon has been implemented. This will be the fault>of the underlying program and not the addon.I don't agree with that view. If FS9 doesn't cause a memory leak when used by itself then if a memory leak occurs with an add-on it surely must be due to the add-on. MS can't be expected to have designed FS to cope with the myriad unknown ways in which an add-on might be implemented.

[Slick9 24]

I've been having the same problem for the last few weeks with the PMDG 744F and it is really frustrating. Thanks for this post W6KD very informative. My OOM CTDs occur on short final but only during the night, I can land at any US airport during the day, but night landings are a different story. I finally deleted my FS9 config and let FS rebuild it and the problem seemed to go away on a flight from KBOS to KLAX, i landed swtiched views over and over with no problems. I then did a flight from KLAX to KJFK and i was able to land but after parking as soon as i tried to swith to outside view, OOM. The strange thing though is i can take off from any of the airports at night fly around and return to the same airport with no issues, as soon as the flight takes more than 3 hours the problem surfaces. Now my question is: is there a way to identify which products use which modules, without trial and error? The addons that i use are: (i don't fly on-line)PMDG 747-400 (Pass. & Freight)EagleSoft Citation X Carenado Cessna 177Radar Contact (thru wide FS I don't use the radar)AS6.5 (thru wide FS)Ultimate TrafficFS HotFXTrack IR 3.0Game Commander 2.0Walk & Follow DBSThanks for your help!!!

[Bob Scott 7,715]

If you mouse over the various DLL files in the FS9 modules directory, you can see (in most cases) who the author is. If it's Microsoft, the DLL needs to stay put. I created a folder called Module Locker that has all the non-standard DLLs that are not to be universally loaded (all the MS ones, plus FSUIPC.dll and pfc.dll). The I wrote some batch files for each add-on that has a unique dll (i.e. C421NJ6.dll for the F1 Cessna 421) and moves it to the locker when I want to run that add-on.In most cases it's fairly clear which ones need loaded...for the PMDG you need PMDGOptions.dllBut...some experimenting may be required, especially if you have modules that don't have header info that identifies what they're for or who wrote them.The PMDG 744 was the worst offender for me. With good management of these extraneous .dll files I haven't had any more problems with OOM.RegardsBob ScottATP IMEL Gulfstream II-III-IV-VSantiago de Chile

[Bob Scott 7,715]

>Just a note on the 2Gb limit. This is because of the>maximum>addressable size for a 32 bit OS architecture, ie all>versions>of win except 64bit XP. There are complex ways about this,>but>FS will not be using this, and for all current programs there>is absolutely no way a 32bit Architecture OS can allow a>single>process to obtain more than 2.147 bytes of memory. >>I therefore doubt you will get more memory.No, the max address with a 32-bit address is 2^32 - 1, or 4GB - 1. WinXP made the arbitrary call to allocate 2GB to user processes, and 2GB to the kernel...and many programmers make the assumption that the highest order bit will always be 0 on that basis. Thus, an address between 2GB and 4GB can cause issues due, for example, to an overflow when doing pointer manipulation such as adding two pointer adresses together.But the /3GB flag will cause the OS to allocate adresses between 2GB and 3GB without doing anything heroic.CheersBob ScottATP IMEL Gulfstream II-III-IV-VSantiago de Chile

[Brian Doney 127]

Bob,You've been running this way for awhile now, yes ?What has been your experience with I/O, system cache, etc. ?The /LARGEADDRESSAWARE flag ? Or should I say, the lack thereof ?Bit 31 ?Video drivers ? <---often they require more address space than the 1 GB kernel address space permits.Finally, what on earth are you doing to get FS up over 2GB in the first place ?

[psolk 3,037]

>Bob,>>You've been running this way for awhile now, yes ?>>What has been your experience with I/O, system cache, etc. ?>>The /LARGEADDRESSAWARE flag ? Or should I say, the lack>thereof ?>>Bit 31 ?>>Video drivers ? <---often they require more address space than>the 1 GB kernel address space permits.>>Finally, what on earth are you doing to get FS up over 2GB in>the first place ? >>>>Hey Brian,I just thought I would chime in with my .02 here. Bob, I don't want to speak for you so please don't take it that way...1. About the /3gb switchI know that FS9 is not a /largeaddressaware app not trying to disagree at all BUT for some reason (inexplainable but true you can try it real quickly yourself) if you add the /3gb switch to your boot.ini and monitor your mem usage that switch changes FS behavior completely. Now with that being said Brian's point about video drivers taking up more than a gig by themselves is ABSOLUTELY true. I run dual 8800 GTX cards and with the /3gb switch enabled I lose all AA/AF in the drivers and 1/2 the time nv4disp.dll crashes... Not such a useful switch although at one time I thought it was... You can just feel the video cards crying for help even if you add more physical memory because it is not available to your drivers..BUT again, with the /3gb switch enabled you will see ZERO PF usage. Even if you only have 2 gigs physical all of a sudden FS stops using the pagefile and loads everything to physical memory. I still don't know why but it does! Typical Microsoft I suppose.Right now for example I am doing KJFK-PANC-RCTP real time (With Brian's amazing help I have the right amount of fuel) in the PMDG 744 using DXT textures default JFK, add-on PANC and RCTP. Using ASV6, ActiveCamera, UTUSA, GE, FSPassengers and FS2Crew OF COURSE. I am running on 3 monitors 1920*1200 on 1 and 1280*1024 on the others all 3 Full screen at 16XAA/16XAF. (I see ~150-200 megs difference in memory usage between full screen and windowed mode AND resolution AA &AF all affect available memory as well.) Right now I am somewhere off the West Coast of Alaska and I show mem usage at 678 megs available/2046 megs. My O/S and background apps eat up ~210 megs which means FS is consuming a little over a gig of memory. My PF usage is 2765 available/3942 megs. With the /3gb switch enabled I will use almost all my physical memory but PF will show 3924/3924 available as long as that switch is enabled and FS will not look for the p/f at all so something is telling FS to switch its behavior just by the /3gb switch alone... Ask MS why on this one!2. Now for why FS uses so much memory to begin with. Everything above is quite standard BUT (#### I use that word a lot) a few weeks ago I was flying the same way I always did for years with my entire scenery library enabled, same exact trip and my memory numbers were showing close to 300 megs LESS available memory the same way they always did. I used to fly at exactly 400 megs available for the entire flight and land with ~ 325-275 megs available at add-on airports so my FS usage with all my add-ons was regularly 1.3 gigs and change AFTER a complete flight. Memory usage with FS when you first load and when you shut down is usually at least 3-400 megs less because of everything that gets loaded during the flight, not because of memory leaks like so many people claim. Now I only enable the scenery I am using for that flight and I hover around a gig of memory usage the whole time. If you run Filemon while FS is loading you will see it load your ENTIRE active scenery DB. By reducing that or increasing that the amount of memory used by FS will change accordingly hence why some users might be seeing such varying memory usages.WOW. Sorry for the long and drawn out .02 but I hope some if it made sense. Brian, not sure why FS is affected by that /3gb switch and it certainly hoses video drivers like you said but it is definitely affected and it changes the memory consumption and paging behavior quite drastically. As for memory usage look at the size of someones scenery installations, add-ons,DXT or 32-bit textures, .dll's in the modules directory, screen resolution, AA/AF + basic Windows memory usage and you get why some people use SO much memory...I think the OOM comes down to a memory management issue and we are really pushing FS to its limits as an application in terms of what XP as an O/S can deliver as available resources. By limiting our FS installations and activating only what is necessary we can "try" to keep FS and its add-ons within the limits of our O/S.So sorry again for the long and drawn out post but hope it contains some useful points...-Paul

[Brian Doney 127]

Paul,Please, please add more, you are one of the rare few who have extensively tested this, I'm sure Bob, and I know myself, certainly appreciate what you have to add.I have actually experienced the OOM myself, though not since removing UT completely.I use almost everything else you do as well, except FS2Crew.I have recently migrated to a WideFS setup though, so my experience today isn't applicable.The /3GB switch certainly does have an effect on caching, paged pool size, etc., but by definition it should make things worse, but maybe...Either way, it isn't really meant for a system that will be running high res graphics, etc, anything that will require extensive driver level memory allocation, so anyone looking at trying this, be careful, please.EDIT: Just to add Paul, and I'm certainly not trying to get you to guinea pig here, but did you try using /USERVA to set an amount slightly lower than 3GB, to give the driver room to breathe ?http://support.microsoft.com/kb/319043/en-us

[psolk 3,037]

Thanks Brian,I appreciate it especially from one of the people who has been around these forums for a while. I agree on the /3gb switch and at one point I was one of the people pushing to use it. It does not do good things to our systems in these setups... The only reason I know my memory stats so well is due to a little external LCD that shows me the %cpu usage, Phys. mem usage and PGFile usage of my system in real time via USB. (www.pertelian.com) That and A LOT of hours trying to figure this out and eliminating every single hokey pokey potential reason for the OOM under the sun is what leads me to my earlier conclusion. It can't be the add-ons or the code they were written in. I used to get it all the time with the LVL-D, PSS, PMDG on my long hauls, the planes didn't change but the errors did by managing FS and the amount of memory it uses.I have greatly reduced the OOM error with the 1.30 patch from UT but still disable all of my HL*.bgl's with UT installed. The patch alone did not seem to solve the problem. So far so good with the 1.30 patch installed and the HL*.bgl's disabled though. FS2Crew will add some additional overhead but for me the biggest part is that it means you sit on the tarmac for 30-45 minutes depending upon the version before your flight departs. I love that for the realism and interaction but what I notice is that FS does not recover the memory it uses while you are on the ground very well...For example, I mentioned above I had 680 megs available memory right now. When I started at KJFK upon initial load I had close to 1 gig available and FS was only using ~800 megs total with the PMDG and all my add-ons loaded at the terminal. After 45 minutes on the tarmac and opening/closing the FS2Crew dialogs I was down to ~575 megs available at KJFK. I then did the 31L departure from JFK over NYC and ~ 50 miles from JFK my memory jumped to the 680 number it is at now and it will stay there for as long as I can keep the PMDG 744 in the air and then it will start to increase memory usage from that 680 number on arrival, not from the original load number of a gig available. So where did that 300 megs I used setting up the A/C go? It isn't just FS2Crew either. Load up the pMDG 744 and do a ctrl-alt -del to see your memory usage. Now go through your pre-flight and as you are pushing back again see what your usage is. Where does all that memory go not moving anywhere sitting at the gate? Why don't I get it back when I leave the gate? Now comes your add-on airport wanting its 100 megs at the end of a flight and sometimes we just exceed the apps limits within the O/S. Moving to a 64-bit O/S doesn't help because of the app design. It is a true combination of app +O/S limitations, not one or the other...I can not figure out a more logical explanation with everything I have tried and tested other than it is just an overload of the app against available resources in a constrained O/S. MS never intended for FS9 to be pushing the limits of available resources on XP. By default with FS9 and XP running on a machine with 1 gig you will have almost 400 megs available. Vanilla FS9 was a long long way from where we are today. Good to see a nice bantered discussion like this though instead of the typical "this plane has a memory leak" when most don't even know what a true leak is :) The only KNOWN leak in FS that I know of is a LandClass file associated with an empty texture folder. This includes adding a LC.bgl to your addon scenery/scenery directory. Wow, time to change my sig, no more FX chip,:)Cheers everyone,-PaulPrimary RigLiquid CooledIntel C2D E6600 @3.2 gigsAsus P5N32SLI-Plus2 gigs Corsair XMS PC6400 4 4 4 12 @810Dual OC'd XFX 8800GTX @ 2 gigs24 inch Widescreen LCD 16XAA/16XAFDual 19 inch LCD'sRaid-0+1PCPower and Cooling 1k Quad SLIhttp://home.comcast.net/~psolk/3monitorsa.htmlBackup RigAMD 4000 San Diego @ 2.72 Gigs Kingston Corsair XMS CL2XFX 7900 GTX Raid-0

[Brian Doney 127]

Paul,Are you running everything (ActiveSky, etc.) on the FS machine ?Not trying to diagnose, I just tend to forget, even though I mentioned it above, that since I have offloaded most all applications I run with FS to the WideClient machine, our experiences will of course be different.On the WideServer/FS machine, I absolutely struggle to get over 800-900MB usage in any situation, not that I'm complaining, just an important difference.

[psolk 3,037]

Hey Brian,I'm not actually running Wideserver or running anything on my other box. I run everything on the FS box directly. Do you see a performance gain by offloading certain apps to a separate box? Is it worth it? I just never took the time but I know FDC, ASV, FSEarth can all be run networked.Maybe it is something worth looking into :-hmmm Bob, are you runnig everything on your FS box?Thanks Brian,-PaulPrimary RigLiquid CooledIntel C2D E6600 @3.2 gigsAsus P5N32SLI-Plus2 gigs Corsair XMS PC6400 4 4 4 12 @810Dual OC'd XFX 8800GTX @ 2 gigs24 inch Widescreen LCD 16XAA/16XAFDual 19 inch LCD'sRaid-0+1PCPower and Cooling 1k Quad SLIhttp://home.comcast.net/~psolk/3monitorsa.htmlBackup RigAMD 4000 San Diego @ 2.72 Gigs Kingston Corsair XMS CL2XFX 7900 GTX Raid-0:-hmmm :-hmmm