Breaking the 4Gb Barrier w/ a Ram Drive

[Noel 4,492]

Just for fun I used the nice IMDisk freeware Ram Drive software driver to create a 8Gb ram drive and assigned my swap file to it w/ a fixed size of 7gb.  By paging to the ram drive I noticed maximum memory use went over 4030Mb, whereas when I left the page file on my C:/ drive SSD maximum memory committed would peak out at about 3780Mb.   The thing I haven't yet tried is disabling the page file.  Methinks WIndows would allocate even less to FSX in that instance, however I may not understand how it functions.  Some have conjectured using a swap file in ram can help reduce writes to your SDD if used correctly, which might matter some for longevity, or not.

 

Edited July 19, 2013 by firehawk44
Image removed. Exceeds 400KB weight limit.

[TheGrunt 44]

Of course your total memory consumption can go over 4GB with 64-bit OS. Problem is within FSX itself and as it is a 32-bit program, it can't address more than 4GB of memory. Using 32-bits you can't just make any bigger number than 4,294,967,296. Law of the mathematics apply here. Period. Besides, more important is FSX process virtual size, which you can't see from task manager. It reaches 4GB limit far more earlier, than the process itself. Then boom, and you have yourself an OOM error.

 

What is possible though, is that 32-bit OS can use over 4GB using so called physical address extension, where tables are used to map 4GB memory areas to a larger memory chunk up to 64GB. In 32-bit desktop Windowses 4GB limit is imposed and PAE is not supported, but I've read somewhere that desktop Windowses can be hacked too to utilize up to 64GB. Never tried that though, because I've been running 64-bit OSs since 2006 when 1-2GB of memory was pretty common in home use and problem didn't exist. PAE doesn't magically make 32-bit programs act like they would be 64-bit programs and 32-bit OS with PAE can't run 64-bit programs. It is just remapping 32-bit memory addresses in processor level.

[Noel 4,492]

Yes, but it was interesting to note the difference, albeit small, when I used the ram drive.  Next will try no pagefile which could be the best option of all unless OOM's occur.  That the system doesn't have to allocate resources to page seems like the best option of all.

[lodestar 186]

use process explorer for that: http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

[TheGrunt 44]

Next will try no pagefile which could be the best option of all unless OOM's occur.  That the system doesn't have to allocate resources to page seems like the best option of all.

Disabling pagefile doesn't most likely affect in anything when you have plenty of RAM. If there is room for the program to fit in the main memory, no data is written in page file. Only exception may be some very, and I mean very, legacy programs which can require a pagefile for it to function at all. In these cases program was written to write some data straight to page file instad of the memory. I would say, that it is very unlikely that you use anything like that.

 

Page file is relevant only when your amount of RAM is lower than address space and your program actually uses more than that amount. Page file is understandably slow, even if it is on SSD and makes using something like modern game completely useless and most probably it would end up in a crash or program may be even written so that it refuses to work if there is not enough physical RAM. As a general rule of thumb, page file is quite useless with todays common RAM amounts of 8GB or 16GB with 64-bit OS: there is plenty of room for (at least) one large 32-bit application, like game, and still more than enough for all backround programs to run in their own address spaces.

 

If you see more memory consumption with ramdisk one reason is the ramdisk software itself, which requires some memory. Total memory consumption varies also a lot depending what your OS and all backround processes are actually doing. By the way, I just noticed that in the screen shot your total memory consumption is 4030MB and even that is not over 4GB.

 

As I said, it is pointless to look FSX memory consumption through task manager, because you can't see its virtual address space and usage of virtual memory it actually needs. When I once suffered from OOMs with my FSX I noticed, that in general, when VAS-usage hit 4GB and program crashed, FSX process memory usage was around 3.7GB. Those few hundred MBs is consumed mainly by all the drivers and essential OS components that FSX requires to function. They need to be in the same virtual address space of FSX, which is bound to 4GB.

 

One more thing, whether you are talking of main memory or page file, adding page file doesn't help in any way for the program address space. Let's say, that you have a situation where you have 2GB of consumed physical RAM and 2GB of page file for the program. That is a 32-bit address space consumed, totalling 4GB. In that sense, page file is exactly similar memory compared to RAM from software and memory address perspective. You can't add to address space by using page file.

[goates 576]

That the system doesn't have to allocate resources to page seems like the best option of all.

 

...

 

Some have conjectured using a swap file in ram can help reduce writes to your SDD if used correctly, which might matter some for longevity, or not.

Except that without the page file the OS can no longer move lower priority data out of RAM (including things like the registry). Many background tasks that get loaded at system start, for example, don't actually do anything until they get called upon, in which case the OS moves their data to the page file. If you have far more RAM than anything you do with the system will require, you can probably get away without the page file, but in most cases it is best to just leave it alone (or at most just set a fixed size).

 

Trying to save your SSD by moving the page file to RAM also doesn't really make much sense. If your system is heavily using the page file, it means you need more RAM, period. Moving the page file to RAM would just exacerbate the situation. This holds true with hard drives as well. SSDs are also designed to last many years and aren't going to wear out from page file usage. One would have to do something like delete and write to every sector in an SSD 24/7 for a year or more before the drive began to wear out. Under normal usage, an SSD will last a few years. They wouldn't offer 3+ year warranties if the drives were going to die after one or two years.

 

Moving the page file to a RAM disk doesn't really make much sense either as the point of the page file is to free up RAM, however, if the page file is moved to RAM, it can't really do this. It would be far better to make as much RAM available as possible for applications to use directly.

 

Here is a link to some Microsoft articles on how Windows memory is managed.

 

http://blogs.technet.com/b/markrussinovich/archive/2008/07/21/3092070.aspx

[TheGrunt 44]

If you have far more RAM than anything you do with the system will require, you can probably get away without the page file, but in most cases it is best to just leave it alone (or at most just set a fixed size).

This. Every now and then you see on some forums extremely long discussions about what to do with the page file. SSD drives becoming more common have lifted this subject up again and some are claiming that pagefile writes are considerably taking hours away from SSDs and better way is to disable it or set it to fixed size. Whole discussion is moot IMO, as you said. SSD drives are used succesfully in far more demanding environments than regular home desktops, where writes occur more often than OS page filing. OP with 32GB can run his computer fully without any page file. I once ran my Core i7 920 with 6GB of RAM for a while without any issues, untill I realized the uselessness of all that to my new SSD. Fixed size? Yeaaah, but I'd still keep it OS managed because then you have in every possible scenario enough of page file, if it is needed. Keeping it fixed doesn't take away any writes to that allocation, so if PF is used, writes will occur whether the size is fixed or not.

[goates 576]

This. Every now and then you see on some forums extremely long discussions about what to do with the page file. SSD drives becoming more common have lifted this subject up again and some are claiming that pagefile writes are considerably taking hours away from SSDs and better way is to disable it or set it to fixed size. Whole discussion is moot IMO, as you said. SSD drives are used succesfully in far more demanding environments than regular home desktops, where writes occur more often than OS page filing. OP with 32GB can run his computer fully without any page file. I once ran my Core i7 920 with 6GB of RAM for a while without any issues, untill I realized the uselessness of all that to my new SSD. Fixed size? Yeaaah, but I'd still keep it OS managed because then you have in every possible scenario enough of page file, if it is needed. Keeping it fixed doesn't take away any writes to that allocation, so if PF is used, writes will occur whether the size is fixed or not.

A fixed just keeps the OS from using up 16GB of hard drive space for a page file that doesn't get used much (especially if you using 128GB or smaller SSD as your boot drive). I had one system where Windows allocated a huge page file on an SSD for some reason. Guess I must have run something that requested a lot of address space. However, for most people, I agree, just leave the page file settings alone unless you really do have a reason to change them.

[Noel 4,492]

goates, on 19 Jul 2013 - 07:54 AM, said:

Trying to save your SSD by moving the page file to RAM also doesn't really make much sense. If your system is heavily using the page file, it means you need more RAM, period.

What's the best way to evaluate paging activity?

 

What value has a page file in a 64-bit system running a 32-bit app (FSX), where there is ample physical RAM? I'd like to see exactly how often memory is paged using either a SDD or RAM based pagefile.sys. I ran this evaluation 5 years ago or so in a system w/ 4GB of RAM in Vista 64. At that time when FSX was running paging was virtually constant. I think it was perfmon that I used to observe this.

 

Right now I have a 16GB page file set up in RAM using imdisk. Works well, but may not be worth the trouble. The key for me would be watching the write activity going to the page file. I understand some applications (maybe FSX) were designed to page more actively than others, though that was internet hearsay.

[Noel 4,492]

 

 

Except that without the page file the OS can no longer move lower priority data out of RAM (including things like the registry). Many background tasks that get loaded at system start, for example, don't actually do anything until they get called upon, in which case the OS moves their data to the page file. If you have far more RAM than anything you do with the system will require, you can probably get away without the page file, but in most cases it is best to just leave it alone (or at most just set a fixed size).

 

This supports my hypothesis, however irrelevant it may be in the real world, unless as I say FSX tends to page more frequently be design.  Putting the page file on a RAM drive in a box w/ mega amounts of RAM should ultimately be the best theoretical solution:

• Frees up all available ram by being occupied by 'lower priority items'--unless this is obviated in 64 bit memory management w/ lots or RAM installed
• Eliminates any paging to the SSD, which can't hurt the SSD
• Is faster than paging to an SSD or HDD, though as I say, may be completely irrelevant in the system described

[JimmiG 244]

Theoretically, having a page file (on disk) would allow the OS to page out unused data, allowing RAM to be used for stuff you actually run. The system doesn't just page to disk because it has run out of physical RAM, it might page out data that hasn't been used for ages even if you have plenty of free RAM.

However, computers today have such ridiculous amounts of RAM (8GB is minimum, many have 16GB or even 32GB) that it really doesn't matter. A couple of years ago (ie. when Windows 7 was released), a system might have had 4GB of RAM plus a 6GB paging file or 2GB plus a 3GB paging file (using the arbitrary 1.5x RAM rule for the paging file) - total address space would be 10GB or 5GB - much less than the 16GB of actual RAM many systems have today. I don't think anybody back then foresaw computers with 16GB of RAM.

Basically, it's not worth it to mess with the default settings unless you're running desperately low on disk space. If you are, it's probably time to upgrade your primary SSD or HDD anyway, and the new one will no doubt have at least 2x the capacity of your current drive, removing any concerns about disk space. Leaving the paging file alone will guarantee you won't run into weird compatibility problems with poorly coded applications and gives you the benefit of the crash dump, which can be analyzed to find the offending driver or software.

 

RAM disks are, of course, a total waste of time regardless of your setup. It's like trying to pull yourself up by your bootstraps.

[TheGrunt 44]

This supports my hypothesis, however irrelevant it may be in the real world, unless as I say FSX tends to page more frequently be design.  Putting the page file on a RAM drive in a box w/ mega amounts of RAM should ultimately be the best theoretical solution:

• Frees up all available ram by being occupied by 'lower priority items'--unless this is obviated in 64 bit memory management w/ lots or RAM installed
• Eliminates any paging to the SSD, which can't hurt the SSD
• Is faster than paging to an SSD or HDD, though as I say, may be completely irrelevant in the system described

Just turn the page file off alltogether if you are going to test "all in RAM solution". Using some ram drive for page file really isn't theoretically fastest solution because OS has to write the data to page file format and then you have ram drive software somewhere in the middle handling that data. Optimal way to get everything 100% in the memory is just to turn it off. Just a couple of mouse clicks and you are good to go.

[Noel 4,492]

RAM disks are, of course, a total waste of time regardless of your setup.

 

How so?  As long as paging activity occurs, and as long as you have enough RAM so that dedicating a portion to become the location for a page file doesn't pull address space away from all applications & processes running, I don't see how RAM disks are a total waste.   As I mentioned when I tested this out many moons ago in a 4Gb system I saw continuous write to disk operations happening when FSX runs, using perfmon I believe it was.   I moved the page file to a 3rd HDD that had only data on it, w/ OS running on #1 HDD and FSX running on #2 HDD.  What I witnessed was virtually constant write activity in chunks from 20 or 30 kb up to 50-100mb per write.   When I then moved the page file to #2 the write activity commenced on #2 HDD and stopped on #3 HDD as expected.  I'm assuming resources to do this are greater on a physical HDD than in RAM as a RAM drive.  I plan to do the same test now but need to find the correct tool to identify page file write activity.  Perfmon may work I haven't looked to see if it's in WIn 7.

[goates 576]

You may be interested in this thread over on FlightSim.com.

 

http://www.flightsim.com/vbfs/showthread.php?257691-RAMDisk-vs-SSD-tests&highlight=ssd+ramdisk

 

The conclusion was that the RAM disk wasn't worth the trouble, at least in his setup.

As I mentioned when I tested this out many moons ago in a 4Gb system I saw continuous write to disk operations happening when FSX runs, using perfmon I believe it was.

With 4GB I would expect there to be lots of paging activity as FSX could easily use up to 4GB on its own. Add in the OS and a couple background tasks or other programs, and 4GB would fill up pretty quick. With 8GB of RAM there is plenty of room for FSX, Windows and a few other tasks or programs. FSX will still be limited to 4GB itself, but the rest of the RAM would be free for Windows and whatever else is running.

 

Perfmon is still included in Windows. One of the enhancements to Windows 8 was an improved task manager as well, though it still doesn't quite match perfmon or some of the Sysinternal utilities.

[TheGrunt 44]

How so?  As long as paging activity occurs, and as long as you have enough RAM so that dedicating a portion to become the location for a page file doesn't pull address space away from all applications & processes running, I don't see how RAM disks are a total waste.

If you simply take the page file away, there is nothing for the OS to page anymore! It is all in the memory even wihtout ram drive. From software standpoint PF is same memory as main RAM and OS handles what and if it puts something in there, usually when main memory gets full. Programs read and write to the page file without "knowing" which one it is. Thus, if you disable PF, you have only your RAM as memory.

 

But if you don't want to listen what other people here are trying to say, by all means use the ram drive. It is useless nonethless.