New project: Tile server for MS FSX based on Google Map...

[Guest cbuchner1]

The MS IFS kit is not free, I do not intend to spend 110 dollars for the "privilege" to write file system drivers for windows. But as you noted, the FileMon tool from Sysinternals hooks into API calls for filesystems and could be used as a basis for development of a tile pre loader.My worries are, since this is nitty gritty kernel mode driver development, that I will bluescreen and crash my trusty old PC more than I would like to.VmWare to run the tile server is an excellent idea, as networking between the host and the virtual pc should be quite fast.Let's see how that IEEE1394 (Firewire) link performs.Christian

[Guest cbuchner1]

I am considering this thread a technical project blog ;-) Read along if you're interested.I was able to solve some performance issues with high resolution (1m) texture loading by setting the MAX_BANDWIDTH_MULT parameter to 5000 and additionally increasing the fiber time parameter to 0.25. The tile server is connected to the simulator via Firewire and I have added a read cache feature to the modified Samba server. It now keeps a lot of tiles cached in RAM because Flight simulator is one greedy son of a ####. It keeps reloading the same tiles over and over, requesting the entire file even if it just needs a low resolution mipmap. So if you only have 600 MB of tiles on hard disk for a particular region, it may actually request around 5 GB of data over the network before the plane can even start taxiing. During flight, the sim keeps updating tiles at rates between 2 and 6 Mbytes/sec whereas during the initial loading phase, the Firewire sometimes bursts at 14 MBytes/sec (by courtesy of the new read cache).The 1m ground texture resolution is mindblowing. During a test flight today loading of tiles was keeping up with my aircraft most of the time. I have picked one Pacific island location for testing and when looking out of that airplane cockpit everything seemed so real. It's frightening. Almost spiritual. I'll have to invite some friends over just to show them the deal. Posting the images on this forum (at reduced size) wouldn't do them justice. Next stop: Grand Canyon. In the main FSX forum some people were actually questioning whether my latest screen shots were doctored. Nope, it's all for real.Regarding further development, I might try running the Samba server inside a virtual machine on the simulator on the 2nd CPU. Less networking overhead when using a virtual network connection.

[Guest dabagboy]

RE: the rest of us seeing - You could try posting the screenshots somewhere else? I think we'd all like to http://www.imageshack.us/

[Guest cbuchner1]

A bit more of an update. One major problem was that always the entire tile file would be read by FSX, even if only a 4x4 pixel bitmap was required. This created some insane network load when using 1024x1024 textures - 650 kb per tile. And thousands get loaded, yikes!I created some code to dynamically serve lower resolution tiles from the read cache by changing the BMP header and removing higher resolution bitmaps. This causes no CPU overhead as the tile does not need to be recomputed or rescaled and the network load goes down significantly.I created a small program using the SimConnect SDK that reads out the plane's current coordinates periodically. I would like to to report the current position to the tile server through some particular I/O operation, e.g. a write call to a magic file or an open () call specifying the data as part of the filename. Then the tile server could know which tiles are close by to the aircraft and pick an appropriate tile resolution.The part that tells Samba about the plane's position is not done yet as well as choosing the best resolution for each tile. For sure the tile should not exceed the mip level that FSX intends to load. And it should not have a much lower resolution either. Quite a challenge to "guess" what MIP level the simulator intends to load for a specific tile. ;-)The benefit would be a much increased performance with this Samba thing and hopefully no more blurries even with 1024x1024 textures.

[Guest cbuchner1]

Project update!Performance problems of the network connection are solved! A combination of caching and delivering lower resolution tiles (depending on the tile's distance from the aircraft) proved to work. I can now fly in a Learjet over my pacific island location and enjoy every spot in crisp texture quality.Some bad news: I found out that MS FSX always loads BMP tiles fully, even when accessing just low resolution mipmap levels. Don't believe it? Run filemon.exe from Sysinternals.com and check what the Read calls say. That would explain some performance issues and long load times on FS2004 style phototerrain textures.Luckily with Samba I can simply serve less data over the network and the file will appear to be "virtually shorter" without causing a read error on the receiving end. Hence I can provide the lower mip levels while FSX thinks it's still getting the full deal. Ha! ;-)The next step will be to dynamically lower the resolution loaded off the Internet depending on distance to the aircraft. It simply doesn't make much sense to get everything at the max detail level - loading time being the main reason.I should now check out and test the latest incarnation of Gaia. It has been cleaned from Google code and now accesses NASA's World Wind servers. Much more legal.Christian

[Guest cbuchner1]

Here you go: Enjoy the photorealism.http://www.flickr.com/photos/71882461%40N0...57594407535533/

[ibanez737 2]

This is incredible work, I never thought that something like this would be possible with current technology. Can't wait to see where it ends up!-Scott

[Guest cbuchner1]

Some tests in larger areas showed problems with tiles far out from the plane (the lowest resolution 4x4 MIPs) - they appear black. I did not notice this happening in the Oahu scenario at all - obviously because all terrain was close enough to be loaded at higher detail level. Of course the black tile issue must be solved to allow an immersive flying experience in larger photorealistic areas.Someone noted there is no haze in my screenshots at all - making everything appear quite unrealistic. This could be because I disabled Shader model 2.0 and fly at the best weather setting. I shall try some rain or fog soon ;)On the technical "front", I have now devised several ways of serving lower resolution tiles to reduce the network load - all because the tile loader of FSX is so wasteful with resources.1) - This method provides a modified bitmap header (lower resolution, less mips) and skips the bitmap data for higher MIPmap levels. The file then appears as a low resolution tile to FSX.2) - The second way is to serve the original tile file, but in a "sparse" way, zeroing out the parts of the file which I want to skip. This is deep trickery with Samba and is only tested with Windows XP as client so far. XP reads files in 64k blocks and entirely skips a block if you return a read operation with 0 bytes - thus allowing me to not having to transfer the uninteresting parts of a file across the network.3) - I can also serve the BMP header only when FSX accesses a tile for the very first time. This works on all mipmap levels other than 4x4 - the 4x4 level appears black though because FSX does not access the 4x4 tiles for a second time. This is the fastest way of serving out tiles though and the preferred method in the initial loading phase of a flight. FSX then caches how many MIP level each known tile has, allowing me to use method 1) on the subsequent tile reads.A combination of 3) and 1) has worked for Oahu, but I may have to use method 2) for the 4x4 tiles in the far distance.I will also further investigate the LOD quadtree structure that FSX maintains internally such that I can better predict when FSX will load tiles and at what resolution.

[Guest cbuchner1]

Hey, thanks for your encouraging comment.I've had a look at your webpage, in particular the Ni'ihau project and I wonder how you masked out water vs. land texture. Did you create kind of an alpha mask for the land shape and applied it with RESAMPLE.EXE? Did you use the FSX SDK or the FS 2004 SDK when doing your project?The reason I am asking is that I will have to come up with ways to prevent my project from displaying water as a land texture. I might have to use an autonomous water recognition algorithm (image processing). Currently I have no idea how to store the alpha information in the DXT1 texture.There is one SDK sample with water masks in the FS 2004 SDK - I shall try that.Christian

[scott967 62]

Am I understanding that you are using the FS2004 resample to create the bgls for your photo tiles? Or is there a way not to embed the tiles in the bgl in FSX resample?As far as the water masking, it seems like one solution would be to read the cvx files to get the hydro polys, and use this to build the masks. Extracting shorelines from the image I guess is possible, but more of a problem for inland water and bridges. Of course if the cvx data don't match the image there is a problem. For the US, the UT USA quality data seem to match very well. scott s..

[Guest cbuchner1]

Yes I used the FS2004 SDK RESAMPLE tool for my experiments so far.I am planning to reverse engineer the BGL format such that I can create BGLs for all of the world's landmass in a single go. A few hundred BGLs maybe. The texture is then filled in by the tile server whereever I fly.. that's the theory. ;)

[Guest cbuchner1]

One more update.All my previous speed optimizations concerned the data flow between tile server and flight simulator PC and made sure textures could be loaded at sufficient speed. This applied when the textures were already cached on hard drive.I have just revisited the part that loads textures from the Internet. I think I now have it done well enough to allow helicopter flights over large photo sceneries and the textures are getting streamed from the Internet. I am flying over San Diego, and I can see right where my friends live (Hi Jesse!). Everywhere I go, the textures get downloaded from the Internet at high resolution.Future work will focus on:- getting tile backends ready for MSN virtual earth and NASA World Wind servers- ensuring GLOBAL coverage of all land mass (not just in limited test areas)- multithreading the network access modules to get multiple downloads going in parallel (more loading speed, allowing for faster flights at low altitude)- porting this to Linux and packing this into a virtual machine module (e.g. for VMWare Player or Virtual PC)...and possibly talking to interested parties (Microsoft, Google, MSN).. and whoever wants to see this stuff demonstrated by me in person. I think this prototype now kicks some serious a**Christian

[Guest cbuchner1]

Images of the latest progress. Enjoy! Note that you can get 1920x1200 pixel versions using the "All Sizes" button on the individual photo pages.Slide show link:http://www.flickr.com/photos/71882461@N00/...413142755/show/Photo set link (use "all sizes" button for the full versions):http://www.flickr.com/photos/71882461@N00/...57594413142755/The enjoyment of flying in this visual quality more than rewards me for the work hours invested into this project.

[JPmiami 0]

Hi Christian!I sent you a PM, please check.. ;)Cheers!JP

[Guest cbuchner1]

Small update:Samba performance was biting me when directories grew large (~15000 files). Turns out Samba has got to keep a filename cache for offering the Windows-typical case insensitivity on the Mac's case sensitive filesystem. When too many files are used, the cache spills and filename lookups require a full directory scan. A quick fix is to force uppercase filenames in the samba share.Grand Canyon looks quite fabulous from a helicopter. But it turns out the Robinson R22 has a few problems flying at that high altitude in hot air ;) I had to dump some fuel and empty my passenger seat to make this baby climb. But overall: FABULOUS EXPERIENCE. Especially because Grand Canyon National. Park is pretty much a no-fly zone in reality for environmental protection reasons.On the programming front, I've devised some experimental code to use the "multi" interface of libcurl instead of the "easy" interface, allowing for faster tile access on the Internet, especially on the higher resolution tiles (512x512, 1024x1024).I wonder what happens if I try some 2048x2048 tiles. Will FSX crash?Christian