Sign in to follow this  
Guest

New SRTM_3ArcSec_To_BGL - Fixes holes in SRTM data

Recommended Posts

Just uploaded v1.02 to the library.It now interpolates any missing SRTM data. So that means no more holes in your SRTM mesh files :-jumpyFor those who don't know what this program does... all you have to do is place this program in a directory with some SRTM 3arc second (ie 90m) files, and it will create BAT and INF files to automate the production of FS2002 mesh from the SRTM data.NASA SRTM mesh data can be downloaded herehttp://edcsgs9.cr.usgs.gov/pub/data/srtm/ Matthew.

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

Freakin' Awesome man!I've got form N43 to N47 done.Now I can complete all of North America today with your update!Woooo Hoooo!

Share this post


Link to post
Share on other sites

Glad you like it...The interpolation (ie hole fixing) could be better. I'll improve it later.Also you may want to build LOD8 mesh as well... Since the LOD8 mesh is used at longer distances than the LOD9 mesh can be displayed. So this means your mesh will look better from a distance. Just edit the .INF file, and change the LOD = 9 to LOD = 8 and rerun the batch file. Make sure you rename the existing LOD9 .BGL first.Also working on a SRTM_1ARCSEC_TO_BGL utility for the 30 meter SRTM data of the USA.BTW, build some mesh for the Caribbean... makes a big difference! Now all those islands actually look rugged (volcanic even) :)Matthew

Share this post


Link to post
Share on other sites

Might just try the LOD8. Any iea if it's a higher detail terrain? I'm curious about whether any of you have made a complete freeware mesh of north america/south america yet? I'm currently working toward that goal. Each BGL is compiled by degrees latitude. (ie N43W060 to N43W126) Each BGL is in the neighborhood of 50MB each.So it's gonna be a space hog :-lolSo far the mesh looks awesome! I'm trying to complete Canada and northern USA first. Then on to soutern USA. etc etc.It's a bit time consuming, but it's well worth it. Any one else interested in helping to compile data together?

Share this post


Link to post
Share on other sites

Howdy:LOD8 is lower resolution than LOD9 (153 vs 76m). However, since the SRTM resolution is somewhere in between (depending on latitude) there's little or no noticable difference with respect to display accuracy. The big advantage is that LOD8 tiles will save you a lot of disk space.Maybe you know this already...Be aware that your tiles have to overlap somewhat to fit with the underlying LOD grid (and the LOD8 grid is different from the LOD9 grid). If you're not aware of this, check out Steve Greenwood's website; it also provides a spreadsheet tool for doing the LOD grid calculations: http://www.fs-traveler.com/fs-traveler.htmlInterpolation will NOT replace the missing data. For small holes it will do a good enough job but for larger ones expect disappointing results. For example, most of the steep canyons in Utah and Arizona are AWOL in SRTM (including the Grand Canyon), as are some of the fjords and mountains along the BC and Alaska shoreline. Simple interpolation will make a mess out of these places. For the US areas, your best bet is to download the USGS tiles of the worst areas and merge the two DEMs in MicroDEM to fill the holes.One thread to check out is this (also, do a search for "SRTM" in this forum):http://forums.avsim.com/dcboard.php?az=sho...g_id=626&page=2Finally, like all add-on meshes, there will be problems with lake elevations, rivers, and shorelines due to a horizontal offset between the FS default hydrography and the SRTM data. This often leads to sunken or raised lakes (and airports) and ugly cliffs on shorelines. Fixing it requires either tons of flatten switches (as I did for my BC meshes) or changing and recompiling the FS hydrography .bgl files, like Gilles Gauthier did for his Quebec and Maritimes meshes (search under his name in the AVSIM library).These issues are meant to be helpful, not to discourage you ;-)Cheers, Holger

Share this post


Link to post
Share on other sites

Hi David,Expanding on Holger's comments, I'd like to suggest a different approach.> Each BGL is compiled by degrees latitude.The height of an LOD8 quadrant is 0.35 degrees. This means that about 35% of the area between your nominal one-degree longitudinal bands will still be default mesh. You can confirm this using the TmfViewer, which should be used to check all mesh.If you construct your mesh in smaller blocks you can avoid this problem, produce smaller files, and make it easier for folks to download smaller areas of interest. It will also make it easier for you to fill large areas of missing data with lower resolution source data, if desired. (However, this will be VERY difficult to do if the missing data has been interpolated - you may want to use the earlier version of SRTM_3ArcSec_To_BGL if you decide to fill in this data.)For example, a nominal 5-degree square block can be constructed from 6x6 1-degree hgt files (36 total). You will lose some data along 2 sides and have excess on the other two, but when combined with adjacent mesh files, the coverage will be complete. This crude approach will create some redundant mesh, but avoiding the overlapping mesh will require a great deal more effort.E.g.:[pre] N65 N65 N65 +---+ +---+ +---+ W075 | | W070 W070 | | W065 W065 | | W060 ... +---+ +---+ +---+ N60 N60 N60 [/pre]And the next row would include hgt files N65 to N70, ....Good luck!Stevewww.fs-traveler.com

Share this post


Link to post
Share on other sites

(However, this will>be VERY difficult to do if the missing data has been>interpolated - you may want to use the earlier version of>SRTM_3ArcSec_To_BGL if you decide to fill in this data.)>I'll add a command line switch in the next version to disable my interpolation. Matthew.

Share this post


Link to post
Share on other sites

I think we should ask AVSIM to let us host a "Project Mesh" site (eg www.avsim.com/projectmesh) on the AVSIM site, just like they do with ProjectOpenSky, TerraFish and others."Project Mesh" could be a colaborative effort to build high quality mesh for the entire world from SRTM 3 and 1 arc second data, and other data.We could break the world up into various areas and each person compile FS2002 / FS2004 mesh for their area and then upload it to Project Mesh webiteWe would continue to use the current Mesh forum for our discussions and load our work into the AVSIM file library. The Project Mesh site would link to the file library and forum.We could have a clickable map of the world that would allow users to easily download mesh files (from the AVSIM library) that interest them.I could continue to enhance the SRTM_3ARCSEC_TO_BGL and SRTM_3ARCSEC_INTERPOLATE utilities to make our jobs easier and to add new features to improve the mesh quality... including merge missing data with GTOP data instead of using interpolation for large holes.What do you think?Matthew.

Share this post


Link to post
Share on other sites

Hi Matthew,>I'll add a command line switch in the next version>to disable my interpolation. Excellent solution. Apparently I've spent too much time working with the data in MicroDem today! At the moment, my approach is to auto-fill holes up to a specified radius and then fill the larger holes with GTOPO30 data. I don't know how difficult this would be for you, but an alternative might be to allow the user to specify the radius/size of the gap to auto-fill. A value of 0 could disable the interpolation, while other values would provide more control over the process. (And perhaps something like -1 to replace all missing data, as I assume your program does now.)You've created a promising program; I look forward to seeing how far you can go with it. Cheers,Steve

Share this post


Link to post
Share on other sites

I've now made a change that allows the interpolation to fill holes from the outside of the hole to inside of hole and upto a specific radius.So small holes are filled nicely while larger holes have their middle left unchanged.What I should do next is fill those remaining (large) holes with GTOP30 or sample resampled lower LOD SRTM dataI'll try and upload something today/tomorrow. Right now though I better get back to work :)Matthew.

Share this post


Link to post
Share on other sites

Matthew;Is the data derived from STRM only used for Mesh? Is their no way to use the Water, Urban, and Transportation data?Richard;

Share this post


Link to post
Share on other sites

I don't think that SRTM has any other information included in it other than elevation data. But i'm a newbie at it so I could be wrong.

Share this post


Link to post
Share on other sites

Only elevation data in the SRTM files.However I guess you could derive the coast line since that would be where elevation is 0. But from what Ive seen of the initial SRTM data is would be quite inaccurate.Matthew.

Share this post


Link to post
Share on other sites

Hi Matthew:yup, coast line extraction wouldn't work because the darned radar is so accurate that it measures the exact water elevation, which is anything but zero. I noticed in some of the tiles along the BC coast that if you change your color scheme to highlight small differences at low elevations you can actually see the wave patterns in the ocean and fjords - pretty cool. In fact, ship spotting (by their wakes) is one of the applications of radar satellites, though not of this particular Shuttle system.Cheers, Holger

Share this post


Link to post
Share on other sites

1. If there are already some .HGT_INTERPOLATED filed in the directory, they become .HGT_INTERPOLATED_INTERPOLATED and so on... Perhaps the program should process only .HGT files2. [MultiSource] with only one source does not work, resample's fault... The workaround is to remove the MultiSource section and rename [source1] to SourceOtherwise your software is great!Interpolation works like a charm for me (south Brasil), where all voids are under lakes/rivers or are very small dots.Thanks a lot!Rafa

Share this post


Link to post
Share on other sites

How do you merge SRTM and GTOPO30? Are you using MICROdem?Also... what about things like lakes and so on. In some northern parts of Canada, they seem to be using their own altitude, which in turn makes them look like big buckets of water in the middle of nowhere. Any ideas?

Share this post


Link to post
Share on other sites

Howdy:yes, I believe that the best way for merging is to use MicroDEM; you'll have to resample the GTOPO30 DEM to the same resolution as the SRTM file prior to merging.as to the buckets of water: I mentioned that problem in the last paragraph of my previous post here. When MS matched their water linework and polygons to the default mesh they seem to have lowered (or raised) the lakes to the surface, regardless of the true elevation. Along comes a new add-on mesh with correct terrain elevation, only to end up with lots of "holes" and "mesas". I believe that Gilles Gauthier's fixes for Quebec and the Maritimes are currently the most elegant method whereas my flatten switches were the brute-force approach. Either method requires a lot of tedious work and good topographic maps.The ideal solution would be to acquire hydrographic vector and polygon data that matches both in resolution and quality the SRTM meshes and use those to compile your very own set of hydrography .bgl files.Cheers, Holger

Share this post


Link to post
Share on other sites

Hi David,Yes, I do use Microdem. There are several steps involved and the results are far from perfect, especially along the edges. But I've encountered areas of missing data almost 1-degree square, and this is better than interpolation for such large areas. I still go in and manually fix any small remaining problems if I see them.If Matthew can develop a way to resample and merge the GTOPO30 or Globe data and smooth the transitions between the data sets, it would provide much better data to work with. :-)1) extract a small subset of the GTOPO30 data for the area2) load the subset and save as a DTED file, resampling to 3 arcsec3) merge the data, loading the DTED file(s) first, then the SRTM data4) you may have to take a subset of the merged data, if your DTED data extends beyond the boundaries of the SRTM data.I turn off auto-filling and do it after the merge, but this does not work perfectly, and I still have to set missing values to sea level. I plan to try leaving it turned on for my next test.Microdem Options:* Import/Export - Auto-fill holes on DEM merge (I'm using 100 as max gap to fill, but no sound basis for this value)* Views - Missing values set to sea levelYou may find Globe data easier to work with; it is available in one-degree files. I've recently downloaded the Globe data and nasa jpegs (2 files, about 250MB each) from:http://globe.unibuc.ro/mapdata_zipped/These also work well with the Blue Marble option in Microdem. This feature is not really useful for our purposes, but is fun to experiment with!The lakes are a problem. The elevations can be altered with flatten bgls but it is tedious work when done on a large scale. Holger created "tons of flatten switches" for his BC mesh. My flatten utility creates flatten bgls fairly easily:www.fs-traveler.com/flatten.htmlSteve

Share this post


Link to post
Share on other sites

Hi all.There is one other method for handling lakes... although I do hesitate to suggest it.The default LWM bgls could be replaced with CFS2 style LWM bgls. This would be accomplished by decompiling the default LWM BGLs with and then using using Edgar Knobloch's [link:library.avsim.net/esearch.php?DLID=&Name=&FileName=cfs2conv.zip&Author=knobloch&CatID=Root]CFS2Conv, then recompiling as CFS2 LWMs.CFS2 LWMs have no elevation. They cling to the mesh. If the mesh is sufficiently detailed, there is little need to use flattening.This would cause problems for uninstalling the scenery, as the defaults would have to be reinstalled.This might have to wait for a 'corrected' shuttle DEM set, as waves, swells, and other things are now present in the water.Dick

Share this post


Link to post
Share on other sites

Thanks for that... Ill address those bugs in the next versionMatthew.

Share this post


Link to post
Share on other sites

Nice!Actually I tried it before i started writing the SRTM_3ARCSEC_TO_BGL utility.My goal with SRTM_3ARCSEC_TO_BGL is to make it as simple as possible... just run it and nothing else to do or configure!I might add a simple GUI to make it more accessible to people who are not comfortable with the DOS prompt and batch files.Really I just want it to be a "Mesh for Dummies" type of program. I think i will add in ability to download SRTM from Net, and ability to install BGLs into FS2002/2004... all from a simple GUI "Wizard". Then there will be no need to work with DOS prompt at all.BTW - Merging GTOP30 with to fix holes is a great idea. Must try to put it in soon.Matthew.

Share this post


Link to post
Share on other sites

Hello Matthewyour work is so incredible do get fast results. this way big meshes are possible on an easy way.Only be careful when you want to divide the mesh in small bgl's because your programm reads the mesh not completly but restrict to LOD9 cell's so there will be always blank strips around. I solve this by generating bgl's out of 16 *.hgt files (4 x 4) where the neighbour - one's overlap one degree width. hmm... this creates double redudant data, but the work is fine!i run your program inside a batch which select certain hgt's out of a folder so it looks like this:del *.hgtdel SRTM_3ARCSEC_TO_BGL.batdel SRTM_3ARCSEC.infcopy ..sourceperu9*.*SRTM_3ARCSEC_TO_BGL.execall SRTM_3ARCSEC_TO_BGL.batrename SRTM_3ARCSEC.bgl qowimesh_9.bgldel *.hgtdel SRTM_3ARCSEC_TO_BGL.batdel SRTM_3ARCSEC.infcopy ..sourceperu10*.*SRTM_3ARCSEC_TO_BGL.execall SRTM_3ARCSEC_TO_BGL.batrename SRTM_3ARCSEC.bgl qowimesh_10.bgl...so i just seat back and let the computer work!what would be interesting is:- implementing whether to get error-hole data out of GTOPO30 (must be in same directory... but what format?) or to interpolate data - as it does now.- while creating the *.bat adding an option to create either LOD8 or LOD9 files, which would automatically create the accurate *.inf.now, big work must be done by reflatten polygon -rivers and lakes, deleting default rivers and streets, and puting newones!Thanks again Matthew for your work!SaludosFrancis

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this