USGS Seamless NED Data and MicroDEM

[Guest jkvato]

I've been studying Steve Greenwood's thesis on the errors that Microdem has reading NED data from the USGS seamless data site. This post refers to his writings, so I refer you to his website if you're not familiar with these potential errors:http://www.fs-traveler.com/sdds-test.shtmlI started playing around with numbers on a block of NED data I downloaded from the USGS seamless server. The northwestern corner of the data that I requested was 40N 107W. After loading the data into Microdem, I noticed that it was reporting a northwestern corner of 39.99972N and 107.00000W (I got this from the following menu path: Calculate/Map Window Corners). The TFW file reported the northwest corner as 39.999861112231109N and 106.99986111167104W.I hope Steve doesn't mind, but I'm linking one of his images here to illustrate my point:As Steve theorizes, the TFW file reports the point at the center of the northwestern cell as the northwest corner (point #2 on the drawing). Microdem reports the southwestern corner of the northwestern cell as the northwestern corner (point #3 on the drawing). (I know, it took me several readings and some math to get this all straight, and I still get confused). But to my knowledge, and someone correct me if I'm wrong, we need to have point #1 listed as the northwest corner in the .inf file for the SDK resampler, and neither Microdem nor the TFW file gives us that coordinate.Getting quite confused about all these points, I made an Excel file to calculate the differences between them, and my findings are right in line with Steve's. I'm attatching my Excel file to this post for anyone who is interested in picking it apart. If my calculations are correct, and if Steve's theory is correct, then this Excel file could be used to help pinpoint the true and correct northwest corner that goes into the .inf file.The link to the Excel file:http://forums.avsim.net/user_files/43910.zipNow, for 1 arc-second data, the offset may seem minimal. But if data is run through Microdem a few times before it goes to the resampler, the error could increase with each round. Also, I ran some non-scientific tests of Grand Canyon data, and found that streams feeding into the Colorado river in the Grand Canyon area fell into the very bottoms of the canyons nearly perfectly. I was amazed how accurate the default stream data is in that region. When I used the other two points in my .inf file, the streams were a bit off and probably not very noticeable to the casual flier. As for me -- well, I hate misplaced streams and lakes, and I was quite pleased with the results.Anyway, I hope someone will take a look at the Excel file and pick it apart to see if I'm doing anything wrong. And I welcome any input that someone may have for it.I think later on I'll post my theory on how maybe this same offset problem applies to the SRTM data.

[Guest jkvato]

OK, I've already found one error in my spreadsheet.If you look only at the decimal degrees, it's OK. My error is in converting decimal degrees to degrees and minutes. In fact, the error is only with negative numbers (I think). I'll have to work on that.The decimal degrees look OK, though.

[Justin Toposim 16]

HI John,Instead of using the .tfw figures, why not just use the same figures you you plugged into the "define area by coordinates" of the SDDS?One could easily move the entire Rocky Mountains chain to, say, Kansas, just by adjusting your inf file.If you downloaded your data from SDDS using 40N 107W, use that same figure in your INF file.-------Justinhttp://www.fsgenesis.netHigh Quality Scenery for FS200x

[Guest sgreenwood]

Hi John, " my findings are right in line with Steve's."That's good to hear. ;) "But to my knowledge, and someone correct me if I'm wrong, we need to have point #1 listed as the northwest corner in the .inf file for the SDK resampler, and neither Microdem nor the TFW file gives us that coordinate."We're dealing with elevation values and their point locations on the earth, however they are derived. The USGS DEM specification is described in these terms, and the SDK, using dem terminology, requires the NW corner specified as pair of coordinates. The notion of a point representing an area is an artifact introduced by graphical mapping software. The ESRI specification for the BIL file header format provides a rule for translating between these intrepretations. Neither Microdem nor Manifold seem follow this convention correctly. Fortunately, The USGS provides us with the BIL/TFW header information (along with other sets of coordinates which are of less immediate use to us). Ultimately we must rely on the values provided by the USGS as being accurate. As demonstrated, we can make the calculations required to adjust for anomolies introduced by Microdem, assuming we are aware of them. (Manifold is more problematic.) But the simplest approach is to use the values in the BIL/TFW header. (Note: Manifold keeps changing these each time the data is saved!)So, the location of point #2 is the best value we can provide in the inf file.Hope this helps; it does make things simpler!Steve

[Guest sgreenwood]

Hi Justin,"Instead of using the .tfw figures, why not just use the same figures you you plugged into the "define area by coordinates" of the SDDS?"Of course you could do that, if you're not concerned with positioning your data accurately. The error can be as much as 720m+ when working with 30" srtm data. ;)As I understand the situation, the problem lies with the way the USGS extracts the data. Your requested coordinates are interpreted as a request for data that maps to an Area that includes the Area you've requested. The actual location of the NW corner returned in the BIL/TFW header may lie inside or outside the boundaries you requested. Without understanding how this works, you will not be able to makes sense of the variable relationship between your requested coordinates and the BIL/TFW information returned.The probability of your requesting a NW corner that lies exactly at the location of the data point returned is virtually 0, so your approach guarantees an error that will average 1/2 the square root of the source resolution.-----Stevewww.fs-traveler.comHigh Quality Mesh and Information about Mesh for FS200x

[Justin Toposim 16]

>The probability of your requesting a NW corner that lies>exactly at the location of the data point returned is>virtually 0, so your approach guarantees an error that will>average 1/2 the square root of the source resolution.>. . . maybe 3 meters, which will be rounded anyway by the resampling process. We're not working with anything requiring that fine of a tolerance. The end result will still "snap to" the LOD grid.-------Justinhttp://www.fsgenesis.netHigh Quality Scenery for FS200x

[Holger 0]

Hi there:interesting discussion!I do agree with Justin that for 10-m and even 30-m data the potential misplacement is probably negligible, particularly with respect to the fact that the rest of the FS surface (vectors and polygons) comes with an margin of error at least one order of magnitude larger.On the other hand, since we know about the cause of this displacement and how to adress it in different ways, there's absolutely no reason not to use the correct values either. It's conceivable that add-on designers provide scenery with highly accurate vectors and polygons or, even better, that one day we'll have a generic tool that allows us to read high-quality source data directly into FS format. It would be prudent for us to provide the add-on meshes ready for those data.As for SRTM data, they are referenced as follows:"File names refer to the latitude and longitude of the lower left corner of the tile - e.g. N37W105 has its lower left corner at 37 degrees north latitude and 105 degrees west longitude. To be more exact, these coordinates refer to the geometric center of the lower left pixel, which in the case of SRTM-3 data will be about 90 meters in extent.(...)SRTM-3 files contain 1201 lines and 1201 samples. The one additional row of data at the north edge and one additional column of data at the west edge of each tile overlaps and duplicates data in the adjacent tile. SRTM-1 files contain 3601 lines and 3601 samples, with similar overlap."For any SRTM tile Microdem reports full degrees (e.g., N 37.0 W 105.0) as map extents, which to me indicates that Microdem reads the geometric center of each pixel! Doesn't this contradict what you point out about the NED files? Anyway, if resample.exe assumes the NW corner of a pixel, then the SRTM data will be offset by 1/2 pixel to the SE. In the case of 1-arcsec and 3-arcsec data the difference is less than 1 unit/pixel of the LOD they would be compiled at (e.g. 45-m offset vs. 76-m LOD9) but, again, there's nothing wrong with using the correct values.In either case, I suggest that the final word on the value of the lat/long coordinate pair in the .inf file should be based on a quality check in TMFViewer, with the corresponding HYL* or STR* files as overlay. This will quickly reveal any obvious and systematic displacement of the mesh relative to the FS vector/polygon data.Cheers, Holger

[rhumbaflappy 113]

Hi Holger.The SRTM one degree tiles available from:ftp://edcsgs9.cr.usgs.gov/pub/data/srtm/are HGT files and are identical in format to a BIL file. Each point is a 2-byte data representing elevation in meters.There is no offset. The point does not represent a pixel or a cell. It is an absolute height at an absolute point. The span of the cells is only the distance between equally spaced absolute points.The Seamless NED data from:http://seamless.usgs.gov/in BIL format is exactly the same type. The only confusion is that what you request is not necessarily what you will receive! The NW corner point may be different, and the cell span may also be different than what you'd expect.When creating the INF file for resample, the BLW file and the HDR file contain the info needed for the source. Don't use what you requested, as that is not what you received! The only remaining problem with that data is that it is in NAD83 format ( or NAD27 for Alaska ). If you can accept the Datum "as is" then resampling without reprojection is fine. As I showed in an earlier post, the errors caused by the incorrect datum are actually less than the errors in the data itself.=========================TIF files ( and GeoTiff ) may actually be different than BILs or DEMs. They may represent each pixel as a physical cell or span, and measure from the center of the pixel to the next. That would account for the problems encountered with that format. This would make sense as TIFs are a legacy from raster viewing programs.Converting a TIF file to a DEM or BIL may cause problems, as that data would definitely treat the points as absolute points... not cells. Manifold and MicroDem may be attempting to compensate for this difference in formats by moving the absolute point to the corner of the span when it saves to a different format. That would be unfortunate, as we all want WGS84 as the datum, and we need to load the BILs into a GIS program to change the datum... and the resulting save may become misalligned. This is really a problem for the Alaska NED data as I believe you pointed out in an earlier post.GlobalMapper also seems to suffers from the above problem. A possible solution would be to open the data as BIL format in MicroDem, change the datum, then save as MDEM. Then use BigBSQ to reconvert to the BIL ( BSQ ) format used by FS. BIGBSQ is available as:http://www.terrainmap.com/downloads/bigbsq.zipThis is speculation, but it would account for some problems. And this may be a workaround...avoiding the TIF and ARCGrid formats.==================An interesting note is that the seamless SRTM data is already in WGS84 datum, and would require only hole filling to make this data usable. One other problem with that data set, is that it has not yet been verified or corrected. The intention of the USGS is to do just that, then use the data as a basis for a new NED data set... hopefully in WGS84 datum.DickDick

[Holger 0]

Hi Dick:"There is no offset. The point does not represent a pixel or a cell. It is an absolute height at an absolute point. The span of the cells is only the distance between equally spaced absolute points."Interesting point. So you're saying that resample.exe reads the BSQ file as a height field and thus the height values provided will be placed at the center of each LODx grid vertex. If that's the case then there is no offset but it seems to me that people claim that that's not the way resample.exe works - or maybe I'm just a bit dense today x( ---"A possible solution would be to open the data as BIL format in MicroDem, change the datum, then save as MDEM. Then use BigBSQ to reconvert to the BIL ( BSQ ) format used by FS."I tried this in Microdem with the NAD27 Alaska data but it doesn't work as intended: after BIL import, Microdem incorrectly reports 'NAD83' in the Header file. Obviously, simply editing the Header won't do because that triggers Microdem's NAD83-to-NAD27 algorithm and it will rewrite the file. Steve provided me with information about how to change the header with a hex editor but I haven't tried that yet. Even if it works, that's not a very user-friendly approach :( Cheers, Holger

[Guest sgreenwood]

Hi John,Perhaps we should summarize this thread by returning to the original topic (the location of the NW corner of USGS NED seamless data) from another perspective. * No matter where your requested coordinates fall in the gray area surrounding the red dot, the red dot, and it's coordinates, are what are returned by the USGS site in the *.BIL and *.blw files.* The amount of positional error you introduce into the final mesh depends on how far your inf coordinates are from the coordinates of the red dot.* The potential magnitude of that error depends on the source resolution. (Assuming your inf coordinates lie somewhere within the gray area - otherwise all bets are off.)* The importance of the error is entirely subjective. Developers can use the above information to position their mesh more accurately. At least vis-a-vis the source data, which is at least somewhat representative of the real world, and the best information most of us have to work with.But there is no accounting for personal taste. :)Steve

[Guest sgreenwood]

Hi Holger,"Steve provided me with information about how to change the header with a hex editor but I haven't tried that yet."I didn't mean to suggest you should try changing the value there, only that it is a way to see what Microdem put there after the file was saved. This can be reassuring because Microdem displays everything in Map view using it's default format. I agree with your original opinion that is is very risky to change the header information under most conditions. Sorry for any confusion.Steve