Sign in to follow this  
Holger

Downloading Data

Recommended Posts

I've been trying to download NED data from the USGS seamless data distribution site (http://seamless.usgs.gov/), but I have a very low success rate because quite often a download will start and then timeout. Sometimes I'll only get 1% or 2% downloaded, sometimes as much as 92%. But only occasionally will I get a completed download.I have DSL, and when the files are downloading I get an average rate of about 138 kB/s. (That's pretty good; or at least good enough to not induce any timeouts.) Anyway, out of the clear blue, data stops coming in and I have to go back and try the download again -- where the frustration is compounded because the data has to be extracted and archived yet again before the doomed download even starts again. It's a waste of bandwith for me and a waste of bandwidth and CPU time for the USGS.Does anyone have any better luck than I do? Is there anything that can be done to improve this process? There's no way to resume a download -- not that I'm aware of.Maybe I'm old school, but I'd rather FTP than use their new view map thingie. I know you can get SRTM via FTP, but does anyone know if other formats are available that way, too?Thanks in advance for your input,

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

Hi John,I haven't downloaded data in the last few weeks, so there may be a problem I am unaware of. But I have probably downloaded far more data from the site than anyone else in the sim community, with virtually no problems. I started back when we had to place orders, receive e-mail confirmation with an ftp address, and download with ftp. But the new system is much more convenient. Performance does depend on time of day, however, especially on weekdays. By 7AM EST I can begin to detect a slowdown in response. As the day progresses and more of the country (US) begins the workday, the delays increase exponentially so that by noon EST, the delays are quite apparent. By late afternoon performance can fall to 1/3 that of early morning hours, and I begin to experience failed transfers and other error messages. Time of day matters! (Other factors you may want to look at include your FTP software and TCP/IP settings, but these are not very likely to cause major problems.)I find the most effective way to request data is to use the "Define Areas By Coordinates" option, below the blocks of icons on the left side of the Map screen. This allows you to request data quite precisely, based on the LOD you will be using, minimizing downloaded file sizes.Tip: If you do get to the point where you specify areas based on LOD coverage, plan carefully. If there is any doubt about the LOD you will use, specify the areas based on the LOWEST LOD you may use. You can always use a higher LOD with the same source data, with some overlapping of mesh resulting. But If you try to create mesh with a lower LOD from source data requested assuming use of a higher LOD, Resample may leave gaps you cannot fill easily. (Especially along coastlines, where there is no data for the adjacent offshore region. Large percentages of islands can be lost this way.)Stevewww.fs-traveler.com

Share this post


Link to post
Share on other sites

Thanks for your input.I've tried downloading mainly at off-peak times, typically late at night on the west coast. There was one section I tried downloading last night, and I gave up after five or so attempts -- each attempt getting less than 1 KB downloaded before stopping. After a few more attempts this afternoon, I finally got the whole thing. Now my next block just stopped at 7%. I guess I'll contact the USGS and let them know of my problems.While I have your attention -- how do you convert BIL files to MicroDem? I know that MicroDem can read the GeoTiff files, which are larger, but I'd rather deal with BIL files. That reminds me -- please check out another thread I'm starting regarding MicroDem's spacing problems with NED data.Thanks again,

Share this post


Link to post
Share on other sites

Hi John,I don't know what's causing your problems. Did you request the data using coordinates as I suggested? If you are using an FTP program, you might try another. You might want to check out www.dslreports.com for info re DSL and tweaking your TCP/IP settings as well.BIL files are certainly more convenient! I've had mixed success with loading them directly into Microdem, and since identifying the problem with coordinates I've stopped trying. I wrote my own software to help with mesh creation and it included the ability to read Microdem DEM files and convert them to SDK format. It was pretty easy to add another module to reverse the process and convert the BIL data into a Microdem DEM, with the correct coordinate information. Some day I may take the time to pull out this code and make a stand-alone application others can use, but I can't do it any time soon. :(Sorry I can't be more helpful here. Steve

Share this post


Link to post
Share on other sites

Hi John and Steve,just two points that I assume you know about but haven't been mentioned here:1. I don't know why these timeouts happen to you, John, but if you bookmark the Raster Data Extraction Summary Page (for each different block of data) all you have to do is reload that page and try the download again. Good luck!2. The BIL format can be read directly by FS resample.exe. If you do that you just have to follow Steve's advice about downloading a block that is of the correct extent for your desired LOD (Steve has a LOD calculator utility on his website that helps with figuring out the needed lat/long). One caveat about the direct use of BIL data: it doesn't work for Alaska because the NED data have a different datum (NAD27 instead of NAD83).Cheers, Holger

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this