OrthoXP for P3D tool and questions on grouping bgl files?

[theskyisthelimit 48]

Im curious for those that are creating new ortho using the new ortho4xp p3d tool.. (or converting old dds files to bmps and doing the conversions)..

How are you grouping the resultant bgl files that are produced.

They are produced in subfolders of the orthophotos folder.. under the zl label of say "ZL17" for a given tile.. so this means that for scenery.cfg a new entry is needed for every single tile (and to be able to make it so if you need to re-render a tile it takes less time to reprocess what was already done)..

If you are doing say 8TB worth of bgl files in hundreds of tiles, this means the scenery.cfg file will have hundreds of new line items (yes you can use sceneryconfig editor to help, but its still a bunch).. originally i thought about merging them into one folder but then it takes away re-render possibilities on problem tiles (and the average is at least 45 minutes with the resample time factored in, if the cmd window doesnt stick like they often do, another issue not to mention they steal focus from other windows while going on)..

Any experiences so far?

Thanks in advance

Edited October 22, 2018 by theskyisthelimit

[theskyisthelimit 48]

I'm guessing just leaving them as is and manually adding them is probably best.. it would be nice if sceneryconfigeditor supported drag and drop to create entries.. this would make the process easier.

[John_Cillis 2,406]

Bgl's can be combined in a scenery subfolder, as I have done with FSEarthtiles for years.  There is no need to create a scenery.cfg entry for every one.  Just copy and paste what is created to a common scenery subfolder and point the scenery.cfg to it.  I usually organize my work by state or region if I am working a smaller area.

John

Author, Landclass Assistant, Colorado Photoreal scenery for FSX and P3D

[klamal 282]

2 minutes ago, Cactus521 said:

Bgl's can be combined in a scenery subfolder, as I have done with FSEarthtiles for years.  There is no need to create a scenery.cfg entry for every one.  Just copy and paste what is created to a common scenery subfolder and point the scenery.cfg to it.  I usually organize my work by state or region if I am working a smaller area

Yep.  Exactly.  Just create a scenery directory somewhere.  For example, I plan to create a scenery folder for each continent I do.  So Europe\scenery and place all bgls for Europe in that Europe\scenery directory.  Then just add that single entry to your scenery.cfg file and done.

[theskyisthelimit 48]

1 hour ago, klamal said:

Yep.  Exactly.  Just create a scenery directory somewhere.  For example, I plan to create a scenery folder for each continent I do.  So Europe\scenery and place all bgls for Europe in that Europe\scenery directory.  Then just add that single entry to your scenery.cfg file and done.

Yep that would work, but as i think i mentioned, to redo a given tile will force the bgl's to be redone which wont live in the same space.. i guess i could just copy them and overwrite the big folder's bgls.. assuming it didnt remove some (then space is lost).. i dont know if a redo of a tile is 1 to 1 exactly as far as the count and naming of the bgls goes, with ortho4xp P3d version.

[klamal 282]

1 hour ago, theskyisthelimit said:

Yep that would work, but as i think i mentioned, to redo a given tile will force the bgl's to be redone which wont live in the same space.. i guess i could just copy them and overwrite the big folder's bgls.. assuming it didnt remove some (then space is lost).. i dont know if a redo of a tile is 1 to 1 exactly as far as the count and naming of the bgls goes, with ortho4xp P3d version.

True.  Good question.  I think if the zoom level and other properties are the same in Ortho4XP are all the same, it would be fine because it's basically a tile that should cover the same area on the planet and thus would have the same coordinates for the tile that gets created?  But I am not 100% sure.

[theskyisthelimit 48]

17 minutes ago, klamal said:

True.  Good question.  I think if the zoom level and other properties are the same in Ortho4XP are all the same, it would be fine because it's basically a tile that should cover the same area on the planet and thus would have the same coordinates for the tile that gets created?  But I am not 100% sure.

I think ill experiment.. create a tile with one given set and then the bgl conversion.. then change a parameter redo the tile (after moving the bgls) then create the new bgls and at least do a file count comparison/copy overtop and see if it matches up.  Either way i'm sure ill have to break these down by at least country (which is hard since tiles overlap).. because load times may be quite high after so many are added.

Edited October 24, 2018 by theskyisthelimit

[theskyisthelimit 48]

Well the first experiment at a big directory didnt go well at all.. not sure if i had a glitch or p3d just cant handle this size.. what i did was did a search for all bgl files from about 70 or so directories of ortho that were created.. did a copy to a new location.. at the end of the copy it prompted to overwrite 2000 files of the same name (unsure why), so i just hit cancel on that.. then added the new massive 900gb scenery directory in sceneryconfigeditor..

RIght away there was an issue.. some 10 or more minutes later it was stuck on indexing scenery.. 99%.. so i had to kill the task.. restart p3d.. this time it got past that point but got stuck at 6% terrain building.. had to kill it again (this after 1 hour too)..

Finally on the 3rd attempt it did load but took at least 14 minutes or more.. whereas without this directory it was 2.

So either i did in fact have something mangled on the copy or maybe duplicates... or this simply wont work, at least not in one big directory, which in the end would be 4-12TB in size eventually (900 now).. i may retry this putting separate entries for each tile.

Has anyone tried anything this large and succeeded?

 

[theskyisthelimit 48]

Well i think i had some malformed folders.. went through making sure the tiles\terrain folder count matched bgl count for each one, weeding them out.. instead i manually put each entry in for this test..

Still 937gb.. it took 3min 10 sec to load the same area that normally took 2min 15sec with ortho off (this area having ortho tiles) (about 55 seconds longer)

trying an area without ortho tiles before/after:

with ortho disabled in scenery.cfg:  tried kfll as an example:  2min 0 sec

Then with ortho back on, same area (but no ortho for this area):  2min 30sec (only 30 sec longer)

 

It appears there is less of a hit if the area you are loading doesnt have tiles already made (and ortho is enabled in scenery).. Not sure how this will pan out with say 4 or more times the data, but i'm going to keep going and find out at least to the 2TB point and maybe group by country/continent.

I should mention i have the ortho on a 7200 drive and the OS/main P3D drive are SSDs.

 

 

Edited October 30, 2018 by theskyisthelimit

[cnc3 14]

I don't alter the scenery.cfg file at all. Instead I use the addon.xml method.

I group my BGL files by lat/lon since that is native to the ortho4xp program. So, for example, I will have folders for +50-119, +50-120 etc. I could put these all under a 'North America' or 'Canada' or 'British Columbia' folder if I wanted to (or all three in a hierarchy). The bgl files are located in the scenery folder under the lat/lon folder. That is, +50-119 will have a folder named 'scenery' and the scenery folder will contain the BGL files.

Then I simply add these to an add-on.xml file located in the '...\Documents\Prepar3D v4 Add-ons\Ortho Scenery' folder. The add-on.xml file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<SimBase.Document Type="AddOnXml" version="4,0" id="add-on">
  <AddOn.Name>Ortho Scenery</AddOn.Name>
  <AddOn.Description />
  <AddOn.Component>
    <Category>Scenery</Category>
    <Path>G:\Ortho Scenery\+50-119</Path>
    <Name>Ortho Scenery - +50-119</Name>
    <Layer>301</Layer>
  </AddOn.Component>
  <AddOn.Component>
    <Category>Scenery</Category>
    <Path>G:\Ortho Scenery\+50-120</Path>
    <Name>Ortho Scenery - +50-120</Name>
    <Layer>301</Layer>
  </AddOn.Component>
</SimBase.Document>

If I decide to regen a tile in ortho4xp, I simply shift-click on the desired tile, select the new zoom level and regenerate. The I replace the existing lat/lon BGL files with the new ones.

[theskyisthelimit 48]

1 hour ago, cnc3 said:

I don't alter the scenery.cfg file at all. Instead I use the addon.xml method.

I group my BGL files by lat/lon since that is native to the ortho4xp program. So, for example, I will have folders for +50-119, +50-120 etc. I could put these all under a 'North America' or 'Canada' or 'British Columbia' folder if I wanted to (or all three in a hierarchy). The bgl files are located in the scenery folder under the lat/lon folder. That is, +50-119 will have a folder named 'scenery' and the scenery folder will contain the BGL files.

Then I simply add these to an add-on.xml file located in the '...\Documents\Prepar3D v4 Add-ons\Ortho Scenery' folder. The add-on.xml file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<SimBase.Document Type="AddOnXml" version="4,0" id="add-on">
  <AddOn.Name>Ortho Scenery</AddOn.Name>
  <AddOn.Description />
  <AddOn.Component>
    <Category>Scenery</Category>
    <Path>G:\Ortho Scenery\+50-119</Path>
    <Name>Ortho Scenery - +50-119</Name>
    <Layer>301</Layer>
  </AddOn.Component>
  <AddOn.Component>
    <Category>Scenery</Category>
    <Path>G:\Ortho Scenery\+50-120</Path>
    <Name>Ortho Scenery - +50-120</Name>
    <Layer>301</Layer>
  </AddOn.Component>
</SimBase.Document>

If I decide to regen a tile in ortho4xp, I simply shift-click on the desired tile, select the new zoom level and regenerate. The I replace the existing lat/lon BGL files with the new ones.

Yeah i've decided to go with this structure, no merging of bgls..

I was actually about to do a comparison with scenery.cfg and an addon xml option.. however.. thinking about this.. isnt there a limit to the number of entries for the addon xml method?  Ill still give it a go though.. I think the Lorby Tool will make this very easy to import a ton at once which i had forgotten (or convert the scenery entries i've already made).

 

 

[theskyisthelimit 48]

I thought i was having good results with them in separate folders.. well.. if i reboot and start up again.. 9 minutes later still at 6% loading terrain data.. process monitor shows files being read are still drilling through the ortho folders..

what tends to happen is if i kill it and try a few more times even after 5 or more minutes it loads.. then subsequent loads take around 2min50 to 3min20 with the amount i have going on (933GB)..

I'm not sure what to make of this

 

[cnc3 14]

14 minutes ago, theskyisthelimit said:

I thought i was having good results with them in separate folders.. well.. if i reboot and start up again.. 9 minutes later still at 6% loading terrain data.. process monitor shows files being read are still drilling through the ortho folders..

what tends to happen is if i kill it and try a few more times even after 5 or more minutes it loads.. then subsequent loads take around 2min50 to 3min20 with the amount i have going on (933GB)..

I'm not sure what to make of this

 

The first load of new scenery always takes a long time for me too. I suspect that Prepar3D is doing some indexing and merging of the scenery.cfg and add-on.xml files during this pass. If you look at the scenery_add-ons.xml file, you will see that it is generating CRC (cyclic redundancy check) signatures for each SceneryComponent. Equally, I suspect that the CRC generation is single threaded but I have not brought up the monitor to check on that. Even if is it multi-threaded, CRC requires the reading of every BGL file in the folder and performing a calculation on each element. Not having to regenerate the CRCs would explain the lower loading times on subsequent runs.

[theskyisthelimit 48]

13 minutes ago, cnc3 said:

The first load of new scenery always takes a long time for me too. I suspect that Prepar3D is doing some indexing and merging of the scenery.cfg and add-on.xml files during this pass. If you look at the scenery_add-ons.xml file, you will see that it is generating CRC (cyclic redundancy check) signatures for each SceneryComponent. Equally, I suspect that the CRC generation is single threaded but I have not brought up the monitor to check on that. Even if is it multi-threaded, CRC requires the reading of every BGL file in the folder and performing a calculation on each element. Not having to regenerate the CRCs would explain the lower loading times on subsequent runs.

If thats the case.. it would be nice to know what gets flushed on a reboot.. so this could maybe be preserved to speed things up the next time.. ill take a look in there and see... 

I went through the trouble of moving everything to an addon xml.. of course this didnt help on load times.. another run after this.. 5 minutes in, still loading only splash screen up so far (i chose to purge the scenery index when i exited the lorby addon manager too).

[theskyisthelimit 48]

I see the CRC's.. i rebooted looking to see if these files get deleted.. they did not.. unless there is some temp files somewhere.. then launched again and the painful first load.. by painful on mine for 65 folders / 933gb its 14minutes 50 second on first load, then second load around 3min ... 15 minutes seems excessive