Jump to content
Sign in to follow this  
tonywob

Speed up and further development for OSM2XP

Recommended Posts

OSM2XP is a great piece of software which is used to generate X-Plane scenery from OpenStreetMap data. 

 

I know the simheaven website provides scenery for OSM, which also supports multipolygon forests amongst other things, however for people who like to play round with osm2xp and also test their own changes without waiting for a new download. I've been playing round with a few scripts to try and speed up the processing and add in the forest relations, and I've managed to drop the processing time down for the british-isles scenery from about 4 hours to 30 minutes. I'm adding the instructions here for anyone who is clued up on using the command-line, and hope it will be useful. The osmconvert and osmfilter tools are available freely from the osm wiki site.

 

Firstly convert the downloaded osm.pbf file to o5m using osmconvert

 
osmconvert --drop-author british-isles-latest.osm.pbf -o=british-isles-latest.o5m
 
Then run it through a filter, removing all the stuff osm2xp doesn't use, e.g. roads, names, postcodes, etc.
 
osmfilter british-isles-latest.o5m --drop-tags="addr:city= source= source_ref= created_by= addr:housenumber= addr:postcode= addr:post_code= addr:street= phone= amenity=parking name= building:house_number= building:house_name=" --keep="building=* amenity=place_of_worship man_made=lighthouse man_made=water_tower man_made=crane man_made=windmill landmark=windmill man_made=tower man_made=mast man_made=chimney landmark=chimney generator:source=wind natural=wood natural=scrub landuse=forest landuse=allotments natural=forest leisure=garden leisure=park landuse=cemetery" --drop-relations --out-o5m > british-isles.o5m
 
Then, I process the file again to pull out forest relations
 
osmfilter british-isles-latest.o5m --keep= --keep-relations="type=multipolygon and ( landuse=forest or natural=wood or natural=forest of leisure=park or leisure=garden )" -o=forests.osm
 
I then use the following simplistic ruby script to add forest tags to the forests.osm file. It seems to do the trick and gives more woodland, although probably not correctly as it applies the tag to all items.
 
f = File.open("woods1.osm","w")
File.open("forests.osm", "r").each_line do |line|
  data = line.gsub("</way> "," <tag k=\"landuse\" v=\"forest\"/>\n</way> ")
  f << data
end
f.close
 
Finally, I combine the forests.osm file with the british-isles.o5m file, and convert it back to the original pbf file (ready for osm2xp)
osmconvert woods1.osm -o=woods1.o5m
osmconvert --drop-author woods1.osm british-isles.o5m -o=british-isles.osm.pbf
 
The whole process above takes about 40 seconds and is all in a single script so it's automated. On the entire europe download, it takes about 15 minutes to run, but reduces the filesize down quite significantly, and as well as adding forest relations (i.e. More forests), it also massively decreases the processing time inside osm2xp.
 
Unfortunately it dosn't seem anyone is actually developing osm2xp further, and currently the following features IMO are needed to move it forward:
  1. Support for relations/multipolygons, i.e. For large forest areas or buildings with inner court yards.
  2. Support for advanced options, such as changing the building facade or object based on a number of criteria (not just a single tag). e.g. You might want a specific phone mast with a height tag to use another object file. Or for houses under a certain height to use a specific facade.
  3. Scripting. I would like to be able, for example, to tell osm2xp to only add streetlights to roads if the road is in a residential area. This would require the program to know that the current road is inside a residential zone, etc and isn't a trivial thing.

I've had a look at the osmosis c library for OSM, and have written a very simple program to run through an OSM file and fetch forests and woodlands and produce X-Plane DSF files. It also deals with relations and multi polygons, and also forests crossing boundaries. Osmosis is a pretty powerful library and could easily (for someone with the time), be programmed to create some pretty advanced scenery for X-Plane. I'm really hoping development picks back up on osm2xp, because the OSM data has so much potential in well populated areas, and the libraries available are improving everyday.

 

Couple more ideas I will try on my next europe OSM run,

 

1) Only extract residential streets and motorway junctions, and then OSM2XP will only generate street lights just for those. This should make it look more realistic in some instances, where only roads with houses and motorway junctions are lit. This will slow processing down a little bit, but should give a nice effect at night.

2) Add small shrubs to the landuse=residential (treat it as a forest with very low density). This should add the odd small tree in residential areas aka. like orbx do. However, the chances of a tree being inside a house are high, so it might look odd. But worth a try.

3) Often shrubs and trees and found along drains/ditches. Maybe, I can split these up into their individual nodes using osmfilter and place a tree at each node. 

 

 

 

 
 
 

Share this post


Link to post
Share on other sites

Since development on OSM2XP seems to have stalled (Last change was back in July), I decided to see if there is anything I can do get multipolygons and polygons crossing tile boundaries working. Unfortunately I had problems getting the code to run and compile on my Mac, and it would require some work to get it to support the requested stuff. So I have started to write my own tool, concentrating just on forests at the moment.

 

In the below screengrab from OSM, there are a few forests shapes, two of them have holes in the middle, and the border shading indicates DSF tile boundaries inside X-Plane. When using OSM2XP, only the small square forest on the right would work, the others would not show up (after trying various things).

 

Screen Shot 2014-01-23 at 16.05.39.png

 

The tool I'm working on takes a different approach than OSM2XP, in that it makes several initial passes of the OSM file to get geometry information. X-Plane can handle multipolygons for forests, but for items on tile boundaries I had to break the multipolygons up into smaller shapes, which seems to produce the desired effect:

 

Screen Shot 2014-01-23 at 16.03.26.jpg

 

So far, the tool does the following:

 

  • Supports multipolygons and relations in OSM (Lots of forests in Europe are done like this). 
  • After the initial passes of the OSM file, only the tiles that are needed are generated. Empty tiles (without anything drawable in X-Plane) aren't processed. Speeding up processing time for large maps.
  • Allows advanced rules, for example you can determine what type of forest to draw based on a number of tag combinations. This can also be used for using different models for different types of churches, masts, etc based on height, and other criteria.

There is still a lot of work required to get it user friendly and configurable, but hopefully it will come in useful for scenery developers in the future who need much more configurability than currently allowed.

 

Next challenge is multipolygon buildings, which, although the specs say aren't supported, it seems to work :-)

 

 

Share this post


Link to post
Share on other sites

I've now added initial building support in, and have generated central Berlin:

 

Screen Shot 2014-01-25 at 15.24.00.jpg

 

Screen Shot 2014-01-25 at 15.24.26.jpg

 

Screen Shot 2014-01-25 at 15.22.20.jpg

 

Berlin is a good example as there are many buildings with holes, but also lots of the bulidings have detailed information attached to them, e.g. height, material, color. Also, lots of the parks and woodland are built up of multipolygons which have come across quite nicely.

 

However, buildings with more than one hole inside don't render too well inside X-Plane, so at the moment I triangulate them (Which works, but looks ugly). I need to find a better way to do this, so possibly splitting the building into 2 or 3 parts, removing the holes may work (but is complex to do).

 

The OSM data really could do with a cleanup, lots of areas are unclosed, e.g. Forests, or have duplicate coordinates which makes parsing fun.

 

 

Share this post


Link to post
Share on other sites

Man that is some serious stuff! You software engineers never cease to amaze me.

Could your process be able to generate something like say Monument Valley Utah?

Share this post


Link to post
Share on other sites

 

 


Could your process be able to generate something like say Monument Valley Utah?

 

It only generates buildings, trees and points of interest, and not actual terrain. Also, the US is a bit behind Europe in terms of information on OpenStreetMaps. So whilst some areas such as Chicago and New York are really well populated, other cities are almost completely empty. It's improving everyday though :-)

Share this post


Link to post
Share on other sites

Hey man you doing great, seriously too good.


Ryzen 5 1600x - 16GB DDR4 - RTX 3050 8GB - MSI Gaming Plus

Share this post


Link to post
Share on other sites

Tony, I figured if it could make a 3D building out of a flat image then it might be able to generate the individual mesa's in Monument Valley since it is not an entire terrain, just individual models.

Monument valley is an area that has never been done very well in any version of FS. The photoreal stuff is ok but the terrain detail needed to accurately model the mesa's is not there since they are small compared to say a mountain. They are more like extremely large buildings.

Share this post


Link to post
Share on other sites

You could in theory create yourself a facade for a mesa and then map them in OSM using whatever tag is appropriate for it.. You'd set the height, and then the program would draw a solid object to fill the flat polygon you drawed. But, I'd imagine it would look bad, since it would have to be completely vertical, with no slanting edges.

 

The other approach is to model a couple of mesas as 3D objects using the .obj format (Which is pretty easy to do), and then they could be inserted into the scenery. I'm not sure if the mesas are tagged in OSM, but if they are, then the program could be told to insert your object whenever if finds a mesa. But this all depends on how well the information is tagged and the range of models you have.

Share this post


Link to post
Share on other sites

Ahh now I understand why, the slanting edges and shapes kill it.

 

Thanks Tony!

Share this post


Link to post
Share on other sites

Some more screenshots. Reducing the number of red roofs, and replacing some of the facades with real objects.

 

 

1.jpg

2.jpg

3.jpg

Share this post


Link to post
Share on other sites

Ooops wow.

 

Wouldn't it be possible to import google warehouse based on geo location, i know you development does not concern google warehouse but that stuff looks neat  ? Sorry i am asking this as i don't have knowledge about the osm or scenery things don't understand graphics. 


Ryzen 5 1600x - 16GB DDR4 - RTX 3050 8GB - MSI Gaming Plus

Share this post


Link to post
Share on other sites

Yes it could, but the problem with Google Warehouse objects is that each object has multiple texture files, whereas X-Plane only supports one object with one texture sheet. There is a converter for these objects, but it requires the textures to be sorted into one file and the UV coordinates remapped. I don't think there exists an automated way of doing this yet.

 

Also, somebody has already tried this (http://members.ferrara.linux.it/cavicchi/GMaps/). What he does is create multiple objects for each face/texture. Although it works (sometimes), the objects are really bad on the framerates, and huge. Generation also takes forever.

 

Never say never though, maybe one day :-)

Share this post


Link to post
Share on other sites

Tony, first of all I just want to say your works looks really awesome :D!

 

I wondered what kind of objekts does your script link into the generated scenery, just the X-Plane Standard ones or also objects from librarys like OpenSceneryX? Using objects from OpenSceneryX could make the impression much more realistic since most american buildings don't really fit into Europe or even Asia :D.

 

Also it would be really cool if you could tell us what your plans with this tool are? Are you thinking of earning money with it (by selling generated scenerys) or do you consider publishing it as open source on a platform like github (which would enable other coders to send in patchs to support other tags in OSM for example)? The later one would be really awesome and I think this tool could become a very powerful scenery generator to make x plane even more awesome ;).

Share this post


Link to post
Share on other sites

Yes it could, but the problem with Google Warehouse objects is that each object has multiple texture files, whereas X-Plane only supports one object with one texture sheet. There is a converter for these objects, but it requires the textures to be sorted into one file and the UV coordinates remapped. I don't think there exists an automated way of doing this yet.

 

Also, somebody has already tried this (http://members.ferrara.linux.it/cavicchi/GMaps/). What he does is create multiple objects for each face/texture. Although it works (sometimes), the objects are really bad on the framerates, and huge. Generation also takes forever.

 

Never say never though, maybe one day :-)

 

 

Hey, I used to try ModelconverterX and scenproc for Fsx had only succeeded once :) that was last year, don't how it had worked. Maybe you give that a try. Of not scenproc but Modelconverterx.


Ryzen 5 1600x - 16GB DDR4 - RTX 3050 8GB - MSI Gaming Plus

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Tom Allensworth,
    Founder of AVSIM Online


  • Flight Simulation's Premier Resource!

    AVSIM is a free service to the flight simulation community. AVSIM is staffed completely by volunteers and all funds donated to AVSIM go directly back to supporting the community. Your donation here helps to pay our bandwidth costs, emergency funding, and other general costs that crop up from time to time. Thank you for your support!

    Click here for more information and to see all donations year to date.
×
×
  • Create New...