Hybrid photogrammetry areas

[Raymond.Groenendijk 255]

Just a thought. As it seems from what we've seen so far there will be areas filled with detailed 3D objects created from photogrammetry (like Chicago, Seattle and San Francisco) and other areas that will be filled in with plain aerial images and a building/roads/object overlay based on other data (similar like X-Plane does with Ortho4XP). Now technically it should be possible to substitute flat aerial images with 3D objects created from photogrammetry once you've got enough data by using machine learning.

A warehouse will have a distinct look from the skies for example, as do silo's, trees, parked cars/buses and certain buildings. Take the distinct red roofed building blocks with a patio on the inside in Barcelona for example, these occur all around the wide area even where the photogrammetry data ceases to exist (at least in Earth View). A machine learning algorithm should be able to recognize these and replace them with a match from the nearest photogrammetry set. If you combine this technique with building height data you might even let AI create realistic looking tall buildings. 

Now it won't look as good as the areas that are fully created by photogrammetry data, but it will probably look a whole lot better than the generic X-Plane or autogen approaches. 

We don't know if the MSFS team is already using similar techniques, but it might be a nice solution to fill in the large gaps between photogrammetry areas at release or further down the road. The scenery system can always fall back to generic buildings or even autogen when there is just no data.

[bonchie 2,305]

You can see that Dubai in the trailer is using some kind of technique like you describe. It's not photogrammetry and the buildings appear to be generic, but the heights are somewhat right and they are in the right places. Everything is on top of a satellite image. 

They could be hand placed, but it doesn't look refined enough IMO. 

My speculation is that they've found a way to use AI to build some of these areas out in a plausible way. 

Edited August 14, 2019 by bonchie

[Raymond.Groenendijk 255]

1 hour ago, bonchie said:

You can see that Dubai in the trailer is using some kind of technique like you describe. It's not photogrammetry and the buildings appear to be generic, but the heights are somewhat right and they are in the right places. Everything is on top of a satellite image. 

They could be hand placed, but it doesn't look refined enough IMO. 

My speculation is that they've found a way to use AI to build some of these areas out in a plausible way. 

Ah you're right, they do seem to match the shapes that you can see when you zoom in on Bing maps (map view). This means that they probably use a technique similar to the OSM (OpenStreetMap) approach X-Plane uses. It's a nice technique to get some reasonably looking scenery, but it will look a bit generic in contrast to photogrammetry scenes. 

I do hope they will be able to leverage machine learning to create a more hybrid solution between the X-Plane approach and photogrammetry in the future. That way they should at least be able to recognize trees and cars on aerial images and alter the ground mesh based on these to get a more cluttered and detailed look. They should be able to improve this down the line when more data becomes available just like Google does with Maps. 

Anyway, I'm really curious to what they will be able to do with machine learning and AI in the near and further future. But just having streaming scenery with generic looking buildings will be such an improvement over what we have now, especially with the lighting and clouds they've showed us. It would be a dream to be able to just go and fly somewhere just to discover new places. 

[Christopher Low 5,353]

Photoscenery with generic and accurately placed buildings and trees would be a massive improvement over FSX and P3D. I really hope that they are able to do this for the entire planet (if this Azure AI is all that it is cracked up to be), because it would be revolutionary for the PC flight simulation world.

[threexgreen 3,139]

There would still need to be a way to achieve good looking seasons, though. One of the major drawbacks of current photoscenery approaches.

[Raymond.Groenendijk 255]

20 minutes ago, threegreen said:

There would still need to be a way to achieve good looking seasons, though. One of the major drawbacks of current photoscenery approaches.

Yeah that's really hard to do. You might get a little progress in that direction with color filters on the landscape mesh, replacing green with more brown-like colors. Machine learning might be able to differentiate between grass fields and trees, leaving the grass green and making the trees brown. It might even be able to differentiate between loaf and pine trees (to leave the latter green during winter). 

Snow simulation is also possible as the scenery system knows where roads are, so they could be left partially open and for buildings only the roofs will need to get snow on top of them. 

Winter with bare trees is hard to recreate in photogrammetry scenes as it transforms the actual shape of the trees. 

[dtrjones 356]

4 hours ago, Christopher Low said:

Photoscenery with generic and accurately placed buildings and trees would be a massive improvement over FSX and P3D. I really hope that they are able to do this for the entire planet (if this Azure AI is all that it is cracked up to be), because it would be revolutionary for the PC flight simulation world.

The technique is of course used by several scenery companies to create buildings on top of satalite imagery - you can see this technique quite clearly used in Orbx TrueEarth the white town center buildings in the UK for example all appear the same colour and height but quite obviously follow the floor pattern on the satalite image which are often an odd shape.

For sure MS could include this in the photogrammetry stream but Kevin Miller has already highlighted clear areas that use the more classical created landclass textures and autogen scenery so we know that exists as well, and then there's the 3d grass.

Theorectically yes you could create the whole earth like this, the processing to create the mesh, objects etc would be done upfront and stored in the cloud, but you'd need to stream this to a PC online as these areas don't really work that well without the context of the expensive satalite imagery underneath.So given we know Microsoft have already used alternative rendering techniques it makes this unlikely to be seen for the entire planet.

[dtrjones 356]

4 hours ago, threegreen said:

There would still need to be a way to achieve good looking seasons, though. One of the major drawbacks of current photoscenery approaches.

Unless you keep the photoscenery as it is and change everything else...

 

[Christopher Low 5,353]

There is a risk in using both techniques. Some of those "3DM" buildings do not look all that great (IMO) in the ORBx TrueEarth GB South package, but then "generic landclass and autogen" does not exactly inspire me with confidence either. I would really prefer to get away from that approach if at all possible. Generic terrain textures and autogen (IMO) rarely provide a truly convincing landscape (and the ORBx FTX scenery is a good example of this; it is better than default, but urban areas in particular still look nothing like the real world).

Edited August 15, 2019 by Christopher Low

[Paraffin 2,197]

5 hours ago, Raymond.Groenendijk said:

Yeah that's really hard to do. You might get a little progress in that direction with color filters on the landscape mesh, replacing green with more brown-like colors.

Or the reverse, from brown to green if the AI knows what should be there at a given time of the year.

One of the few things I don't like about the Orbx TE Washington scenery is that every cleared area is brown. The overall look of the state is brown, brown, brown in areas not covered with forests or other vegetation. That's what it looks like now in mid-August, because it's the dry season out here in coastal WA where I live, and the satellite imaging is done when the skies are clear. But it only looks like that for 3 months! The other 9 months it's green everywhere.The state motto is "The Emerald State" after all. It's depressing to see this area not looking properly green for most of the year.

I don't blame Orbx for what they had to work with. It would have required a ton of work to hand-edit all the underlying images for the seasonal change. But this is one area where default landclass in XP11 actually looks better for most of the year (although the autogen and custom builds look much better in Orbx TE). It's the reverse situation where the landclass textures don't shift to brown in the summer, but it could do that with procedural programming for seasons. Which is why I still lean a bit more towards the landclass method for scenery instead of satellite orthos. It will be interesting to see how the new MSFS handles this.

[F737MAX 5,603]

Flat photo scenery / ortho without 3D objects is dead - that tech is at least 10 years old (probably 15 now) and we've seen products that use machine recognition to place objects on their correct photo footprint for at least the past 2 years.
The initial trailer footage and photos of Dubai and LAX give me a lot of confidence that 'hybrid' photo scenery plus generic 3D buildings is in use.

On 8/15/2019 at 11:20 AM, Raymond.Groenendijk said:

Winter with bare trees is hard to recreate in photogrammetry scenes as it transforms the actual shape of the trees. 

If Azure AI can recognise trees, then all you need is a lookup library tree object that changes each season. That's not that dissimilar to what has been achieved by Turbulent Design's TerraFlora and Orbx's Global Trees.

20 hours ago, Christopher Low said:

There is a risk in using both techniques. Some of those "3DM" buildings do not look all that great (IMO) in the ORBx TrueEarth GB South package, but then "generic landclass and autogen" does not exactly inspire me with confidence either. I would really prefer to get away from that approach if at all possible. Generic terrain textures and autogen (IMO) rarely provide a truly convincing landscape (and the ORBx FTX scenery is a good example of this; it is better than default, but urban areas in particular still look nothing like the real world).

Completely understandable. That said, the landclass in the new sim appears to be a whole lot better than what we currently have in P3D - I loathe flying near landclass mountains, cliff faces and cities as the textures look far too generic for my liking.
The new sim appears to have done much better with the texture sets.

As much as I want to have the entire globe covered by photo scenery, the lack of quality definition in large parts of the world means that landclass will be the only suitable alternative until such time as the imagery improves.

18 hours ago, Paraffin said:

Or the reverse, from brown to green if the AI knows what should be there at a given time of the year.

One of the few things I don't like about the Orbx TE Washington scenery is that every cleared area is brown. The overall look of the state is brown, brown, brown in areas not covered with forests or other vegetation. That's what it looks like now in mid-August, because it's the dry season out here in coastal WA where I live, and the satellite imaging is done when the skies are clear. But it only looks like that for 3 months! The other 9 months it's green everywhere.The state motto is "The Emerald State" after all. It's depressing to see this area not looking properly green for most of the year.

This is the biggest problem I had with the Horizon / Playsims UK VFR photo scenery. Aerial photos were taken in late July, August and early September for the best weather, yet the countryside was golden yellow, brown and yet more brown for vast swathes of farmland at harvest time - that's not right for about 10 months of the year.

Shader technology will be the solution. Rather than 4 or 5 individual seasons' worth of textures, original source images ought to have colour filters applied to match the appropriate time of the year.
Where this might work really well, is that as it is just applying numerical values to the colour filters, you could define early Spring, late Spring, height of Summer, mid-Autumn, Winter, deep Winter. Forget 4 seasons, you could have a different colour palette based on which week you choose to fly in.

[nickhod 425]

It'll be interesting to see how MS handle areas that don't have photogrametry (which is actually most of the Earth, they are way behind Google in that respect).

You can break down creating a non-photogrametry building into its various components:

Footprint: Bing maps has great building footprint data. Much of this machine learning generated, not hand edited. There should be no problems with the missing building areas you experience with OSM data.

Height / Floors: OSM data is limited in this respect, so Lidar data is usually the way forward for cities. It can be challenging to tie up lidar data to exact buildings though.

Roof: Limited OSM data. This is normally region specific; British houses have different roofs to Japanese houses. They could do something smart and get the roof texture from the satellite imagery. That would be really interesting.

Texture: Without photogrametry you're limited to stock textures, unless, again, they did something smart and grabbed a texture from a nearby photogrametry model of a similar building type. (As @Raymond.Groenendijk says)

My other thought is whether the SDK lets developers bring in their own photogrametry data. If so, expect to see a Google maps photogrametry add on by a 3rd party developer.

[scottb613 1,188]

On 8/15/2019 at 5:50 AM, threegreen said:

There would still need to be a way to achieve good looking seasons, though. One of the major drawbacks of current photoscenery approaches.

Hi Folks,

While I haven't messed with scenery design in a while - it's pretty easy to convert satellite images into seasonal textures with image editing software... Here the base image was in summer - but I color matched everything to GE Pro seasons - Spring, Summer, Autumn, Winter, and Hard Winter... Hard Winter took a few extra edits - aside from color matching  - to make snow - but it was relatively easy with great looking results... I'm sure it would be possible to automate this on a large scale given the resources MS has... When loaded in the sim it was nearly impossible to see where the satellite imagery ended and the landclass tiles began... 

 

Summer

 

Autumn

 

Hard Winter

 

Regards,
Scott

Edited August 16, 2019 by scottb613