Removing baked shadows in Bing images by AI?

[GlideBy 324]

10 minutes ago, Claviateur said:

Maybe I am mistaken, the AI does not only fetch the data (similar to OSM) but has image processing and 3D generating capability in its deep learning process. 

The devs have stated that Azure is used for buildings and trees.  Here is a video where they talk about using Azure to create the buildings.  https://youtu.be/Q4pKRjNhQKs?t=430

 

Edited February 20, 2020 by GlideBy

[Claviateur 789]

1 minute ago, GlideBy said:

The devs have stated that Azure does building and trees.  Here is a video where they talk about using Azure to create the buildings.  https://youtu.be/Q4pKRjNhQKs?t=430

 

Yes I saw this video and I just heard again his explanation. I think he is saying exactly what I explained in my post above, Azure AI analyzes and brings the data. I don't think he said the AI creates the assets.

You know, if the Azure AI retouches imagery and creates 3D elements over a map, MS would have already used it to make Bing imagery seamless and have worldwide 3D infrastructure, forests, powerlines etc etc....And score some points against Google Maps 🙂

 

[GlideBy 324]

3 minutes ago, Claviateur said:

I don't think he said the AI creates the assets.

Here are links, and articles showing that Azure and/or AI to create building footprints.  These are not specific to MSFS though.

https://azure.microsoft.com/en-us/blog/how-to-extract-building-footprints-from-satellite-images-using-deep-learning/

https://github.com/microsoft/USBuildingFootprints

https://blogs.bing.com/maps/2018-06/microsoft-releases-125-million-building-footprints-in-the-us-as-open-data

In other interviews the MSFS devs have talked about how they can know a buildings height based on the shadow.  There was another interview where they mention about Azure creating the buildings and trees. 

So all that, and more which I am not going digging for, leads me to believe they are using AI to create buildings, trees, and probably more. 

 

[TechguyMaxC 516]

20 hours ago, n4gix said:

MSFS has a binary star system modeled!

Guys, this is just a preview of the DLC:

Microsoft Flight Simulator: Tatooine

Edited February 20, 2020 by TechguyMaxC

[bonchie 2,305]

41 minutes ago, Claviateur said:

Yes I saw this video and I just heard again his explanation. I think he is saying exactly what I explained in my post above, Azure AI analyzes and brings the data. I don't think he said the AI creates the assets.

You know, if the Azure AI retouches imagery and creates 3D elements over a map, MS would have already used it to make Bing imagery seamless and have worldwide 3D infrastructure, forests, powerlines etc etc....And score some points against Google Maps 🙂

 

The AI is definitely color correcting, removing shadows, removing clouds, and creating buildings dynamically based on ortho footprints. We’ve seen all that in action. It’s impossible that it’s all been done by hand.

MS has no reason to do that for Bing maps, though perhaps they will one day.

 

 

[divide 97]

Bing Map will probably prefer to keep the original untouched data as much as possible, as long as MSFS algorithms are experimental.
It's great to have unwanted shadows automatically removed for a flight sim, but the AI may also occasionally erase dark areas it mistakes for shadow - such artefacts are fine within a flight sim, but it's not for Bing Map which has to remain as accurate as possible, even if that means some data is partially occluded, as long as artefacts aren't artifically introduced. For the same reason, Bing Map may prefer to keep the original, non color-contiguous areas, as smoothing those colored borders could potentially lead to artefacts or loss of information.
I would not be surprised if those great algorithms make their way to Bing Map eventually, but not before several years of refinement and testing.

Edited February 20, 2020 by divide

[ChaoticBeauty 6,692]

Regarding shadows being removed, it also depends on the dynamic range of the imagery. Especially under harsh lighting conditions, the shadow details could be totally clipped, or lots of noise would be introduced when bringing up the brightness in those spots, which would create unsightly artifacts. Until AI is trained to completely reconstruct those areas (perhaps using other versions of the imagery at different times of the day), there will not be a perfect solution to this.

[GlideBy 324]

20 minutes ago, ChaoticBeauty said:

Until AI is trained to completely reconstruct those areas

They use AI to create/recreate areas in the source satellite images that are hidden behind clouds.

[ChaoticBeauty 6,692]

6 minutes ago, GlideBy said:

They use AI to create/recreate areas in the source satellite images that are hidden behind clouds.

Thanks for the info. Since clouds can cover a much larger part of the imagery, it will be interesting to see how accurately the AI can reconstruct those areas.

[goates 576]

55 minutes ago, ChaoticBeauty said:

Thanks for the info. Since clouds can cover a much larger part of the imagery, it will be interesting to see how accurately the AI can reconstruct those areas.

For an example of how this might work you can read about how Google did it. Basically, if you have enough images of an area taken at different times, you can work out what is cloud and what isn't.

https://www.smh.com.au/technology/google-maps-removes-clouds-puts-ground-in-focus-in-700trillion-pixel-upgrade-20160628-gptnuh.html

On a smaller scale, you can use image stacking to remove people from photos by taking multiple shots. A number of image editors these days will automate it for you.

https://photoshoptrainingchannel.com/remove-tourists-stack-mode/

[Claviateur 789]

3 hours ago, GlideBy said:

Here are links, and articles showing that Azure and/or AI to create building footprints.  These are not specific to MSFS though.

https://azure.microsoft.com/en-us/blog/how-to-extract-building-footprints-from-satellite-images-using-deep-learning/

https://github.com/microsoft/USBuildingFootprints

https://blogs.bing.com/maps/2018-06/microsoft-releases-125-million-building-footprints-in-the-us-as-open-data

In other interviews the MSFS devs have talked about how they can know a buildings height based on the shadow.  There was another interview where they mention about Azure creating the buildings and trees. 

So all that, and more which I am not going digging for, leads me to believe they are using AI to create buildings, trees, and probably more. 

 

I shared these links already in multiple posts 🙂

And I think we are saying the same thing here.

Yes, this is what the AI does, analyzes and fetch the footprint data and its metadata (height, type etc).

But the 3D assets construction out of this data is done procedurally (not manually) most probably with Asobo's in-house tools or with the engine built in procedural tools.

I use similar tools to generate autogen from footprint data for XP11. For an area that cover 4 tiles (i.e: 10500 km2) It takes about 3 min to generate a complex ready to use autogen package with all you can immagine: buildings, houses, forests, roads, railways, power lines etc and all these sit perfectly on the photoreal scenery you generate with another tool... 

A similar tool Arno G. created for FSX as far as I can I remember it was called Sceneproc. It uses OSM data as well to create procedural autogen for FSX.

Now the only difference here is that Azure AI brings all the things you mentioned above with metadatas and Asobo's tool create the 3D world with it...

[ChaoticBeauty 6,692]

11 minutes ago, goates said:

For an example of how this might work you can read about how Google did it. Basically, if you have enough images of an area taken at different times, you can work out what is cloud and what isn't.

https://www.smh.com.au/technology/google-maps-removes-clouds-puts-ground-in-focus-in-700trillion-pixel-upgrade-20160628-gptnuh.html

On a smaller scale, you can use image stacking to remove people from photos by taking multiple shots. A number of image editors these days will automate it for you.

https://photoshoptrainingchannel.com/remove-tourists-stack-mode/

Thanks for the links. It is impressive that this could be done when the articles were published four years ago.

[goates 576]

1 hour ago, ChaoticBeauty said:

Thanks for the links. It is impressive that this could be done when the articles were published four years ago.

The concepts and functionality go back further than that. Here's one article from 2007 about removing people from images, for example.

https://photoshopnews.com/2007/03/27/image-stacks-in-photoshop-cs3-extended/

What makes the new sim impressive is that they're combining a number of different technologies, plus modern hardware, and applying it at a much larger scale.

[GlideBy 324]

I see double shadows on some building in this image.  Look at the plane tail, you just see a short shadow that doesn't even cover the fuselage.  Then look at the taller buildings in the lower right, not the ones on the water.  I see a short shadow at a 3:30 o'clock (115 degree), and a long shadow at 5 o'clock (150 degree).  The plane shadow seems to correspond to the short shadow at 3:30.

 

 

---

[Claviateur 789]

Just to add a little note based on the latest (Airport) Video from Asobo.

The dev mentioned the word "Algorythms" during his explanations. This is exactly what I was refering to... In-house tools programmed to process the data and generate content/assets for the engine.

Then he talked about the thousands of airports they had to work on somehow "manually"... The AI did not do that...

But there are chances the AI detected the airports and airfields on the Bing Imagery and built an additional data for Asobo to work with (in addition to the FSX data) just like it did for footprints, vegetation zones etc etc...

The AI is already doing a super job of dectection and data collection on the world scale... Imrpessive.

If we think about it, it's the equivalent of many years of OSM mapping by (human) volunteers and the OSM data is not consistent across the globe since not every conuntry has the same dedicated volunteers to map their region...