DECEMBER 7TH, 2023 – DEVELOPMENT UPDATE

Posted by: Microsoft Flight Simulator Team

This week’s Screenshot Challenge winner is Forum user sidthekid871397! If you’re interested in participating in our weekly Screenshot Challenges, check out the details in the Community section below!

Earlier this week, we released Sim Update 14, the latest major upgrade to Microsoft Flight Simulator. We are pleased to see that sim pilots are enjoying the many bug fixes, improvements, and new features, with the new atmospheric colorization and the Working Title Cirrus SR22T G6 being highlights of this release.

Unfortunately, the launch of Sim Update 14 did not go as smoothly as we would have liked and our players expect. The initial release on December 5th was marred by download issues that prevented many players from installing and enjoying the new update for a period of about six hours while our support teams worked to resolve the issue.

While the incident is now fully resolved and players are able to download and install the update normally, this is the second upgrade in recent months (along with Sim Update 13 in September) where players experienced download issues. We know this has affected your trust in us, so in the interest of full transparency, we have written an incident recap report outlining what happened to cause the download issues with the SU_13 and SU_14 releases and the steps we took to both resolve the incidents in the moment and prevent them from happening again.

We understand that this experience was frustrating for many players who were eager to jump into Sim Update 14 at its release, and we truly appreciate everyone for their continued support. Thank you all, and we look forward to what is to come in the new year.

– MSFS Team

 

FEEDBACK SNAPSHOT

Click here for the web-friendly version of the Feedback Snapshot.

The Feedback Snapshot will return next week.

DEVELOPMENT ROADMAP

Click here for the web-friendly version of the Development Roadmap.

During the November 29th Developer Livestream, Jorg gave a sneak preview of the upcoming roadmap for 2024. You can see this new Roadmap at this link.

MARKETPLACE UPDATE

This week, we are pleased to debut an exciting new release to the Marketplace for both PC and Xbox, a collaboration from developers IndiaFoxtEcho and Heatblur Simulations: the legendary F-14 Tomcat!

IndiaFoxtEcho has created several military jets for Microsoft Flight Simulator previously, but this is a first-time MSFS release for Heatblur, a developer who is well-known in the military flight sim community for their high-quality add-ons available on the DCS platform. For more information about the IndiaFoxtEcho+Heatblur F-14 Tomcat, check out our feature article:

Now Available: Heatblur & IndiaFoxtEcho F-14 Tomcat 

 

MARKETPLACE RELEASES

Click here for the web-friendly version of the Marketplace Update.

There are 104 new and 36 updated products in today’s Marketplace Update.

MARKETPLACE BACKLOG

Current Backlog: 311 Total

BACKLOG: NEW PRODUCTS

	Testing Status
Content Type	Initial Processing and Validation by MS	Testing in Progress by MS	Testing in Progress by 3P	Total
Aircraft	17	3	37	57
Mission	4	2	42	48
Scenery	2	0	8	10
Livery	18	1	6	25
Airport	26	19	17	62
Other	13	5	7	25
Total	80	30	117	227

BACKLOG: UPDATED PRODUCTS

	Testing Status
Content Type	Initial Processing and Validation by MS	Testing in Progress by MS	Testing in Progress by 3P	Total
Aircraft	7	5	11	23
Mission	0	0	0	0
Scenery	3	8	1	12
Livery	0	0	6	6
Airport	10	11	10	31
Other	3	2	7	12
Total	23	26	35	84

 

SDK UPDATE

ASOBO

• No Update this week.

WORKING TITLE / MSFS AVIONICS FRAMEWORK

• No Update this week.

THIRD PARTY UPDATES

• No Update this week.

 

COMMUNITY FLY-IN FRIDAY

December 9 is celebrated as Tanzania’s  Independence Day! On this date in 1961, the former British colony in East Africa, known as Tanganyika at the time, gained independence and self-governance. A few years later in 1964, Tanganyika merged with the nearby island of Zanzibar to form the new nation of Tanzania.

Today, Tanzania is known throughout the world for its magnificent wildlife preserves and grand natural splendor. Popular tourist destinations include Serengeti National Park, Ngorongoro Conservation Area, and Mount Kilimanjaro, Africa’s highest point. We will be visiting all of these locations and more during our community fly-in this week!

Event details are available here. All are welcome to participate!

TUBELINER TUESDAY

It’s officially Winter, and where else better to visit to celebrate than Innsbruck (LOWI), Austria?! This airport will be familiar to many Microsoft Flight Simulator pilots, due to its iconic views and tricky approaches. Tubeliner Tuesday Episode 7 will see us depart Vienna (LOWW) and we will be flying this flight in the Fenix Airbus A320 on the VATSIM Multiplayer Network.

Event details are available here. Feel free to follow the flight on our Twitch channel!

COMMUNITY VIDEO

Today will see the release of the brand new highly anticipated F-14 Tomcat for Microsoft Flight Simulator developed by Heatblur and IndiaFoxtEcho, which players can purchase now via the Marketplace!

YouTuber Steve’s Sim Station has been lucky enough to receive an early access copy of this iconic fighter jet, and dives right into showing off some of the neat features that players can expect to use. This includes “Jester”, an AI Radar Intercept Officer (RIO) to help manage all of the secondary functions within the cockpit.

After watching this early access video we definitely feel the need…the need for speed!

SCREENSHOT CHALLENGE

This week’s screenshot challenge: You Can Be My Wingman Anytime!

This past week’s Screenshot Challenge was The Jet Age

Weekly Screenshot Challenge Winner: sidthekid871397 (Forums)

Screenshot by @SioRAaMEN_sim (Twitter)

Screenshot by HolmesCZ6127 (Forums)

Screenshot by Linska495th (Forums)

Screenshot by @ZillNiazi (Twitter)

Screenshot by DaveBGaming (Forums)

Screenshot by THECHIZINATOR (Forums)

Screenshot by @ukaka5656 (Twitter)

Screenshot by mrjmt2625 (Forums)

Screenshot by @KotetuP (Twitter)

Screenshot by BanZer9680 (Forums)

---

NEXT DEVELOPMENT BLOG UPDATE – December 14th, 2023

Sincerely,

Microsoft Flight Simulator Team

Edited December 8, 20232 yr by David Mills

Haven't read the details, but very much appreciate the post-incident report. Kudos devs, and similar transparency with significant problems in future please!

I'm keeping an eye on the game awards just in case lol

Oh, according to the incident report we are just 'players', not users or simmers. It seems the infrastructure is just in a beta/alpha testing stage? But players don't really mind, we are used to it!

Keep up the good work MS engineers!

57 minutes ago, Silicus said:

Oh, according to the incident report we are just 'players', not users or simmers. It seems the infrastructure is just in a beta/alpha testing stage? But players don't really mind, we are used to it!

Keep up the good work MS engineers!

Lol ok then.. from that detailed and technical report you're hung up on their use of the "players" terminology? And where does it say the infrastructure is in beta/alpha stage? It's the Azure cloud infrastructure, been around for years. The issue here was a specific bug in specific servers which impacted SU13, and then another bug around caching and SAS tokens that impacted SU14. Complex infrastructures like Azure with myriad complex ways to setup/configure can still reveal bugs, that doesn't mean the infrastructure is in beta/alpha stage.
 

16 minutes ago, lwt1971 said:

Lol ok then.. from that detailed and technical report you're hung up on their use of the "players" terminology?

Flight simmers can be weird, man lol.

The post incident report shows that they need to upgrade their network infrastructure significantly before releasing FS2024, especially if they want to deliver it with a "thinner client" (=more stuff to download as you fly).

Nice of them to apologise and explain in detail, maybe they're finally realising they need to improve in certain areas, I hope this sets a precedent for the future for better QC across the board (one can dream eh!?).

In the meantime download speeds are still incredibly poor (currently downloading world updates this morning after a reinstall of the Sim last night), another area they still haven't rectified and it's not even peak load time.

2 hours ago, MrFuzzy said:

The post incident report shows that they need to upgrade their network infrastructure significantly before releasing FS2024, especially if they want to deliver it with a "thinner client" (=more stuff to download as you fly).

What do you mean by "upgrade"? If you're talking about throwing more hardware at the issue, it sounds to me like it's just a coding fix that was required.

11 minutes ago, Krakin said:

What do you mean by "upgrade"? If you're talking about throwing more hardware at the issue, it sounds to me like it's just a coding fix that was required.

They need more bandwidth to support even the updates, imagine when FS2024 will be live with less data on the local storage and more to be downloaded during the flight.

Coding is not a solution: if you allow each user to establish fewer parallel connections - as described - you are just distributing your bandwidth more equally, resolving what for a certain number of users was previously a total outage, but you don't increase your bandwidth just by coding. So everybody manages to download data, downloads are not stuck, but they become slower for everyone.

What described in the incident report is nothing but server overload: 

503 errors = storage refusing requests from the CDN (Service Unavailable) 

"Note that immediately following the SU_14 launch, there was a large spike in 200 and 206 codes, indicating that the first players to download SU_14 were able to do so successfully. However, this was shortly followed by a surge in 503 errors, showing where the problems started. 

When the attempted fix of reducing parallel downloads from 8 to 1 was implemented, you can see that the number of 503 errors was reduced by about half, but the number of successful downloads (shown by the blue and red 200 and 206 lines) didn’t have a corresponding increase. Once the final fix was deployed, 503 errors dropped to almost zero and downloads were once again completing successfully."

Downloads were once again completing successfully, even though more slowly for everyone - I would add.

I appreciate the report from Microsoft, and that people trying to clarify it in this thread in simpler terms.  I am still not sure if I understand it fully.  I will have to read over it again a few times I fear.  

Is the word 'infamous' being misused again in the Innsbruck Twitch Video title?  Do they think Innsbruck is "well known for some bad quality or deed, wicked; abominable?".  I quite liked Innsbruck when I was there! 

Edited December 8, 20232 yr by bobcat999

4 hours ago, MrFuzzy said:

They need more bandwidth to support even the updates, imagine when FS2024 will be live with less data on the local storage and more to be downloaded during the flight.

Coding is not a solution: if you allow each user to establish fewer parallel connections - as described - you are just distributing your bandwidth more equally, resolving what for a certain number of users was previously a total outage, but you don't increase your bandwidth just by coding. So everybody manages to download data, downloads are not stuck, but they become slower for everyone.

What described in the incident report is nothing but server overload: 

503 errors = storage refusing requests from the CDN (Service Unavailable) 

"Note that immediately following the SU_14 launch, there was a large spike in 200 and 206 codes, indicating that the first players to download SU_14 were able to do so successfully. However, this was shortly followed by a surge in 503 errors, showing where the problems started. 

When the attempted fix of reducing parallel downloads from 8 to 1 was implemented, you can see that the number of 503 errors was reduced by about half, but the number of successful downloads (shown by the blue and red 200 and 206 lines) didn’t have a corresponding increase. Once the final fix was deployed, 503 errors dropped to almost zero and downloads were once again completing successfully."

Downloads were once again completing successfully, even though more slowly for everyone - I would add.

You're not looking at the full picture I think. I'm no expert but a bug in the system can cause the servers to not manage the load properly. The fix you outlined above is what they did in the interim while the root cause of the issue was being identified..

 

"After a more detailed analysis of the errors, the PlayFab team realized that their previous fix deployed for SU_13 was insufficient. Fixes for the remaining issues (broken shared cache and SAS token generation every 15 minutes) had to be deployed to restore normal service.

Playfab engineers deployed the final fix on their servers at approximately 2045 UTC. This new version restored the shared cache, generated only one SAS token per content and renewed an SAS token only when the previous one is about to expire (approximately once every 48 hours)."

 

"While it is regrettable that this issue occurred over two major releases, we are confident that a permanent fix to the SAS token generation bug is now in place which will prevent this from happening again."

 

As you can see, the eventual fix was not throwing more bandwidth problem but fixing how the existing bandwidth is managed. Reducing the parallel downloads was just a stopgap.

 

 

6 hours ago, MrFuzzy said:

They need more bandwidth to support even the updates, imagine when FS2024 will be live with less data on the local storage and more to be downloaded during the flight.

Coding is not a solution: if you allow each user to establish fewer parallel connections - as described - you are just distributing your bandwidth more equally, resolving what for a certain number of users was previously a total outage, but you don't increase your bandwidth just by coding. So everybody manages to download data, downloads are not stuck, but they become slower for everyone.

What described in the incident report is nothing but server overload: 

In addition to what Krakin says above, I'd draw a similarity here to other coding bugs for example, like those that can slowly cause leaks in memory.. that problem can always be masked by throwing more hardware memory resources at it (i.e. similar to increasing bandwidth in this Azure cloud case), but eventually the underlying bugs will still manifest symptoms, and overall it's best fix the bugs.

A SAS (Shared Access Signature) token at a high level is like a permit to authorize a request/api/etc (https://www.smikar.com/azure-sas-tokens/) .. in this case the bug appears to have been a generation of too many tokens that "flooded the system" (will need to re-read again more carefully). 

There's no worry here about whether enough Azure resources are available for MS to allocate to MSFS, their cloud infrastructure is vast, and the cost for them to scale/allocate its resources to their various applications (both gaming and enterprise) is trivial. Also, Azure's enterprise applications and customers use way way more bandwidth and cloud resources than the gaming/entertainment software including MSFS, so the infrastructure can more than handle MSFS and MS can easily allocate what's needed for it (and MS/Asobo will know best about how to handle v2024).
 

Edited December 8, 20232 yr by lwt1971

10 hours ago, MrFuzzy said:

They need more bandwidth to support even the updates, imagine when FS2024 will be live with less data on the local storage and more to be downloaded during the flight.

This logic is a bit flawed (and as others mentioned, not really the root cause of this issue). But re: bandwidth, an update can max your internet connection. Streaming data during a flight is more like streaming a Netflix movie. It's also more steady/predictable. If GeForce NOW can stream Cyberpunk with no issue, I'm sure MS/Asobo can figure it out for FS2024. The entire sim won't be running in the cloud like GeForce NOW, but I believe Asobo mentioned some of the processing will be performed server-side.

12 hours ago, Funky D said:

This logic is a bit flawed (and as others mentioned, not really the root cause of this issue). But re: bandwidth, an update can max your internet connection. Streaming data during a flight is more like streaming a Netflix movie. It's also more steady/predictable. If GeForce NOW can stream Cyberpunk with no issue, I'm sure MS/Asobo can figure it out for FS2024. The entire sim won't be running in the cloud like GeForce NOW, but I believe Asobo mentioned some of the processing will be performed server-side.

When I fly, the LAN traffic I see for MSFS is 1-10 Mbit/s (intermittent, only when loading tiles), when I download updates it's 2-3 Mbit/s on average. Not much different.

BTW it's your opinion that the network doesn't need more bandwidth and that it was only a "software bug" that is now fixed. We will see on the occasion of next update.

In the meantime, you can tell that to the thousands of users who complain about 10 min flight loading times, painfully slow download speeds, network disconnections, etc. There is a 500 page thread on the official forum.

Another thread here posted 12 hours ago when all was "fixed" already: