I completed my Ph.D. in 2020, and the subject of my thesis was "Relational Effects of Data Networks: How Strategically Aligned Data Networks Impact Digital Transformation." If you find yourself unable to sleep on any given night, feel free to click on the link and give it a read.

For the dissertation, I did multiple case studies of Fortune 500 companies (anonymized for the thesis) building critical applications that relied on the cloud or their own private data network. Every company that either underperformed or completely failed ignored (for some strange reason) the importance of the data network infrastructure as key to its success. This is because, until now, most applications were not real-time, and investments in the application provided the largest ROI with little need to invest in the network. However, real-time applications like MSFS 2024, Streaming Video, etc., rely on a robust data network infrastructure built to scale to support the user base. 

What confounds me is that CPUs are cheap, and bandwidth is cheap. Additionally, this is all now elastic and automatic! With virtual servers and networks, you can spin up new resources in real time and then scale them back down when they are no longer needed, so you're not wasting that one cost that can get out of hand: the cost of power and cooling. You don't want to power and cool resources that you don't have to, but with resource elasticity, there is no reason to. And Asobo has access to -- wait for it -- Azure! Someone has to explain this one to me.

Outside of that, I think the Asobo team suffers from a noble intent issue. Their intent was, of course, honorable. But somewhere along the line they tried to "boil the ocean." For any of us who have tried it, it doesn't work. Ever. You can't. Unfortunately, as the old saying goes - "the road to hell is paved with good intentions." They have accomplished something never before seen in the history of computing - they built a digital twin of the entire planet in unsurpassed detail. That feat is historic and awesome on its own. What they should have done is opened up AI aircraft and ships, ATC, weather, local detailed scenery, etc to those who can do it better. They should have opened the pre-release alpha stress test to EVERYONE who is willing to test it to get a true understanding of the breaking point. Also, calculating capacity is usually done on a 10:1 rule. If you have 1 million users, assume 100,000 using it at a time, etc. How they calculated 20 million unique users (their numbers) but built for 200,000 users on release day is simply malpractice (sorry.) Most importantly, there was no need to boil the ocean with some of the most brilliant 3rd party developers being ready, willing, and able to jump in. Indeed they did some of this with scenery and some aircraft, but alas it wasn't enough.

Either way, I will write an academic paper on this in another couple of years to give the process time to play out. FS 2024 will undoubtedly be in a better spot, and the recovery process (or not) will be even more fascinating.

One last comment. Multifaceted, complex, and technically forward projects like MSFS 2024,with lots of moving parts, require a strong OPM (office of project management) to be successful. If there was an OPM for this project - they failed and they failed miserably. If there was no OPM, well, then there's your problem.

Either way, it's cool to see my two loves come together - for better or for worse.

Mike T.

Edited December 3, 20241 yr by Mike T

30 minutes ago, Mike T said:

How they calculated 20 million unique users (their numbers)

Interesting. I'm wondering if the calculation was for concurrent users. Do you have a link?

Yes, concurrent users for a given population. For real-time apps like VoIP 10:1 was always a good starting point and you tweak according to user patterns from there. For instance, 1000 call center users will be on the phone at a higher rate than an office with random calls coming in and going out. So in the former you may decide to use a 5:1 ratio vs the latter which you decide to use a 10:1 ratio. Because what you DON'T want is -- you guessed it -- a busy signal!

But this is why you do a traffic study to lock in the appropriate ratio; the deviation from the mean (assuming 10:1) can be dramatic.

To be fair, Asobo did attempt a limited traffic study with the short alpha test. But then it was missing most of the content that had to be streamed, no store, and only a limited subset of the population. That's like doing a traffic study for a new major highway by observing traffic at 3 a.m. and not considering what happens at rush hour. So you build a two lane highway when you actually needed an 8-lane highway (um, hello, Atlanta?) And the materials you used to build the 2-lane highway don't support the weight of 18-wheelers! So now you've got to widen the highway AND fix all the potholes and collapsed bridges on the existing ones. That says nothing of the budget required.

Edited December 3, 20241 yr by Mike T

1 hour ago, Mike T said:

And Asobo has access to -- wait for it -- Azure! Someone has to explain this one to me.

I think it's pretty easy to read between the lines on this one. Asobo's access to Azure is still coming out of Microsoft's bottom line. Asobo probably asked Microsoft for x number of servers and they had to make do with less for budgetary reasons. Also, the launch day issues were due to caching which is likely disk dependent, and datacenter storage is neither cheap nor available in huge excess.

Other real-time applications such as video streaming or corporate SAAS applications (including Microsoft's own applications) depend on ad revenue and subscriptions to motivate a high SLA. MSFS is ultimately a game supported by a marketplace that isn't even up and running yet. You can bet MS would rather deal with a day of bad publicity versus spending hundreds of thousands of dollars on extra servers that were needed for one or two days.

1 hour ago, Mike T said:

If there was an OPM for this project - they failed and they failed miserably.

You may be in for a rude awakening once you witness project management for "Multifaceted, complex, and technically forward projects" in the real world.

LOL Funky D, I may have stumbled upon one or two projects being managed in the real world, but thanks for the advice! 

This is clearly a marketing vs factory tug of war inside MS. Marketing always over promises and wants things delivered faster and before the factory approves it. In most cases, if the damage is minimal, ie, isolated bugs and ctd, it's manageable and the money keeps rolling in. MSFS 2020 was a good example of it. In the 2024  roll out case, there was a gross miscalculation: the building structure, while not totally collapsed, turned into the leaning tower of Pisa. The best idea came from someone who suggested that MS provide users with two options: mostly downloaded and installed on the pc, or on the cloud.

1 hour ago, Funky D said:

You may be in for a rude awakening once you witness project management for "Multifaceted, complex, and technically forward projects" in the real world.

Indeed. I studied and wrote a paper on "the conspiracy of optimism" that exists with complex projects like this when I did my Masters in Project Management. In short, very senior management agree to aggressive schedules, low budgets and high scope, then toss the medicine ball to the "OPM" and are surprised when the project fails to meet these ridiculous requirements.

One more thing: I was executive VP of manufacturing during the Y2K scare. Many manufacturers decided to switch to new "enterprise" or ERP computer software systems to manage all company's businesses-- from supplier to customers. With insufficient funds, time, skilled personnel, etc, etc, many companies trusted their entire business to consultants who tried to fit a  company's unique businesses practices into their erp's "best business practices". The end result was a disaster for many companies. Business literally stopped cold. Classic case of not understand how to manage very complex systems. In my company, as the lead executive for our erp implementation, I would report to the board once a month. It took us two years to get some parts of the ERP implementation done, and when the board asked me how the transition went, I said: "the lights flickered, but the plant did not go dark". They smiled and I kept my job. We'll see if this Asobo- MS experiment represents a flickering of the lights or the building going dark. Some reviewers  feel that things will only improve. Let's hope so.

I perceive MS's ultimate goal is to stream games completely to a display unit e.g. your local TV, handheld or PC/Laptop where the local hardware is not the defining factor in running the game.

https://www.rollingstone.com/culture/rs-gaming/xbox-console-future-cloud-ceo-phil-spencer-1235166597/

Cheers

35 minutes ago, flyforever said:

This is clearly a marketing vs factory tug of war inside MS. Marketing always over promises and wants things delivered faster and before the factory approves it. In most cases, if the damage is minimal, ie, isolated bugs and ctd, it's manageable and the money keeps rolling in. MSFS 2020 was a good example of it. In the 2024  roll out case, there was a gross miscalculation: the building structure, while not totally collapsed, turned into the leaning tower of Pisa.

You have absolutely know idea as to this being fact!

Do you have direct knowledge of allocation of resources within Microsoft that are dedicated to the game???

If not your purely making a very bad assumption.

 

16 minutes ago, tpete61 said:

You have absolutely know idea as to this being fact!

Do you have direct knowledge of allocation of resources within Microsoft that are dedicated to the game???

If not your purely making a very bad assumption.

 

Just FYI - Using 'know' in the first sentence implies the author has correct knowledge of the topic, and you're agreeing with him, which, from your tone, is not what you meant. 

Speaking of 'your' (last sentence), I think you mean 'you're', as in 'you are'. 

English lesson over. As you were.  

52 minutes ago, Reset XPDR said:

Indeed. I studied and wrote a paper on "the conspiracy of optimism" that exists with complex projects like this when I did my Masters in Project Management. In short, very senior management agree to aggressive schedules, low budgets and high scope, then toss the medicine ball to the "OPM" and are surprised when the project fails to meet these ridiculous requirements.

To your point. When I was in technology consulting, we would start off a project with a LOE (level of effort), which is exactly as it sounds. You want to do X? It will take Y resources and it will cost an estimate of Z. A big portion of Z was project management at which point the customer would shake their head and ask why do we need it and why is it so expensive. Well - PMO pays for itself since it keeps sources and costs in check. It also checks progress and dependencies and foresees technology and resource gaps. A good PM is a major point of contact for all stakeholders, especially the CFO who holds the purse strings and project sponsor whose butt is on the line.

Many see PM as a luxury, but in truth, it's a necessity. I'm an engineer at heart, and I don't want to waste time with meetings and structure—I used to be the archetypical cowboy. Let me do my job and stay out of my way. Now that I am senior leadership I fire cowboys. They're brilliant but we can't afford them or the chaos they create anymore.

We would all like to pretend there's some mean scrooge in a suit sitting in an office somewhere messing everything up, but in truth it's not the bean counters, its often the beans.

 

 

4 hours ago, Mike T said:

I completed my Ph.D. in 2020, and the subject of my thesis was "Relational Effects of Data Networks: How Strategically Aligned Data Networks Impact Digital Transformation." If you find yourself unable to sleep on any given night, feel free to click on the link and give it a read.

For the dissertation, I did multiple case studies of Fortune 500 companies (anonymized for the thesis) building critical applications that relied on the cloud or their own private data network. Every company that either underperformed or completely failed ignored (for some strange reason) the importance of the data network infrastructure as key to its success. This is because, until now, most applications were not real-time, and investments in the application provided the largest ROI with little need to invest in the network. However, real-time applications like MSFS 2024, Streaming Video, etc., rely on a robust data network infrastructure built to scale to support the user base. 

What confounds me is that CPUs are cheap, and bandwidth is cheap. Additionally, this is all now elastic and automatic! With virtual servers and networks, you can spin up new resources in real time and then scale them back down when they are no longer needed, so you're not wasting that one cost that can get out of hand: the cost of power and cooling. You don't want to power and cool resources that you don't have to, but with resource elasticity, there is no reason to. And Asobo has access to -- wait for it -- Azure! Someone has to explain this one to me.

Outside of that, I think the Asobo team suffers from a noble intent issue. Their intent was, of course, honorable. But somewhere along the line they tried to "boil the ocean." For any of us who have tried it, it doesn't work. Ever. You can't. Unfortunately, as the old saying goes - "the road to hell is paved with good intentions." They have accomplished something never before seen in the history of computing - they built a digital twin of the entire planet in unsurpassed detail. That feat is historic and awesome on its own. What they should have done is opened up AI aircraft and ships, ATC, weather, local detailed scenery, etc to those who can do it better. They should have opened the pre-release alpha stress test to EVERYONE who is willing to test it to get a true understanding of the breaking point. Also, calculating capacity is usually done on a 10:1 rule. If you have 1 million users, assume 100,000 using it at a time, etc. How they calculated 20 million unique users (their numbers) but built for 200,000 users on release day is simply malpractice (sorry.) Most importantly, there was no need to boil the ocean with some of the most brilliant 3rd party developers being ready, willing, and able to jump in. Indeed they did some of this with scenery and some aircraft, but alas it wasn't enough.

Either way, I will write an academic paper on this in another couple of years to give the process time to play out. FS 2024 will undoubtedly be in a better spot, and the recovery process (or not) will be even more fascinating.

One last comment. Multifaceted, complex, and technically forward projects like MSFS 2024,with lots of moving parts, require a strong OPM (office of project management) to be successful. If there was an OPM for this project - they failed and they failed miserably. If there was no OPM, well, then there's your problem.

Either way, it's cool to see my two loves come together - for better or for worse.

Mike T.

Anyone in this industry knows what a poorly managed project looks like, and this is exactly what it looks like. 

What is clear in these type of conversations is the fact that some of us come from businesses that required a very strong PM experience to complex projects. Building a rocket ship that does not blow up on take off is one example of a complex project.  This is due to the fact that complex projects, such as building airplanes, rockets and other types of complex systems require the best of the best in experience , competence, and true and tried practices. Good leadership is essential, too. I want MSFS2024 to succeed because I am a long time simmer. We will never know how decisions and choices were made through out the new sim project,  but the laws of complex projects do not change. If these laws or rules are violated, the project will be partially successful or fail outright.

boiling the ocean reminds me of time I tried installing MSFS2024. joking, anyway, I agree with your summary. I have high hopes that FS2024 will be ready for me within a year and until it is I am thoroughly enjoying 2020.