MSFS Testing & Hubble Space Telescope

[David Mills 4,729]

It appears that Asobo's testing process for each sim update is very reminiscent of the pre-launch testing of the Hubble Space Telescope (HST). If you'll recall, the HST was launched back in 1990 after years of development by brilliant men and women of science. Each individual part of the telescope was meticulously designed and built -- and tested and re-tested before launch. Then, after the HST was placed in orbit by the Space Shuttle Discovery, astronomers were dismayed to see that the images transmitted to the ground by Hubble were a disheartening blur. After laborious analysis, it was determined that the 94" main mirror aboard the telescope had been ground to the wrong specifications years beforehand back on earth. The entire project was in serious jeopardy, and Congress held emergency hearings to determine how such a fundamental, mind-boggling error could have escaped years of pre-launch testing.

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit. Each individual part of the telescope had been overseen individually. But a complete, end-to-end testing of the telescope was never done (though Kodak offered to do so but was rebuffed). It was only after launch that the telescope, as a whole, was finally switched on for its mission, to disappointing results.

I think you can probably see where I'm going with this analogy. We in the AVSIM community have wondered how it is possible that Asobo, like the brilliant team who built the Hubble Space Telescope, can release updates to the sim that are so fundamentally flawed -- the most recent example being that clouds now appear at ground level. Many on this forum, including me, have asked aloud the puzzling question, "Doesn't Asobo test these updates before launch?" The answer, like the HST, is obviously no, they don't. I have no doubt that individual components of MSFS, again like the HST, are tested and re-tested by sub-groups of highly skilled programmers and engineers and artists. But there is no prolonged period when Asobo sits down and just plays and enjoys the sim as end-users.

This naturally bring us to the question of Beta testers or, if you'll recall from the pre-launch days of MSFS, the Alpha testers. (Like many of you, I was an Alpha tester from day one.) I don't think it's fair to blame Alpha or Beta testers because Asobo, it seems to me, has a habit of making last-minute changes to sim updates without giving the testing community a chance to examine, and provide feedback to, the final code. I'm willing to bet that literally 100% of you early Alpha testers were blindsided by Asobo's surprise announcement that the sim would be released to the public in only six weeks, on August 18, 2020. Without getting into the weeds here -- there were many important differences between the final Alpha release of MSFS and the final gold code released to the public in August of 2020. (And I'm not just talking about the watermarks being removed.) On my machine, the multi-threading performed very differently in the final release as compared to the final Alpha. My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

[GSalden 4,655]

David, make a flight in MSFS or a nice walk around your house, instead of these kind of threads ….

🧐

[Tuskin38 6,451]

20 minutes ago, David Mills said:

My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

According to a leaked internal message, SU7 testing was cancelled because they couldn't get the build out in time for any bug reports to make a difference on the final release.

Edited December 2, 2021 by Tuskin38

[somiller 694]

25 minutes ago, David Mills said:

It appears that Asobo's testing process for each sim update is very reminiscent of the pre-launch testing of the Hubble Space Telescope (HST). If you'll recall, the HST was launched back in 1990 after years of development by brilliant men and women of science. Each individual part of the telescope was meticulously designed and built -- and tested and re-tested before launch. Then, after the HST was placed in orbit by the Space Shuttle Discovery, astronomers were dismayed to see that the images transmitted to the ground by Hubble were a disheartening blur. After laborious analysis, it was determined that the 94" main mirror aboard the telescope had been ground to the wrong specifications years beforehand back on earth. The entire project was in serious jeopardy, and Congress held emergency hearings to determine how such a fundamental, mind-boggling error could have escaped years of pre-launch testing.

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit. Each individual part of the telescope had been overseen individually. But a complete, end-to-end testing of the telescope was never done (though Kodak offered to do so but was rebuffed). It was only after launch that the telescope, as a whole, was finally switched on for its mission, to disappointing results.

I think you can probably see where I'm going with this analogy. We in the AVSIM community have wondered how it is possible that Asobo, like the brilliant team who built the Hubble Space Telescope, can release updates to the sim that are so fundamentally flawed -- the most recent example being that clouds now appear at ground level. Many on this forum, including me, have asked aloud the puzzling question, "Doesn't Asobo test these updates before launch?" The answer, like the HST, is obviously no, they don't. I have no doubt that individual components of MSFS, again like the HST, are tested and re-tested by sub-groups of highly skilled programmers and engineers and artists. But there is no prolonged period when Asobo sits down and just plays and enjoys the sim as end-users.

This naturally bring us to the question of Beta testers or, if you'll recall from the pre-launch days of MSFS, the Alpha testers. (Like many of you, I was an Alpha tester from day one.) I don't think it's fair to blame Alpha or Beta testers because Asobo, it seems to me, has a habit of making last-minute changes to sim updates without giving the testing community a chance to examine, and provide feedback to, the final code. I'm willing to bet that literally 100% of you early Alpha testers were blindsided by Asobo's surprise announcement that the sim would be released to the public in only six weeks, on August 18, 2020. Without getting into the weeds here -- there were many important differences between the final Alpha release of MSFS and the final gold code released to the public in August of 2020. (And I'm not just talking about the watermarks being removed.) On my machine, the multi-threading performed very differently in the final release as compared to the final Alpha. My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

The dead horse is now beaten...okay?!

[Republic3D 1,229]

Very good post, and I agree. 

I think Microsoft is flogging Asobo to meet deadlines, and there isn't enough time (or resources) to test properly before launch. 

[leprechaunlive 1,852]

52 minutes ago, Tuskin38 said:

According to a leaked internal message, SU7 testing was cancelled because they couldn't get the build out in time for any bug reports to make a difference on the final release.

SU7 was cancelled due to short deadline. then we were called back for it, with the specific mention that any bugs found, would not be fixed before release, but instead, put on a list for post launch fixes and/or SU8

Edited December 2, 2021 by leprechaunlive

[leprechaunlive 1,852]

13 minutes ago, Republic3D said:

Very good post, and I agree. 

I think Microsoft is flogging Asobo to meet deadlines, and there isn't enough time (or resources) to test properly before launch. 

its exactly that

[pmb 5,137]

I think the gist of David's analogy isn't just insufficient testing. They may do quite some isolated testing of different modules in - from what I heard - quite a number of different builds in parallel. The point is missing final testing of the complete release with all the modules cooperating (or not).

Or, as some suggested: They should task a couple of their fellows with just normally flying the complete pre-release version for at least one day before release.

Kind regards, Michael

[Tuskin38 6,451]

Jorg said last Q&A that World Updates will no longer include big code changes, it will just be whatever they need for the updated scenery.

There's also going to be a Sim Update in the future that is only bug fixes, no new features. Not sure when that is going to be, might be SU8 in February?

Edited December 2, 2021 by Tuskin38

[Dominique_K 10,133]

12 hours ago, David Mills said:

 

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

Stimulating post, David, but the overwhelming feeling we now have from some sources confirming what some have been saying here out of reasoning and experience,  is this is not a development issue (ie Asobo not doing a proper job) but a distributor issue (MS releasing something they know is buggy to keep a marketing momentum). 

I have not read the leaked message referred upthread  but this kind of leak reveals a tense situation in any project due to a flawed management. Not good. 

 

Edited December 3, 2021 by Dominique_K

[TheJAG 57]

11 hours ago, Tuskin38 said:

Jorg said last Q&A that World Updates will no longer include big code changes, it will just be whatever they need for the updated scenery.

There's also going to be a Sim Update in the future that is only bug fixes, no new features. Not sure when that is going to be, might be SU8 in February?

You're right, an update solely to fix bugs, and to give the team a well earned rest.

It's a shame we're again in a speculative thread on how Asobo conducts their business. Mistakes happen, it's a living beast that Asobo / MS is pushing forward. Whilst some errors happened for whatever reason, a gazillion other things that *could* go wrong, didn't. Clearly we all want a perfect simulator from day 1, but that's just not how this sim works. We have gotten something that was a good way there, we're now on the journey with them to perfection, C'est La Vie - accept and move on.

 

[Matchstick 1,885]

The thing is there ARE beta tests, even SU7 had a 1 week beta test with pre-selected people but Asobo aren't able to incorporate the feed back from these tests prior to release.

It's the same story we've seen since the pre-release Alpha and Beta and it's down to release dates being written in stone long before anyone knows when the release will actually be ready.

So please direct your ire at the people responsible for the ridiculous scheduling but PLEASE stop thinking the problem are due to a lack of beta testing - it's already happening it just isn't able to help because of managment decisions.

Edited December 3, 2021 by Matchstick

[hansb57 492]

Maybe this can help us through the holidays:

 

https://forums.flightsimulator.com/t/blog-december-2nd-2021-development-update/478125

[pmb 5,137]

I don't enjoy issues either, and the VR mouse bug hurts me as well. But from a perspective, I recall all flight sim versions by different makers I've seen over the last 30 years having been released with issues, sometimes minor, but often more serious.

Even the recently released Prepar3d5.3 is reported to have a bug where even a small amount of car traffic incurs a hard framerate hit, not to mention the popping autogen still not having been cured. In no way is this intended as an insult on LM, it just shows these beasts are hard to tame because of the level of complexity they reached - all of them.

The difference is, you can forego updates in the other sims, which isn't an option for MSFS. But that's the price for the server-based goodies we get. And on the other hand, I still would be on FS4 if I rejected all later more or less issue-plagued FS releases.

Kind regards, Michael

[MrFuzzy 2,117]

15 hours ago, David Mills said:

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit.

Yeah, the difference between a space telescope and MSFS though, is that to fully test the first you have to bring it into space, for MSFS the same PC used to make it is sufficient. Many bugs could have been detected just by starting a flight... or even in the world map.

I agree on the conclusions.