Jump to content
Sign in to follow this  
David Mills

MSFS Testing & Hubble Space Telescope

Recommended Posts

It appears that Asobo's testing process for each sim update is very reminiscent of the pre-launch testing of the Hubble Space Telescope (HST). If you'll recall, the HST was launched back in 1990 after years of development by brilliant men and women of science. Each individual part of the telescope was meticulously designed and built -- and tested and re-tested before launch. Then, after the HST was placed in orbit by the Space Shuttle Discovery, astronomers were dismayed to see that the images transmitted to the ground by Hubble were a disheartening blur. After laborious analysis, it was determined that the 94" main mirror aboard the telescope had been ground to the wrong specifications years beforehand back on earth. The entire project was in serious jeopardy, and Congress held emergency hearings to determine how such a fundamental, mind-boggling error could have escaped years of pre-launch testing.

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit. Each individual part of the telescope had been overseen individually. But a complete, end-to-end testing of the telescope was never done (though Kodak offered to do so but was rebuffed). It was only after launch that the telescope, as a whole, was finally switched on for its mission, to disappointing results.

I think you can probably see where I'm going with this analogy. We in the AVSIM community have wondered how it is possible that Asobo, like the brilliant team who built the Hubble Space Telescope, can release updates to the sim that are so fundamentally flawed -- the most recent example being that clouds now appear at ground level. Many on this forum, including me, have asked aloud the puzzling question, "Doesn't Asobo test these updates before launch?" The answer, like the HST, is obviously no, they don't. I have no doubt that individual components of MSFS, again like the HST, are tested and re-tested by sub-groups of highly skilled programmers and engineers and artists. But there is no prolonged period when Asobo sits down and just plays and enjoys the sim as end-users.

This naturally bring us to the question of Beta testers or, if you'll recall from the pre-launch days of MSFS, the Alpha testers. (Like many of you, I was an Alpha tester from day one.) I don't think it's fair to blame Alpha or Beta testers because Asobo, it seems to me, has a habit of making last-minute changes to sim updates without giving the testing community a chance to examine, and provide feedback to, the final code. I'm willing to bet that literally 100% of you early Alpha testers were blindsided by Asobo's surprise announcement that the sim would be released to the public in only six weeks, on August 18, 2020. Without getting into the weeds here -- there were many important differences between the final Alpha release of MSFS and the final gold code released to the public in August of 2020. (And I'm not just talking about the watermarks being removed.) On my machine, the multi-threading performed very differently in the final release as compared to the final Alpha. My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

  • Like 10

Share this post


Link to post
Share on other sites

David, make a flight in MSFS or a nice walk around your house, instead of these kind of threads ….

🧐

  • Like 6
  • Upvote 1

Gerard

12900K (8 cores @ 5.1-5.5 GHz / 8 cores @ 3.9 GHz ) - Asus ROG Strix Gaming D4 - GSkill Ripjaws 2x 16 Gb 4266 mhz @ 4000 mhz / cas 16- Inno3D RTX3090 X4 iCHILL 24 Gb - 1x SSD M2 2800/1800 2TB - 1x SSD M2 2800/1800 1Tb - Sata 600 SSD 500 Mb - Thermaltake Level 10 GT case - EKWB Extreme 240 liquid cooling set push/pull - 2x 43” Sony 4K tv's in NVSurround

8700K : all 6 cores @ 5.0 GHz - Asus ROG 370 - 2x 8 Gb 4000 MHz @ 3600 Mhz - Inno3D RTX 3090 X3 iCHILL 24 Gb  - 1x SSD M2 2800/1800 2TB - 2x  Sata 600 SSD 500 Mb - 1x 55” Sony 4K tv

My flightsim vids :  https://www.youtube.com/user/fswidesim/videos?shelf_id=0&sort=dd&view=0

 

Share this post


Link to post
Share on other sites
20 minutes ago, David Mills said:

My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

According to a leaked internal message, SU7 testing was cancelled because they couldn't get the build out in time for any bug reports to make a difference on the final release.

Edited by Tuskin38
  • Like 4

Share this post


Link to post
Share on other sites
25 minutes ago, David Mills said:

It appears that Asobo's testing process for each sim update is very reminiscent of the pre-launch testing of the Hubble Space Telescope (HST). If you'll recall, the HST was launched back in 1990 after years of development by brilliant men and women of science. Each individual part of the telescope was meticulously designed and built -- and tested and re-tested before launch. Then, after the HST was placed in orbit by the Space Shuttle Discovery, astronomers were dismayed to see that the images transmitted to the ground by Hubble were a disheartening blur. After laborious analysis, it was determined that the 94" main mirror aboard the telescope had been ground to the wrong specifications years beforehand back on earth. The entire project was in serious jeopardy, and Congress held emergency hearings to determine how such a fundamental, mind-boggling error could have escaped years of pre-launch testing.

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit. Each individual part of the telescope had been overseen individually. But a complete, end-to-end testing of the telescope was never done (though Kodak offered to do so but was rebuffed). It was only after launch that the telescope, as a whole, was finally switched on for its mission, to disappointing results.

I think you can probably see where I'm going with this analogy. We in the AVSIM community have wondered how it is possible that Asobo, like the brilliant team who built the Hubble Space Telescope, can release updates to the sim that are so fundamentally flawed -- the most recent example being that clouds now appear at ground level. Many on this forum, including me, have asked aloud the puzzling question, "Doesn't Asobo test these updates before launch?" The answer, like the HST, is obviously no, they don't. I have no doubt that individual components of MSFS, again like the HST, are tested and re-tested by sub-groups of highly skilled programmers and engineers and artists. But there is no prolonged period when Asobo sits down and just plays and enjoys the sim as end-users.

This naturally bring us to the question of Beta testers or, if you'll recall from the pre-launch days of MSFS, the Alpha testers. (Like many of you, I was an Alpha tester from day one.) I don't think it's fair to blame Alpha or Beta testers because Asobo, it seems to me, has a habit of making last-minute changes to sim updates without giving the testing community a chance to examine, and provide feedback to, the final code. I'm willing to bet that literally 100% of you early Alpha testers were blindsided by Asobo's surprise announcement that the sim would be released to the public in only six weeks, on August 18, 2020. Without getting into the weeds here -- there were many important differences between the final Alpha release of MSFS and the final gold code released to the public in August of 2020. (And I'm not just talking about the watermarks being removed.) On my machine, the multi-threading performed very differently in the final release as compared to the final Alpha. My understanding is that the last Beta testers for SU7 were sent packing long before SU7 was finalized. I was not personally a part of that process, so I'll stand corrected by those more knowledgeable about this last sim update.

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

The dead horse is now beaten...okay?!


Steven_Miller.png?dl=1

i7-6700k Gigabyte GA-Z170X-UD5 32GB DDR4 2666 ASUS ROG-STRIX GTX1070

Share this post


Link to post
Share on other sites

Very good post, and I agree. 

I think Microsoft is flogging Asobo to meet deadlines, and there isn't enough time (or resources) to test properly before launch. 

  • Like 3
  • Upvote 1

R9 5900X | RTX 3090 | 64 GB 3600 | 3440x1440 G-Sync | Logitech Pro Throttles Rudder Yoke Panels | Thrustmaster T.16000M FCS | TrackIR 5 | Oculus Rift S
Experience with Flight Simulator since early 1990s

Share this post


Link to post
Share on other sites
52 minutes ago, Tuskin38 said:

According to a leaked internal message, SU7 testing was cancelled because they couldn't get the build out in time for any bug reports to make a difference on the final release.

SU7 was cancelled due to short deadline. then we were called back for it, with the specific mention that any bugs found, would not be fixed before release, but instead, put on a list for post launch fixes and/or SU8

Edited by leprechaunlive
  • Like 4

Share this post


Link to post
Share on other sites
13 minutes ago, Republic3D said:

Very good post, and I agree. 

I think Microsoft is flogging Asobo to meet deadlines, and there isn't enough time (or resources) to test properly before launch. 

its exactly that

  • Like 2

Share this post


Link to post
Share on other sites

I think the gist of David's analogy isn't just insufficient testing. They may do quite some isolated testing of different modules in - from what I heard - quite a number of different builds in parallel. The point is missing final testing of the complete release with all the modules cooperating (or not).

Or, as some suggested: They should task a couple of their fellows with just normally flying the complete pre-release version for at least one day before release.

Kind regards, Michael

  • Like 2

MSFS, P3D Professional 5, AeroflyFS2, XP11; Beta tester of SimStarter, SPAD.neXt, and FS-FlightControl

Intel i7-6700K 4.0 GHz / Asus MAXIMUS VIII RANGER / Kingston 32 GB DDR4 / Samsung SSD M.2 500 GB + Samsung SSD 1 TB +  4 TB + WD HD 6 TB / EVGA GTX 1080Ti 11 GB / LG 34UM95 3440 x 1440 /  XTOP/Saitek hardware panel / HP Reverb / Win 10/64

 

 

Share this post


Link to post
Share on other sites

Jorg said last Q&A that World Updates will no longer include big code changes, it will just be whatever they need for the updated scenery.

There's also going to be a Sim Update in the future that is only bug fixes, no new features. Not sure when that is going to be, might be SU8 in February?

Edited by Tuskin38
  • Like 1

Share this post


Link to post
Share on other sites
12 hours ago, David Mills said:

 

My summary point is this: Asobo, like the team who build the Hubble Space Telescope, needs to do extensive, end-to-end testing before launch. Individual geniuses working in isolation can nonetheless produce a partially dysfunctional product upon final assembly.

Stimulating post, David, but the overwhelming feeling we now have from some sources confirming what some have been saying here out of reasoning and experience,  is this is not a development issue (ie Asobo not doing a proper job) but a distributor issue (MS releasing something they know is buggy to keep a marketing momentum). 

I have not read the leaked message referred upthread  but this kind of leak reveals a tense situation in any project due to a flawed management. Not good. 

 

Edited by Dominique_K
  • Like 1

Dominique

Simming since 1981 -  4770k@3.7 GHz with 16 GB of RAM and a 1080 with 8 GB VRAM running a 27" @ 2560*1440 - Warthog HOTAS - MFG pedals - MFS Standard version with Steam

 

Share this post


Link to post
Share on other sites
11 hours ago, Tuskin38 said:

Jorg said last Q&A that World Updates will no longer include big code changes, it will just be whatever they need for the updated scenery.

There's also going to be a Sim Update in the future that is only bug fixes, no new features. Not sure when that is going to be, might be SU8 in February?

You're right, an update solely to fix bugs, and to give the team a well earned rest.

It's a shame we're again in a speculative thread on how Asobo conducts their business. Mistakes happen, it's a living beast that Asobo / MS is pushing forward. Whilst some errors happened for whatever reason, a gazillion other things that *could* go wrong, didn't. Clearly we all want a perfect simulator from day 1, but that's just not how this sim works. We have gotten something that was a good way there, we're now on the journey with them to perfection, C'est La Vie - accept and move on.

 

  • Like 3

Share this post


Link to post
Share on other sites

The thing is there ARE beta tests, even SU7 had a 1 week beta test with pre-selected people but Asobo aren't able to incorporate the feed back from these tests prior to release.

It's the same story we've seen since the pre-release Alpha and Beta and it's down to release dates being written in stone long before anyone knows when the release will actually be ready.

So please direct your ire at the people responsible for the ridiculous scheduling but PLEASE stop thinking the problem are due to a lack of beta testing - it's already happening it just isn't able to help because of managment decisions.

Edited by Matchstick
  • Like 1
  • Upvote 1

Share this post


Link to post
Share on other sites

I don't enjoy issues either, and the VR mouse bug hurts me as well. But from a perspective, I recall all flight sim versions by different makers I've seen over the last 30 years having been released with issues, sometimes minor, but often more serious.

Even the recently released Prepar3d5.3 is reported to have a bug where even a small amount of car traffic incurs a hard framerate hit, not to mention the popping autogen still not having been cured. In no way is this intended as an insult on LM, it just shows these beasts are hard to tame because of the level of complexity they reached - all of them.

The difference is, you can forego updates in the other sims, which isn't an option for MSFS. But that's the price for the server-based goodies we get. And on the other hand, I still would be on FS4 if I rejected all later more or less issue-plagued FS releases.

Kind regards, Michael

  • Like 2

MSFS, P3D Professional 5, AeroflyFS2, XP11; Beta tester of SimStarter, SPAD.neXt, and FS-FlightControl

Intel i7-6700K 4.0 GHz / Asus MAXIMUS VIII RANGER / Kingston 32 GB DDR4 / Samsung SSD M.2 500 GB + Samsung SSD 1 TB +  4 TB + WD HD 6 TB / EVGA GTX 1080Ti 11 GB / LG 34UM95 3440 x 1440 /  XTOP/Saitek hardware panel / HP Reverb / Win 10/64

 

 

Share this post


Link to post
Share on other sites
15 hours ago, David Mills said:

Ultimately, the answer turned out to be that the Hubble Space Telescope had not actually been tested before launch -- at least not as a complete, working unit.

Yeah, the difference between a space telescope and MSFS though, is that to fully test the first you have to bring it into space, for MSFS the same PC used to make it is sufficient. Many bugs could have been detected just by starting a flight... or even in the world map.

I agree on the conclusions.

  • Like 1

i9 10850K / 32 GB 3200 MHz CL16 / RTX 3090 / Acer Predator X34P G-Sync

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Tom Allensworth,
    Founder of AVSIM Online


  • Flight Simulation's Premier Resource!

    AVSIM is a free service to the flight simulation community. AVSIM is staffed completely by volunteers and all funds donated to AVSIM go directly back to supporting the community. Your donation here helps to pay our bandwidth costs, emergency funding, and other general costs that crop up from time to time. Thank you for your support!

    Click here for more information and to see all donations year to date.
  • Donation Goals

    AVSIM's 2020 Fundraising Goal

    Donate to our annual general fundraising goal. This donation keeps our doors open and providing you service 24 x 7 x 365. Your donation here helps to pay our bandwidth costs, emergency funding, and other general costs that crop up from time to time. We reset this goal every new year for the following year's goal.


    53%
    $13,405.00 of $25,000.00 Donate Now
×
×
  • Create New...