I've had my 2nd CTD yesterday since the 2024 release and with daily flying.

Is that shoddy or exceptional QC?

I can guess who will say what. 😏

dd

 

 

When I read these posts, many people are lacking the knowledge how QA works and how it affects the shipped product.

You guys are aware that a perfectly working QA can be in place and the software still ships with lots of evident bugs?

 

58 minutes ago, fsiscool said:

When I read these posts, many people are lacking the knowledge how QA works and how it affects the shipped product.

You guys are aware that a perfectly working QA can be in place and the software still ships with lots of evident bugs?

 

Having been part of a couple of QA teams (as a volunteer-ish) and having been involved in countless gaming/simulation Betas since the 1990s not to mention being involved in the "gaming scene" since the 1980s(!), I can safely say I think I know a shoddy QA setup when I see one and I see one here & have done for five years, to be fair it has got marginally better in some areas but they still make absolutely unnecessary regressions as witnessed with SU4 🙂

52 minutes ago, MarcG said:

Having been part of a couple of QA teams (as a volunteer-ish) and having been involved in countless gaming/simulation Betas since the 1990s not to mention being involved in the "gaming scene" since the 1980s(!), I can safely say I think I know a shoddy QA setup when I see one and I see one here & have done for five years, to be fair it has got marginally better in some areas but they still make absolutely unnecessary regressions as witnessed with SU4 

Sorry, you seem to not understand how it works.

QA only reports issues which are managed in huge bug tracking and managing tools. Whether and to what degree they are fixed is not in their responsibility.

Jorg mentioned in one of the early dev streams that they had 30k or so open bugs at release of 2024. Who has found them? Who reported them and managed them?

Calling that a shoddy QA setup more than anything else just reveals one thing: you misjudged them and have no clue how QA or testing is handled professionally in big projects.

Edited October 15, 2025Oct 15 by fsiscool

15 minutes ago, fsiscool said:

Sorry, you seem to not understand how it works.

QA only reports issues which are managed in huge bug tracking and managing tools. Whether and to what degree they are fixed is not in their responsibility.

Jorg mentioned in one of the early dev streams that they had 30k or so open bugs at release of 2024. Who has found them? Who reported them and managed them?

Calling that a shoddy QA setup more than anything else just reveals one thing: you misjudged them and have no clue how QA or testing is handled professionally in big projects.

Ok

3 hours ago, PlumCrazy said:

When you see REGRESSIONS like the mouse curser leaving a never-ending tra

That is why I said "may well actually be their intent", when looking at specific, especially glaring, issues, like the one you're referring to.  Perhaps it happened after QC in the hands of someone who decided to make a minor edit they forgot to do, or what have you. 

Important to factor in the scale of the required QC in an operation to support MSFS.  I'm sure that none of us have a clue.

Where I worked it was 1 qc to 1 dev.

dd  

Edited October 16, 2025Oct 16 by Sky_Pilot071

8 hours ago, Sky_Pilot071 said:

Important to factor in the scale of the required QC in an operation to support MSFS.  I'm sure that none of us have a clue.

Where I worked it was 1 qc to 1 dev.

The symptom that at release 30k bugs remained open, indicates several areas which were lacking, but QC is not among them. Where I worked, it was also like that. More than qc not being supportive enough, devs were not able to address the flood of issues. If the bug burndown rate barely matches the rate of new bugs being detected (or even exceeds it, a dreaded situation in which I personally was for 2-3 years) you are watching at an organization with a good oiled QC which is potentially lacking in many other departments.

I mean, just think about the large number of dev teams who work for MSFS and the over the top complex nature of the sim. I have nothing but the highest respect for what they achieved no matter what.

QC is generally overworked anyway.

dd

14 hours ago, fsiscool said:

The symptom that at release 30k bugs remained open

This is one of those things that has been distorted by internet telephone. What Jorg actually said in the Developer Livestream which became the original source of the 30k number was that the QA team logged about 50,000 issues during the course of MSFS 2024's development. Prior to shipping, about 20k of those bugs were fixed. Of the remaining 30,000 logged issues, some were determined to be duplicates (i.e. multiple entries for the same bug or multiple issues that share a common root cause) and others were investigated and determined to not actually be valid bugs. The internet being what it is, people then started repeating that MSFS 2024 shipped with 30,000 known bugs, which isn't what Jorg said at all.

That's not to say that MSFS 2024 shipped with zero bugs; that clearly wasn't the case either (and, indeed, it's exceedingly rare for any software project to ship with zero bugs), but the 30,000 known bugs number that keeps being repeated isn't accurate.

Thanks,

SeedyL, MSFS Community Manager

Edited October 16, 2025Oct 16 by SeedyL

@SeedyL Nice to see some facts from the source straighten things out!

Good to hear from a knowledgeable source. 

The folks who claim that there is no quality control at Asobo/Microsoft are simply spreading misinformation.

8 hours ago, SeedyL said:

This is one of those things that has been distorted by internet telephone. What Jorg actually said in the Developer Livestream which became the original source of the 30k number was that the QA team logged about 50,000 issues during the course of MSFS 2024's development. Prior to shipping, about 20k of those bugs were fixed. Of the remaining 30,000 logged issues, some were determined to be duplicates (i.e. multiple entries for the same bug or multiple issues that share a common root cause) and others were investigated and determined to not actually be valid bugs. The internet being what it is, people then started repeating that MSFS 2024 shipped with 30,000 known bugs, which isn't what Jorg said at all.

That's not to say that MSFS 2024 shipped with zero bugs; that clearly wasn't the case either (and, indeed, it's exceedingly rare for any software project to ship with zero bugs), but the 30,000 known bugs number that keeps being repeated isn't accurate.

Thanks,

SeedyL, MSFS Community Manager

Disregarding the 30k mythical number (totally agree with you there), has any attempt been made by the team to further improve Quality Control over the past five years? 

I ask as someone who has clearly seen (and is capable of seeing) regressions in some/most updates that are very obvious and should've been dealt with, some of which never get fixed.

Let's take a look at the mouse issues in VR for example; 2020 SU7 was a dreadful release in this regard, there was no public beta and the update shipped with a disastrous bug that prevented controller usage when the mouse was "over" a toolbar or garmin screen. It was semi-fixed some 5/6 weeks later but still - to this day - is not working the way it was prior to SU7 (ergo not fixed).

Now this wasn't a user issue, this wasn't happening in just one or two HMDs, it was 100% happening across the board for all VR users, so any member of the internal team using VR must've known about it, it's as simple as that. But yet you shipped with a major game breaking bug and took far too long to half fix it.

Fast forward to SU4 beta, currently a major issue with the mouse/controllers not working together, so why, like the issue above, are these regression bugs not being picked up and dealt with internally in a timely manner?

Those are just a drop in the ocean of the number of regression issue examples over the past five years, which points to poor Quality Control - IMO, far too many obvious issues creep through almost every update that constantly begs the question "how did they miss that?!" and it brings with it the fear that an update will break more than it fixes was has become a meme in certain quarters.

Just to appease those that cannot stand anyone having an opinion that doesn't reflect the team in a golden light with rainbows and unicorns (!), Yes I perfectly understand there will be bugs, I'm not naive to think it'll ever be bug free, Yes I understand regressions will sometimes happen and that they cannot all be fixed during the same beta period, Yes I understand new bugs will appear.

My question relates to the reasons as to why we still have to suffer with obvious regressions, time & time again when ideally they should be fixed ASAP and not swept under the carpet like some have been (see Shutdown Bug in 2020 for a prime example of that thank you Martial...).

I firmly believe there needs to be better Quality Control internally, I want nothing but the best for this title (I wouldn't be here, there and everywhere whilst enjoying the product itself otherwise!), it just pains me to see the obvious issues which no doubt cause the team more work to do.

Thanks.

 

Edited October 17, 2025Oct 17 by MarcG

13 hours ago, SeedyL said:

That's not to say that MSFS 2024 shipped with zero bugs; that clearly wasn't the case either (and, indeed, it's exceedingly rare for any software project to ship with zero bugs), but the 30,000 known bugs number that keeps being repeated isn't accurate.

Oh, I did not want to say that there were exactly or even close to 30k bugs. I presented the 30k figure as a rough number how I remembered it (and you did confirm that I remembered correctly).

In the context of QC which we were discussing it does not matter. Removing the duplicates or the not-a-bug-cases in no way invalidates my statement about QC.

So let me rephrase the part about QC once more very clearly: Managing tens of thousands of issues is not indicating a shoddy QC how @MarcG claimed. These number prove the opposite. QC guys know, that a project is very large, very complex and QC is running at full speed if the ID's in the bug tracking tool become five-digit numbers or are even approaching six-digits. In fact the 50k of managed issues totally, which you mentioned, are a great testimony to the MSFS QC operations. The 50k indicate a very large and efficient QC organization (because I ask once more, how could an inefficient QC raise and manage that many issues?).

6 hours ago, MarcG said:

I firmly believe there needs to be better Quality Control internally

Do you think, if QC had raised 70k issues at release (a total number which could well be reached in their bug tracker by now, listing issues in any state like open, duplicate, fixed, deferred, not-a-bug) the released product would have been better?

In aviation terms, they were clearly behind the power curve, so the dev side failed to close known issues fast enough. In that situation, QC is not to blame. QC is only to blame, if they would not have known the state of the software. But they knew. Which basically was the essence of Jörgs admission in that dev stream.

Marc, don't forget that given these dimensions, it is very very likely that every single issue that is plaguing you exists in their bug tracker. You need to be aware that there are many reasons why known bugs, including regressions, won't be fixed for a release. Sometimes the risk of damaging even more important parts is larger than the benefit of fixing a comparibly minor issue. When public betas start, it is not the time anymore for large scale code changes. Trade offs to be made from that point onwards are e.g. "avoid fixing issues that affect less than 5% of users if it requires architecture changes", "don't touch code anymore which requires large efforts to stabilize it", "minimize changing particular code which is known to cause significant ripple effects", "we can't address these bugs anymore because it would require mandatory updates from all addon devs", "feasibility to fix is not given", "this is not a bug but a new feature request -> deferred", "fixing bug A unmasked bug B, which impossibly can't be addressed anymore for this release".

Btw. I am neither blaming the dev side. What we have is lacking in parts, because it (mainly) is the result of one of the most complex software projects I could think of. In absolute terms. Across all industries. And I have a worked on a lot of incredibly complex software projects. Considering all that, 2024 overall is working exceptionally well.

Edited October 17, 2025Oct 17 by fsiscool

QA and regression testing is expensive ... release to Beta users is nice option if one has the resources to manage.  But Beta users (we/us/you) aren't the best sources for bug replication procedures and narrowing the issue down to a very specific operation that a software engineer can replicate.  Don't mean this in a harsh way, but end users make terrible Beta testers ... I mean it's not your/our job.  Dedicated QA teams will be far more efficient and effective at bug discovery, validation, and replication. 

Unfortunately, obtaining the resources for large QA teams is difficult, it's often the corner that is cut in development projects (I've been begging for dedicated QA for the projects I'm working on, but funding just isn't there even on a 26,000 employee sized company).  NOT, saying this is the case with MS/Asobo as the bugs I've run into for MSFS 2024 have so far been pretty minor (except the saving of controller settings in the earlier release and LOD/culling of objects that should stay in view).  There are some UI designs that were implemented for the benefit of console uses and not as fluid as they could be for desktop users, but it does work.  

As pointed, considering the complexity of this software, it's doing pretty good and I would dare say MS/Asobo have a QA team in place.