Asobo/MSFS need to fix the code base…

[RobJC 2,069]

1 hour ago, mrueedi said:

Can you tell Asobo's defect escape rate (the ratio between caught and uncaught bugs)? No, you can't. And therefore your rant is unsubstantiated speculation.

I am not making any statement about MSFS codebase. Because I can't. I just say, you can neither.

I do say however, that a project needs to be organized pretty good to get so far. The ratio of broken things vs working things would be much worse otherwise. Don't fool yourselves. If you count function points, the lacking parts form a very small percentage of the whole package. There is a vast amount of stable and working functionality.

Also don't forget, not everything depends on the code of MSFS. A lot depends on configuration settings (server and client side). Also, as a distributed system MSFS depends heavily on local and server-side infrastructure.

I run software projects for a living. If we were putting our clients through this over and over I would call a meeting and figure out what the heck is going on. A regression can occur for a variety of reasons, but the same regression over and over should not happen, because the first time it occurs you put that on a test list to double-check before release. This is basic software development 101. Basic steps are being overlooked. The guys need to tighten things up. 

[fogboundturtle 2,708]

15 minutes ago, RobJC said:

I run software projects for a living. If we were putting our clients through this over and over I would call a meeting and figure out what the heck is going on. A regression can occur for a variety of reasons, but the same regression over and over should not happen, because the first time it occurs you put that on a test list to double-check before release. This is basic software development 101. Basic steps are being overlooked. The guys need to tighten things up. 

It easy to point the finger but have you build something has complex as MSFS before that has tons of legacy code and new code ?  I understand your point, yes they should do better but there is a millions configuration and so many mods out there, it's really hard to regression test for all possibilities. Than there is people like you who run OC and play with voltage that could cause all sort of issue.  

Does Asobo needs to do better ? Absolutely, Are people blowing things out of proportion ? yes.  There other alternative like Xplane and P3D are worst and they only release update/content every 6 months or so and there is bug that have been in the game for years. At least we have a company that is willing to work with us to improved the product. I also would like to remind you that there was no beta testing for WU5 and so far it doesn't seem to be any for SU6. Every time Asobo does this, its always worst.

[MattNischan 3,778]

3 minutes ago, RobJC said:

This is basic software development 101. Basic steps are being overlooked.

I understand the frustration, especially from industry folks, but this is a naïve view.

The product is rapidly evolving because the customers demand (loudly) that it rapidly evolve. But it isn't your normal business app, and any comparison to that kind of project isn't going to quite hold up. This is a highly complex piece of real-time software with distributed server-side component, a completely unbounded problem set, 30 years of legacy baggage, and multiple millions of lines of code. The test surface is effectively infinite, and the rate of change is necessarily high.

Folks who have been in the front-line business apps business, even for a very long time, rarely quite grasp what it means to test a real-time simulation. It's the functional definition of chaos theory, because even small changes in the starting state can have large changes in the ending state, and thus even with the most rigorous test plans humanity can construct, they fail under the smallest of timing changes by a tester, or even by an automated testing program. The number of available configurations is huge, and where you are in the world plays a huge part in what errors you can see, but that also changes based on what data comes down from the servers, which is an unknown from the user side.

It requires a much more analysis-driven approach than just simply saying "well then just make the test plan more complete" or "well, just hire more testing resources".

-Matt

[MarcG 2,904]

15 minutes ago, MattNischan said:

I understand the frustration, especially from industry folks, but this is a naïve view.

The product is rapidly evolving because the customers demand (loudly) that it rapidly evolve. But it isn't your normal business app, and any comparison to that kind of project isn't going to quite hold up. This is a highly complex piece of real-time software with distributed server-side component, a completely unbounded problem set, 30 years of legacy baggage, and multiple millions of lines of code. The test surface is effectively infinite, and the rate of change is necessarily high.

Folks who have been in the front-line business apps business, even for a very long time, rarely quite grasp what it means to test a real-time simulation. It's the functional definition of chaos theory, because even small changes in the starting state can have large changes in the ending state, and thus even with the most rigorous test plans humanity can construct, they fail under the smallest of timing changes by a tester, or even by an automated testing program. The number of available configurations is huge, and where you are in the world plays a huge part in what errors you can see, but that also changes based on what data comes down from the servers, which is an unknown from the user side.

It requires a much more analysis-driven approach than just simply saying "well then just make the test plan more complete" or "well, just hire more testing resources".

-Matt

...but then they forget to add in the simple UI terrain option into the VR section even though they added it in the PC section, those are the schoolboy errors that should not be happening in the first place!

I think most of us are savvy enough to realise it's a monumental task in developing this simulation, that much to me is obvious. But they're making the same mistakes, over and over again which even the most un-savvy Joe Bloggs can see clear as day - the tree line Rendering for one, i mean what the hell is going on in the background that causes that to happen - again - it's bewildering and thusly as you say frustrating because it shouldn't be happening.

I personally find it hard to believe at times that they have a direction of where they're taking this Sim, it seems very convoluted from the outside with no apparent direction in fixing the obvious vs improvement. Of course you're on the inside and i trust your word (to a degree!), but from mine and perhaps others points of view it looks like a mess, which is only proven every time a new update comes out that breaks something which was working perfectly fine before.

I don't think we're far wrong with our concerns, even though we're on the outside.

[RobJC 2,069]

30 minutes ago, fogboundturtle said:

It easy to point the finger but have you build something has complex as MSFS before that has tons of legacy code and new code ?  I understand your point, yes they should do better but there is a millions configuration and so many mods out there, it's really hard to regression test for all possibilities. Than there is people like you who run OC and play with voltage that could cause all sort of issue.  

Does Asobo needs to do better ? Absolutely, Are people blowing things out of proportion ? yes.  There other alternative like Xplane and P3D are worst and they only release update/content every 6 months or so and there is bug that have been in the game for years. At least we have a company that is willing to work with us to improved the product. I also would like to remind you that there was no beta testing for WU5 and so far it doesn't seem to be any for SU6. Every time Asobo does this, its always worst.

I have been involved in pretty complex projects throughout my 20+ year career, but the problems aren't solely related to complexity. The permutations are pretty high on this project, and I get that this causes issues. But when people are having a nightmare just downloading this software, that tells you something is amiss. Can you think of any other software that causes people to lose days of their lives just trying to get the software updated? We write these episodes off as "they must have been overclocking, or their hardware is bad" but come on. These stories litter this forum and the main MSFS forums. And that is just to install the application. Now let's consider the fact that pretty much every single feature of this project has serious bugs. Every one of the aircraft have bugs, many game breaking. Weather has been fixed and broken multiple times. ATC has issues. Literally, ever single feature of the application has some elements broken. AI traffic? It's nerfed. Lights work sometimes, don't others. Heck, I turn off my AI traffic and then they show up, yet with it turned on they do not show up. 

Again, I like MSFS overall and think it has a great future. I really do. But the team needs to get control of this project. And to be fair, this is a big project. Not saying otherwise. They just need to focus more on how these things are being managed. I would have a list of regressions that are double and triple checked before every release. And..why no beta for WU5? I think the answer to that is simple. A release date was set and that date must be met. The team are working right up to the last second to make those dates, and it shows. 

It doesn't have to be like this.   

[fogboundturtle 2,708]

1 minute ago, RobJC said:

I have been involved in pretty complex projects throughout my 20+ year career, but the problems aren't solely related to complexity. The permutations are pretty high on this project, and I get that this causes issues. But when people are having a nightmare just downloading this software, that tells you something is amiss. Can you think of any other software that causes people to lose days of their lives just trying to get the software updated? We write these episodes off as "they must have been overclocking, or their hardware is bad" but come on. These stories litter this forum and the main MSFS forums. And that is just to install the application. Now let's consider the fact that pretty much every single feature of this project has serious bugs. Every one of the aircraft have bugs, many game breaking. Weather has been fixed and broken multiple times. ATC has issues. Literally, ever single feature of the application has some elements broken. AI traffic? It's nerfed. Lights work sometimes, don't others. Heck, I turn off my AI traffic and then they show up, yet with it turned on they do not show up. 

Again, I like MSFS overall and think it has a great future. I really do. But the team needs to get control of this project. And to be fair, this is a big project. Not saying otherwise. They just need to focus more on how these things are being managed. I would have a list of regressions that are double and triple checked before every release. And..why no beta for WU5? I think the answer to that is simple. A release date was set and that date must be met. The team are working right up to the last second to make those dates, and it shows. 

It doesn't have to be like this.   

Well, in case you didn't know, most of the download issue are related to Amazon CDN services. I  experienced the download loop myself during the beta so  I opened up my wireshark capture and start looking at the TCP packet only to find out that the Amazon CDN was basically couldn't handle the transaction. It was sending TCP Reset windows Size to 0. Which basically completely stop the download. Now the client seems to be unable to perform any type of resume which prompt the file to start downloading again. hence the download loop.

I'm a solution architect and I work with developer every day. I've been doing this for 20+ year as well. I understand where you are coming from. It seems to me that Asobo has too many builds and  when they merge things, things happen that shouldn't. In this world of agile development, not enough company uses automated regression testing and rely on good old human QA testing which is slower and prone to more error. I really hope there is another beta for SU6 as I feel like when the community gets involved early in the code, the GA release is in better shape.

[urthgental 181]

1 hour ago, RobJC said:

I run software projects for a living. If we were putting our clients through this over and over I would call a meeting and figure out what the heck is going on. A regression can occur for a variety of reasons, but the same regression over and over should not happen, because the first time it occurs you put that on a test list to double-check before release. This is basic software development 101. Basic steps are being overlooked. The guys need to tighten things up. 

if you are so familiar with development you know that developers ask for defects or requirements from their product management team. And definitely not some comments by uninformed customers to fix the code base, just ridiculous…

Edited September 22, 2021 by urthgental

[RobJC 2,069]

3 minutes ago, urthgental said:

if you are so familiar with development you know that developers ask for defects or requirements from their product management team. And definitely not some uninformed comments by customers to fix their code base, just ridiculous…

Then why have Zendesk? BTW, on that topic how many times have we read testers saying they submitted reports of bugs, but felt those reports were ignored? I am not saying Asobo are incompetent, bad programmers, or lousy product/project managers. But there are areas that they really need to tighten up, and hopefully they do. Again, I really like MSFS. They have delivered a really cool project. But some areas of how this project is being managed, specifically around regression, needs to be looked at more closely. It's really obvious. 

[mrueedi 702]

22 minutes ago, RobJC said:

Now let's consider the fact that pretty much every single feature of this project has serious bugs.

That does not mean anything. Because any software has hundreds if not thousands open bugs at any time. Please take that into consideration as well. Check out how the rather simplistic flight simulator FlightGear right now has 376 open bugs.

I would also dispute the "serious"-bugs remark. Because, to be honest, I am not bothered or affected by any of the issues you mentioned. So to me, they are not serious.

That does not mean, that your requirements are less valid than mine.

But it means, that judging the criticality of these issues from and end-user prespective is hard (if not impossible) and we again don't know, how widespread people are impacted by these issues. Out of 100 users, are more impacted like you or more like me? Are you the exception out of 100 or am I? Or is the ratio unaffected/affected somewhere in between? It is natural, that you feel, the whole world is having your issue, a.) because you tend to easily spot others posting the same (but even 20 threads reporting the same issue can come from a extremely small minority) or b) because CTDs keep you from flying so you have time to read complaining threads (which again is not representative for the majority). You need to allow for the possibility, that the issues for the majority are minor or not existing.

And you need to refrain from judging Asobos proficiency. Because for that, you and I lack the data.

 

[RobJC 2,069]

40 minutes ago, fogboundturtle said:

Now the client seems to be unable to perform any type of resume which prompt the file to start downloading again.

Yeah, those are the things that need to be looked at. I see some people are still having these issues. I had it one update, but never again. 

[urthgental 181]

4 minutes ago, RobJC said:

Then why have Zendesk? BTW, on that topic how many times have we read testers saying they submitted reports of bugs, but felt those reports were ignored? I am not saying Asobo are incompetent, bad programmers, or lousy product/project managers. But there are areas that they really need to tighten up, and hopefully they do. Again, I really like MSFS. They have delivered a really cool project. But some areas of how this project is being managed, specifically around regression, needs to be looked at more closely. It's really obvious. 

Again, you have no clue if there are areas they need to tighten up and nothing is obvious.

[MarcG 2,904]

12 minutes ago, urthgental said:

Again, you have no clue if there are areas they need to tighten up and nothing is obvious.

Have you played the Sim since release?! It's obvious they need to tighten up quality control prior to an update release for one, if you can't see that then you must have sand up your nose.

Explain to me why we get the regressions/UI issues/bugs/broken features that worked fine previously etc

Edited September 22, 2021 by MarcG

[HiFlyer 9,338]

9 minutes ago, MarcG said:

Explain to me why we get the regressions/UI issues/bugs/broken features that worked fine previously etc

As soon as you explain why many of us (often) have nothing of the sort, while using the exact same software.

Even now, I see on the asobo oriented facebook flightsim groups people reporting things like "Autopilot doesn't work!!! This is trash!!!" or "7000 CTD's this morning!!! Asobo has no quality control!!!

And while I'm reading that my A320 is cruising flawlessly over PA (or wherever) soon to come in for a flawless landing.

Again.

It goes on and on.

[MarcG 2,904]

3 minutes ago, HiFlyer said:

As soon as you explain why many of us (often) have nothing of the sort, while using the exact same software.

Even now, I see on the asobo oriented facebook flightsim groups people reporting things like "Autopilot doesn't work!!! This is trash!!!" or "7000 CTD's this morning!!! Asobo has no quality control!!!

And while I'm reading that my A320 is cruising flawlessly over PA (or wherever) soon to come in for a flawless landing.

Again.

It goes on and on.

Uumm ok;

Tree line broken again, affects everyone (global issue).

Sensitivity page missing from UI, affected everyone at the time (global issue).

VR pre terrain caching missing in UI, granted only VR users but regardless, affected all VR users.

Flaps issue, global

Terrain spikes, global

The list goes on!

 

 

Edited September 22, 2021 by MarcG

[Bert Pieke 7,584]

2 minutes ago, MarcG said:

Flaps issue, global

I must not be "global" then... 😉