FS2024 - Beta Team?

[Tuskin38 6,625]

50 minutes ago, g-liner said:

How long before release did the alpha go out last time? 

September 13th 2019 was when they opened the first sign ups, I'm not sure when the first build went out though.

So around 11 months before release.

Edited December 27, 2023 by Tuskin38

[Drumcode 1,907]

Considering the 2020 testing was a joke last time around, hopefully they will take community’s feedback seriously. I reported at least 20 issues in the last alpha/beta and all of them were released to production in August of 2020. 

[turbomax 3,299]

2 hours ago, Tuskin38 said:

I'm not sure when the first build went out though.

I think it was January 2020

[Tuskin38 6,625]

2 hours ago, Drumcode said:

Considering the 2020 testing was a joke last time around,

Many bugs have been fixed because of testing. It wasn't a joke.

Just because they didn't fix the issues you reported, doesn't meant they weren't doing anything.

Edited December 27, 2023 by Tuskin38

[Matchstick 1,898]

8 hours ago, Tuskin38 said:

Many bugs have been fixed because of testing. It wasn't a joke.

Just because they didn't fix the issues you reported, doesn't meant they weren't doing anything.

No, the reponse to issues raised in original alpha and beta testing was a bad joke.

It's easy to forget just how rough the original release was and how Asobo had to rush patches out post launch to fix things that had been reported long before I joined the beta.

There was general incredulity withing the testing community when MS announced the release date because it clearly wasn't ready for release.
Many testers believed there had to be a secret later build that included fixes we hadn't seen, but no, all the faults were there in the release.

Edited December 27, 2023 by Matchstick

[MarcG 3,031]

8 hours ago, Tuskin38 said:

Many bugs have been fixed because of testing. It wasn't a joke.

Just because they didn't fix the issues you reported, doesn't meant they weren't doing anything.

There were many reports from Alpha testers post release asking why certain bugs had still not been fixed for release, this went on for a fair few Sim Updates as well.

[MarcG 3,031]

Overall the Beta Testing or "flighting" (ridiculous term) as it's called has been largely shambolic post release let alone pre-release and here's why;

1. Jorg Neumann originally wanted to release an update every month without beta testing, no Lead Developer would ask his team of something so stupid on a game of this scale. It was extremely naive of him to think this could happen, a major lack of actually planning ahead which leads us too...
2. Initially there was no Public Beta periods at all, just like adding VR it came from a community outcry to provide a way to help better test the Sim Updates before release as it was clear releasing without public betas was harming the product with new bugs & issues.
3. Whilst very slowly the beta periods did improve it was nowhere near good enough and from a company like Microsoft that was very strange, the whole fiasco over SU7 is a prime example of extremely poor beta planning & application which led to the release of an update clearly not ready for public consumption - an update from which there are still bugs present today. (IIRC SU7 didn't have a pre-release beta at all, it only came about post-release when MS realised it was a failure of an update)
4. One Way Communication - A little history from my perspective; I've been around gaming since the 1980s, I've been on many Public Betas not to mention Private Betas for games/simulations over those decades - therefore I can say without fail that I have never known the communication between Developers & Testers to be so poor as it is with MSFS up to this very day - well it's Zero. The Moderators on the official forums acting as "go betweens" is the most ridiculous system I've ever seen in place, you cannot guarantee that they've forwarded on the issues correctly, or if they have whether they get properly picked up on the other side, or whether they just get ignored altogether (many threads get archived at the end of beta even though the bugs have not been fixed for example). It's simply not working effectively enough and that is very very clear, in all those aforementioned betas I've been in there's been direct chat about bugs & issues, language was never a barrier, it helped towards fixing the most obvious issues and it made you - the tester - feel like you were helping properly as part of the team, with MSFS it does not as it seems like you're constantly talking to a brick wall.
5. List Of Objectives; Again another area that had to be highlighted by the community was a lack of actual directives to test certain areas, this eventually happened but then vanished again for the past 2 or 3 Sim Updates.
6. During SU14 Beta they did indeed have a Multiplayer Flight directive to test visible planes, it was incredibly poorly attended by less than 10 people (so I'm told). In a game with millions of users you have to ask why so few are willing to help? Well it coincided with the Assistance Bug Reset during this beta, which took 4 weeks to fix by which time it was too late to properly test anything as the update rolled out to the public soon after - again poor management of the beta - if there is a detrimental bug keeping testers numbers low then they need to add more time to the end of beta not to mention fix the bug in a more timely manner, a bug incidentally that has been around since the very beginning at that and one that still exists today.

Overall the Betas need a vast makeover and from experience there's a simple few key areas - that a company like Microsoft should have knowledge on - that would make them a whole lot better for MSFS2024;

• Developers active in the beta forums; The actual team coding the game need to be in communication with the testers, asking the questions, requesting the feedback. It's worked very effectively elsewhere and there is no reason it won't work here.
• Better Bug Reporting Procedures; Even the smallest of development studios have better means of which to report bugs, an in game Crash/Bug Report System (widely available) is one simple way. Better categories for specific bugs on the forum, it's a mess at the best of times currently and all threads with bugs not fixed moved to the main bug forum, not archived.
• Proper Directives every beta; Start off with a list of testing directives, gather feedback, update those directives where necessary, add new ones where needed. They did start to do something like this but as above they suddenly stopped without explanation.
• Longer Beta Periods; Many games/simulations have a semi-persistent Beta running and there's no reason why MSFS couldn't have something similar, or if not then they simply must extend the betas for as long as possible to catch as many regressions as they can. Sure they have delayed public release from time to time but it doesn't happen often enough when required.

Personally I think there's been far too many errors over these years with regards to beta practices that I don't see them ever getting it "right", far too many regressions and bugs slipping through betas into the public build, years go by and still bugs remain from past betas/sim updates which is incredibly word not allowed. I've very rarely met a developer or software management team that are so blasé to regression bugs, it's coding 101 to fix your mistakes & errors as best you can before going to public builds with the acceptance that some will slip through the net, but you fix those thereafter in a timely manner, but with MSFS it feels like that should be an every update thing and to sweep those issues under the carpet for another day. That's not to say the beats haven't helped as they have in one way or another, at least we have them to help them out with but how much help overall is the question which is where they could improve.

In conclusion the betas since release have simply been poorly managed, poorly directed and poorly communicated but there is one good thing....they can only get better.

[Matchstick 1,898]

23 minutes ago, MarcG said:

Overall the Betas need a vast makeover and from experience there's a simple few key areas - that a company like Microsoft should have knowledge on - that would make them a whole lot better for MSFS2024;

• Developers active in the beta forums; The actual team coding the game need to be in communication with the testers, asking the questions, requesting the feedback. It's worked very effectively elsewhere and there is no reason it won't work here.
• Better Bug Reporting Procedures; Even the smallest of development studios have better means of which to report bugs, an in game Crash/Bug Report System (widely available) is one simple way. Better categories for specific bugs on the forum, it's a mess at the best of times currently and all threads with bugs not fixed moved to the main bug forum, not archived.
• Proper Directives every beta; Start off with a list of testing directives, gather feedback, update those directives where necessary, add new ones where needed. They did start to do something like this but as above they suddenly stopped without explanation.
• Longer Beta Periods; Many games/simulations have a semi-persistent Beta running and there's no reason why MSFS couldn't have something similar, or if not then they simply must extend the betas for as long as possible to catch as many regressions as they can. Sure they have delayed public release from time to time but it doesn't happen often enough when required.

The thing I'd add to the recommendations is detailed logging that is accessible by the user.

Being able to see what was happening when a crash occurs isn't just helpful to Asobo it's vital to 3rd Party Dev and End Users in helping provide detailed information to help Asobo.

MS & Asobo want people to volunteer their time to make MSFS a better product - give them the tools to do that PROPERLY.

[Drumcode 1,907]

On 12/26/2023 at 8:17 PM, Tuskin38 said:

Many bugs have been fixed because of testing. It wasn't a joke.

Just because they didn't fix the issues you reported, doesn't meant they weren't doing anything.

It was a joke.