World Update IV Testing Opportunity

[Chock 17,769]

1 hour ago, kaosfere said:

This place really can be special at times.  A chunk of the community routinely goes "Beta testers!  We need beta testers!  Get with it, Asobo!".   Now they announce a beta testing program, albeit a small one.  It may not be perfect, but it's a step in the right direction.  What do we get?   A thread full of "This isn't good enough!  You only picked random plodders, not Serious Simmers!   Get with it, Asobo!"

Some of y'all could be given a puppy, and you'd complain that its ears weren't floppy enough for your tastes.

That would only be true if the people demanding a beta test like this, were the same ones saying this was a cack idea. But that's not me. I never said a word about needing a beta test, I just commented here because as a quality control function, this is a cack-handed half @rsed way to go about testing stuff, and it most likely won't yield any useful benefits.

Above anything, if they were going to take this approach, they needed to pick a broad cross section of hardware users with different bandwidths and such; not go: 'roll up, roll up!' to the first three hundred people who were up and about at the time the gates opened. It's preposterous.

If I wanted three hundred people of varying ages to test read a book, I'd at least have the good sense to pick people who liked to read, of varying ages, not go for the first three hundred people who I saw in the street.

Edited April 4, 2021 by Chock

[Langeveldt 279]

1 hour ago, kaosfere said:

This place really can be special at times.  A chunk of the community routinely goes "Beta testers!  We need beta testers!  Get with it, Asobo!".   Now they announce a beta testing program, albeit a small one.  It may not be perfect, but it's a step in the right direction.  What do we get?   A thread full of "This isn't good enough!  You only picked random plodders, not Serious Simmers!   Get with it, Asobo!"

Some of y'all could be given a puppy, and you'd complain that its ears weren't floppy enough for your tastes.

If you're going to use the puppy analogy, it's like being given a puppy, but it's been deliberately injected with a serum by the people who gave it to you that makes it less and less loving and less and less obedient as time goes on so that cat lovers may one day be tempted to get one also, and that it may be compatible with cat flaps

[Dyno152 51]

31 minutes ago, Chock said:

That would only be true if the people demanding a beta test like this, were the same ones saying this was a cack idea. But that's not me. I never said a word about needing a beta test, I just commented here because as a quality control function, this is a cack-handed half @rsed way to go about testing stuff, and it most likely won't yield any useful benefits.

Above anything, if they were going to take this approach, they needed to pick a broad cross section of hardware users with different bandwidths and such; not go: 'roll up, roll up!' to the first three hundred people who were up and about at the time the gates opened. It's preposterous.

If I wanted three hundred people of varying ages to test read a book, I'd at least have the good sense to pick people who liked to read, of varying ages, not go for the first three hundred people who I saw in the street.

Oh man, if I had a dollar for every expert that knows all the answers....

Progress, not perfection. 

[Dominique_K 10,133]

1 hour ago, kaosfere said:

This place really can be special at times 

 

Why are you so aggressive toward "this place" (how nice...)  ? The MSFS community manager herself has acknowledged that it was indeed legitimate to raise questions about this test.

Quote

Thanks for this feedback! I will certainly pass it on to the rest of the team.

I definitely understand and acknowledge some of the noted pitfalls about this test, it is very short and our pool of testers is very shallow with only 300. However, this test is almost like a good “test” for “testing” as redundant as that sounds, to hammer out proper processes for future testing. Our goal is of course to give more time for these tests and nail down more targeted crowds in the future! We see this as step one to see how a very small version goes.

Regardless, understand these points completely and will make sure they are voiced. Thanks!

Explain us , you are a professional you should know, 

- why the way the test is implemented, is what is needed at that time 

- Why the largest software house in the world and, I think, the largest game publisher in the world, needs to test a testing ? MS is no indie, after 50 years they should know how to test a game, shouldn't they ? 

 

 

 

Discussion: April 1st, 2021 Development Update - Community / General Discussion - Microsoft Flight Simulator Forums

Edited April 4, 2021 by Dominique_K

[Chock 17,769]

 

20 minutes ago, Dyno152 said:

Oh man, if I had a dollar for every expert that knows all the answers....

Progress, not perfection. 

Well in the first place, I'm not claiming to know all the answers, nor even suggesting that what I suggested was going to lead to perfection, so nice straw man there; but if progress is your desire, you don't need to be a rocket surgeon to see that this approach they've taken is not going to lead to much of that.

Edited April 4, 2021 by Chock

[Dyno152 51]

7 minutes ago, Chock said:

 

Well in the first place, I'm not claiming to know all the answers, nor even suggesting that what I suggested was going to lead to perfection, so nice straw man there; but if progress is your desire, you don't need to be a rocket surgeon to see that this approach they've taken is not going to lead to much of that.

Not claiming to know all the answers.....this approach they’ve taken is not going to lead to progress. Says you  🧐

I think if all of us took a glass half full approach we would be a lot more civil and helpful instead of always complaining. 

 

[MarcG 2,929]

Whether finally having a beta test of 300 is important or not, what is important is to note that its still taken this long for Asobo to pull their collective fingers out of their nether regions and do something that should've been in place from the very start.

The late decision to include VR, the late decision to include a beta test team, the poor application of previous patches all smacks of extreme naivety on their part. Yes it's great it's happening but why's it taken so long!?! Rhetorical question of course.

 

[Chock 17,769]

19 minutes ago, Dyno152 said:

Not claiming to know all the answers.....this approach they’ve taken is not going to lead to progress. Says you  🧐

I think if all of us took a glass half full approach we would be a lot more civil and helpful instead of always complaining. 

 

Do you witness me always complaining? Nope. But you do see me pointing out something is a cack-handed way of going about things, when it is very evidently a cack-handed way of going about things.

If you imagine randomly picking the first three hundred people who happened to see an offer of a freebie preview of some scenery - instead of intelligently selecting a broad cross section of suitable testers with a wide variety of hardware, including numerous different GPUs, CPUs, Motherboards, RAM etc - then having forgone the opportunity to actually do some meaningful testing, decide that keeping a sunny disposition about things will mean it all works out fine and is definitely the way to go about testing for a multi-million Dollar enterprise, well, good luck with that approach.

This isn't having the glass half full, this is completely missing the glass altogether and ending up with the drink all over the floor.

Edited April 4, 2021 by Chock

[ADamiani 470]

14 minutes ago, Dyno152 said:

Not claiming to know all the answers.....this approach they’ve taken is not going to lead to progress. Says you  🧐

I think if all of us took a glass half full approach we would be a lot more civil and helpful instead of always complaining. 

 

Considering uncivil those who complain has never produced anything good on this planet. On the other side, there are nations where your approach is officially approved by the government.

Those are the same nations where ALL citizens are strongly (quite strongly) encouraged to take a half full glass approach no matter what happens. For the citizens safety, of course.

You probably didn't mean it, right?

A.

[Dominique_K 10,133]

38 minutes ago, Dyno152 said:

Not claiming to know all the answers.....this approach they’ve taken is not going to lead to progress. Says you  🧐

I think if all of us took a glass half full approach we would be a lot more civil and helpful instead of always complaining. 

 

Hi Mike

The issue is not to know whether the glass is half full or empty but to assess whether the methodology chosen will or not deliver an update free of the type of bugs or nerfing which has plagued the last updates.  

Edited April 4, 2021 by Dominique_K

[kaosfere 3,115]

1 hour ago, Chock said:

That would only be true if the people demanding a beta test like this, were the same ones saying this was a cack idea. But that's not me.

No, Chock, you're usually very reasonable in your criticism, with sincere intent, and even when I disagree with you I can always understand where you're coming from.  But you're not the only person in threads like this, and not everyone's criticism is as thoughtful as yours. 🙂

[scotchegg 4,170]

I wonder if the kerfuffle after the last SU persuaded them to roll out a quick ‘n dirty beta for the next update. Definitely agree that some user’s feedback will be more useful than others, but appreciate betas being introduced in any form after the last update.

[Chock 17,769]

8 hours ago, kaosfere said:

No, Chock, you're usually very reasonable in your criticism, with sincere intent, and even when I disagree with you I can always understand where you're coming from.  But you're not the only person in threads like this, and not everyone's criticism is as thoughtful as yours. 🙂

I don't think some of the criticism even needs to be that thoughtful on occasion. Some of the decision-making and processes, and the clumsy lack of even basic testing at Asobo on occasion, borders on being farcical, and this really doesn't inspire a lot of confidence. To quote just a couple of examples:

We've seen a patch which busted the flaps, yet was still rolled out. This was a glitch which didn't really need extensive testing to discover, literally one practice circuit in a Cessna would have shown that something was amiss when you deployed the flaps. I didn't even need to take off in the Carenado PA28 to discover this problem, I simply dropped one stage of flaps whilst taxying to the runway and the next second my aeroplane was doing a sommersault, and that's a product which is on the built-in marketplace. Yet this erroneous sim update was still rolled out in spite of the fact that we saw a community fix was able to correct the problem easily, and presumably could have been fixed in similarly short order at the Asobo offices, since it was literally a typo which caused it and as noted, it could be spotted on simply loading up a basic flight and commencing a take off roll.

Prior to that, we saw another patch which resulted in the avionics conking out after ten minutes on every flight, for the vast majority of users. Again, this wasn't something which required extensive testing to discover; literally a ten minute flight in pretty much any aeroplane had it showing up, but again, this was a showstopper in a patch which was rolled out in spite of the fact that this one really did make the simulator pretty much unusable for a couple of weeks.

I appreciate that some things can be hard to nail down in software when there are so many potential hardware combinations the software is expected to run on, so it's inevitable there will be some cock ups occasionally in a program which quite literally replicates the entire planet. But when the above to examples are clear evidence of the mind-boggling lack of care and attention going on, it seems to me that testing stuff externally is a needless and lazy jump to the second line of defence, when the first line of defence at the coal face of the PCs in the Asobo office, is not even being used to the extent that a two yard taxy in an aeroplane in the sim would reveal an error.

It's this which makes me question the decision making process behind this latest endeavour in terms of what needs to be addressed. Fixing the crack itself, as opposed to papering over it if you will. Because clearly there's something which needs fixing in terms of basic quality control at a point long before a potential release candidate ever gets near a download server.

I'm as enthusiastic as anyone for the potential of where MSFS can go, and on the whole I'm impressed with the product in spite of the occasional issues, because I can appreciate what the pandemic did to make life difficult in hitting the deadline for release. But this is not the issue at hand now, when there is a usable product on people's hard drives which has in spite of that happy state of affairs, repeatedly been altered detrimentally by what are supposed to be improvements. Frankly, if I was running a project where someone kept knocking stuff out with showstopper problems which could have been easily rectified, I would not be recruiting end users to test stuff as a means to correct this, I'd be finding out which plonker was behind the cock ups in the first place, and telling them to put their stuff in a cardboard box and leave the office.

But whether this is or is not done, what is also somewhat farcical, is the insistence on an NDA for the testers they are set to employ in this task. I suspect that is more to do with industry force of habit than an actually practical benefit. After all, we are not talking about nuclear launch codes here, it's a plane game/simulator, and if anyone who is testing it encounters an issue, surely if they were able to post on numerous flight sim forums about the issue they were having whilst testing, the potential for suggestions from the massive numbers of knowledgeable people out there, could only serve to be of assistance. Effectively you have thousands of additional people able to come to, or suggest, a sensible solution.

If we were talking about a software company not wishing to show its hand to guard against competition then it would make sense to enforce an NDA, but seriously, is there another multi-billion Dollar company sat there just waiting to see if a beta-tester lets slip some trivial detail about some scenery before they can greenlight their own world-replicating flight simulator MSFS-beating product? What great secret is so necessary that volunteer testers could not even mention there's some lovely scenery of Nice in a terrain update for France and post a screenshot of it? I'm pretty sure we can all work out that France is going to look a bit better after an update of the French scenery, there's no real benefit in making a secret of it and one might even claim that the opposite is true in terms of a promotional benefit.

This is my issue with all this stuff. It's not with the notion of testing stuff, it's a lack of confidence in some of the management decisions, based on what we've seen transpire with some frankly very slapdash quality control and auditing. There is clearly a lot of talent and skill at Asobo, but there's clearly someone who is a total plonker as well.

Edited April 5, 2021 by Chock

[marsman2020 1,310]

I routinely do long flights in GA aircraft and have produced lengthy bug reports for ZenDesk including in depth YouTube videos comparing AP behavior across 4 simulators and references to the FAA documentation to update airports (not that it helps, Orbx on contract to Asobo to do airport updates totally mangled one of the airports despite having perfect documentation availability and Bing having perfect aerial images...obvious no one bothered to spend even 5 minutes double checking the work that was done)

I flew community Endurance Race across Australia this weekend, 16 hours straight, and despite closing and restarting the sim at each refueling stop, I still ended up having 2 CTDs about 10 hours into a 16 hours flight, which cost me ~30 min of time on my final time.

But instead of finding a way to make a Beta available to people with the time and skills to produce good bug reports, we get "whoever clicks on the link first gets in".

And they had no way for Steam users to participate (enough though Steam has built in features for delivering Beta channels and we know that MS uses those features in some respects because various third party websites that track Steam package data show multiple alternative branches available via Steam) so as a Steam user I wouldn't have been welcome even if I had been one of the 300 lucky people to click the link first.

I guess there is a silver lining - I have my time open to do other stuff and not to provide free QA services to Microsoft, an organization that should be more then capable of producing appropriate coverage and test cases with paid testers. 

Edited April 5, 2021 by marsman2020

[Kilo60 660]

10 hours ago, Dyno152 said:

Not claiming to know all the answers.....this approach they’ve taken is not going to lead to progress. Says you  🧐

I think if all of us took a glass half full approach we would be a lot more civil and helpful instead of always complaining. 

 

Yeah, cause their track record of past World Updates screwing up the sim and Porking FPS has been so productive and stellar thus far!