Alpha and Beta Testing

[Chock 17,769]

after Alpha and Beta testing, there is then Charlie testing, which is a cocaine party. :( Al

[NoahBryant 26]

Beta: Aircraft is feature complete and essentially ready for release as far as we the developers can tell. We give it to a handpicked team consisting of average simmers, real life GA pilots, and a few partners in the addon industry to basically go and use like they would had they just bought it as a customer. We're also looking to evaluate performance here, as our testers have everything ranging from bleeding edge systems down to machines that are 4 or 5 years old. If problems arise, we address them. If none do, we release!

Meanwhile Tabs' email box fills with requests...I wouldn't want to be a beta tester. I can't imagine how frusterating it must be to test an aircraft where you may have to do a whole flight just to find out it crashes when the autobrakes turn on or something. Then start all over to try to reproduce.

[magentajim 10]

after Alpha and Beta testing, there is then Charlie testing, which is a cocaine party. :( Al

Which is followed by Delta Testing which involves lot's of Liquor. :))

[Cras 6]

Meanwhile Tabs' email box fills with requests...I'll be the first to officially say I wouldn't want to be a beta tester. I can't imagine how frusterating it must be to test an aircraft where you may have to do a whole flight just to find out it crashes when the autobrakes turn on or something. Then start all over to try to reproduce.

I second that. I beta tested for a game once and I hated it. Never wanted to do that again.

[MrPlaneDriver 41]

I also think I'd hate to be a beta tester. Would be good because you'd be one of the very first people to see the new add-on, but for me this doesn't really matter. You can't really enjoy the add-on, cause your work as a beta tester is to search for malfunctions.

[cslawrence 5]

I also think I'd hate to be a beta tester. Would be good because you'd be one of the very first people to see the new add-on, but for me this doesn't really matter. You can't really enjoy the add-on, cause your work as a beta tester is to search for malfunctions.

Partially true, but you get to use the product as you would normally use it. The best beta testing is the kind that tests the product as it's likely to be used by an end-user. The real frustration as a beta tester is when things don't work and you have to find a workaround. It isn't a treasure hunt for bugs as much as some might think, you sort of just stumble upon them.

[Chock 17,769]

Testing a flight sim product is a bit like being a test pilot in the real world in some respects; it sounds like a cool job but in reality is often rather tedious because it involves very fastidious study of things in a controlled environment. If you ever meet anyone who is a real test pilot, you find they are finicky and meticulous rather than a dashing 'gung ho' type as is portrayed in movies. So with beta testing, you'd typically fly the plane a lot, but generally always on the same test route with the same weather settings so that experiences between several beta testers will be consistent and can be compared to other similar flights to see if a glitch can be repeated. Then you have to accurately report your findings and such.It's not always exactly like that, but most of the time it doesn't involve flying the thing around anywhere you feel like going and just playing with the thing, which is a mistake a lot of people make when volunteering to beta test, thinking it's just a free way to get a product early. You do indeed get it early, but there is certainly work involved in the deal, so you find you are 'singing for your supper'. That's a bit like doing product reviews too in some respects, people think that's a great way to get freebies too, and then they realise how much research and testing of thing you have to do, followed by writing a big long article about it LOL.So as with the army, think carefully before you volunteer for anything like that.Al

[RPRIEVER 52]

you find they are finicky and meticulous

Considering that I have underwear on the floor from 3 days ago, I kinda figure I'm out of the running.

[MrPlaneDriver 41]

It isn't a treasure hunt for bugs as much as some might think, you sort of just stumble upon them.

I used to think that (you may have noticed by what I said in my post).

Testing a flight sim product is a bit like being a test pilot in the real world in some respects; it sounds like a cool job but in reality is often rather tedious because it involves very fastidious study of things in a controlled environment. If you ever meet anyone who is a real test pilot, you find they are finicky and meticulous rather than a dashing 'gung ho' type as is portrayed in movies. So with beta testing, you'd typically fly the plane a lot, but generally always on the same test route with the same weather settings so that experiences between several beta testers will be consistent and can be compared to other similar flights to see if a glitch can be repeated. Then you have to accurately report your findings and such.It's not always exactly like that, but most of the time it doesn't involve flying the thing around anywhere you feel like going and just playing with the thing, which is a mistake a lot of people make when volunteering to beta test, thinking it's just a free way to get a product early. You do indeed get it early, but there is certainly work involved in the deal, so you find you are 'singing for your supper'. That's a bit like doing product reviews too in some respects, people think that's a great way to get freebies too, and then they realise how much research and testing of thing you have to do, followed by writing a big long article about it LOL.

I've recently watched a documentary about a test pilot. He spoke exactly as you said. Many people who came to talk to him usually started a conversation by saying that his job must be awesome, thrilling and exciting, but in the documentary you see that the thing is not so funny. Every flight is meticulously planned to the smallest details. The conference made by the test pilot, the engineers and other staff members before every lasts for hours. The flight itself is not randomly made. There are a bunch of steps that have to be followed, each at a time. If one step is not accomplished, the whole flight has to be interrupted and whatever problem that shows up has to be immediately fixed. Once its done, all the process has to be accomplished again.

So as with the army, think carefully before you volunteer for anything like that.

Will do, thank you Al for the advice.

[KCA9304 0]

Answers as they pertain to us:Alpha: Features are still being coded in. Part of our alpha process other developers would probably call beta - near the end of alpha, we send the airplane out to our internal tech team that consists of real life pilots on the aircraft in question, mechanics, design engineers and so on. This is to make sure we haven't missed anything that their expertise would notice before we consider the plane feature complete.Beta: Aircraft is feature complete and essentially ready for release as far as we the developers can tell. We give it to a handpicked team consisting of average simmers, real life GA pilots, and a few partners in the addon industry to basically go and use like they would had they just bought it as a customer. We're also looking to evaluate performance here, as our testers have everything ranging from bleeding edge systems down to machines that are 4 or 5 years old. If problems arise, we address them. If none do, we release!

That's very clear Ryan,My item being MD11,I have not been part of the Alpfa team as a real pilot that has an MD11 rating, and not been able to indicate imperfections (some call it bugs) in the Alpfa fase.Also for the Beta part of the production I had no way to indicate improvement options.(or it must be the lateral FPV presentation on the PFD which is correct on the MD11 now, compared to the 747)Once the product was released, I signed the license agreement to download and have the 'right' to use the product as it is.Since using the product (including patches) as it is, I admire PMDG for producing a real non-Boeing, but American made product, and see very detailed conformity to the real plane.So Ryan: If I make remarks on the forum after the product is released, I now understand I'm much too late.From the above I understand not to make remarks unless invited. Thanks for making this clear.Business must go on indeed, check the last forum reference on facebook is december 02.Regards,Harry

[markadeane 272]

Beta testing for PMDG is doubtless the most professional such experience I've had, and I've beta tested for a lot of outfits over time, many in the FS realm. (FS2Crew was another highlight, Bryan knows his stuff!)That said, it IS still beta testing. So unlike you guys who get the end product on release day and can use it largely problem free from day one, we often have to spend a little more time and effort to do so. It's like getting a new toy home from the store as a kid, only to find there's a problem or imperfection with it, it takes some of that giddy "buyer's buzz" out of the experience, if that makes sense?Still, I do it because I'm interested in the future of home flight simulation in-and-of-itself, and I strongly believe PMDG are setting new standards for how future commercial operations will be conducted in this industry. Formal co-operation with a giant company like Boeing is a BIG deal, and really redefines the idea of what a 'top shelf' add-on developer is. Being a small part of that enterprise is immensely satisfying work.And for the record, beta testing hasn't started. Well either that or I've been fired...Ryan? :)

[aceridgey 330]

[...]And for the record, beta testing hasn't started. Well either that or I've been fired...Ryan? :)

Once you get onto the beta team do you do every single following release? it just says MD11, so perhaps you were key to a md11 project? I don't know speculating :D

[wildwally 0]

Not sure about PMDG - but some or most beta tester are under contract to not disclose any images, features, etc.It's like having a new toy and you want to show it off - but you can't/ :(

[Guest 413X3]

That's too bad if it were in Beta now you might have a chance getting it out by the end of the year... Oh well, keep up the good work PMDG and we'll all be patient :)