Payware quality review

[marcom 139]

Hello fellow developers!This might be a little long, but please bare with me!First of all, let me give you a small introduction about myself:My name is Mark and I am a young software developer without a lot of experience. I am also a very active flight simmer and have been for the last 10 years. Together with a mate of mine, I am also in the process of developing a flight sim (stand alone, not a MSFS addon).Apart from flying gliders, I have spent a lot of time in level D sims (watching pilots perform their checkrides, flying the sims and troubleshooting snags I had found :-))What's my point? I have probably tested every addon (*) that claims to be "as real as it gets" or products by developers who claim that their addon has reached realism beyond anything available sofar... I guess you know the ads. Basically addons like the level D 767, PMDG 747, FeelThere ERJ/737, Flight One ATR...(*) I must confess, I don't have any experience on the payware dash 300.While they are all unique in their ways, I have not seen one product that didn't have "serious" bugs that were never fixed. Not small issues like the wiper blade being the wrong size, but basic logic error that is fixable and can really narrow down the fun you get out of these products. Things like a spoiler inhibit function that stops the spoiler from extending in certain conditions but instead of inhibiting the actual control surface, it stops the spoiler handle from moving on a fbw aircraft...The problem, as I see it, why these things don't get caught is that its hard to find good beta testers. Many developers think that having real life pilots on their beta team will cure this problem but in fact they don't. Why? Because they probably just follow the standard procedures they follow in real life and many logic problems won't occur as long as you are doing everything by the book. The die hard simmer, who plays around for ages won't pick it up either because he is not aware how the real aircraft would perform... You'd need an engineer who developed the real aircraft to test the sim but let's stay realistic :-)Many products have common basic bugs - just try this:on your addon's comm panel, change the decimal portion so that it "overflows" (i.e going from .97 to .00 or from .00 to .97) - you will see that very often the whole integer part changes aswell - not so in reality (at least not on the ones I have tested) I guess every developer would agree that fixing that bug might take about 30 minutes or even a lot less if you know how to fix it.Other problems include lack of system knowledge. Most flightsim developers don't work in the aviation business and its often quite tedious to figure out how a specific system works. If you solely rely on system schematics (they are often wrong!) or on hear-say, you will have logic errors (naturally). This includes stuff like:-- on IRS equipped aircraft having a wind readout while being stationary on the ground.- unrealistic autopilot operation (especially things like VNAV/LNAV)- GPWS radio altitude callouts with the radio altimeter not being powered...Yes, these things are not noticed by the normal user and might also go unnoticed during beta testing. But I do hope that developers will have the will to perfect their products (I know, there is no such thing like perfect software)My motivation behind this posting is the following:I want to create a small organization (not commercial!) that consists of a pool of very experienced simmers/pilots/developers who can test a release candidate and give the product an "approval" if the level of realism is high enough. The developer wouldn't have to look for its pool of testers but instead could turn to this organization which would then provide the developer with "certified betatesters".As mentioned above, I am pretty new to the IT sector (from a professional side) and therefor I don't really know how exactly the legal portion would work (the license to use that software). As I said, the organization would be non commercial but, as betatesting is very time consuming, the beta testers would not be charged for using the product (I guess that's how it usually works?)These are just a few general thoughts and I am very curious as to what other developers think about this? Would companies like Flight1/PMDG ask for this type of "criticism"? Would this "approval" be worth the extra effort / time?I am well aware that my expectations are incredibly high for the price tag, yet I hope that some/most developers do strive for perfection and that they want to be able to say that they have developed the best addon ever seen...Kind regards,Mark

[rcbarend 37]

Hi Mark,Well, besides a freeware designer for FS, I AM a professional in ICT development.Although I cannot comment on every statement you make (like about the individual product/companies) I DO agree with your general observation: beta-testing FS addon in general certainly could be improved !! But probably for a very different reason.Main reason IMO, is in the term "beta-testing".Now, without a doubt, there are more definitions of what "beta-testing" really is. I define it as: "'blackbox' testing of a SW product against a testspecification".But I think such a testspecification is hardly ever made; hence the term "beta-testing" is used in a wrong way.A simple example: compare the default FS9 B747 with that of the PMDG product. Now, which is a "better" tested product (i.e. which product contains the least bugs) ?? Do you have the answer ??I don't (in fact, IMO: the default B747) !!The only thing I'm trying to say is: proper "beta-testing" has nothing to do with realism; it only says something about "how well did a designer succeed in implementing a product he specified to make". But since that specification is usually very unclear in FS addons, there's a very large grey area between "bugs" and "simplifications/limitations". If you get my drift.Hence, don't confuse "bugs" with "lack of realism". These are unrelated terms. Personally, I have made gauges that where very "unrealistic" but were perfectly "beta-tested". AN example from your list:"- GPWS radio altitude callouts with the radio altimeter not being powered..."Which has nothing to do with the product being properly beta-tested, unless you know what the designer intended to implemented.To be even more extreme: in your defintion of "beta-testing", all FS95 addon must be a lot of rubish, right ?? But I'm sure you don't agree :-)Again, testing for realism has nothing to do with proper beta-testing; which, in turn, is something completely different what a user expects of a product. But in that respect you are right: sometimes even the simplest bugs remain undetected by betatesters; just because of the complexity of a product.Let alone the statement of some developpers, that claim that their product has been tested by a real-life pilot; it has nothing to do with beta-testing. I hope you understand where I'm going :-) : striving for realism, and proper beta-testing has nothing to do with eachother.Now, what is it you are after ?????Cheers, Rob Barendregt

[marcom 139]

Hi Rob!"A simple example: compare the default FS9 B747 with that of the PMDG product. Now, which is a "better" tested product (i.e. which product contains the least bugs) ?? Do you have the answer ??I don't (in fact, IMO: the default B747) !!"I get what you are trying to say and I agree, from a "is" and "should be" perspective."To be even more extreme: in your defintion of "beta-testing", all FS95 addon must be a lot of rubish, right ??But I'm sure you don't agree :-)":-) You can bet I'm gonna buy the next level D / PMDG product :-)"I hope you understand where I'm going :-) : striving for realism, and proper beta-testing has nothing to do with eachother.Now, what is it you are after ?????"Actually, I have two goals:1) I want to provide developers with an easy access to qualified testers who can test a release candidate and have a reliable statement whether or not they think the product is ready for release. If no big issues are found, AND the attained level of realism is really high, the organization will give their "approved by XXXX" which would act as a proof of realism. (Of course, not every developer will intend their product to be approved, especially if the target customer is the eye candy lover)2) I want to provide the customer with an objective measurement of realism. If a product gets the "approved by XXXX" patch, the customer will know that this product is really as good as it claims to be. (ie. as real as it gets etc...). New/small companies could really benefit from this!I am aware that this will take time. The organization has to earn its trust from both the developing companies and the customers. Also, it will take some time until the reputation can rise high enough for it to be taken serious. (which does not mean that until then the required level of realism will be lower to get the "approved by XXXX" patch).I hope you also get my drift :-)Regards,Mark

[Guest rkruijer]

Hi Mark>My motivation behind this posting is the following:>I want to create a small organization (not commercial!) that>consists of a pool of very experienced>simmers/pilots/developers who can test a release candidate and>give the product an "approval" if the level of realism is high>enough.I appreciate your dream but let's have a look at the practical side of it.1. A non commercial developer wants your team to have a look at his product before he makes it available. Would your team take him on?You will not be rewarded a free product as it is free to begin with.Testing is a time consuming job.2. A commercial developer wants your team to have a look at his product. You expect a free final version for your effort. How can you be sure that your team will discover the flaws and bugs? You would have to have a complicated protocol to know where to start en where to end. Still you will not discover what some of the thousands of users will accidentally discover. "There's always one more bug!" I found bugs in my software after 20 years!!! A good commercial developer makes patches because it relies on users to test the product. A greedy developer will not spent too much time on it end goes on to develop the next product.What you are saying is that your team will test the product, award 1 to 5 stars or tell the developer to go back to the drawing table.Very quickly commercial developers will make you an offer that you can't refuse. How do you maintain your integrity?How do you preserve unity in your team?>The developer wouldn't have to look for its pool of>testers but instead could turn to this organization which>would then provide the developer with "certified>betatesters".Why do you think they would change their present ways.Just because your are saying that their beta testers are no good and your team will do a better job?I would suggest not to test release candidates but test a product that has been released. The drawback is that it has to be paid for but if you do a good job you may find sponsors for a site. Stay independent and reward products with 1 to 5 stars. Commercial developers will beg you to review their product and sent you a free copy knowing that if you say it's good it is worth their while. Something like "Toms Hardware Page." Users of the product may give you enough input to keep you going. Just being practical.Roelof Kruijer

[marcom 139]

Hi Roelof,Regarding freeware addons - as long as they are "serious" (i.e. not just a panel bitmap with a few standard gauges) then sure! My main motivation is not the fact that I can get the product for free, but that I can have fun with it (because it behaves close to the real thing).Of course, I can not guarantee that we (let's hope there will be a 'we') can find each and every bug. That's not my intention either. I just want to get the big things out of the way (again, big in this regard means stuff like I mentioned above). Sofar, I have not 'flown' an addon where I didn't find an error within 10 minutes. I know this sounds pretty arrogant and please forgive me for it, but that's the truth! I don't think its a good marketing strategy to sell a product that should still be a beta version, wait for the customers to find the errors and then release an update (which itself wasn't tested thoroughly). In my view, that just annoys the customer."What you are saying is that your team will test the product, award 1 to 5 stars or tell the developer to go back to the drawing table."Well, I haven't thought up a grading system yet, and it might just consist of "tested by XXXX" and "approved by XXXX" - the first meaning that we got rid of the big bugs but aren't satisfied with the overall realism while the second statement means that this is one product you shouldn't miss out on...It will be the developer's decision to go back to the drawing table. All we can do is write a recommendation. Its also up to the developer if he wants to mention our efforts if he isn't satisfied with our outcome."Why do you think they would change their present ways.Just because your are saying that their beta testers are no good and your team will do a better job?"Let me just say that I don't think that all present flightsim beta testers are no good! I know a few personally who I think do everything possible to make a product as bugfree as possible - sometimes even more so than the developer himself :-)Anyway, for those developers who do want to compile a project as bugfree as possible, we provide them with a team of 'professional' betatesters. They don't have to worry about finding the right people (which I bet requires quite a bit of time aswell).Testing a product once it has been released is of course possible aswell. The problem here is thata) the customer has no objective way of finding out how good the product is before he buys it (as long as no review has been written which usually takes a couple of weeks):( The developer misses out on the opportunity of our recommendation (ok, as long as our reputation isn't high enough, this doesn't count that much).BUT repudation goes a long way: I bet that if PMDG were to release an aircraft, a lot of people would buy it even if they don't favor the aircraft in question all that much... (nope, I am not aiming at their Airbus 'cos everyone loves Airbus, right? :-) )Regards,Mark

[LAdamson 240]

>:( The developer misses out on the opportunity of our>recommendation (ok, as long as our reputation isn't high>enough, this doesn't count that much).>BUT repudation goes a long way:IMO, you'd just be another "middle man", trying to get a bit of the pie. This is done all the time in the building trades. Someone comes up with a website & advertising to make a consumer believe that all business should be done with someone on this particular "approved list". Yet these outfits constantly come and go!The well known flight sim developers have quite a diverse group of beta testors, as it is. Most of us already know what to expect from them before purchase. We don't need another middle man for a supposeable stamp of approval. Of course, this is all my own opinion.L.Adamson -- beta testor, pilot, and "all that stuff"

[Au MaV 0]

Hiheres a good idea for any designer/beta tester http://www.sim-outhouse.com/sohforums/showthread.php?t=20633 Wozza

[n4gix 4,947]

Just my 2 cents worth in way of an observation. Unless you have a comprenensive and demonstrable knowledge of the gauge and panel system in FS, including all of its foibles, errors and limitations, you're opinion won't carry a lot of weight. ;)Just as a brief example, let's take your observation regarding the perceived "error" of the "whole digit rollover" while tuning the decimal portion of the COM or NAV radios.Are you aware that this is the way ACES has hard-coded the radio tuning system? If a developer wishes his product to be compatible with 3rd party developer's products (such as Go Flight modules), then they must stick with SDK methods when programming things like the radio stack, simply because using "custom variables" and "custom code" will break compatibility with such products.Just as a real example, a customer has asked if it would be possible for him to program a key-combination to access the pressurization system on my Citation II, so he can assign it to one of his hardware control modules. The answer unfortunately must be "NO," since FS doesn't model a pressurization system! I have to use custom variables and program the logic within the gauge itself.Understand that all developers must take into consideration myriad factors when setting forth the various criteria, methodologies, and goals for any project's development. Sometimes a perceived "bug" or "shortcoming" is simply a side-effect of being forced to compromise a particular "feature" because of compatibility issues, or constraints of the sim's panel & gauge paradigm.Of course, just like Freud said regarding cigars, sometimes a "bug" really is a bug! But, without an indepth understanding of the entirety of the various SDK's issued my MS, how can you know which is which? :)

[WarpD 824]

I particularly like the comment about VNAV. Do you realize that FS doesn't support any autopilot VNAV capability whatsoever?Any plane utilizing an actual VNAV autopilot has custom software coding in place. It's 'accuracy' is limited to the accuracy of FS and it's flight dynamics at that point.I don't think you grasp enough of the intricasies and limitations of FS itself to be able to provide a true opinion regarding "as real as it gets".

[WarpD 824]

This is another 'beaut' showing your relative lack of actual FS knowledge:"on your addon's comm panel, change the decimal portion so that it "overflows" (i.e going from .97 to .00 or from .00 to .97) - you will see that very often the whole integer part changes aswell - not so in reality (at least not on the ones I have tested) I guess every developer would agree that fixing that bug might take about 30 minutes or even a lot less if you know how to fix it."That's not a coding flaw, it's an FS design. FS has it hardcoded that when the rollover is hit it increments the integer part of the frequency.Sure, there are ways around the design limitation of FS... but that's not a bug.

[Guest iholrf]

That's not to say his idea is without merit. I think there might be a place for a credible standards compliance testing organization. However the keyword there was credible. It might better come in the form of some sort of ACES FSASQL (FLight Sim Add-on Quality Labs).A non-aligned organization could be started, but it would require persons with a proven track record to be part of the *testing procedure standards* body (vs the addon standards testing body themselves) with oversight on the testing and testors.The work involved would be rather enourmous to do right I suspect. The two to three year turn around times for FS might prove challenging in keeping the standards current unless ACES has a clearly defined long term roadmap. Sort of like treating FS as an OS.I suspect it would take at least one 2 year FS release cycle to actually get up to speed and iron the wrinkles out, so if there are lots of changes to the engine, it may be back to the drawing board before it really ever gets going. If not then it might actually prove useful. Of course then there is the whole legal issue...Shad

[Guest iholrf]

>This is another 'beaut' showing your relative lack of actual>FS knowledge:>>"on your addon's comm panel, change the decimal portion so>that it "overflows" (i.e going from .97 to .00 or from .00 to>.97) - you will see that very often the whole integer part>changes aswell - not so in reality (at least not on the ones I>have tested) I guess every developer would agree that fixing>that bug might take about 30 minutes or even a lot less if you>know how to fix it.">>That's not a coding flaw, it's an FS design. FS has it>hardcoded that when the rollover is hit it increments the>integer part of the frequency.>>Sure, there are ways around the design limitation of FS... but>that's not a bug.>The man had an idea. He wanted to discuss it. Why the above? It did not need to be said. The man was posing some examples with a clear statement of his own limitations. Is it possible the self-appointed thought police in this place can give the man a break and reply to his idea and not look at what *you* deem to be his credentials (like they mean or prove anything).ThanksShad

[WarpD 824]

>The man had an idea. He wanted to discuss it. Why the above?>It did not need to be said. The man was posing some examples>with a clear statement of his own limitations. >>Is it possible the self-appointed thought police in this place>can give the man a break and reply to his idea and not look at>what *you* deem to be his credentials (like they mean or prove>anything).>>Thanks>ShadFirst, the man (your term) is offering himself up as a starting point for a group of 'experts' who decide what is and isn't "as real as it gets". The first requirement for that is knowing the limitations imposed on those who create the aircraft and panels. If he doesn't know some of the simpler issues... he's in no way credible when offering himself up as a so-called 'expert'. Especially when he references an actual DESIGNED behavior within FS as a bug in third-party products. He simply isn't knowledgable enough to lead a group that makes such a call. As for the thought-police comment... because I post a dissention doesn't make me a member of such concepts. I never said he couldn't or shouldn't offer his opinion. What I did say is that he's not knowledgeable enough on the topic at hand to be in a position to make the call for "as real as it gets". There's a difference. His example is based on a real limitation of FS and has absolutely nothing to do with anything a third-party developer creates. His example about VNAV not functioning within FS is another perfect example of my point. He simply doesn't know that FS literally does not support things he's considering to be bugs and/or shortcomings of third-party products.Until he knows the difference between what FS can suppport and what FS can not support... he's not in a position to offer up any 'authority' regarding whether a third-party product is "as real as it gets" for FS.

[marcom 139]

Hi!I'm happy to see ideas and opinions coming in! Thanks to everyone who did post!Regarding my knowledge / experience:Well, first of all - you are right! I haven't created a gauge yet. What I did do was create a small FBW flight controller that accessed FS from the outside via FSUIPC (which is a really magnificent addon - thanks Pete!) and an EADI that could be run on a second computer.I have also been simming since sublogic's ATP. So I do think I have a little bit of experience in this field (just not from a developer's perspective, I agree).Regarding the limitations on FS: I am very well aware that they exist and what they look like - and so is everyone else. In my opinion, if you are a serious developer and want to create an addon that surpasses everything known sofar, you should not rely on FS for systems modelling! FS is good in regards of scenery and the flight model can be quite right (except for the obvious flaws such as ground control). Heck, you could forget about the entire SDK (well almost, you still need it for the output) and use FSUIPC to set your flight control surfaces...3rd party addons such as the GoFlight MCP would't work out of the box but there are ways around that limitation aswell... Its all a question of how creative/experienced the programmers are and how much time/energy are they ready to invest. Almost anything is possible, you just need someone to do it for the first time. (the first radio altimeter, the first realistic electrical system, the first more or less realistic autopilot, the first functioning FMC, the first TCAS, the first weather radar)... I remember developers stating that none of these items could ever be modelled due to FS's limitations. Well, except for the weather radar (that is still not natively supported by FS), these systems just required some "thinking out of the box". I'm sure one of these days someone will program a decent yaw damper...In a nutshell: many of the standard systems FS uses (the radio panel, the default autopilot, the yaw damper) are extremely "dumbed down". But no developer is required to use them!Again, I'm not a developer so I could be totally wrong here. But I doubt it :-)Kind regards,Mark

[Guest JeanLuc_]

Mark, you must be commended for your courage to come in this "expert" board and offer some "super-expert" advises! No wonder some of the reaction.Further reading the thread, I guess I understand some of the issues you are refering to. you are right some radios rollover, and this is a bug (although I wonder if there are some radio tuning units doing just that as well rolling over), and it is as true to state that this is a FS limitation and not a bug. In your particular case, I understand the point: it is not because it is a FS limitation, that developpers cannot go around and implement their own (and in this case, the Radios, it is very easy for anyone to code the tuning logic).As for VNAV, it depends on too many factors (FDE, precision of the A/P systems code, update rate of the systems, user FPS...). Yet, it could be coded, and without FSUIPC, you can bend FS to comply to your own A/P orders on the control surfaces for example.As for the weather radar, you may be surprised from the results of this:http://www.avsim.com/pages/0506/RealityXP/RealityXP.htmNevertheless, why in this market do you think we need such an "accreditation" entity? Why is it not needed for operating systems, graphics/painting programs, music programs, etc... ?