The Missing Link - QUALITY

[nudata 10]

I find the "dumbing down of software quality" disturbing. The customers that accept anything less than a properly functioning product are only setting the stage for even worse conditions in the future, as software companies take advantage of the "peaceful customers".Taking the high moral ground and urging other customers to have patience with a defective product is not a good approach. It may be ok to tell a lynch mob to calm down but we are not at that stage - YET!. If you take a broad view of such a moral position you become a victim of sloppy workmanship and even worse fixes. Some vendors are just a distributor and probably spend little time or money testing what they receive from their contractors. Somewhere in the agreement with these contractors is language to cover bugs. However, I bet there is none on quality! Until there is a realization that sub-contractors have their own issues they will continue to receive defective software which they simply distribute to their customers. There is a cost of doing business and some have not accepted that cost

[Mike S KPDX 597]

... I would love to find out what your last FS related purchase was :-)

[Guest danowat]

Maybe if simmers didnt keep asking "is it done yet, is it done yet", and simmers didnt jump to buy the latest release as soon as possible then maybe the developers would look at the testing system in more depth.Truth is most of us are mugs who will buy the "next big thing" at the drop of a hat, as soon as it is released, and the developers know this.Dan.

[kerosene31 349]

Right or wrong, things are not going to change anytime soon. No payware companies have anywhere near the resources to do what you propose, without raising prices considerably. Anyway, the responsibility lies with the consumer. Do your research before you buy. All these steps should be followed before buying anything:- Do NOT rush out and buy something as soon as it is released. Especially for more complex add-ons. Do this and you are just begging for trouble. Even simply waiting a day or two can provide you with a lot more information so you can make an informed decision. - Read the support forums for the company before you buy. Look at the issues people are having and look for major problems.- Read the forums here. Ask questions. Whenever something is released, there is almost always a thread about it within hours. - Download and read all available documentation ahead of time. Most companies offer manuals free from their website. So many complaints could have been answered by simply reading the documentation that is freely available out there.- Keep an open mind, and understand that 100% bug free products are rare for complex addons. If that bothers you, than wait a while until the eventual service release(s) come out.

[Guest]

Maybe if customers weren't screaming bloody hell whenever a week goes by without an announced product being released and they'd be willing to pay a reasonable price (as offset against the cost of development) for the product they'd get something with less bugs.And add another factor: the massive number of different hardware/software configurations out there.With 2 million copies of FS in the wild there are 2 million different configurations it's running on.When I create a piece of software for our customers at work we have full control over the hardware and OS it will run on (we deliver both and the customer contracts us to maintain both).That makes it a lot easier to create the software, let alone to test it.And even that's not enough. I've been campaigning (so far fruitlessly) for professional testers.Testing software, unless what many seem to think even high up in management of IT companies (which FS addon producers typically are not), is a specialist profession that requires dedicated training and a certain mindset.You can't expect that from the people that usually test FS addons, who are amateurs in every respect (they're typically not employed in IT, aren't checked out on the characteristics of the product they're about to test, etc. etc.).

[Guest Zevious Zoquis]

Don't know what add-ons you're buying but I'm literally dumbfounded almost every time I load up the ones I've bought recently by the quality, depth, and just insane amount of detail they include. If a company like MS with all its vast resources is unable to release a bug-free product (and that's no indictment of MS becuase I've quite literally never owned a piece of complex PC software that didn't contain some bugs) then how can anyone possibly expect a smaller developer to manage it (particularly in light of the complexity and amoount of data involved in many of the add-ons nowadays).

[quantumleap 1]

As a former Software QA Manager where the quality and performance of our product was our most important concern (especially when you go up against the big-boys in the industry who have marketing machines which can crush you), I cannot agree more with what you say. I will also add that quality goes far beyond the software itself, and extends to documentation, installation, customer support, etc..In fact, I read your posting and it immediately brought to mind Laminar Research who develop X-Plane. Many have expressed that their current version 8 was rushed to market and they hadn't even addressed the many complaints from the previous version. They seem to have no formal software testing inhouse as far as any reasonable measure of what that really means, and rely on tossing over low quality version after low quality version to a small group of passionate users who just seem to accept the obvious flaws as "well, that's what you've got to expect from the developer!".It has been widely reported that the developer has no source control process, and cannot simply undo problems he created in previous versions, except by going back in and recoding it from scratch. I can truly believe this as in one case where he broke the code for behaviour of aircraft sat on the ground, such that they started spinning around even with the engine off, he was unable to fix it immediately and took many versions to address it. I fully appreciate that you cannot deliver a bug-free software product (don't let anyone tell you they can), but you should not think it is acceptable to deliver one with such obvious flaws, and you should be testing it to make sure it doesn't have them.Through all of this, these passionate users just kept accepting that this was the way the product was developed and that it was a way of life you had to get used to. They attacked those who pointed out the faults. If they had purchased a consumer electronics product or a car with these level of defects and rough edges, I'm pretty sure they'd be complaining vociferously to who they purchased it off demanding a refund. Why therefore accept this from a piece of software?In summary, if you purchase a piece of software which has obviously not been adequately tested, or demonstrates flaws which should reasonably have been addressed before release, you should let the developer know that it is unacceptable.

[Guest Paul Middleton]

I fully agree with what you say about the need for suppliers to ensure "right first time, every time" quality. It's something I always focus on when doing reviews. In a former life I was responsible for delivering banks' Internet Banking services to customers. The services had to be 100% correct when implemented. Saying "It's got a few bugs but you can live with it while we work on some fixes" just wouldn't cut it.Having said that, my view of this "industry" is that quality has steadily improved over the years. Yes, "Caveat emptor" still applies in some cases, but there are also some excellent products out there that are both complex but well-tested and robust. And it's also possible to add layer upon layer of software and still get something that is virtually error-free. As an example, (and this is of course not an exclusive list, apologies to those excellent products not quoted), I can add on top of FS9 itself:- PMDG 737- ActiveSky- FSUIPC- ActiveCamera and- FS2Crewwhich all sit on top of each other, in an airspace populated by- Ultimate Trafficyet all work virtually error-free. I think that's remarkable and not something we would have seen a few years ago.I also believe that as time moves on, quality will improve even more, simply because those who deliver less quality will go to the wall. Two reasons:- customers will demand better quality and avoid those who don't provide it- poor quality actually costs more, not less. It costs far more to fix a bug, once released, than to prevent it in the first place. Fixing unnecesary bugs cost companies money and makes them less profitable. They will therefore produce fewer products than those who can bring them out error-free.So overall, I'm optimistic about the future of product quality, as long as we all buy wisely.

[Guest Zevious Zoquis]

I too agree that devs need to strive to catch as many bugs as possible before release. But we're not talking about banking here (I'm sure if the avg developer of software add-ons was making billions in profits every year they would be more likely to hire a couple thousand beta testers to work full-time bug hunting :) ), we're talking about add-ons for a computer game and it's important to keep things in perspective. And as in any area of business, some companies are better than others. But one or two bad apples don't spoil the barrel. It's not the notion of striving for perfection I have an issue with, it's the tone of the title of this thread ("The Missing Link - QUALITY") that I take issue with becuase it makes it sound like we're just drowning in a sea of bug-riddled add-ons. That just isn't the reality of things (at least not as far as I've experienced). Over the past year, I've bought all the RealAir planes, the Dreamfleet Archer and 727, the Flight1 ATR and Piper Meridian, and the Flight Factory RV-7. The only one that dissappointed me was the RV-7 and then only becuase the company basically folded shortly after I got it. I've had no serious issues running any of them (can't recall a single ctd at all actually) and while there have been some problems after release with one or two, those were addressed quickly. Now that's not to say that some others haven't had issues with those planes of one sort or another, but judging by what I've read at the relevant forums, I'd say the vast majority of users of the ones I listed are very happy with them. BTW, up here in Canada, we've had one or two incidents over the past couple of years where major bugs have cropped up in software used by various of our banks that have caused quite a bit of trouble, so even the banks aren't immune to bad code... :)

[Guest yarvelling]

From my experience, most of the payware add-ons that I have purchased for FS have worked fine, and as expected...there may be some bugs that affect certain aspects of the system if a certain something over and above the norm is done, but mostly, I've not had these problems. One payware supplier though has in the past put out add-on aircraft that are not merely bugged, just downright defective straight out of the box; essential switches on the panels not operating, or activating something different to what they should, for instance. This is pretty inexcusable as it proves that neither beta testing nor any final 'fitness for release' testing was done...unless it was being done by the office cat!!I do feel that the odd bug creeping in can be excused as it may not show up on my PC at all, but it's there on 2 others, and with the millions of varieties of PC's out there, some leeway has to be given. ...but just general sloppy and faulty workmanship leading to virtualy unusable products on ANY PC should be strongly chastised, berated and made fun of publicly in these forums...LOL!!

[Guest Paul Middleton]

But we're not talking about banking>here (I'm sure if the avg developer of software add-ons was>making billions in profits every year they would be more>likely to hire a couple thousand beta testers to work>full-time bug hunting :) Indeed not. In my final project, we had 6 alpha testers, 4 beta testers, and a one-month field test with 20 volunteer customers before going live to approximately 300k customers. Probably a smaller team than some aircraft add-on suppliers use! :DThere's often a misconception (and I shared it at one time) that good QA is very expensive. It's not, as long as it's built into each stage of the development process. And to misquote that other famous saying about education. "If you think quality assurance is expensive, try multiple catastrophic failures!"

[nudata 10]

It appears that there are some who decry testing because it has as a cost that cannot be recovered given the restricted market. Probably true. However, in my original post I offered, that should the low cost approach be taken (free testers) that a testing regimen be created that the beta testers had to follow reporting on each step of the procedure. The cost is additional time and for one-person vendors that is no cost. Also, as I mentioned, it is better to spend that time in a managed setting preparing the process rather than staying up late at night trying to find a bug that may cost you some revenue or at the least your peace of mind.The market will influence the quality of commercial software. However, this does not mean that the little guy will be left in the dust. If the small developer follows good quality processes (testing each stage) and preparing a beta test regimen, the quality will rival the output of larger organizations. I build complex electronic kits for my other hobby, Amateur Radio. All of the good kits have you build in stages with test objectives for each stage. Quality can be improved. Perfection is beyond reason. Dick Boley @KLBE

[Guest Zevious Zoquis]

How do you know developers aren't already using methods like what you've suggested? I mean they all beta test (at least any that I've purchased from). The problem is that they could have 25 testers testing for a year, and still within a day of actually releasing the product there'd be people finding things wrong. There's way to much variety of hardware systems and configuration, and way to many different FS setups for them to possibly forsee every combination. Nobody is saying they shouldn't test the stuff before they release it. They already do - and I'd be willing to bet they have testing regimes setup much like what you are talking about. But if they didn't release until they new they had perfection, they'd never release.

[mnmon 0]

Paul, your list of the truly quality add-ons is great; mainly because it agrees with my own ;-) I think testing goes beyond just checking for bugs with regard to FS, especially with payware aircraft. I have 3 payware planes that are bug free. They run flawlessly, no problems in the VC, and reasonable frame rates. However, their flight models are awful. Two are impossible to land by hand flying (see Peter Sidoli's comments in another thread), and one even defies all the physics of lift and gravity. If one is producing an add-on for a "flight sim" one should should produce a plane that actually comes close to simulating flight. Unfortunately even MS is guilty here as we all know what the quality of the default planes are like. I over look this as I view FS as a platform for others to produce the good aircraft. I would still buy FS even if MS left off the default planes. Mike

[Guest Zevious Zoquis]

holy smoke, just re-reading my post above and I see an English prof would have a field day with it. Just replace most of the instances of "to" with "too" and replace "new" with "knew" and we're closer to the goal line. :)