Jump to content

Archived

This topic is now archived and is closed to further replies.

Guest robains

3rd party providers and reviewers seriously lacking...

Recommended Posts

what products and addons to include?at this exact moment there are 155273 files in the avsim library. which ones to test? what about flightsim.com's library?there are likely more than 200 payware offerings for FSX ... double that for FS9. which ones to test?then there is the 'base hardware' to test discussion ...--
That's why the need to open up beta testing to a wide variety of users. There are plenty of people out here that are willing to test products. Developers choose to only include their inner-circle of highly knowledgeable colleagues. Sure, they may be able to give good technical feedback but they don't actually test in the environment where the product will be used - on the average users computer that contains a bunch of add ons.

Windows 7 Professional 64bit; P3DV4, ASP4, REXSF, UTL, ProATCx 

I7-4790k @ 4.4; Asus Maximus Formula VII; Asus Strix GTX 1080ti; 16GB G.Skill Trident X DDR3 2400; Corsair H110; EVGA Supernova 1000P2, Single 32" 4k Monitor 

Save a Life: adopt a retired racing greyhound

 

Share this post


Link to post
Share on other sites

I'm a photoreal software developer and I beg to differ with some of the comments posted on this topic. I began using FSX as a flight training tool. I was not a simmer. I discovered how and made some local scenery for myself. Then I got carried away and made a whole lot more. I'm a 55 year old, married, lower-middle class entrepreneur who has tried dozens of money making schemes/scams. I saw an article on the owner of MegaScenery and he was flying around in a jet. So lets talk numbers. The AvSim advertising flyer says they have on average 60,000 visits daily. And their library has 750,000 members. I don't think it's such a small market at all. So lets call it a cool million and figure I'll sell to 1% of the market. That's 10,000 sales WOW. Let me just count on 10% of that and make a thousand sales. OK, it's been 1 year now and I've sold 60. Had to buy some software and hardware and advertising and web-site costs and gee whiz I made 4K and spent 8K. But I can't stop cause Now I'm a simmer. My flight training? Got halfway and crapped out and haven't been up in over a year. My income? This is it, and my wife won't buy my cigarettes and beer. So here I am responding to a forum topic but really hoping to drum up a sale. Simulation technology in our time reminds me of automotive technology a generation past. Like the grease-monkey tinkering in his garage building a Hot Rod, we get to tinker with a lot more, and have the advantage of worldwide communication. I believe eventually, we will unite into an industry of our own.Sim-Savvy

Share this post


Link to post
Share on other sites
But like I said, when's the last time you saw any 3rd party developer log errors to the OS event logs (or anywhere for that matter)? I haven't seen one do it to date ... it's like they perform ZERO error trapping in their code, just crash and bring down FSX along with it.
I'm just going to point out something... this isn't something we can do. It's an OS limitation and there's no way around it. I know, I've tried... even talked with Microsoft and ACES themselves. Just can't be done.

Ed Wilson

Mindstar Aviation
My Playland - I69

Share this post


Link to post
Share on other sites

I have, not 5 seconds ago, managed to get on to the megasceneryearth website, so, I cannot agree with the original poster's comments about his problem with the website access, which could have been attributed to a number of issues, anyway, the website is fully functional at this precise moment.I partly agree with his comments with reviews, but, as has already been stated, you don't get paid to do it, and what qualifications / experience / knowledge do you need to possess to give an impartial review of any content which may slate the Payware supplier's assets to such a degree that their sales will suffer, and of course, I personally have all of the various texture and environment enhancers, thousands of aircraft and hundreds of airports on my computer which MAY give false readings when playing with any product and it's likely that even if it works fine on my PC, it might not work on yours, in which case you would feel completely miffed at having spent your money on an item that doesn't give you satisfactory results, in which case, WHO IS TO BLAME, me for giving you a duff review, OR, the Payware vendor for issuing you with an inferior product ??This is possibly another reason why Joe Public doesn't give public reviews, as, anything inferior would make him lose his street cred on various Flight Simulation forums around the world, some of us have a reputation to maintain.......LOLCheerz. Transport Steve.

Share this post


Link to post
Share on other sites
That's why the need to open up beta testing to a wide variety of users. There are plenty of people out here that are willing to test products. Developers choose to only include their inner-circle of highly knowledgeable colleagues.
Its not as easy to get good beta testers as you think. I can't even count how many times I've had someoneinsist they will be a great beta tester, only to put them on the team and barely hear anything from them.In my experience you're lucky if 1 out of 10 of your Beta testers give you userfull regular feedback.So after a while you learn to value the few really good testers you have and retain them. As far as Beta testers go, more is not necessarily better.
Sure, they may be able to give good technical feedback but they don't actually test in the environment where the product will be used - on the average users computer that contains a bunch of add ons.
Most of the Dev team members are likely to have many of those addons, and your Beta testers will have them as well. But even if you have those addons, many of them are so configurable, it can be very difficult to see an issueduring Beta testing. Often its only when the product is in the hands of 2,000 customers that the issue will then surface. It doesn'tmatter how vigurous your testing is, certain issues simply cannot be anticipated. There are just too many variablesat play regarding O/S, 3rd party addons, and user configurations. Regards.Ernie.

ea_avsim_sig.jpg

Share this post


Link to post
Share on other sites
Guest robains
I'm just going to point out something... this isn't something we can do. It's an OS limitation and there's no way around it. I know, I've tried... even talked with Microsoft and ACES themselves. Just can't be done.
Sorry sir, but the information you were provided is absolutely NOT correct. I've been developing software for 30+ years (back in day when I started on a TRS-80 Color Computer) -- even back then, Error trapping was a primitive feature available for me to use. I have yet to code in a language that does NOT support error trapping. Fast forward to today and most of the dev's appear to use VC++ and/or .NET framework and I can assure you that both of these development platforms support error trapping. There certainly is no OS limitation (just bring up the Event Viewer in your OS, you'll see many many many logged events, not all error, most probably just informational - you hope). Interaction with FSX is a little more tricky but still very easy to implement and certainly doable.Ernie,Your message is somewhat confusing, you say you don't want many Beta Testers yet you go on to say that problems only surface when you have 2,000 customers? Isn't that kinda contradicting yourself? I do agree that it's hard to manage huge volume of testers, but there is a solution -- you pick Beta Testers of wide variance is proficiency -- not necessarily many of them, just a group that consists of highly experienced FS simmer with real world knowledge down to someone with very little knowledge that just wants to fly and crash into things. It only takes a handful of people to cover the entire spectrum.What you have done, is limited your scope to only experienced beta testers with lots of knowledge because they provide you the best feedback -- which is good for you, but not really accomplishing the overall goal. No matter how crazy or difficult the in-experienced beta tester maybe, I find there is always something to be gained from them ... even if it's "gosh, I just never thought anyone would do something like that". Steve,Agree, that's why I no longer read "official" reviews - if the reviewers aren't getting much for their efforts then there is no way they are going to be motivated to do a full exhaustive review -- even if the reviewer gets the reviewing product for free (lets say $50 value) it would take 8 hours to test the product on their system, 3 hours looking at forums, 4 hours to actually write the review, capturing file installs and any registry entry changes - 1 hour, testing with various FSX images in different states using popular products 8 hours -- so we're looking at 24 hours of effort by the reviewer for $50 ... that equates to $2.08/hr (well below minimum wage). Hard pill to swallow when you know the folks asking for the review are making considerably more than that. When I can gain access, I go to the forums (per other's suggestions). But some vendors/devs got smart and lock you out of the key forums (the ones with folks pointing out problems) if you have not purchased the product (this is unfortunate).Rob

Share this post


Link to post
Share on other sites
if the reviewers aren't getting much for their efforts then there is no way they are going to be motivated to do a full exhaustive review -- even if the reviewer gets the reviewing product for free (lets say $50 value) it would take 8 hours to test the product on their system, 3 hours looking at forums, 4 hours to actually write the review, capturing file installs and any registry entry changes - 1 hour, testing with various FSX images in different states using popular products 8 hours -- so we're looking at 24 hours of effort by the reviewer for $50 ... that equates to $2.08/hr (well below minimum wage). Hard pill to swallow when you know the folks asking for the review are making considerably more than that.
Ever hear of the term "volunteering"?http://forum.avsim.net/forum-121/announcement-23-from-avsim-happy-holidays/ :(

Bert

Share this post


Link to post
Share on other sites
Sorry sir, but the information you were provided is absolutely NOT correct.
Rob, have you ever programmed any gauge for FS? If not, then you really have no basis for your statement. If you have, then kindly share (in the appropriate forum here of course) your method(s).As for XML type "gauges," error trapping and recording are most assuredly not possible either.

Fr. Bill    

AOPA Member: 07141481 AARP Member: 3209010556

Interests: Gauge Programming - 3d Modeling for Milviz

Many Thanks to All That Donated To Our Server Drive!

Share this post


Link to post
Share on other sites

If you think it is possible to adequately review the average FS product in 24 hours, then you're wildly off the mark. I spend literally weeks doing it, thus the hourly rate is nothing to do with it, which is just as well when I consider the amount of time I put in. In fact it usually ends up costing me money to do a review even disregarding the 'time is money' aspect, because I almost always end up buying books and other stuff in order to assist in researching the subject material properly and I very often make visits to aircraft or interview people in person in relation to a product.Most products I review get uninstalled and reinstalled numerous times on more than one computer with different operating systems to test their robustness, and this occasionally for both FSX and FS9. Then they are tested with both payware and freeware versions of FSUIPC and other utilities which might have a bearing on performance, then with numerous add-on aircraft/sceneries or other add-ons for FS that might cause a clash. This sort of installing and swapping stuff about is fairly typical for most reviews, and it's usually done on a clean install of FS too on a separate drive, just to make sure there are no clashes from the remains of other files. If you honestly imagine that can be done in 24 hours, you're dreaming, and that's not even including writing the review, then proofing and editing it, not to mention collecting screenshots etc. If I wanted paying for all that work, I'd be charging a fortune, because when do my real job of training people for a living, my daily rate is 400 quid.Some examples of additional expense when doing Avsim reviews...ASE review: Two books purchased on meterology (cost 35 quid) plus a visit to the national weather centre in the UK to ask some questions of the people there. This in addition to the fact that I'm pretty good on meteorology anyway by virtue of being a glider pilot. Alphasim F-104 review: Three books on the F-104 purchased, including a flight manual for the F-104 (total cost 75 quid).A2A P-40 Warhawk review: An original Curtiss P-40 manual purchased (can't recall how much that was) and a very long interview conducted with one of the few surviving pilots who had flown the P-40 in combat against the Japanese in WW2 in order to determine how the real aeroplane handled in combat with regard to roll rates.Flight Replicas Me 262 review: One book on the Messerschmitt 262 purchased (cost 29 quid) plus an interview with someone involved in producing a modern replica of the 262.Captain Sim B727 review: Two books on the 727 purchased, one an original TWA Flight manual for the 727, the other the Boeing 727 Flight Master Manual (total cost 47 quid), plus a visit to interview an ex Dan Air B727 pilot, plus several questions asked of one of my other airline pilot friends in relation to proper VOR navigation procedures for an airliner.Ariane B737-900ER review: Didn't even get a free copy of this, I bought it to be able to review it because there were many people curious about it. I then bought several B737 manuals (cost about 85 quid, plus 80 quid for the add-on itself), plus interviewing several B737 pilots about various aspects of the FMC's VNAV and LNAV capabilities, plus a visit to a B737 to check out the cockpit and galley.Current CS 707 review: Bought the aircraft myself at full price (so again no freebie), spent 65 quid the other day on an original TWA B707-300 flight training manual to check the systems fidelity. Will also be taking a trip to visit a preserved B707 cockpit soon with a bit of luck.That's the reality of doing a proper review. It is what I do, but you will be hard pressed to find many people willing to go to that kind of trouble. From that you should see that I don't do it for monetary gain, I do it in an attempt to be helpful and to contribute to the flight sim community based on the fact that I have some time to spare and enjoy writing and researching stuff. Don't assume all people are all about money, some of us like to volunteer to help make flight simming better, after all, you are on a site which does that, and it costs you nothing to be here.Al


Alan Bradbury

Check out my youtube flight sim videos: Here

Share this post


Link to post
Share on other sites

Ernie got it in one. No product is flawless. Even with a thousand testers, there will be issues.Beta testing requires patience, meticulous observational skills, an ability to write detailed reports and a broad-based knowledge about the particular aircraft/add-on/utility being developed. Beta testing is not about ensuring the product being tested works with other products. In the later phase of testing this is something that can be addressed, but, quite frankly, there are so many products out there, it is IMPOSSIBLE to test for ALL the popular products. Service packs can deal with that.Unfortunately, most simmers - who want to be testers - have a very narrow frame of reference. They focus on a particular area of interest. Testers who want to make sure the product works on THEIR system, regardless of the whole, are useless. If they encounter any problems during testing with THEIR system, they either complain endlessly about THEIR problem, or stop testing altogether. Then there are the testers who's sole focus is to IMPROVE the product with "suggestions". If a product is in BETA, suggestions are noted, but for the most part, suggestions are for the alpha builds. Add the tester who seems unwilling or unable to file adequate bug reports and the beta process gets mired in quicksand as the developers try to extract a coherant explanation from the tester.Testing is HARD.Reviewing Flight Simulator products is much like best-testing. You're going to need patience, meticulous observational skills, an ability to understand the requirements to operate the product, and a broad-based knowledge about the particular aircraft/add-on/utility being reviewed. Unless the reviewer has hands-on experience with the product they are reviewing, the review is bound to miss the mark. For instance, the Concorde-X from FSLabs - I was a beta-tester and wrote the documentation for the product - has yet to be reviewed by AVSIM. This is probably due to the fact that anyone writing a review about Concorde-X better be ready to get their hands REALLY dirty and spend many hours exploring the product. Concorde is a massive simulator. Its systems are daunting, sprawling, magnificent and decidedly old-school. Good luck to the reviewer who has accepted the assignment to write a review on Concorde-X.Good thread, but keep in mind that the flight simulation industry is a NICHE market. We aren't talking about sales in the MILLIONS (like PS3, XBox, Wii games). Try HUNDREDS and perhaps THOUSANDS, if the product is good. Developers can't have hundreds of beta-testers, as that may be their entire market, and sales would be ZERO! ;-)Merry Christmas!


The SUPPORT FORUM for Level-D Simulations products: http://www.leveldsim.com/forums

LVLDF1.gif

Share this post


Link to post
Share on other sites
Sorry sir, but the information you were provided is absolutely NOT correct. I've been developing software for 30+ years (back in day when I started on a TRS-80 Color Computer) -- even back then, Error trapping was a primitive feature available for me to use. I have yet to code in a language that does NOT support error trapping. Fast forward to today and most of the dev's appear to use VC++ and/or .NET framework and I can assure you that both of these development platforms support error trapping. There certainly is no OS limitation (just bring up the Event Viewer in your OS, you'll see many many many logged events, not all error, most probably just informational - you hope). Interaction with FSX is a little more tricky but still very easy to implement and certainly doable.Ernie,Your message is somewhat confusing, you say you don't want many Beta Testers yet you go on to say that problems only surface when you have 2,000 customers? Isn't that kinda contradicting yourself? I do agree that it's hard to manage huge volume of testers, but there is a solution -- you pick Beta Testers of wide variance is proficiency -- not necessarily many of them, just a group that consists of highly experienced FS simmer with real world knowledge down to someone with very little knowledge that just wants to fly and crash into things. It only takes a handful of people to cover the entire spectrum.What you have done, is limited your scope to only experienced beta testers with lots of knowledge because they provide you the best feedback -- which is good for you, but not really accomplishing the overall goal. No matter how crazy or difficult the in-experienced beta tester maybe, I find there is always something to be gained from them ... even if it's "gosh, I just never thought anyone would do something like that". Steve,Agree, that's why I no longer read "official" reviews - if the reviewers aren't getting much for their efforts then there is no way they are going to be motivated to do a full exhaustive review -- even if the reviewer gets the reviewing product for free (lets say $50 value) it would take 8 hours to test the product on their system, 3 hours looking at forums, 4 hours to actually write the review, capturing file installs and any registry entry changes - 1 hour, testing with various FSX images in different states using popular products 8 hours -- so we're looking at 24 hours of effort by the reviewer for $50 ... that equates to $2.08/hr (well below minimum wage). Hard pill to swallow when you know the folks asking for the review are making considerably more than that. When I can gain access, I go to the forums (per other's suggestions). But some vendors/devs got smart and lock you out of the key forums (the ones with folks pointing out problems) if you have not purchased the product (this is unfortunate).Rob
I think you have to kind of keep all this in perspective... I have over 400 aircraft - most of them payware, over 300 sceneries - most of them payware airports but with freeware as well, 3 different weather programs, texture add-ons, mesh add-ons for most of the planet, Ultimate Terrain everything, RXP gauges - all of them for FS9 and all available for FSX, AI traffic all over the place, real time disk defragging, and backup software, and probably other stuff I've forgotten about. Number of hours spent troubleshooting - none; number of hours lost to developer's errors - none, number of FS or FSX crashes - none over the past couple of years at least; number of hours in FS and FSX - in excess of 2000. I'm having a real hard time seeing serious quality issues in the FS product lines. Could developers be a bit neater in some of their work? Sure. Could users spend some time learning to use and take care of their computers? Sure. Could folks spend some time actually reading documentation and manuals? Absolutely. All in all, I'm not seeing any burning issues here.DJ

Share this post


Link to post
Share on other sites
I think you have to kind of keep all this in perspective... I have over 400 aircraft - most of them payware, over 300 sceneries - most of them payware airports but with freeware as well, 3 different weather programs, texture add-ons, mesh add-ons for most of the planet, Ultimate Terrain everything, RXP gauges - all of them for FS9 and all available for FSX, AI traffic all over the place, real time disk defragging, and backup software, and probably other stuff I've forgotten about. Number of hours spent troubleshooting - none; number of hours lost to developer's errors - none, number of FS or FSX crashes - none over the past couple of years at least; number of hours in FS and FSX - in excess of 2000. I'm having a real hard time seeing serious quality issues in the FS product lines. Could developers be a bit neater in some of their work? Sure. Could users spend some time learning to use and take care of their computers? Sure. Could folks spend some time actually reading documentation and manuals? Absolutely. All in all, I'm not seeing any burning issues here.DJ
Here is the Answer to all our problems!! No errors...no trouble shooting....no computer crashes.....Lets all through a dollar or two into a kitty and hire him to write the book on Flight Sims. He didn't say but I know he must have MS word also.

Sam

Prepar3D V4.4/8086K@5.0/EVGA 1080TI SC Black/850W PSU/Windows 10/40" 4K Samsung Monitor@3840x2160/ASP4/ASCA/ORBX/ChasePlane/General Aviation/A2A/Vertx/

Share this post


Link to post
Share on other sites

To be fair though, I largely agree, I actually don't seem to have that many problems with FS add-ons either, and on pretty much every occasion that I have had an issue, which truly is not that often, a quick post on the relevant product support forums has usually elicited a fix pretty quickly.I really can't think of any truly insurmountable issue I've had with anything of that nature. Maybe I'm lucky, but almost everything I buy for FS seems to work just fine on my computer, it really does.Al


Alan Bradbury

Check out my youtube flight sim videos: Here

Share this post


Link to post
Share on other sites
Here is the Answer to all our problems!! No errors...no trouble shooting....no computer crashes.....Lets all through a dollar or two into a kitty and hire him to write the book on Flight Sims. He didn't say but I know he must have MS word also.
Yup - and it's never crashed either... I'd write the book, but nobody would bother to read it - they'd just come here to complain.DJ

Share this post


Link to post
Share on other sites
Yup - and it's never crashed either... I'd write the book, but nobody would bother to read it - they'd just come here to complain.DJ
Exactly.:(

Best Regards,

Ron Hamilton ASEL

Forumsig16.png

Share this post


Link to post
Share on other sites

×
×
  • Create New...