Sign in to follow this  
n4gix

Avsim reviews a case for a rethink?

Recommended Posts

Do you agree with the way reviews are scored, can you think of a better more accurate way?I was personally shocked a while back at the review and scores given to the Aeroworx B200 (In my opinion one of the most "realistic feel" aircraft in MSFS.I tend to agree there are outside pressures which influence the reviewer and it takes a bold reviewer to recieve a free addon be lavished with attention by the producer to then rubbish the product.an addon aircraft is a mass of code and programmimg compiled to give a simulation of a real aircraft.What makes a poor simulation? a good one and an outstanding one?Is an outstanding simulation an aircraft of outside visual perfection and one which hits the numbers or can a poor outside visual rendering of an aircraft which may not hit the numbers or may have some minor flaws still be an outstanding simulation.I argue for the latter as whether an aircraft simulation is good is more of a celestial thing.A chef can cook by the book adding each ingredient measured to the gram.Another chef can instinctively throw in ingredients yet his product is one of unique flavours and an eating delight while the product of the first chef somehow doesnt cut it. His product might lack " soul"For me a review should indicate how closely all that code creates an experience of flying the real aircraft and that should be the priority marking.What gives an aircraft that X plus for you? How would you change the marking system? would you change it at all?I may have up to 20 payware aircraft on my system yet I still come back to flying the same 3.May I add that this is a general topic and not a reference to PSS in an earlier thread who make superb products nor is it a criticism of the reviewers at Avsim whos dedication and efforts are much appreciated.Peter

Share this post


Link to post
Help AVSIM continue to serve you!
Please donate today!

Whilst in many respects I agree with you, please remember many of he review people have NO real world flying experience.More and more, things tend to be rated on how pretty they are and how many toilets you can view, which for some is all they can go off due to the above.What narks me about some is how out of scale they are in comparison to others, I wont mention names as it would not be fair, but it does sometimes remind me of the story of the emperors new clothes.I think also having read what you say, should products affiliated to site management be reviewed by total independants ?Not seens any pics for a while, I enjoyed them.Ash

Share this post


Link to post

>I was personally shocked a while back at the review and scores>given to the Aeroworx B200 (In my opinion one of the most>"realistic feel" aircraft in MSFS.You and me both. I recall that one of the reviewer's gripes was that the B200 "ballooned" on landing. It's well known for doing this in real life. The B200 deserved a 4.5 imho.It would be nice if avsim added a "member rating" to each review. Each of us would get the opportunity to rate the product on a 1 to 5 scale. Many game review sites do this.

Share this post


Link to post

>It would be nice if avsim added a "member rating" to each>review. Each of us would get the opportunity to rate the>product on a 1 to 5 scale. Many game review sites do this.which would only lead to massive flaming of addons from manufacturers some people have a gripe with...Such features always do, as is shown time and again.

Share this post


Link to post

>Whilst in many respects I agree with you, please remember many of he review people have NO real world flying experience.More and more, things tend to be rated on how pretty they are and how many toilets you can view, which for some is all they can go off due to the above.

Share this post


Link to post

Trouble is with reviews is that everybody has a different opinion on what consitutes a good product.I think people just have to take the reviews at face value, pick through the reviews for solid facts, not the reviewers opinion, and just use them as a guide, and not a decider for purchasing the product.No matter what AVSIM or their reviews do, there will always be people who agree and disagree with reviews.Dan.

Share this post


Link to post

How about just removing the scorecard bit alltogether?It's easy to forget everything else written in the review and just look at the score at the end of the article. With no score at the end, people would be forced to read through the review themselves and get their own idea of what score the product should receieve based on what *they* consider to be important features and potential issues.

Share this post


Link to post

I rarely even read the reviews anymore. Why? Two main reasons. First, I feel that the scores are IMHO very inconsistent. But I also feel the scores are pretty high, and rarely, if ever, is a product rated below 4 stars. If you scan the Commercial Products review pages, there are only 3 or 4 products below 3-stars, most of those are several years old (the most recent very low rating almost two years old). At least 70 - 75% of all of the reviews are 4-stars or above. That leaves about 40 reviews 3-1/2 stars and under, and only about a dozen are 3 or under. That tells me that either that the ratings system is inflated, or, "bad" product reviews are shunned. I don't believe this is a conspiracy, but I do feel it is there - whether its a "good news only" syndrome, marketing pressures, or just coincidence.Secondly, by the time a review comes out anyone can gather a better feel for the product just by "listening" in the forums. The info is much better, and, drawn from a wider base than a single review. There is a better chance to make an informed opinion off the forums.BTW, I have commented on reviews in the FEEDBACK area where I disagreed greatly, it is a option that is available.Joe Lorenc

Share this post


Link to post

Yeah, I think to remove the scorecard is the way to go, as I mentioned in the "other" thread. Just an occasional AVSIM award when something really really calls for it..rgrds fredrik

Share this post


Link to post

>How about just removing the scorecard bit alltogether?I never thought that I'd get into this thread but I must agree with JimmiG. I would prefer just a review without the reviewers subjective "score." R-

Share this post


Link to post

Yep! Review without scorecard and for really, really outstanding products a speacial award. That's the way to go, also IMO.Wolfgang

Share this post


Link to post

Peter, You have real world experience and are a long time simmer, please don't think I'm being rude but what about you doing a review ? I'm sure it would carry a lot of credibility ?RegardsAsh

Share this post


Link to post

I used to do the occasional revue as a guest reviewer for Avsim.Now the complexity especially of some of the avionics is so high you almost require each part having a review in its own right.I personally cannot see how you can do justice to an aircraft like Concorde unless you are a type rated pilot or an engineer or use a team of simmers to cover different aspects of the sim.All these sim addons are flavours of the real thing and in that way a high part of the scoring should evolve around how much of a feeling of reality the particular aircraft holds not whether its a couple of kts short here or there.The second part which should hold high marks is useability. Can you set up a small IFR trip with a departure climb cruise approach and ILS to land. All this should be handflown within IFR tolerances and all the Navigation features should be readely at hand.If this is almost impossible to do then the flight model is flawed or the cockpit systems etc and the way they are presented are flawed.Finally no aircraft is any good if it stutters and chugs around the screen so my third point would be framerates and smoothness.Tom Wood did a superb job on such a complex subject and as I said at the start this isnt directed at his review or the PSS Concord which I dont as yet have.Peter

Share this post


Link to post

WolfgangMaybe "Impressions" would be a better way ahead for single author "reviews" I also agree that in some ways an impression without scoring would be better.An impression would cover distinct areas and would come across more as a recommendation than a review.If the reviewer (impressionist) was so impressed then it could be handed over to others at Avsim to try and they would decide whether the aircraft should be presented for a Gold Silver Bronze award.Peter

Share this post


Link to post

Brronze, Silver and Gold awards, or maybe just silver and gold awawrds, would be good, as long as they are used with care...it shouldn't be something that every other product receives.

Share this post


Link to post

"Can you set up a small IFR trip with a departure climb cruise approach and ILS to land. All this should be handflown within IFR tolerances and all the Navigation features should be readely at hand".Well, this brings me back to the point of how many reviewers would actually know how to do this never mind set it up.As you say and rightly so, Tom did his best and all credit to him for having a go. As we dont know any of the reviewers pilot / flight skills (please..this is not directed at Tom or anybody else)It makes things a little difficult to judge exactly what is a "review." Ash

Share this post


Link to post

Whats the difference between a gold award and a five star rating? I think it's important to remember that this is still a recreational hobby and that most of the people involved are simply guys who are into aviation as opposed to type rated pilots. To that extent, I don't have much of a problem with the reviews ftmp. Any of the planes I've purchased that have been reviewed at Avsim or Flightsim I've pretty much agreed with the review. I do think that the recent rash of 5 star ratings is a bit iffy, but otoh I think there has also recently been a bit of a surge in the overall quality and level of detail of the add-ons that have been released. I mean some of the stuff that's come out in the past few months is just remarkably good. I also don't find it that odd that the review scores tend to be pretty high since the reviews are by fellow hobbyists and they are going to naturally want to use good stuff just like we all do. I don't think reviewers need to be pilots. All they really need is a good knowledge of the sim and a good range of experience with different add-ons for the sake of comparison. Most of the time the reviewers specifically make mention of the fact they don't have any real world experience with the plane anyway...

Share this post


Link to post

>Whats the difference between a gold award and a five star>rating? Gong from memory, the gold award is given to outstanding freeware, and the stars are given to payware.

Share this post


Link to post

Really there is nothing that says that a real world flyer can make better reviews. If he have flown a Concorde he may have easier to tell if it flies correctly yes but otherwise it

Share this post


Link to post

>>Whats the difference between a gold award and a five star>>rating? >>Gong from memory, the gold award is given to outstanding>freeware, and the stars are given to payware.>Nope,Not in this case.http://www.avsim.com/pages/0105/concorde/concorde.htmThe Concorde received both a Gold medal and the stars, so it is available to both freeware and payware from what I can infer.I also feel that folks should use a review as just one aspect of evaluating a purchase. What is most important to me is after the release, how others feel about the package, how bugs (or whatever PC term is used), are addressed by the seller, etc...I only buy an add-on after it has been out for a while. I don't need to be the first one to publically beta test an aircraft. ;-) I'll let the others do that for me, and see how it evolves. That way, when I do purchase, I will most likely get a better product and without the hassles.The biggest mistake people make in being dissapointed in a product is running out and buying it right after release. IMHO. They would be better served if they wait for the reviews, user feedback, etc....With regards to the reviews, the reviewer is basically just another user expressing their opinions. AFter a while you can make a value judgment on certain reviewers reviews as to whether they can be judged as objective or not.I know one who calls themselves a reviewer, and basically they are a total JOKE. They might as well put a bought and paid for sign on their reviews, for basically they are as worthless as can be. (Not here by the way.) Subjective is not in their vocabulary, and the worst part is when these so-called reviews show a link to the product which is an affiliate relationship link, IE the website is in partnership with the product. Check out the links as well.For example, in the Concorde review here at Avsim:http://www.avsim.com/pages/0105/concorde/concorde.htmLook at the link for the product, clean as a whistle, no partnership buy link exposed here:http://www.phoenix-simulation.co.uk/Now, go look at some other reviews at other sites, and compare the links to the product, and look for the numbers folks, they have a NON-DISCLOSED relationship with the producers, and IMHO, their review is WORTHLESS, and shows absolutely ZERO Integrity, and should be IGNORED for what it is, a sales pitch.I like the reviews here, and I don't see as much bias as I do at other places. But as always buyer beware, practice a little delayed gratification and don't buy anything right after release, and you might just end up with quality products, which I have for the most part, I only regret one purchase I've made in this hobby.One more thing, I always take with a grain of salt any member review of a payware product where their signature states they were a beta tester. This applies only for payware. I've seen quite a few examples of so-called objective users who rant about a product where they were a betatester.If they beta test a Freeware product, no harm done, IMHO, for the worst you have to do is delete it.Anything I beta test I pretty much keep to myself anyway, for I feel its purpose is to perfect the product, and not gloat about being an insider. And yes, I understand all too well the marketing strategy involved here with those signatures. I just ethically have a concern with the concept, since many of these folks are also the ones bragging about a product right after a release, or shortly before to drum up business, IMHO.One more addition, these are only my opinions, and I respect that yours may vary.Best Regards,JoeJOIN The AVSIM RTW RACE FLIGHT TEAM****************Grab My FREEWARE Cessna 172 Voice recognition Profile here:[a href=http://library.avsim.net/esearch.php?CatID=fs2004misc&DLID=58334]Cessna 172 Voice Profile[/a].You will need the main FREEWARE Flight Assistant program to use it, get it here:[a href=http://library.avsim.net/esearch.php?CatID=genutils&DLID=39661]Flight Assistant 2.2[/a]

Share this post


Link to post

Peter,Avsim does not use absolute scale in their reviews - go read how their scoring works.I have no problem with their reviews and the fact they may be subjective, but I think that since they use a "relative" scale they should tell us what are these other products they used in their comparison. Michael J.WinXP-Home SP2,AMD64 3500+,Abit AV8,Radeon X800Pro,36GB Raptor,1GB PC3200,Audigy 2

Share this post


Link to post

MichaelFirstly this is not a critic of the hardwork and dedication of people who put a lot of time and effort into producing reviews, Having done a little in the past I only know too well the labours involved and the pressures.The whole point of a review should be to give an expert opinion on a product showing its strengths and pitfalls and having a system of marking that product which means that there is some consistancy between reviewers.People often spend their hard earned cash on the recommendation or otherwise of reviews in the same way you may read a test report on a car before you buy it.There are two angles to this. Firstly protecting the consumer from purchasing junk or being taken for a ride by empty claims from manufacturers.Second, Giving a fair review to the manufacturer as often these reviews have a direct impact on their sales.Hence there is a special responsability in making a review not just to the consumer but also to the producer.An example where in my opinion something went very wroung was in the review of the Aeroworx B200. This was worthy of a gold award but recived a poor review. This appears to be the feeling amongst other users who have given this addon high acclaim.Take a group of enthusiasts who give their time for free and by the nature of that will tend to be a mixed bunch of expertise and it is more vital that a strict format is followed to try and achieve some consistnacy.There are very real pressures on the reviewers to go gently on a product by the producers and it takes a strong personality to resist those pressures.Hence a reviewer has to have special qualities.Maybe as suggested a marking system should be abolished and the review put forward as an individual impression. Should an impressionist :-) find an aircraft of particular merit then it can be handed back to others to try as a possible award subject.With these aircraft and their systems forever becoming more complex differing parts of the aircraft could almost warrant a review in their own right and group reviews using people of different expertise areas are probably more needed now than before.It could also be an option to allow forum members to review aircraft they purchase by filling in an online questionaire on the product.I appreciate there would have to be some inbuilt form of stopping such a system being abused by multiple form posting or malicious posting.In a way that happens in the forums now post purchase but again this way there would be consistancy by set questions and a true picture of where the problems are.Peter

Share this post


Link to post

>With these aircraft and their systems forever becoming more>complex differing parts of the aircraft could almost warrant a>review in their own right and group reviews using people of>different expertise areas are probably more needed now than>before.That is an excellent point, Peter. It also suggests to me that if any reviewer has a question about the product, or is unclear about something they "think it should" have and doesn't, they should contact the producers of the product and ask... rather than simply "assume" facts not in evidence... ;)

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this