Can You Trust iTunes App Store Reviews?

— -- Browsing Apple's App Store in search of new applications for your iPhone or iPod Touch, you find an interesting-sounding program and start reading its user reviews. Many are overwhelmingly upbeat, their wording eerily similar.

Are the reviews fake? Were they written by the developer (and family and friends) to inflate the app's overall rating? For other programs, you may find a string of 1-star reviews that trash the app and recommend a competitor. Were those penned by a devious rival?

With more than 25,000 programs in the App Store, at least some competitive shenanigans are bound to happen--and a few developers have already accused others of planting negative reviews.

Njection, the developer of NMobile, an app that alerts drivers to speed traps, has accused competitor Trapster of posting reviews that criticize NMobile and praise Trapster. When contacted by PC World, a Trapster spokesperson, who asked not to be identified by name, e-mailed this response: "We don't have time to post petty slams against competitors, nor respond to accusations that we do."

Complaints are flying even over frivolous programs. Air-O-Matic, maker of Pull My Finger, says that rival InfoMedia, which sells iFart Mobile, posted bogus comments that praise iFart and slam competing flatulence apps.

InfoMedia CEO Joel Comm counters that he doesn't see anything wrong with a developer's posting an anonymous review of a competitor's app. "If people purchase the app, they can do whatever they want within the guidelines of the iTunes review system," Comm wrote via e-mail. "I see no reason to disclose that you are a competitor."

Whether fake or biased reviews of iPhone/iPod Touch apps are common is difficult to ascertain. Verifying the legitimacy of a review is nearly impossible, although in browsing reviews for this story we did come across more than a few that appeared suspicious. We also discovered that it's easy for reviewers to pass judgment on apps they haven't used, despite Apple's recent efforts to curtail that practice. (Apple didn't respond to PC World's inquiries for this story.)

Take, for instance, the reviews for ViewTi Golf, a golf rangefinder currently on sale for $25. In early March it had 19 reviews, including 14 by people who gave the app 5 stars (best on a scale of 1 to 5). Of those 14 reviewers, 10 had reviewed apps only by ViewTi LLC, the developer of ViewTi Golf. We contacted the company; in response, representatives noted that 4 of the 5 other reviewers (those who gave ViewTi Golf 1 to 4 stars) had also reviewed only ViewTi Golf and no other app.

Then there's Brick Breaker Revolution 3D, a $6 game from Digital Chocolate. Within hours of its February 27 release, the app had four 5-star reviews. Certainly, that alone wasn't a big deal. But one reviewer, "Snorkydog," had given 5 stars to six Digital Chocolate games, and just 2 stars to a game from another developer (the only other game Snorkydog had reviewed). So was Snorkydog a plant? Or just a devoted fan?

Digital Chocolate CEO Trip Hawkins acknowledges that user reviews are often less than reliable. "Many reviews are pretty idiotic or hilarious or biased in one direction or another," he wrote via e-mail. "It is like reading any forum or chat log. Of course you have to assume that every developer that loves their mother's cooking should be expected to love their own games."

But a fake review, pro or con, doesn't invalidate the system, he says: "Frankly the presence of these ‘lover' votes only seems to be enough to offset the even more ridiculous hater votes."

One review we found was written by an employee of the company that sells the program. Reid Carlberg, who works for Model Metrics, gave a 5-star rating to his employer's Search2GO for Salesforce CRM. The first paragraph of his review read:

"Downloaded last week and have been loving it. It totally saved me this AM when I was walking to the office from the train and needed a phone number. Fired it up and 10 seconds later I had it and was not late for the conference call I forgot about. AWESOME."

Carlberg's original review didn't disclose that he worked for Model Metrics; he added information later, after we contacted him. Carlberg says that since he didn't try to hide his personal identity--his App Store user name is "ReidCarlberg.com"--he wasn't trying to deceive anyone. But it's unlikely that the average reader would have known that Carlberg worked for Model Metrics until he stated so in the amended review.

The 5-Star Nonreview

Lots of developers use the App Store review section to interact with customers. They might, for instance, announce future upgrades or offer a rebuttal to user gripes. And many use the rating system to pad their app's overall score, even though they're not reviewing it.

When Chris Meyer posted a rebuttal to a critical, 2-star review of his AirPhones audio-streaming app, he gave his own program a 5-star rating. "Sure, I gave myself 5 stars, but that was in knowing that the average of mine and the other 2 star [review] would be reasonable," Meyer told PC World via e-mail. "Also, I only have one opportunity to tout my app, so if there are dozens of comments, it won't matter much."

Meyer added that AirPhones has received several "harsh and unwarranted" 1-star ratings, and that he suspects that some of those reviews were "intended to hurt the product." But despite the imperfections of the App Store review process, Meyer sees it as a "wonderful example of a democracy at work."

In recent months, news reports have stated that some iPhone developers are offering to pay App Store users for positive reviews. In November 2008, for instance, Wired ran a screen shot of an ad on Amazon's Mechanical Turk job site that promised $4 to any App Store user who would post a 5-star review of the SantaLive application. (The ad was later removed.)

To find out if such a practice is common, I searched 560 review-related postings on Mechanical Turk. I encountered only one App Store developer offering to pay for reviews. The fee: 30 cents for any review, good or bad. I submitted two reviews of the product, one 5-star (very positive) and the other 1-star (very negative), and received payment for each.

I also searched Craigslist job posts for 25 major U.S. cities, but I didn't see a single instance of an App Store developer offering to pay for reviews. Though anecdotal, my informal survey suggests that reviewing iPhone apps for pay isn't a huge industry--but that doesn't address how often family/friend/employee reviews distort ratings.

Apple has taken steps to limit the number of questionable App Store reviews. In September 2008 it changed its review policy: Now, only users who download a particular app can review it. And earlier this year Apple deleted comments from customers who hadn't actually used the programs in question. (In an unrelated but also helpful move, Apple recently began sorting user reviews by version as well, so that at least you have some context when you're reading a review.)

While those measures have eliminated many suspicious reviews, it hasn't prevented users from reviewing apps they haven't tried--users need only purchase and download them. Using the Windows version of iTunes, I downloaded two App Store games, Tap Tap Revenge 2 and Wrangle Lite, to a laptop PC. And although I never actually installed the games on an iPhone or iPod Touch, I was able to post reviews of each title.

For App Store customers, a healthy dose of skepticism is a good thing. "I haven't put much stock in the reviews from the App Store," wrote blogger Nick Santilli in an e-mail interview with PC World. Santilli, who has blogged about questionable App Store reviews for TheAppleBlog, would rather hear about useful apps "from the Twitter cloud," or by speaking directly with developers or fellow bloggers. "These methods seem much more reliable than the mass of random users in the App Store review system," he wrote.

How can you determine whether an App Store review is credible? One way is to seek out and read other reviews the same customer has posted; for instance, if someone has posted only 5-star reviews for apps from a certain developer, that's a red flag. Unfortunately, this investigative work isn't possible if you browse the App Store via an iPhone or iPod Touch, but you can do it in the iTunes desktop interface. Apple should address this shortcoming, since downloading and installing apps over the air is much easier than doing so via a PC. "The majority of people are downloading [apps] through the App Store interface on the device itself, just because of the seamless behavior," says ABI Research senior analyst Jeff Orr, who recently studied the spending habits of smart-phone owners.

Despite the system's flaws, however, App Store reviews still provide valuable insights. "Reading reviews on some of the apps I've used personally, a random sampling seemed spot on and generally fair," wrote Santilli. "People are so used to Apple getting things right the first time...so when every angle hasn't been considered right out of the gate, the frustration is made known from the rooftops."