Part Two: Gaming the Amazon System

Update 6/1/2012: In the past day or so, Amazon.com deleted all of the reviews for all of Robert Stanek’s books (both positive and negative). Why they would do that, I can’t fathom. Maybe they have some incontrovertible evidence that most of the positive reviews for Stanek’s books were fake, written by Stanek himself, or paid shills? I’ll leave the below article as-is for the historical perspective.

Update 3/7/2010: Shortly after this article went up in January 2010 (I believe they disappeared in mid-February) nearly two hundred of the five-star reviews for The Kingdoms and the Elves of the Reaches vanished off of Amazon.com. This intrigued me, because there are only two people who can delete a review:

1. The author of the review
2. Amazon.com

I’m pretty sure it wasn’t #1, because what are the odds that two hundred authors would all get online and simultaneously decide to delete their glowing review of one specific book within about a week of each other. However, #2 seems quite plausible, except that Amazon.com generally speaking doesn’t delete reviews unless…well, here are their terms and conditions:

I’m pretty sure the reviews weren’t illegal, obscene, threatening, defamatory, invading of privacy, infringing on intellectual property rights, injurious to third parties, and they did not contain software viruses, political campaigning, chain letters, or mass mailing. The other bits, though, I found very interesting. Commercial solicitation, using a false email address, impersonating someone, or misleading as to the origin of the content? Sounds exactly like what Stanek was accused of doing.

So, as far as I know, the only reason Amazon deletes reviews is because they are off-topic (which these weren’t) or they violated one of the above reasons…and they deleted two hundred of Robert Stanek’s reviews. That’s rather interesting.

*                           *

Anyone who has read one of Robert Stanek’s books and then spent a few minutes reading through the five-star reviews on Amazon would find them suspicious. Generally speaking, reader response is a reasonably accurate gauge of how good a book is. While poorly written books can often be extremely popular, they will also quickly gather their fair share of critics.

For example: Pull up Cormac McCarthy’s The Road on Amazon. Spend a minute or two browsing through the five-star reviews. They have similarities. The authors talk about things they liked, using specific examples from the text, citing McCarthy’s use of language, the bleakness he paints with words, the themes woven throughout the story. Now browse through the one-star reviews. You’ll see authors talking about what they disliked, using specific examples from the text and other examples of how the book failed to appeal to them. Overall, the reviews average out to a four-star rating. For a popular book with critical acclaim, everything about this logically makes sense.

Now pull up Robert Stanek’s The Kingdoms and the Elves of the Reaches on Amazon, and browse through the five-star reviews. They also have similarities. The authors talk about things they liked, naming characters within the text, frequently comparing Stanek to Tolkien and labeling the book as one of the best books they have ever read. But instead of this being a critically acclaimed, award-winning novel that won the Pulitzer Prize for fiction, it’s a self-published novel by an author whose books do not appear in any bookstores, and have not won any awards. What’s more, using Amazon’s handy “Click to Look Inside” feature, anyone can read the first few pages of the book and realize that Stanek is a very poor writer.

For many people, this is evidence enough that the hundreds of glowing five-star reviews are fake. But I decided to delve a little deeper. After all, there are plenty of things that on the surface look a little suspicious, but with a little research and an adequate explanation, it turns out to make perfect sense. Stanek’s reviews were suspicious, certainly. But were the things that I found suspicious also present in the reviews of other authors, authors that we objectively know are popular or well-written?

Obviously, extremely popular titles such as Twilight or Harry Potter have far more reviews than a more obscure book, so comparing the number of five-star reviews will not reveal anything more than who would win in a popularity contest. Since Amazon uses the average rating to determine how to rate a particular title out of five stars, and since Stanek’s business is primarily through Amazon, I decided to figure out what percentage of a book’s reviews on Amazon were positive. The formula I used is:

[5-star reviews + 4-star reviews] * 100 / [total number of reviews].

As an example, let’s take Neil Gaiman’s first Sandman collection, Preludes and Nocturnes, since we know that Sandman is objectively good: it was on the New York Times bestseller list, it was nominated for a World Fantasy Award and for a Hugo Award. Preludes and Nocturnes has seventy 5-star reviews and forty-eight 4-star reviews out of 140 total reviews. Using the formula of [70 + 48] * 100 / 140, that gives us a positive percentage of 84.2%, which sounds perfectly reasonable. 84.2% means that less than sixteen percent of the people who read it found it average or disliked it.

I went through and used the following formula on a number of popular books and also on a number of Stanek’s books and compiled the following list. The books in bold were written by Robert Stanek.

• 98.9% The Kingdoms and the Elves of the Reaches IV
• 98.9% The Kingdoms and the Elves of the Reaches III
• 96.5% The Kingdoms and the Elves of the Reaches II
• 95.3% Elf Queen’s Quest
• 93.9% Harry Potter and the Sorcerer’s Stone
• 92.8% The Kingdoms and the Elves of the Reaches
• 92.4% In the Service of Dragons
• 91.5% Ender’s Game
• 90.6% Keeper Martin’s Tale
• 90.3% The Hobbit
• 88.2% The Princess Bride
• 87.8% The Hitchhiker’s Guide to the Galaxy
• 87.1% Catch-22
• 86.4% The Fellowship of the Ring
• 85.9% A Game of Thrones
• 84.2% Preludes and Nocturnes
• 79.1% Artemis Fowl
• 76.3% Twilight
• 66.4% Eragon
• 57.6% The Da Vinci Code
• 33.3% The Fifth Sorceress

Clearly, something is going on here. And this is not just a product of small sample size – The Kingdoms and the Elves of the Reaches has 352 total reviews. Keeper Martin’s Tale has 268. Either a relatively unknown self-published author has somehow managed to write books good enough for his rabid fans to rate them better than one of the most, if not the most popular series of all time, or there’s something suspicious going on.

At the same time, I wondered about the lack of 1-star reviews. After all, if a book has been on Amazon for over seven years and it’s a terrible book, shouldn’t it have dozens of bad reviews?

I believe that there have been: there are many testimonials online from people who wrote negative reviews of Stanek’s books which were later deleted. I suspect they were deleted at Robert Stanek’s request, since he’s the only person who would want them deleted. I also suspect that some were deleted because they were off-topic. A review written by a furious customer talking about Robert Stanek committing fraud could very well be deleted as off-topic because it’s not about the book itself, and whichever Amazon employee who is looking at that review doesn’t necessarily know that Stanek is a scumbag.

Many critics have also pointed out that all of Stanek’s five-star reviews sound suspiciously familiar, as if they were written by the same person. When I read through them myself, I noticing the exact same thing. In many of these reviews, the reviewers said the exact same things, made very similar comparisons, and had very similar histories. However, that itself is not proof of anything: it could just indicate that a certain book attracts a certain kind of reviewer. It’s possible that if we looked at a completely separate book’s reviews, we would see the same similarities within them.

I opted to use Christopher Paolini’s Brisingr for my control group. Both series are high fantasy, both are marketed towards children, both authors are connected to self-publishing, and both authors have their fair share of vocal critics.

At the time of writing, The Kingdoms and the Elves of the Reaches had 299 5-star reviews on Amazon. I decided to ignore all of the ones labeled “A Kid’s Review” as these were all submitted anonymously, they did not have additional data I wished to record, and nearly all of them were extremely short and did not say anything of interest beyond repeating how awesome the book was. This left me with 180 reviews. I therefore took the first 180 five-star reviews Amazon had listed for Brisingr (also ignoring any “A Kid’s Review”) and began to compare the two groups, specifically looking to see if the similarities I saw among Stanek’s reviews were also present in Paolini’s.

In brief, they weren’t.

The first similarity I saw in Stanek’s reviews were positive comparisons to Harry Potter or J.K. Rowling. In every single case the reviewer stated that ‘Kingdoms’ was as good as or better than Harry Potter. A comparison to Harry Potter was present in 22.7% of the Stanek reviews, compared to 1.6% of Paolini’s.

The second similarity present in Stanek’s reviews were positive comparisons to Lord of the Rings, The Hobbit, or Tolkien. Here they tended to be more respectful: they rarely stated Stanek was better than Tolkien, usually opting to call them equals or to call Stanek the “American Tolkien”. This was present in 16.1% of the Stanek reviews, compared to 2.2% of Paolini’s.

The third similarity was positive comparisons to Narnia. Twenty-four separate reviews compared ‘Kingdoms’ favorably to Narnia or C.S. Lewis, or 13.3%. None of Paolini’s 180 reviews mentioned Narnia or Lewis.

I noticed that in Stanek’s reviews, they often refer to him as ‘Mr Stanek’. This is especially blatant among his most vocal ‘supporters’ on the forums, who are of course the man himself, posting as one of his many sockpuppets. I believe Stanek does this because it sounds more respectful and he’s trying to give himself some credibility and seem more mature. Either way: in his reviews he’s referred to as ‘Mr Stanek’ 16 separate times. Paolini is called ‘Mr Paolini’ exactly twice.

Many of the sockpuppets that posted five-star reviews also have lists of books on their profiles, usually containing several popular book series (such as Harry Potter, Artemis Fowl, and, ironically, Eragon) along with several of Stanek’s books. This is part of the aforementioned effort to manipulate Amazon’s recommendations into constantly directing people back to Stanek’s books. 33.3% of Stanek’s five-star reviewers have lists that contain Stanek’s books on them. Only 1.1.% of Paolini’s do.

This is actually crucially important to Stanek’s business. As an online company, Amazon has developed a variety of creative ways to lead customers from product to product. One of these methods is their recommendations tool, which says “Customers who bought this product also bought ___”. The idea being if you buy several books within the epic fantasy genre, Amazon will start recommending other epic fantasy books in the hopes of getting you to buy one. Amazon draws this data from information on other customers’ purchase history, but also the lists that customers can create, and books that other customers recommend. So if a high number of customers highly rate both Harry Potter and Robert Stanek, for example, eventually customers who are purchasing Harry Potter books will see advertisements around Amazon’s page for books by Robert Stanek. And if they then click on that link and read some reviews for some of Robert Stanek’s books, and see that a high number of reviewers compare Stanek favorably to, say, Rowling or Tolkien, they are then that much more likely to purchase one of Stanek’s books to see what all the fuss is about.

It’s rather ingenious, actually, but people who commit fraud are usually pretty sharp.

Another common trait of Stanek’s reviews is to list out the entire title, along with the complete title of some of the sequels. This is especially ridiculous when you realize that most of Stanek’s titles are horrendous mouthfuls like “The Kingdoms and the Elves of the Reaches” and “In the Service of Dragons” – not even counting the multiple subtitles and taglines such as “Reader’s Choice Edition” and “Signature Illustrated Edition”. I started recording the data – and the results were surprising, to say the least. Within Paolini’s reviews, you would expect to get a lot of people just randomly saying the names “Eragon” and “Eldest” – and Stanek still beats Paolini in the number of reviews that list the entire name of a work by that author. 40.5% of Stanek’s reviews took the time to type out the entire nine-word title, and many of those took the time to put in several more. Compared to just 15% of Paolini’s.

I couldn’t help but notice that 37.2% of Stanek’s reviews just go through and list most of the characters by name because they couldn’t think of anything new and creative to say…compared to 10% of Paolini’s.

One of my last conscious thoughts before my brain started shutting down due to information overload was that it seemed that Stanek’s “fans” didn’t have a lot of reviews to their names – almost as if they were shell accounts created purely to review a couple of Stanek’s books and then discarded. I went through and tallied up the number of reviews in the interests of getting an average, even though that average was skewed when I found what I believe to be the one actual real five-star-review of Stanek’s work. Stanek’s reviewers averaged 8.09 reviews apiece. Which seemed respectable, until I tallied up the numbers for Christopher Paolini’s reviewers, who average 35.2 reviews each – and that was NOT counting Midwest Book Review, who has over 50,000 reviews on Amazon.

And one final, interesting number for you: Amazon has a tag on a review that’s called an “Amazon Verified Purchase”. Simply put, Amazon checks to see if you have actually purchased this book on your account and then it gives you the option to add this tag to your review, as a little extra bit of credibility.

20 of the people who reviewed Paolini has this tag.

Only 2 of Stanek’s did.

Part Three: A Professional Review