New study suggests that Wine Spectator advertisers get higher ratings

The lead paper in the new issue of the Journal of Wine Economics is a study by Jonathan Reuter arguing that Wine Spectator wine ratings for advertisers were about one point higher than ratings for non-advertisers, when controlled against ratings from Wine Advocate. This is in spite of the magazine’s stated policy of tasting wines completely blind.

This from the abstract:

“In markets for experience goods, publications exist to help consumers decide which products to purchase. However, in most cases these publications accept advertising from the very firms whose products they review, raising the possibility that they bias product reviews to favor advertisers…Although the average Wine Spectator ratings earned by advertisers and non-advertisers are similar, I find that advertisers earn just less than one point higher Wine Spectator ratings than non-advertisers when I use Wine Advocate ratings to adjust for differences in quality.”

These are wine ratings, not the restaurant Awards of Excellence, which I’ve written about in the past; the applicants for those awards are advertisers by definition (having submitted a $250 fee to be considered).

Karl Storchmann has also posted an interesting blog entry about Reuter’s paper on the AAWE website.

Reuter later retreats to a statement that he “finds little consistent evidence of bias…at worst, the tests for biased ratings suggest that Wine Spectator rates wines from advertisers almost one point higher than wines from non-advertisers. However, selective retastings can explain at most half of this bias and then only within the set of U.S. wines rated by both Wine Spectator and Wine Advocate. Given Wine Spectator’s claim that it rates wines blind, the remaining difference in ratings may simply reflect consistent differences in how the two publications rate quality, which leads to predictable differences in advertising. This interpretation is consistent with the fact that tests for biased awards provide no additional evidence of bias. Therefore, despite the fact that Wine Spectator is dependent on advertising revenue, the long-run value of producing credible reviews appears to minimize bias.”

I think this conclusion is softer than it need be. Even if selective retastings explain only half of the one-point bias, that’s still pretty damning; it means that if you advertise in Wine Spectator, you might well get the benefit of a selective retasting that gets you, on average, an additional half-point. Translation: advertising influences ratings.

With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.

Wine Spectator awards logo

The more important issue, perhaps—especially if you’re a small wine producer—is how difficult it is to get magazines like Wine Spectator to even review your wines at all. And this is where, anecdotally, bias might play an even larger role. “Unsolicited samples,” states the Wine Spectator website, “may not be tasted.” Advertise in the magazine, and that problem seems to go away.

And then there’s the matter of the selection of a wine (Columbia Crest Cabernet Sauvignon Reserve) from a Wine Spectator advertiser (Chateau Ste. Michelle) as this year’s Wine Spectator wine of the year.

Although proving bias in every such case is a complicated, difficult point, the obvious conclusion of all such research is the simplest:

We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized.

Wine critics should not accept advertisements from wineries.

Period.

Tags: , , , , , , , , , ,

12 Responses to “New study suggests that Wine Spectator advertisers get higher ratings”

  1. Alex Benes 2009 11 December at 12:41 pm #

    Robin,

    I am a long-time consumer of all news about wine. Needless to say, I’ve consumed more than my share of great and not-so-great wine. So, I’ve been interested in how a lot of wine writers and publications seem to move in lock-step on some wines.

    My question, do you ever take paid press trips to wine regions, or accept wine samples from wineries or stay at wineries when you are traveling or reporting for or about wine?

    Thanks,
    AB

  2. Robin Goldstein 2009 11 December at 3:57 pm #

    No, I haven’t ever accepted press trips to wine regions or free accommodations from wineries whose wines I’m reviewing. I have accepted invitations to speak at or attend large-scale conferences such as Spain’s Fenavin, at which I was a speaker last summer. (Fenavin is an independent wine industry convention, akin to Vinitaly or Vinexpo.) But I have turned down all invitations to trips that are sponsored by individual wine producers, groups of wine producers, or regional wine promotion/marketing boards.

    Obviously, with respect to The Wine Trials, my independent guide to wines under $15, we accept wine samples–that is the only way to review wines that are not yet on the market–but we do not accept any payments, in-kind gifts, or other accommodations of any sort from wine producers.

    RG

  3. Thomas Pellechia 2009 12 December at 12:42 pm #

    Matthews at WS responded to the report by saying that the magazine’s writers taste all wines blind. They do not know the producer or price when tasting.

    In my view, here’s what’s wrong with that response.

    1. A blind tasting means that along with not knowing the producer and price, the evaluators should also not know the region or the variety. That way, they are evaluating the wine on its individual merits–and they might have to do some work to determine if the wine lives up to its claim as coming from a particular place or grape variety.

    2. Perhaps, the magazine writers don’t know the producer or price when they sit down to taste, but do they know which wines have been entered into the evaluation line up? Matthews doesn’t mention that aspect. If they know which wines are in the line up, then they already have a clue to go on.

  4. 1WineDude 2009 14 December at 10:25 am #

    Seems to me that the .5 to 1 point potential bias is the smallest amount possible. Having said that, a 1 point move could mean upwards of tens of thousands in sales.

    So, I’m growing increasingly in agreement with your statement “We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized. Wine critics should not accept advertisements from wineries. Period.”

    I’ve just noticed, for example, that one of my rotating ads is adding wineries… which has me worried… so I’m trying to change it.

    Cheers!

  5. m kizl 2009 14 December at 11:38 am #

    The other interesting point: those in the industry who choose not to advertise and who’s wines will then never see a review. Jess Jackson owns some pretty fantastic properties around the world. Try to find a review for one of his wines in a WS publication.

  6. Thomas Pellechia 2009 14 December at 3:11 pm #

    Good point kizi. But of course WS can simply say that they can’t review the wine if it isn’t submitted–to which we can all ask, why don’t you review wines that are on shelves rather than wait for wineries to submit them; wouldn’t that be a well-rounded selection of wines in the marketplace and wouldn’t that be a service to all consumers?

  7. Ed Green 2010 24 April at 4:37 am #

    I am sure there are the few press paid trips and the odd free case of wine to help the advertising flow. Maybe I’m biased being in the PR business

  8. Alex 2010 05 June at 7:27 pm #

    In defense of WS… this just isn’t scientific at all.

    The numbers are so small that there is no way to call it statistically reliable. How many wines could have possibly been used to create this mathematical model? Certainly the total number of advertisers is quite a small number and if we look at ratings for, say, a 12 month period, we would ALWAYS see a difference between the advertised and non-advertised wines.

    In other words, if you flipped a coin 100 times, and you did not get exactly 50 heads and 50 tails, would you say that God is unfairly biased towards the one you got more of?

    The odds of getting exactly 50-50 are very small.

    I think that a less-than-one-point difference cannot possibly be out of the statistically probably norm in this study. And if the point went the other way and advertisers were lower, we’d have never had any controversy that the magazine was intentionally low-balling advertisers to get the advertisers to spend more and buy bigger ads to make up for their poor showing in the blind tasting!

    But, because this was also a coin flip probability, it happened to come up against WS and created an article about what in all mathematical probability is nothing.

    I for one will continue to trust the Specator’s ratings. And, frankly a less-than1-point spread is not a huge influence over my buying habits except in that irrational case of the point between 89 and 90 (I am, after all, a human subject to those weird cognitive biases that all of us humans struggle with).

    Cheers.

  9. Wine O 2010 06 August at 1:35 am #

    In my opinion, the problem is with the sheer amount of wine companies vying for that purchase. With national publicized magazines like Wine Spectator and Wine Advocate constantly getting inundated with request for reviews…there are always going to be manufactures that feel their not getting a fair shake. When in reality there is just way to many wines out there that really should not be given an honest review. On the flip-side I have read many reviews from both WS and WA that I don’t agree with and think they might be biased to some manufactures.

  10. herbs 2010 30 October at 11:18 am #

    With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.

Trackbacks/Pingbacks

  1. Neuro News Nanos - Ryan Sager - Neuroworld - True/Slant - 2009 11 December

    [...] * Does advertising bias wine product reviews? — short answer: not much — in other words: yes [...]

  2. Advertising and ratings, binge drinking, screwcap suit – sipped and spit : Wine Blog Reviews - 2009 12 December

    [...] here as pdf or a blog reaction from the journal’s editor or a hard-hitting response from Robin Goldstein. The quantitative study looks only at reviews and does not examine the editorial, art, restaurant [...]

Leave a Reply