New study suggests that Wine Spectator advertisers get higher ratings
The lead paper in the new issue of the Journal of Wine Economics is a study by Jonathan Reuter arguing that Wine Spectator wine ratings for advertisers were about one point higher than ratings for non-advertisers, when controlled against ratings from Wine Advocate. This is in spite of the magazine’s stated policy of tasting wines completely blind.
This from the abstract:
“In markets for experience goods, publications exist to help consumers decide which products to purchase. However, in most cases these publications accept advertising from the very firms whose products they review, raising the possibility that they bias product reviews to favor advertisers…Although the average Wine Spectator ratings earned by advertisers and non-advertisers are similar, I find that advertisers earn just less than one point higher Wine Spectator ratings than non-advertisers when I use Wine Advocate ratings to adjust for differences in quality.”
These are wine ratings, not the restaurant Awards of Excellence, which I’ve written about in the past; the applicants for those awards are advertisers by definition (having submitted a $250 fee to be considered).
Karl Storchmann has also posted an interesting blog entry about Reuter’s paper on the AAWE website.
Reuter later retreats to a statement that he “finds little consistent evidence of bias…at worst, the tests for biased ratings suggest that Wine Spectator rates wines from advertisers almost one point higher than wines from non-advertisers. However, selective retastings can explain at most half of this bias and then only within the set of U.S. wines rated by both Wine Spectator and Wine Advocate. Given Wine Spectator’s claim that it rates wines blind, the remaining difference in ratings may simply reflect consistent differences in how the two publications rate quality, which leads to predictable differences in advertising. This interpretation is consistent with the fact that tests for biased awards provide no additional evidence of bias. Therefore, despite the fact that Wine Spectator is dependent on advertising revenue, the long-run value of producing credible reviews appears to minimize bias.”
I think this conclusion is softer than it need be. Even if selective retastings explain only half of the one-point bias, that’s still pretty damning; it means that if you advertise in Wine Spectator, you might well get the benefit of a selective retasting that gets you, on average, an additional half-point. Translation: advertising influences ratings.
With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.
The more important issue, perhaps—especially if you’re a small wine producer—is how difficult it is to get magazines like Wine Spectator to even review your wines at all. And this is where, anecdotally, bias might play an even larger role. “Unsolicited samples,” states the Wine Spectator website, “may not be tasted.” Advertise in the magazine, and that problem seems to go away.
And then there’s the matter of the selection of a wine (Columbia Crest Cabernet Sauvignon Reserve) from a Wine Spectator advertiser (Chateau Ste. Michelle) as this year’s Wine Spectator wine of the year.
Although proving bias in every such case is a complicated, difficult point, the obvious conclusion of all such research is the simplest:
We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized.
Wine critics should not accept advertisements from wineries.
Period.
Tags: advertising, bias, corruption, jonathan reuter, journal of wine economics, wine ratings, wine spectator
December 11th, 2009 at 6:46 am
[...] * Does advertising bias wine product reviews? — short answer: not much — in other words: yes [...]
December 11th, 2009 at 12:41 pm
Robin,
I am a long-time consumer of all news about wine. Needless to say, I’ve consumed more than my share of great and not-so-great wine. So, I’ve been interested in how a lot of wine writers and publications seem to move in lock-step on some wines.
My question, do you ever take paid press trips to wine regions, or accept wine samples from wineries or stay at wineries when you are traveling or reporting for or about wine?
Thanks,
AB
December 11th, 2009 at 3:57 pm
No, I haven’t ever accepted press trips to wine regions or free accommodations from wineries whose wines I’m reviewing. I have accepted invitations to speak at or attend large-scale conferences such as Spain’s Fenavin, at which I was a speaker last summer. (Fenavin is an independent wine industry convention, akin to Vinitaly or Vinexpo.) But I have turned down all invitations to trips that are sponsored by individual wine producers, groups of wine producers, or regional wine promotion/marketing boards.
Obviously, with respect to The Wine Trials, my independent guide to wines under $15, we accept wine samples–that is the only way to review wines that are not yet on the market–but we do not accept any payments, in-kind gifts, or other accommodations of any sort from wine producers.
RG
December 12th, 2009 at 12:04 pm
[...] here as pdf or a blog reaction from the journal’s editor or a hard-hitting response from Robin Goldstein. The quantitative study looks only at reviews and does not examine the editorial, art, restaurant [...]
December 12th, 2009 at 12:42 pm
Matthews at WS responded to the report by saying that the magazine’s writers taste all wines blind. They do not know the producer or price when tasting.
In my view, here’s what’s wrong with that response.
1. A blind tasting means that along with not knowing the producer and price, the evaluators should also not know the region or the variety. That way, they are evaluating the wine on its individual merits–and they might have to do some work to determine if the wine lives up to its claim as coming from a particular place or grape variety.
2. Perhaps, the magazine writers don’t know the producer or price when they sit down to taste, but do they know which wines have been entered into the evaluation line up? Matthews doesn’t mention that aspect. If they know which wines are in the line up, then they already have a clue to go on.
December 14th, 2009 at 10:25 am
Seems to me that the .5 to 1 point potential bias is the smallest amount possible. Having said that, a 1 point move could mean upwards of tens of thousands in sales.
So, I’m growing increasingly in agreement with your statement “We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized. Wine critics should not accept advertisements from wineries. Period.”
I’ve just noticed, for example, that one of my rotating ads is adding wineries… which has me worried… so I’m trying to change it.
Cheers!
December 14th, 2009 at 11:38 am
The other interesting point: those in the industry who choose not to advertise and who’s wines will then never see a review. Jess Jackson owns some pretty fantastic properties around the world. Try to find a review for one of his wines in a WS publication.
December 14th, 2009 at 3:11 pm
Good point kizi. But of course WS can simply say that they can’t review the wine if it isn’t submitted–to which we can all ask, why don’t you review wines that are on shelves rather than wait for wineries to submit them; wouldn’t that be a well-rounded selection of wines in the marketplace and wouldn’t that be a service to all consumers?