The lead paper in the new issue of the Journal of Wine Economics is a study by Jonathan Reuter arguing that Wine Spectator wine ratings for advertisers were about one point higher than ratings for non-advertisers, when controlled against ratings from Wine Advocate. This is in spite of the magazine’s stated policy of tasting wines completely blind.
This from the abstract:
“In markets for experience goods, publications exist to help consumers decide which products to purchase. However, in most cases these publications accept advertising from the very firms whose products they review, raising the possibility that they bias product reviews to favor advertisers…Although the average Wine Spectator ratings earned by advertisers and non-advertisers are similar, I find that advertisers earn just less than one point higher Wine Spectator ratings than non-advertisers when I use Wine Advocate ratings to adjust for differences in quality.”
These are wine ratings, not the restaurant Awards of Excellence, which I’ve written about in the past; the applicants for those awards are advertisers by definition (having submitted a $250 fee to be considered).
Karl Storchmann has also posted an interesting blog entry about Reuter’s paper on the AAWE website.
Reuter later retreats to a statement that he “finds little consistent evidence of bias…at worst, the tests for biased ratings suggest that Wine Spectator rates wines from advertisers almost one point higher than wines from non-advertisers. However, selective retastings can explain at most half of this bias and then only within the set of U.S. wines rated by both Wine Spectator and Wine Advocate. Given Wine Spectator’s claim that it rates wines blind, the remaining difference in ratings may simply reflect consistent differences in how the two publications rate quality, which leads to predictable differences in advertising. This interpretation is consistent with the fact that tests for biased awards provide no additional evidence of bias. Therefore, despite the fact that Wine Spectator is dependent on advertising revenue, the long-run value of producing credible reviews appears to minimize bias.”
I think this conclusion is softer than it need be. Even if selective retastings explain only half of the one-point bias, that’s still pretty damning; it means that if you advertise in Wine Spectator, you might well get the benefit of a selective retasting that gets you, on average, an additional half-point. Translation: advertising influences ratings.
With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.
The more important issue, perhaps—especially if you’re a small wine producer—is how difficult it is to get magazines like Wine Spectator to even review your wines at all. And this is where, anecdotally, bias might play an even larger role. “Unsolicited samples,” states the Wine Spectator website, “may not be tasted.” Advertise in the magazine, and that problem seems to go away.
And then there’s the matter of the selection of a wine (Columbia Crest Cabernet Sauvignon Reserve) from a Wine Spectator advertiser (Chateau Ste. Michelle) as this year’s Wine Spectator wine of the year.
Although proving bias in every such case is a complicated, difficult point, the obvious conclusion of all such research is the simplest:
We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized.
Wine critics should not accept advertisements from wineries.
Period.
Alex Benes
Robin,
I am a long-time consumer of all news about wine. Needless to say, I’ve consumed more than my share of great and not-so-great wine. So, I’ve been interested in how a lot of wine writers and publications seem to move in lock-step on some wines.
My question, do you ever take paid press trips to wine regions, or accept wine samples from wineries or stay at wineries when you are traveling or reporting for or about wine?
Thanks,
AB
Robin Goldstein
No, I haven’t ever accepted press trips to wine regions or free accommodations from wineries whose wines I’m reviewing. I have accepted invitations to speak at or attend large-scale conferences such as Spain’s Fenavin, at which I was a speaker last summer. (Fenavin is an independent wine industry convention, akin to Vinitaly or Vinexpo.) But I have turned down all invitations to trips that are sponsored by individual wine producers, groups of wine producers, or regional wine promotion/marketing boards.
Obviously, with respect to The Wine Trials, my independent guide to wines under $15, we accept wine samples–that is the only way to review wines that are not yet on the market–but we do not accept any payments, in-kind gifts, or other accommodations of any sort from wine producers.
RG
Thomas Pellechia
Matthews at WS responded to the report by saying that the magazine’s writers taste all wines blind. They do not know the producer or price when tasting.
In my view, here’s what’s wrong with that response.
1. A blind tasting means that along with not knowing the producer and price, the evaluators should also not know the region or the variety. That way, they are evaluating the wine on its individual merits–and they might have to do some work to determine if the wine lives up to its claim as coming from a particular place or grape variety.
2. Perhaps, the magazine writers don’t know the producer or price when they sit down to taste, but do they know which wines have been entered into the evaluation line up? Matthews doesn’t mention that aspect. If they know which wines are in the line up, then they already have a clue to go on.
1WineDude
Seems to me that the .5 to 1 point potential bias is the smallest amount possible. Having said that, a 1 point move could mean upwards of tens of thousands in sales.
So, I’m growing increasingly in agreement with your statement “We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized. Wine critics should not accept advertisements from wineries. Period.”
I’ve just noticed, for example, that one of my rotating ads is adding wineries… which has me worried… so I’m trying to change it.
Cheers!
m kizl
The other interesting point: those in the industry who choose not to advertise and who’s wines will then never see a review. Jess Jackson owns some pretty fantastic properties around the world. Try to find a review for one of his wines in a WS publication.
Thomas Pellechia
Good point kizi. But of course WS can simply say that they can’t review the wine if it isn’t submitted–to which we can all ask, why don’t you review wines that are on shelves rather than wait for wineries to submit them; wouldn’t that be a well-rounded selection of wines in the marketplace and wouldn’t that be a service to all consumers?
Ed Green
I am sure there are the few press paid trips and the odd free case of wine to help the advertising flow. Maybe I’m biased being in the PR business
Alex
In defense of WS… this just isn’t scientific at all.
The numbers are so small that there is no way to call it statistically reliable. How many wines could have possibly been used to create this mathematical model? Certainly the total number of advertisers is quite a small number and if we look at ratings for, say, a 12 month period, we would ALWAYS see a difference between the advertised and non-advertised wines.
In other words, if you flipped a coin 100 times, and you did not get exactly 50 heads and 50 tails, would you say that God is unfairly biased towards the one you got more of?
The odds of getting exactly 50-50 are very small.
I think that a less-than-one-point difference cannot possibly be out of the statistically probably norm in this study. And if the point went the other way and advertisers were lower, we’d have never had any controversy that the magazine was intentionally low-balling advertisers to get the advertisers to spend more and buy bigger ads to make up for their poor showing in the blind tasting!
But, because this was also a coin flip probability, it happened to come up against WS and created an article about what in all mathematical probability is nothing.
I for one will continue to trust the Specator’s ratings. And, frankly a less-than1-point spread is not a huge influence over my buying habits except in that irrational case of the point between 89 and 90 (I am, after all, a human subject to those weird cognitive biases that all of us humans struggle with).
Cheers.
Wine O
In my opinion, the problem is with the sheer amount of wine companies vying for that purchase. With national publicized magazines like Wine Spectator and Wine Advocate constantly getting inundated with request for reviews…there are always going to be manufactures that feel their not getting a fair shake. When in reality there is just way to many wines out there that really should not be given an honest review. On the flip-side I have read many reviews from both WS and WA that I don’t agree with and think they might be biased to some manufactures.
herbs
With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.
Bill in Sonoma
File under: THE MORE THINGS CHANGE… I see this is an older post, but in the new Wine Spectator mag, (Nov. 30, 2015) on page 11, there is a full page ad for Terlato Wines’ Markham Vineyard Merlot. On page 28, it happens to be featured in the savvy shopper section as the “Smart Buy”. Coincidence?