People have sometimes (often, maybe) misinterpreted The Wine Trials (and The Wine Trials 2010) as making the claim that no expensive wines are worth the money, or that cheap wine is generally “better” than expensive wine. In fact, I make neither one of those claims in the book.

Rather, my basic points are these:

(1) Evidence has shown that most everyday wine drinkers (not wine professionals) don’t prefer more expensive wines to cheaper wines in blind tastings. This is separate from the question of whether the properties of expensive wines are aesthetically superior in the minds of experts.

(2) Many (but certainly not all) expensive wines, such as the luxury brands from LVMH—which are advertised much like the group’s TAG Heuer watches, De Beers diamonds, Guerlain perfume, or Louis Vuitton handbags—are overpriced because such a large portion of their cost base is spent on marketing. This doesn’t just go for superpremium wines like LVMH’s Château d’Yquem, Krug, and Dom Pérignon; it also goes for brands like Cloudy Bay, a straightforward New Zealand Sauvignon Blanc whose price—without any apparent change in the production method—rose from about $15 per bottle to about $30 per bottle after LVMH acquired the brand in 2003 and began marketing Cloudy Bay as a luxury product.  To me, when the consumer dollar is going more toward advertising than toward materials or production, it’s a paradigm case of overpricing. It bothers me that the mainstream wine media doesn’t take brands to task for this.

(3) There are also wines that are overpriced for reasons other than marketing—reasons like an irrational premium Bordeaux bubble that’s being inflated by indiscriminate demand from rich, unsophisticated consumers in emerging markets like China and Russia. Even if Pétrus spends no money on marketing, $5,000 is an irrational price for a bottle, and this is a demand-side phenomenon.

(4) Then there are the producers who model themselves after Pétrus in an effort to capitalize on that same demand-side phenomenon. These producers make “high-end” wine (with the characteristics typically associated with the 95-and-higher-point wines in wine magazines, e.g. aging in new French oak, high alcohol, extreme concentration) and price it as such. Here, there aren’t necessarily the extreme marketing expenditures of LVMH; rather, there’s simply a price-signalling play: the hope that positioning the product at the top end of the market will speak for itself, and that consumers in search of a luxury good will buy into that notion. In this case, the consumer dollar isn’t paying for lots of advertising and marketing—it’s just sustaining unconscionably high profit margins for the producer.

What situations (2), (3), and (4) have in common is that the cost of production of each of these premium wines is virtually unrelated to the street price.

One might divide wine pricing theory into two rough schools of thought. There is the camp that believes wine should be priced from a supply-side/cost-plus perspective–you take the cost of production of the wine, you add reasonable costs and a modest profit for the producer, you factor in markups for distribution and retail, and you arrive at more or less what the wine should cost. The other camp believes that wine should be priced from a demand-side perspective–that a wine is worth whatever the market is willing to pay for it.

The reason I’m in the first camp, and not the second, is that I don’t subscribe to the neoclassical model of consumer rationality upon which the demand-side pricing theory is built, a counterfactual universe of stingily hypersensitive, quality-sniffing consumers. My sense is that, especially when it comes to hazy markets like wine, real human beings—within certain constraints—generally anchor themselves to market prices that are imposed upon them, and generally pay for things what they’re told those things are worth.

One attempt to justify superpremium wines with modest costs of production is an opportunity-cost-of-land argument—that wine in the Champagne appellation is so expensive that the opportunity cost of that land can justify higher prices. I’m unsympathetic to that argument, because real estate prices track market wine prices, so the price of land is not an independent factor.

So when are premium prices justified in my camp?

When the cost of production is high. The fact that Matthews and Briand mention 1er Cru Burgundy and German whites as examples of expensive wines worth the money suggests that they might be in my camp too, because these are particular examples of wine regions in which grapes are often harvested from small plots with very low yields. In the case of German TBA, for instance, the harvesting is often done on steeply terraced slopes that are extremely difficult to work. Ice wines and botrytized wines—the priciest of German whites—are indisputably more difficult and expensive to produce than almost any other type of wine.

In short, while spending $50 or $75 or even $100 on a good Sauternes, TBA, or top red Burgundy might not always make economic sense for the buyer—particularly if it’s a buyer without much experience in wine—it’s at least justifiable from a supply-side pricing perspective. The $150 you’ll pay for a bottle of Opus One or Krug, meanwhile—never mind the $5,000 you’ll pay for a bottle of 2005 Pétrus—has little to do with the cost of production.

This from the abstract:

“In markets for experience goods, publications exist to help consumers decide which products to purchase. However, in most cases these publications accept advertising from the very firms whose products they review, raising the possibility that they bias product reviews to favor advertisers…Although the average Wine Spectator ratings earned by advertisers and non-advertisers are similar, I find that advertisers earn just less than one point higher Wine Spectator ratings than non-advertisers when I use Wine Advocate ratings to adjust for differences in quality.”

These are wine ratings, not the restaurant Awards of Excellence, which I’ve written about in the past; the applicants for those awards are advertisers by definition (having submitted a $250 fee to be considered).

Karl Storchmann has also posted an interesting blog entry about Reuter’s paper on the AAWE website.

Reuter later retreats to a statement that he “finds little consistent evidence of bias…at worst, the tests for biased ratings suggest that Wine Spectator rates wines from advertisers almost one point higher than wines from non-advertisers. However, selective retastings can explain at most half of this bias and then only within the set of U.S. wines rated by both Wine Spectator and Wine Advocate. Given Wine Spectator’s claim that it rates wines blind, the remaining difference in ratings may simply reflect consistent differences in how the two publications rate quality, which leads to predictable differences in advertising. This interpretation is consistent with the fact that tests for biased awards provide no additional evidence of bias. Therefore, despite the fact that Wine Spectator is dependent on advertising revenue, the long-run value of producing credible reviews appears to minimize bias.”

I think this conclusion is softer than it need be. Even if selective retastings explain only half of the one-point bias, that’s still pretty damning; it means that if you advertise in Wine Spectator, you might well get the benefit of a selective retasting that gets you, on average, an additional half-point. Translation: advertising influences ratings.

With respect to the other half-point, if there are indeed “consistent differences in how the two publications rate quality, which leads to predictable differences in advertising,” then you should try leafing through a copy of Wine Spectator and seeing if you’d trust critics who favor the types of wines that tend to advertise in the magazine. I think the roster of advertisers speaks for itself.

The more important issue, perhaps—especially if you’re a small wine producer—is how difficult it is to get magazines like Wine Spectator to even review your wines at all. And this is where, anecdotally, bias might play an even larger role. “Unsolicited samples,” states the Wine Spectator website, “may not be tasted.” Advertise in the magazine, and that problem seems to go away.

And then there’s the matter of the selection of a wine (Columbia Crest Cabernet Sauvignon Reserve) from a Wine Spectator advertiser (Chateau Ste. Michelle) as this year’s Wine Spectator wine of the year.

Although proving bias in every such case is a complicated, difficult point, the obvious conclusion of all such research is the simplest:

We should be skeptical of criticism whose publication is financially supported by the producers of the products being criticized.

Wine critics should not accept advertisements from wineries.

Period.

Ethics scandals are politico porn. They’re also fertile ground for undeserved scapegoating. But there’s one category in which, across the board, there’s not nearly enough public stoning going on: the world of information intermediaries. On the government side, that means regulatory agencies; in the private sector, it’s the critics, the expert witnesses in capitalism’s de facto justice system.

Information intermediaries, we’re to understand, are society’s check against puffery. They make careers of trustworthiness and accountability. In society’s service, they apply rigor to the claims of corporations and analyze their standards. For this hard work, they’re rewarded by the marketplace and by the United States—sometimes handsomely, sometimes not.

Two bits of recent news bring about two otherwise disparate intermediaries, both preeminent in their niches—Robert Parker’s Wine Advocate, the publication whose critical appraisals are one of the central determinants of a wine’s success or failure on the marketplace, and the Federal Aviation Administration, the agency whose critical appraisals are the primary safety check against America’s airlines—systematically abusing that authority.

Jay Miller: Disfrutando?

Parker’s is one of the few wine publications that don’t accept advertising, for which he deserves praise. And it’s certainly acceptable to take free samples of wine from producers—that’s often the only way to taste new releases before they’ve gone to market. But the recent transgressions of Jay Miller, Robert Parker’s right-hand man, are spectacular indeed. In another classic case of the traditional print media jumping on the bandwagon of a topic that had been exposed quite a bit earlier by an incisive blogger—in this case, Tyler Colman, who goes by “Dr. Vino”—Miller’s series of all-expenses-paid vacation/junkets, financed by wine producers, have finally been reported by the mainstream media in a recent Wall Street Journal article.

Some of rumors about Miller’s behavior in Argentina go quite a bit further in scandalousness than the mere acceptance of free hotels, food, and drinks. But what’s crystal clear, as reported in the Journal and, previously, by Colman—and admitted by Parker—is that the writers of Parker’s Wine Advocate accept lavish free meals from, and are flown around on weeks-long junkets by, the same wine producers whose wines they’re supposed to be critically reviewing—in Argentina and elsewhere.

Parker doesn’t just present himself as a wine writer, and he doesn’t just present Wine Advocate as a wine magazine. No, these are “consumer advocates”: in Parker’s new statement of ethical standards, which was published on eRobertParker.com after the scandal broke, he writes: “I…remain today…significantly influenced by the independent philosophy of consumer advocate Ralph Nader.” Mr. Miller’s only job is to independently criticize wines and assign numerical ratings to those wines in service to, and he is engaged in the systematic, ongoing practice of being taken on vacation by those wines’ producers.

Miller has apologized, and Parker has tried to explain himself, but Parker’s actions (or lack thereof) speak more loudly than his rambling statements on bulletin boards. If it’s even true that Parker didn’t know about Miller’s junkets beforehand (which is extremely unlikely), the fact that Parker didn’t fire Miller when he found out is a loud, clear statement that what he did really wasn’t that bad. In fact, it’s endorsed even in the new code of ethical standards. Incredibly, rather than rejecting Miller’s trips, that statement specifically renders them acceptable:

I…expect [the Wine Advocate critics], as I have done for 30+ years, not to solicit or accept free hotel accommodations or hospitality not directly related to their professional endeavors.

Not directly related to their professional endeavors? Of course a free tasting junket would be related to their professional endeavors. That’s exactly the problem!

Parker finishes his exhausting essay with a poetic flourish: “Wine is, in the final analysis, a beverage of pleasure, and intelligent wine criticism should be a blend of both hedonistic and analytical schools of thought—to the exclusion of neither.” At least the record shows that he’s got the hedonistic part down, anyway.

Meanwhile, the New York Times reports that over at the F.A.A., the airlines being rigorously scrutinized for adherence to strict safety standards—again, the organization’s primary responsibility—are referred to within the agency as “customers”:

In 2008, two F.A.A. inspectors assigned to Southwest Airlines testified before Congress that their managers had let Southwest fly its Boeing 737s without inspections for cracks that the safety agency required. Office managers referred to the airline as the regulatory agency’s “customer.” Top F.A.A. officials eventually conceded that the inspectors were right and the middle managers were wrong.

The Times’ investigative report tells the chilling story of an inspector named Christopher Monteleon, who voiced serious concerns about the incompetence and low safety standards of the pilots of Colgan Air a year before one of the airline’s Bombardier Dash 8-400 turboprop planes crashed in Buffalo, apparently due to pilot error. Apparently, Monteleon’s warnings were not just ignored but actually punished by his superiors:

Three times, he said, the pilots flew the airplane faster than the manufacturer’s specifications allowed, but they initially refused to report this and have the plane inspected for damage…[T]hey tried three approaches to the airport in Charleston, W. Va., and “botched” all of them, failing to get the plane at an appropriate altitude, on the right path and at the right speed for landing. “They got confused,” Mr. Monteleon said…But when he reported problems to his F.A.A. superiors, he was suspended from important portions of his job overseeing Colgan’s acquisition of the Dash 8 and given a desk job, he said…Colgan crews were flying fatigued, Mr. Monteleon said, and were not fully focused on the tasks in front of them, two factors apparently in play in the Buffalo crash. All 49 people on board the flight, which took off from Newark, were killed, along with one man on the ground. Mr. Monteleon said his supervisors were too “cozy” with Colgan, and eager to help it keep its schedule…In one memo retained by Mr. Monteleon, his manager indicates that he was reassigned because of his “conduct during a work-related duty” and because “the matter also required management to immediately respond to the operator’s scheduling needs.” The operator was Colgan.

In aviation, as in wine, our intermediaries have failed in their duty to their real customers: the readers, the consumers, the citizens. And unlike what happens when ethics scandals directly involve politicians, neither Robert Parker nor the F.A.A. is likely to be voted out of power anytime soon.

1. Are most wine critics impartial judges of quality, or are they really serving as public-relations advocates on behalf of producers? With evidence from my own empirical work, expository journalism, and a survey of the industry, I argue that most wine critics are really in the business of advertising wine, not judging it impartially. Critics are for sale.

2. My exposé of Wine Spectator’s “Award of Excellence” program.

3. My research indicating that wine experts don’t prefer expensive wine to cheap wine by a significant margin in blind tastings: In a year-long series of blind tastings that I conducted with wine experts and non-experts from the US, France, and other countries, my colleagues and I poured more than 6,000 glasses of wine from brown-bagged bottles that cost from $1.50 to $150. On the whole, our tasters actually preferred the cheaper wines to the more expensive wines—by a statistically significant margin. Even among the wine experts alone, there was only a weakly positive correlation between price and preference—one that didn’t rise to the level of statistical significance.

4. Blind tasting and what I call the “wine placebo effect”: No scientific blind-tasting study of wine experts has ever shown expensive wines to do consistently well, or cheap wines to do consistently poorly.

5. “The taste of money”: I believe that the sensory experience itself that changes when you know the wine is expensive. 

6. “The honest wine movement”: the way forward. I advocate for a new approach to quantitative wine evaluation that’s based only on double-blind tasting. So-called “single-blind tasting,” in which tasters know the vintage and/or region (and thus general price range) but not the exact producer, is not enough.

1. WS writes: “We make significant efforts to verify the facts…We called the restaurant multiple times; each time, we reached an answering machine and a message from a person purporting to be from the restaurant claiming that it was closed at the moment.”

If it’s true that WS called the restaurant’s number (+39 02 4074 6174) multiple times during their “efforts to verify the facts,” and had trouble getting through, why didn’t they ever leave a message? Or send an email?

The only message WS ever left at the restaurant’s number was left after the award had already been granted, by an advertising salesperson asking if I wanted to buy an ad (starting at $3,090 and going up to $8,860). I’ve posted that message as an MP3.

The ad department also emailed me–again, after I’d already won an award (but, interestingly, before the level of the award was determined). The email read as follows:

In short, the only “significant efforts to contact” the restaurant were made by Wine Spectator’s ad sales department, and were made after the Osteria had already won an award.

2. WS writes: “A total of 102 [out of 256] wines earned ratings of 80 points (good) or better.”

Setting aside the fact that the “reserve list” contained the most expensive and lowest-rated wines, this is their justification for why the list was “excellent” overall? If you look at all wines in Wine Spectator’s database, 167,710 of 190,071 wines ever rated–88%–earned 80 points or higher. Only 40% of Osteria L’Intrepido’s wines did. Even if you look only at the 117 wines on the Osteria’s list that were rated by WS, 102 of 117 (87%) earned ratings of 80 points or higher–lower than the average of all wines that WS has ever rated (88%). Is that their definition of “excellence”? Below average?

3. WS writes: “The restaurant sent us a link to a Web site that listed its menu.”

It would not have been possible to apply for the Award of Excellence without filling out the restaurant’s web site, phone number, street address, menu, wine list, and so on. As I explained in my initial post, I created a basic online presence for the Osteria. This included posting some basic information to Google and making one or two posts to Chowhound in February (though none in August, as the magazine suggests). I wanted to see if Wine Spectator’s level of expert scrutiny in determining award winners went beyond the sort of basic online research that anyone could do in two or three minutes.

Yes, this experiment was mischievous. Deception was required to expose what I saw as a wrong against the readers and the public. But the deception was hardly elaborate. Everything I did took only a couple of hours.

Wine Spectator’s readers and the public have systematically been led to believe that the Awards of Excellence represented an expert stamp of approval for a restaurant with a wine program that has been judged “excellent” by the magazine’s wine authorities, rather than an advertising scheme. You can decide for yourself, based on the evidence set forth here and in my previous post, which it is.

My name is Robin Goldstein, and I’m the author of a new book called The Wine Trials (book here; website here). Lately, I’ve become curious about how Wine Spectator magazine determines its Awards of Excellence for the world’s best wine restaurants.

As part of the research for an academic paper I’m currently working on about standards for wine awards, I submitted an application for a Wine Spectator Award of Excellence. I named the restaurant “Osteria L’Intrepido” (a play on the name of a restaurant guide series that I founded, Fearless Critic). I submitted the fee ($250), a cover letter, a copy of the restaurant’s menu (a fun amalgamation of somewhat bumbling nouvelle-Italian recipes), and a wine list.

In order to make the application appear genuine, I also obtained a Milan phone/fax number, as required by the application, and established a small online presence. Aside from creating the menu and wine list, all of this took less than three hours.

Osteria L’Intrepido won the Award of Excellence, as published in print in the August 2008 issue of Wine Spectator.

Since then, the Osteria’s listing has, not surprisingly, been removed from Wine Spectator‘s website. After the story broke, one of Wine Spectator‘s main claims (aside from calling me names) was that its staff had “called the restaurant multiple times.” However, the only message that was ever left on the restaurant’s voice mailbox (before this story broke) was on May 22, 2008, after Osteria L’Intrepido had already won the Award of Excellence. The message was from the magazine’s ad sales department, asking me if I’d like to buy an advertisement for Osteria L’Intrepido to appear in the August issue along with my listing. You can listen to the voicemail here: GSM format or MP3 format.

The main wine list that I submitted was a perfectly decent selection from around Italy that met the magazine’s basic criteria (about 250 wines, including whites, reds, and sparkling wines–some of which scored well in WS). However, Osteria L’Intrepido’s high-priced “reserve wine list” was largely chosen from among some of the lowest-scoring Italian wines in Wine Spectator over the past few decades.

While it’s interesting that the reserve list would receive such seemingly little scrutiny, the central point is that the wine cellar doesn’t actually exist. And while Osteria L’Intrepido may be the first to win an Award of Excellence for an imaginary restaurant, it’s unlikely that it was the first submission that didn’t accurately reflect the contents of a restaurant’s wine cellar.

Restaurants, like all businesses, have strong incentives to embellish their images online. We turn to experts and awards bodies to help navigate the chaotic world of information and misinformation that results. If Google, Chowhound, and a couple of unanswered phone calls suffice to verify not just the existence of a restaurant but also the authenticity of its wine list, then it’s not clear what role the critic is playing.

I presented this result at the meeting of the American Association of Wine Economists in Portland, Oregon, on Friday, August 15.

Osteria L’Intrepido’s so-called “reserve list” appears in its entirety below (with scores and some excerpts from the Wine Spectator reviews of those wines added here):

I rossi italiani “riserva” della nostra cantina

AMARONE CLASSICO 1998 (Veneto) Tedeschi 80,00 €

Wine Spectator rating: 65 points. “…Not clean. Stale black licorice…”

AMARONE CLASSICO “LA FABRISERIA” 1998 (Veneto) Tedeschi 185,00 €

Wine Spectator rating: 60 points. “…Unacceptable. Sweet and cloying. Smells like bug spray…”

AMARONE CLASSICO “GIOÉ” 1993 S. Sofia 110,00 €

Wine Spectator rating: 69 points. “…Just too much paint thinner and nail varnish character…”

BARBARESCO ASIJ 1985 (Piemonte) Ceretto 135,00 €

Wine Spectator rating: 64 points. “…Earthy, swampy, gamy, harsh and tannic…”

BAROLO 1990 (Piemonte) Az. Agr. GD Vajra 140,00 €

Wine Spectator rating: 64 points. “…Earthy, musty, lacking in charm…”

BAROLO RISERVA 1982 (Piemonte) Bruno Giacosa 250,00 €

Wine Spectator rating: 72 points. “…Agressive [sic] tannins that are sharp and harsh…”

BAROLO “ZONCHERA” 1994 (Piemonte) Ceretto 120,00 €

Wine Spectator rating: 74 points. “Quite disjointed…a coarse, chewy texture and an astringent finish. Hard to tell if it will ever come around…”

BRUNELLO DI MONTALCINO RISERVA 1996 (Toscana) Gianfranco Soldera 235,00 €

Wine Spectator rating: 74 points. “…Turpentine. Medium-bodied, with hard, acidic character. Disappointing…”

BRUNELLO DI MONTALCINO “LA CASA” 1982 (Toscana) Tenuta Caparzo 200,00 €

Wine Spectator rating: 67 points. “…Smells barnyardy and tastes decayed. Not what you’d hope for…”

BRUNELLO DI MONTALCINO 1993 (Toscana) Tenuta Caparzo 180,00 €

Wine Spectator rating: 80 points. “…A bit lacking in concentration, but with pretty, round tannins and a soft finish…”

BRUNELLO DI MONTALCINO RISERVA 1995 (Toscana) Tenuta Caparzo 135,00 €

Wine Spectator rating: 81 points. “…The palate is light-bodied with a slightly diluted finish. Light for the vintage. Rather disappointing for this producer…”

CABERNET SAUVIGNON “I FOSSARETTI” 1995 (Piemonte) Poderi Bertelli 120,00 €

Wine Spectator rating: 58 points. “Something wrong here. Of four samples provided, two were dark in color, but tasted metallic and odd…”

SASSICAIA 1976 (Toscana) Tenuta San Guido 250,00 €

Wine Spectator rating: 65 points. “…Even Sassicaia could not apparently escape the wet weather of this memorably bad vintage in Tuscany. It lacks harmony, having oxidized…”

SASSICAIA 1980 (Toscana) Tenuta San Guido 280,00 €

Wine Spectator rating: 77 points. “…Light, watery and diluted vanilla and milk chocolate character…”

SASSICAIA 1995 (Toscana) Tenuta San Guido 300,00 €

Wine Spectator rating: 90 points. “…Rich in currant, blackberry, dried herbs and tanned leather…”