The Wine Trials 2010: A Lack of Transparency

Previously, Robin Goldstein scammed the Wine Spectator. Is he now trying to scam the general public with his new book, The Wine Trials 2010?

The other day, on Joe Robert's blog 1 Wine Dude, I read an interview with Robin Goldstein. Robin is the co-author of the newly released, The Wine Trials 2010, a book which lists 150 wines under $15 that allegedly beat $50-$150 bottles in "rigorous brown-bag blind tastings." I was initially skeptical of this book, having numerous questions about the methodology of the blind tastings and wine choices.

I first checked the Wine Trials website but it did not provide a description of their methodology. So I decided to buy a copy of the book to learn more. Unfortunately, my many questions have remained unanswered as the book did not contain what I sought.

The book is not fully transparent about their methodology and there is no explantion for why it was omitted. Without such transparency, I have strong doubts about the validity of the results. It is even more curious when the book provides a list of the members of their "Scientific Advisory board" who allegedly helped interpret the results and review their methods and conclusions. The authors provide "expert" credentials for the book, yet fail to allow the reader to review the methodology on their own. Readers are apparently expected to just trust the experts.

Only a few sparse comments are made in the book about their methodology. First, they used a list of 450 wines under $15, which eventually would be whittled down to 150 for inclusion in the book. Why did they start with a list of 450? Why not 500, or 1000? There is no explanation. Last year, the book only contained 100 wines and they were automatically included as part of this year's 450 wines. Why do that? It would seem to stack the deck a bit for those wines, giving them an added chance of being included in the new book.

To be nominated, and potentially placed on that list of 450 wines, a wine had to have a production minimum of 20,000, reduced from last year's 50,000 case minimum. Such a minimum ignores plenty of excellent, artisan wines under $15 which are not produced at such a high quantity. That seems to give preference for more mass produced wines, not exposing consumers to other possibilities.

The book also instituted a new nomination process this year, permitting wine professionals such as producers, sommeliers, importers, and retailers to nominate wines. First, doesn't that create potential conflicts of interest? Could a producer nominate his own wines? Could importers nominate the wines they import? Or could they nominate the wines of friends? How many total nominations were received? How were the nominations whittled down to 450? We should have much more information on this nomination process.

There were then a series of blind taste tests, but there is no information provided on how they were conducted, the number and demographics of the tasters, the experience levels of the tasters and much more. Which expensive wines were used for all of the blind taste tests? Who chose those expensive wines? How can we trust the results when there is almost no information provided on how the tests were conducted? The lack of transparency in this area really bothers me a lot.

Though the book spends plenty of time supporting the reasons in support of blind taste testing, they would have been better served by supporting their own specific results and providing details on the methodology. They spent time discussing a prior study they conducted, but fail to address the particular taste tests that led to the wines included in their book.

Some blame is placed on the major wine magazines for promoting the idea that expensive wines are better than inexpensive wines. But wine bloggers also receive some blame for their "passionate enjoyment of expensive wine." The book basically calls for all wine reviewers to conduct blind taste tests of the wines they review.

There is also a chapter that addresses some prior criticisms of the Wine Trials, much of it dealing with articles written by Eric Asimov, a New York Times wine writer. One of Asimov's major points is that wine is meant to be drunk with food, so it should be reviewed in that manner. The book actually agrees that "Most wine is better and more complex with food;... " (p.34) But the book does not feel you can "seriously evaluate" wine with food, and that doing so causes the same problems as non-blind tasting. But is that really so?

I guess it depends on what you mean by a serious evaluation, and the goal of your wine reviews. Are your reviews directed to wines people will drink on their own, or wines they will pair with food? The book does agree that a wine you drink on its own may not provide the same experience as if it is paired with food. But it appears the book's position is to provide reviews of wine to enjoy on their own. So, the wine recommendations are not as useful if someone is seeking a wine to pair with dinner.

Though the recommendations in this book are generally supposed to be for wine novices, even the authors have some caveats. First, they state to "...take our blind tasting results with a grain of salt." (p.56) They then continue to state: "To some extent, these choices might reflect the preferences of wine experts more than those of wine novices,..." (p.56) Obviously these caveats increase my skepticism.

I remain very skeptical of the Wine Trials and would like to see much more transparency from the authors. There are too many questions that have been left unanswered. As such, I cannot recommend this book.