Previously, Robin Goldstein scammed the Wine Spectator. Is he now trying to scam the general public with his new book, The Wine Trials 2010?
The other day, on Joe Robert's blog 1 Wine Dude, I read an interview with Robin Goldstein. Robin is the co-author of the newly released, The Wine Trials 2010, a book which lists 150 wines under $15 that allegedly beat $50-$150 bottles in "rigorous brown-bag blind tastings." I was initially skeptical of this book, having numerous questions about the methodology of the blind tastings and wine choices.
The other day, on Joe Robert's blog 1 Wine Dude, I read an interview with Robin Goldstein. Robin is the co-author of the newly released, The Wine Trials 2010, a book which lists 150 wines under $15 that allegedly beat $50-$150 bottles in "rigorous brown-bag blind tastings." I was initially skeptical of this book, having numerous questions about the methodology of the blind tastings and wine choices.
I first checked the Wine Trials website but it did not provide a description of their methodology. So I decided to buy a copy of the book to learn more. Unfortunately, my many questions have remained unanswered as the book did not contain what I sought.
The book is not fully transparent about their methodology and there is no explantion for why it was omitted. Without such transparency, I have strong doubts about the validity of the results. It is even more curious when the book provides a list of the members of their "Scientific Advisory board" who allegedly helped interpret the results and review their methods and conclusions. The authors provide "expert" credentials for the book, yet fail to allow the reader to review the methodology on their own. Readers are apparently expected to just trust the experts.
Only a few sparse comments are made in the book about their methodology. First, they used a list of 450 wines under $15, which eventually would be whittled down to 150 for inclusion in the book. Why did they start with a list of 450? Why not 500, or 1000? There is no explanation. Last year, the book only contained 100 wines and they were automatically included as part of this year's 450 wines. Why do that? It would seem to stack the deck a bit for those wines, giving them an added chance of being included in the new book.
To be nominated, and potentially placed on that list of 450 wines, a wine had to have a production minimum of 20,000, reduced from last year's 50,000 case minimum. Such a minimum ignores plenty of excellent, artisan wines under $15 which are not produced at such a high quantity. That seems to give preference for more mass produced wines, not exposing consumers to other possibilities.
The book also instituted a new nomination process this year, permitting wine professionals such as producers, sommeliers, importers, and retailers to nominate wines. First, doesn't that create potential conflicts of interest? Could a producer nominate his own wines? Could importers nominate the wines they import? Or could they nominate the wines of friends? How many total nominations were received? How were the nominations whittled down to 450? We should have much more information on this nomination process.
There were then a series of blind taste tests, but there is no information provided on how they were conducted, the number and demographics of the tasters, the experience levels of the tasters and much more. Which expensive wines were used for all of the blind taste tests? Who chose those expensive wines? How can we trust the results when there is almost no information provided on how the tests were conducted? The lack of transparency in this area really bothers me a lot.
Though the book spends plenty of time supporting the reasons in support of blind taste testing, they would have been better served by supporting their own specific results and providing details on the methodology. They spent time discussing a prior study they conducted, but fail to address the particular taste tests that led to the wines included in their book.
Some blame is placed on the major wine magazines for promoting the idea that expensive wines are better than inexpensive wines. But wine bloggers also receive some blame for their "passionate enjoyment of expensive wine." The book basically calls for all wine reviewers to conduct blind taste tests of the wines they review.
There is also a chapter that addresses some prior criticisms of the Wine Trials, much of it dealing with articles written by Eric Asimov, a New York Times wine writer. One of Asimov's major points is that wine is meant to be drunk with food, so it should be reviewed in that manner. The book actually agrees that "Most wine is better and more complex with food;... " (p.34) But the book does not feel you can "seriously evaluate" wine with food, and that doing so causes the same problems as non-blind tasting. But is that really so?
I guess it depends on what you mean by a serious evaluation, and the goal of your wine reviews. Are your reviews directed to wines people will drink on their own, or wines they will pair with food? The book does agree that a wine you drink on its own may not provide the same experience as if it is paired with food. But it appears the book's position is to provide reviews of wine to enjoy on their own. So, the wine recommendations are not as useful if someone is seeking a wine to pair with dinner.
Though the recommendations in this book are generally supposed to be for wine novices, even the authors have some caveats. First, they state to "...take our blind tasting results with a grain of salt." (p.56) They then continue to state: "To some extent, these choices might reflect the preferences of wine experts more than those of wine novices,..." (p.56) Obviously these caveats increase my skepticism.
I remain very skeptical of the Wine Trials and would like to see much more transparency from the authors. There are too many questions that have been left unanswered. As such, I cannot recommend this book.
13 comments:
Dear Richard,
Thank you for taking the time to consider the Wine Trials so seriously, and all of us at Fearless Critic would like to take a moment to address your concerns. We have heard a few comments along these lines before, and we can assure you that any extra information on methodology was left out due to space constraints, not a desire to obscure anything. We'll detail our process more thoroughly in the 2011 edition, but in the mean time let me respond to your concerns:
1. We required production volumes of at least 20,000 bottles (down to 10,000 bottles in the upcoming 2011 edition) to make sure that these wines would be available to any reader. Many of our favorite wines are from artisanal producers, and there are indeed great values in this area, but we wanted to make sure that all the wines in the book were made in sufficient quantities that a reader in Topeka could find them as easily as a reader in New York.
2. The list of 450 wines included all wines that were nominated, and we absolutely encouraged producers or importers to nominate their own wines. These are the wines that were initially evaluated, and we wanted to have as large a pool as possible. If we had the resources to begin with10,000 wines, we would have loved to taste them all - and this year we hope to have the initial pool be significantly larger.
3. The $50-150 wines were chosen randomly from bestselling brands in that range in control states according to the NABCA, including Dom Perignon, Caymus, and Beringer Private Reserve. However, it is not the aim of the book to trash specific expensive wines by calling them out--rather, it is to recommend good-value wines under $15. If we wanted to publish a guide to which $50+ wines not to buy, that would be another project entirely, and, we suspect, a much less useful one
4. As for the issue of tasting with food, a book like this simply would not be possible to do with food, nor is it an effective way to taste large groups of wine. Including food in the tastings would add a huge series of variables - are the wines paired correctly, is the food well cooked, is the dish too similar to the dish we had yesterday, etc. The only way to evaluate the wines fairly for our readers is to taste them under the same conditions - that is, alone. If you have a way to efficiently conduct blind tastings of hundreds of wines in a way that eliminates these variables, of course, we would be thrilled to hear them.
5. Our tasting panel and their qualifications are spelled out in detail on p. VI of the introduction. They all have professional experience with wine and include sommeliers, winemakers, wine educators, wine economists, and wine importers.
I hope this clears things up, and we're always eager to hear what our readers think. Happy drinking!
Cheers,
Tyce Walters,
Wine Trials Associate Editor
Thank you very much Tyce for taking the time to address some of my questions.
I can understand the issue of space constraints, but you very well could have published the methodology info on your website, where you would not have any space constraints. And you still could do so now. As I said, without that methodology, there is a lack of transparency which casts a shadow over your results.
1. I understand the availability reason, but the result still ignores all of the excellent artisan producers. Thus, larger producers are getting a preference.
2. How did you decide on a limit of 450 nominations? Though really it was 350, as 100 were grandfathered in? Will you provide a list of all those people and companies which provided nominations? Why would you encourage them to nominate their own wines?
You failed to answer my prior question about conflicts of interest. And the fact you encouraged the nominators to nominate their own wines seems to me to clearly be a conflict of interest. Your book makes a big thing about conflict of interest with wine advertisers, so why does your nomination process also seem to be rife with conflict of interest?
3. I understand that you don't want to bash specific expensive wines, but failing to provide a list of those wines does raise questions about the validity of the the blind tests. Were similar wines pitted against each other? To be fair, such tests should compare very similar wines.
For example, did you compare inexpensive Cava to Champagne? I would not consider that a fair comparison. Instead, any test should have compared an inexpensive Cava to an expensive Cava. What expensive wine did you compare to the Portuguese Vinho Verde? What expensive wine did you compare to the Conquista Torrentes? Which $50+ wine did you compare to the Roses, as it is extremely rare to find any Rose that costs $50+?
4. I can understand the difficulties involved in tasting with food, but that does not deny my conclusion that your wine recommendations are not as useful if someone is seeking a wine to pair with dinner.
5. Are you saying that your tasting panel were the only people who participated in the blind tastings for this book? Did you have any wine novices participate in these blind taste tests? If so, what were the demographics of those novice tasters?
Thanks again and cheers!
I would challenge anyone to go buy a few bottles of the wines recommended in this book. As we have done from the 2009 & 2010 books.
We are quite informed wine fans, having collected wines and enjoyed "fine wines" for the past 20+ years. To sum up our experiences, we are quite picky with our tastes.
"The Wine Trials" does an excellent job of pointing us in the direction of some excellent, and tasty, bargains. Regardless of the exact methods used to create the list, we believe that this book is an excellent resource.
I believe that whatever your background with wines, finding several excellent, enjoyable wines for $15 (or less) for a bottle is well worth the purchase price of "The Wine Trials".
Hi Kevin:
I think the methodology is extremely important, especially considering the stance of the authors. They want to question the methodology of others, such as the wine magazies, yet fail to adequately answer questions about their own.
So there seems little reason to embrace the Wine Trials recommendations for value wines over value wines recommended by a magazine such as Wine Spectator. Or even by a blog. And a blog will give you those recommendations for free.
Are there some good recommendations in the Wine Trials? Sure, but that is true of most any wine guide. But there are also better wine values out there. And without the apparent conflict of interest evidenced by the Wine Trials.
I also think the book is far more geared to the wine novice, than someone with more knowledge and experience.
Hi Kevin, thanks so much the kind words - glad to hear you enjoyed the wines, and I hope you'll find many more you like in the book.
Richard, I'm a bit confused by where you see a conflict of interest. A conflict of interest typically occurs when a reviewer or critic has a financial incentive to be less than objective - for instance, when the producers he reviews also are his main revenue stream by buying advertising space, or when an importer funds a lavish vacation. We are absolutely committed to avoiding anything of this nature (one of the reasons we do not accept advertising), and I'm therefore confused as to where a conflict of interest comes in.
And you're correct that the book is intended to be accessible to anyone, whether or not they have any wine background. We hope that the research we have done and the bottles we uncovered will be helpful to even the most experienced drinkers, but we certainly did keep novices in mind. I hope the book will be at least as useful to a couple in Kansas looking for a great value dinner wine as it is for a New York oenophile.
Hello Tyce:
First, you failed to respond to most of my prior comments and questions, placed in my first comment to this thread. Do you intend to address those matters at all?
Second, I broached the conflict of issue matter in my original post. You did not address it in your first comment, and I followed up again in my initial comment here.
Let me try to explain the issue again for you. 450 wines were nominated, yet you asked wine stores, importers, and such to nominate those wines. Those individuals and organizations had a financial interest in nominating their own wines. And you encouraged them to do so. The nominators had a clear conflict of interest.
So, your list of nominations surely was not independent or unbiased. The deck was stacked in a way.
Your lack of full transparency also does not inspire confidence in the process.
Hi Richard,
I get the sense that many of your concerns stem from our nomination process, and I think you may have gotten the wrong idea about what this process was. As we say in the book (but perhaps not clearly enough)the nominations were simply a way for us to make sure that we were tasting as wide a range of wines as possible. As I'm sure you know, there is no publicly available list of every wine below $15 and with a sufficient production volume. A method was therefore required to find as many of these wines as possible. As we do not know every such wine (and if we had solely chosen the wines to be tasted ourselves there would inevitably have been charges of favoritism, etc.) we asked anyone to nominate wines. There was no "weed-out" process as to who was nominated or who wasn't - every wine that was nominated, whether by a producer, retailer, importer, wine blogger, critic, or anyone else, was included in the tastings.
In an ideal world, every single producer who made a wine that fit our criteria would have nominated it. That way we would have been sure that we were tasting everything. Of course many wineries or importers did not submit - either because we were unable to contact them about submissions or because they chose not to for whatever reason. But by encouraging producers to submit nominations, we were using the power of the marketplace to make sure we were being as thorough as possible - not in any way being corrupted or otherwise beholden to it.
As I said before, a conflict of interest occurs when the REVIEWER has a financial incentive to favor some wines. That is simply not the case here. The fact that wineries have a financial incentive to nominate their own wines was precisely the point, as it allows us to make sure we are starting with as broad a list of wines as possible - not a conflict of interest.
To address a few of your earlier questions about methodology as well:
Wines were not "pitted against each other" individually. Rather, participants tasted a large number of wines and ranked them in a numerical scale. If a taster tasted 15 wines, and among those wines gave Ste. Michelle a "4" and Dom Perignon a "2", I think it's fair to say that the taster preferred the Ste. Michelle to the Dom Perignon. The fact that this is how the tastings were conducted can be found in our descriptions.
Finally, you'll notice in the book that we draw a distinction between our blind tasting panel (who tasted the initially nominated wines to select the wines for the wine trial itself) and the tasters at the finals/award round. Many of the tasters in the finals (the "wine trials" themselves) were everyday wine drinkers. I'm not sure what kind of demographic information you're looking for, but we did not collect data on education levels, income, etc., if that's what you're curious about.
I hope all this helps. You might also be interested in a post that Robin Goldstein just put up on his blog, http://blindtaste.com/2010/02/13/when-are-high-wine-prices-justified/#more-628 , which explores some of the first principles of the book.
Hi Tyce:
I am still skeptical of the nomination process. Somehow, you only got 350 nominations, worldwide, for the 2010 book? (Considering 100 of the 450 were from the prior year.) That means thousands of wines never made the list. How do you know this was a representative list?
It was mainly made of wineries, importers, etc. who wanted to promote their own wines. They were not encouraged to be independent or impartial, and had no incentive to do so. Just the opposite, you encouraged them to nominate their own wines. Just to increase the number of wines you would review.
Though you defined "conflict of interest," that is not the sole definition. It exists in many different forms.
As for the wines pitted against each other: You previously stated the $50+ wines were chosen by you randomly from best selling brands. But how many of those wines were chosen? And why didn't you ask for nominations for those wines as well? You did not want to select the <$15 on your own to avoid impropriety, so why then choose the $50+ wines?
Ok, so the professional blind tasting team first tasted the nominated wines, narrowing down the list for the actual wine trials. As they started with 450 wines, how many wines did they eliminate prior to the wine trials? Did those blind tasters also compare the wines to over $50 wines? Or was that only in the actual "wine trials?"
If you did not keep demographic information on the wine trial tasters, then how can you determine whether it was a valid study or not? How do you know whether the group was representative or not? What information did you obtain on the wine trials tasters?
As for a few other questions I previously asked, that you did not respond to yet.
What expensive wine did you compare to the Portuguese Vinho Verde? What expensive wine did you compare to the Conquista Torrentes? Which $50+ wine did you compare to the Roses? To be fair, there are few $50+ wines in these categories for comparison. So, I want to know what was compared against them, to determine whether it was fair or not.
Any why haven't you published the full details of your methodology on your website? You may not have had space in your book, but you have plenty of space on your website.
Thanks.
Hi Richard,
If you don't understand why wine producers would have an incentive to submit their best wines, then we are talking in circles.In an analogy to law: it is not a conflict of interest for a judge to ask lawyers for both sides to submit evidence, even though those lawyers have an interest in representing their client. On the other hand, it would be a conflict of interest if the judge's salary were being paid by one side or the other.
As we stated previously, the $50+ wines were not chosen randomly. Rather, they were the bestselling wines in that price point in the Control States.
Your other questions are fully answered in the book (see especially the appendix, preface, chapter 1, and chapter 1).
Frankly, my other questions are not clearly answered in the book.
The appendix deals with the 2008 experiment, and not the 2010 wine trials, so it is irrelevant to my questions. Chapter 1 does the same, dealing with the experiment and not specifically the 2010 trials.
Based on the preface and your comments here, it appears your professional blind tasting team tasted the 450 wines, and narrowed them down to 150. All of those 150wines were then added to the book. And it was the professionals who compared those wines to the $50+ wines. Is that correct?
So, the average wine drinkers only taste tested those 150 wines, and helped to choose the top winners on pg.59&60. And they did not compare these 150 wines to the $50+ wines. Correct?
Though I am repeating myself, these questions were NOT answered in the book. If you think they were, please provide me the specific page for the answers.
If you did not keep demographic information on the wine trial tasters, then how can you determine whether it was a valid study or not? How do you know whether the group was representative or not? What information did you obtain on the wine trials tasters?
What expensive wine did you compare to the Portuguese Vinho Verde? What expensive wine did you compare to the Conquista Torrentes? Which $50+ wine did you compare to the Roses? Based on your prior experiment, it appears you did not compare equivalent wines, only very general comparisons. An inexpensive Cava is a very different style from Veuve. I would not say that is a fair comparison.
Any why haven't you published the full details of your methodology on your website? You may not have had space in your book, but you have plenty of space on your website.
First, I must commend both Tyce and Richard for having what might be the most mutually respectful argument I've ever seen in my lifetime. No mudslinging, no namecalling... You two should run for public office!
Anyway, I understand both sides of the issue. Any scientific publication should be held to the highest criticism, while the publisher simultaneously has the right to defend his or her claims. But as an innocent bystander, I feel this argument has gone on a MAJOR tangent, and neither party is picking up on it...
Indeed, the selection process is imperfect for BOTH the inexpensive wines and the expensive wines. This is because, as is true with ANY scientific study, the world we live in is imperfect.
But I'm willing to accept this imperfection because, as I see it, the GOAL of "The Wine Trials 2010" is not to render expensive wines as wasteful conspicuous consumption, while unilaterally praising cheap wine. Rather, I see it as a methodology for the Average Joe to enjoy wine to the same capacity as those that can afford the world's most expensive...
Still, there seems to be this uncomfortable overtone that the book is an attempt to sabotage the expensive wine industry. I just don't think that's true. After all, it's our own perceptions that matter the most, right? And if people perceive expensive wine as more enjoyable, more power to them!
(Last two sentences shamelessly stolen from "The Wine Trials 2010" and paraphrased)
I'm a college student in my mid 20's. As great as some pricey wines may be, I just can't afford them. So if 500 blind tasters said they liked a $6 of Fish Eye Cabernet Sauvignon, I'm inclined to give it a shot :).
That's my two cents. Look forward to hearing from either of you soon.
-Jason M
Greater Hartford, CT Area
Hi Jason:
Welcome to my blog and thanks for your comments and compliments.
I am all for value wines, and have recommended plenty of them on my blog. One of my fav wines of last year was a Portuguese wine that sold for under $6. Plus, there are some excelent boxed wines starting to come onto the market. One certainly does not have to spend tons of money to find delicious and enjoyable wines.
I probably would not have been as critical of this book if it merely were a list of recommended value wines. But when it attempted to make it a more scientific stdy, and the metholody was murky, then it raised questions in my mind.
These books are great for Non-Snobs. Artisan wines my butt, many people just want to go down to the grocery store and get a decent bottle of wine for under $20. I couldn't give a Rat's you-know-what about methodology. I have tried most of the Cabernets in the last 2 books and have found many I like (I don't touch white wine, can't stand it). These books are great for the rest of us.
Post a Comment