The October 15, 2007 issue of U.S. News & World Report has an article on the best cars and trucks to buy. You can read a small excerpt of the article here. You can see the official U.S. News & World Report car rankings here. Their rankings are broken out by category, so you can, for example, find out their recommendation for the “best upscale midsize car”, the “best luxury large car”, the “best affordable compact SUV”, and so on.

They describe the methodology used to determine the rankings here. In short, they combine information from existing reviews and available data from other sources. Then — and this is the important part — based on a survey indicating which factors are most important to consumers, this information is converted into an overall score. For example, the survey results might indicate that on average (i.e., averaged across all survey respondents), style is twice as important as cost when buying an upscale luxury car, but cost is half again as important as style when buying an affordable hybrid. So when compiling the aggregate score for each car, for upscale luxury cars they will count style twice as much as cost, and for affordable hybrids they will count cost 1.5 times as much as style. You get the idea.

My complaints with the U.S. News & World Report methodology and results are as follows:

1. The results are the product of a survey. Consequently the results are only useful for a particular reader to the extent that the reader happens to agree with the mythical “average” consumer’s opinion of the relative importance of the various factors.
2. The average relative importance of these factors (resulting from the consumer survey) are not provided, so the reader can’t even determine the extent to which (s)he agrees with them.
3. Many (if not most) readers will not bother to look at the methodology page. Of those who do, many will fail to realize that the results represent a one-size-fits-all solution for the mythical “average” consumer. The article should provide a 24-point font flashing red warning to explain this to the reader — on the main article page, not the methodology page.
4. The rankings and reviews page does not provide the raw scoring data that was aggregated together, so the reader cannot perform his/her own analysis based on his/her own view of the relative importance of the scoring factors.

Since many readers will fail to fully appreciate the significance of the results being targeted to the “average” consumer, thousands of people will doubtlessly be running out now to purchase the #1 ranked car in their category of interest when in fact this might not be the best car for them. For example, reliability and longevity are probably far more important to me than to the average consumer, so I’d probably be much happier with the #5 ranked Toyota Sequoia than with the #1 ranked Chevy Tahoe.

If U.S. News & World Report had provided the raw data — i.e., the scores (on a scale of 1 - 10) in each of the detailed scoring criteria (interior style, exterior style, performance, reliability, cost, etc.) for each car evaluated, then each consumer could perform his/her own individual multi-criteria decision analysis and find the best car for him/her. (Actually they do provide it, but not in tabular form; you must surf to each car’s review page and gather the data yourself. And even then, the scoring criteria are not very detailed.)

Don’t worry if the phrase “multi-criteria decision analysis” sounds scary. Although very sophisticated methods exist for this that rely on advanced decision theory (see, for example, the Analytic Hierarchy Process), there are also simplified approaches that can work well for the layman. In my next post, I’ll present such a method.