Last November NORC (formerly the National Opinion Research Center) at the University of Chicago published an updated methodological report on college rankings for Vanderbilt University, Tennessee.
The report by Sobhik Barari, Eric Newsom, Ji Eun Park, and Susan M Paddock examines three national rankings: US News and World Report (USN), Wall Street Journal, and Forbes Top College List, and two international rankings, Times Higher Education (THE) World University Rankings and QS World University Rankings.
The focus of the report was on the construct validity of these “illustrative” rankings and it presented a number of criticisms:
- Lack of clarity and consistency of their underlying concepts
- The poor quality of the data, particularly that reported by institutions or collected by third party surveys
- Dependence on subjective decisions by the rankers, the most important of which are the weights assigned to metrics in the construction of composite scores
- The lack of any indication of the degree of uncertainty of scores and ranks.
The report concludes with several recommendations including the disaggregation of metric scores, making methodologies and data publicly available, using ratings rather than rankings, improving the visual presentation of overall ranks, and creating personalized rankings.