Pitfalls in aggregating performance measures in higher education

What are the implications of different methods used to aggregate performance measures in higher education rankings?

We care because aggregating university performance measures into rankings significantly influences perceptions about institutions, impacts funding allocations, and drives decisions at both institutional and national levels. Understanding the implications of various aggregation methods helps ensure that rankings maintain statistical validity and transparency, preventing potential misinterpretations and misuses in their applications. With rankings prevalent in an increasingly competitive global education market, it is crucial for stakeholders to understand the strengths and limitations of these methodologies to make informed decisions.

We evaluated alternative aggregation methods and their sensitivity to weights using published ranking data, with illustrations from the Times Higher Education, Shanghai Jiao Tong, and Universitas 21 rankings. We analyzed the impact of various scales, including the common method of scaling measures relative to the highest performer and the method of standardization through z-scores. We focused on the effects of these methodologies on the rankings of 41 countries in terms of gender balance in tertiary institutions, among other performance measures.

We find that the aggregation method has considerable implications for rankings, particularly highlighting that scaling methods are preferred over data transformation for maintaining interpretability. We find that the choice of included variables is more influential than the weights assigned to them. We also note significant limitations in extending rankings to a large number of universities and that using z-transforms can obscure practical differences in performance, such as when evaluating the percentage of female students and staff.

• Published ranking data from Times Higher Education, Shanghai Jiao Tong, and Universitas 21 rankings for analysis of university performance measures

• The methodology used for constructing rankings accurately captures the relative performances of the universities or countries being compared
• Statistical assumptions regarding the distribution of performance measures (e.g., normal distribution for z-scores) hold true
• The definitions and measures of performance are relevant and applicable across different contexts and types of institutions