The Times Higher Education Impact Ratings have not been as stable as more traditional research rankings, and it is worth exploring why.
There are some technical reasons. The World University Rankings (QS and THE) have two significant biases that slow down movement: reputation and institutional wealth. ARWU has proxies for these, using awards.
The Impact Ratings don’t use reputation, and although there is an advantage for wealthier institutions in some measures, in others there is a clear disadvantage.
But there is movement.
Is upwards movement always good?
As part of their mission to understand how higher education contributes towards the UN SDGs, the Impact Ratings have to assess qualitative data and quantitative data. That qualitative data requires universities to choose evidence to submit.
So when it comes to the Impact Ratings, Victor Gustavo Gomez Rodriguez asked a great question: “how could an external reader tell apart a university that improved its activity from one that improved its evidence selection?”
That is difficult to do in an absolute sense, so I tried to create a system that mitigated against gains that are only based on better evidence selection. I was very aware of the need to ensure that higher education institutions could participate on a (mostly) level playing field.
I wanted a university in Malaysia or Ghana to be able to demonstrate success as readily as a university in America or Germany, despite the greater resources that the latter might be able to bring to bear.
Reducing workload
The first idea I implemented was that universities wouldn’t have to provide answers to every question. We would treat each SDG individually, and allow universities to submit to as many or as few as they were able.
To be in the overall ranking they would have to submit four from seventeen.
Evidence evaluation: trust but verify
Secondly, the scoring approach for qualitative data minimises the penalty for a poor choice of evidence. The questions always start by asking a university if they do something. If they say that they do, then they will get partial credit.
I did expect that universities should be able to back this up with good evidence, but there were always going to be situations where an error was made.
How to improve, effectively
Obviously the choice of evidence is still really important for universities. Choosing weak evidence undercuts the good work your institution is doing.
But I would also suggest a change for THE to implement going forwards to strengthen this a little. If a university fails to provide good evidence to the same question three years in a row then they should probably stop getting any credit for that question… but, of course, that is for the THE data team to think about.
First published on LinkedIn.
