Integrity and trust is vital in the pursuit of truth in research

Integrity and trust is vital in the pursuit of truth in research

October 9, 2019 0 By Stanley Isaacs


Welcome to this episode of Integrity Matters
by Turnitin. And I’m pleased today, to be joined by Phil Baty from Times Higher Education.
He is the Chief Knowledge Officer. Thanks for joining us, Phil. It’s a pleasure.
So I’d just like to start by asking a little bit about your role as Chief Knowledge Officer.
What does that entail exactly? Well, I’ve often get criticised for my hilarious
job title. I was once the Editor-at-Large of Times Higher Education. Everyone asked
me what that meant. I think people thought I was editor at lunch or editor who is never in
the office. Chief Knowledge Officer is really just a way of describing quite a holistic role. I created the World University Rankings’ current
methodology. So I’m responsible for the editorial interpretation and analysis of all of
our world rankings and the development of different rankings. But I’m also responsible
for Times Higher Education’s communication as well. So, the public engagement side and
our comm, side and I also have an advisory role in terms of strategy because I’ve been
working at THE for more than 20 years. So it’s a nice generic job title that kind of
covers a range of activities. Okay. You mentioned that you’ve been there for
nearly 20 years, what attracted you to Times Higher in the first place and specifically
education? Why did you want to be… have a career in the education space? Well funny enough, actually the first articles
I ever wrote as an aspiring journalist were for Times Higher Education. I was
a student journalist passionate about journalism, really as the sort of good old-fashioned
truth-seeking. But the first opportunity I had to write any journalism at all was actually
from within the education field. I did a work camp in Israel and the occupied territories
and I was really passionate about communicating other people’s stories. So that’s where it
all started and I managed, because I had worked for THE, I got my first job, well my first
real job, at THE – Times Higher Education. So I was there as a junior reporter and it would
just occur to me very, very early on in my career that I just am so passionate about
education, you know, it’s transformative. It transformed my life, my opportunity to
go to university was hugely transformative to me personally. And then once you’re in
the sector, you start really appreciating the extraordinary, range of, public
good that the universities provide right through from educating people but through
to solving the world’s grand challenges who through research, it’s just an extraordinary
sector to represent. And then as you see it more globally and forming
and global higher education becoming more powerful collaborations and interactions between
universities globally, you suddenly realise this one of the most positive forces for good
in the world. So, I’m stuck, I’m hooked, I’m here for the long term. Great. Yeah. I mean certainly, I’ve
been involved in education, well with Turnitin, for the past 15 years and I think for me
the global remit as well in terms of having an impact at that scale and improving standards
to that scale is something that that’s still excites me today as well, so it’s kind of interesting. And you get the sense that actually the big
challenges the world’s facing, a) will be tackled primarily by great academics thinking great
thoughts, but particularly through academics collaborating across borders and crossing those
cultural divides, transcending geopolitics, to take on the grand challenges of food, security,
climate change, et cetera, et cetera. So, it’s an incredible sector, absolutely
extraordinarily privileged to work in it. Yep, I agree. So you’ve launched the…
Times Higher has launched the Impact Rankings this year. What was the… what was
the thinking behind that and what are the key takeaways that you’ve got from
the first set of results, would you say? Well, the thinking really was, recognising
that there are different kinds of excellence in global Higher Education. So we’ve
produced the World University Rankings since 2004; 15 years. We’re on the 16th edition
and that really is a very narrow sense of excellence. It’s a traditional sense of excellence.
We look at research excellence. We look at prestige and we do look at the university
holistically, but primarily it’s going to be the Oxford’s and the Harvard’s and the
Stanford’s of this world. Well, the Melbourne’s and Sydney’s that are really
standing out there. Very, very research active and they tend to be more wealthy. Although,
of course, we are seeing developing countries doing very well. China, in particular, is
rising. Some South African universities are doing well. But it tends to be a global North-dominated ranking and obviously the more that gained influenced it’s become the Gold standard
of what a great research university looks like. We’ve wanted to try and celebrate and
demonstrate that universities are excellent in different ways. It’s not all about being rich and having
huge endowments or having nobel prize winners on campus. It’s about the small everyday things,
you know, transforming a student’s life chances; making your contribution to sustainability
through your own stewardship of your own resources; educating or contributing to making the world
a better place in terms of health and welfare but not by curing cancer but by educating
the next generation of nurses and health professionals. So just understanding excellence in different context. So we created the Impact Rankings framed around
the sustainable development goals. And it’s been incredibly well received. It really has been transformative in terms of universities saying it’s a breath of fresh
air that, you know, non-traditional universities in the sense that those don’t always shine
out or get recognised in traditional rankings are able to demonstrate the impact
they have on a different set of metrics, on a different framework. Now, of course, you know, you have
the Impact Rankings and the World University Rankings and a couple
of other rankings as well, and you have an awful lot of metrics go into that and the… a lot of people
look at these rankings and you know, people are hired or fired based on them,
so it’s very important to ensure that the integrity is maintained within those rankings.
How seriously do you take that and how do you ensure that the results are and have integrity and can’t be challenged on that basis? I mean, I guess the results can always be
challenged because the methodology is always based on subjective judgment. So Times Higher
Education, me personally, we’ve taken responsibility for the metrics we choose and the weightings
we give them. So there’s always an interpretation issue. Of course, we do that in a very consultative
way when we created the current methodology back in 2010. I almost had to have worldwide,
roadshow of critical analysis and discussion of methodology. We surveyed users, we
published a draft methodology for comment and we had an expert advisory group. So, creating
the methodology and developing the method is very collaborative and consultative and
very transparent as a major organisation. We openly talked through our processes but
at the end of the day, it’s a subjective judgement to say, well, we weight this element in this
way, we weight the element in that way. But beyond that, we are incredibly precious
about our… the trust people have in us. As you say, you know, you do see Vice
-Chancellor salaries or President salaries pegged to ranking performance, the governing
bodies of many universities across the world, put Times Higher Education’s rankings into
the governing KPI’s for the whole institution. Students make decisions based on rankings.
Academics make career choices that are raised in form by rankings. So our integrity is…
is massively important to us. And of course, we make money from the rankings.
It’s a free process; it’s the universities submit data for free. We publish it all openly
for free. But of course, we provide analytic services and consultancy services and an interpretation
of all the data we have and it’s painful. So integrity is massively important to us. The trust we have in the sector is
everything about our brand, that’s how we are a successful business. I think the most
profound element around rankings is that we made a decision a couple of years ago now,
to have the entire thing externally audited. So we actually brought in Price Waterhouse
Coopers and they have a process where they audit our data handling and data collection.
But they also then recreate the rankings independently of us. So they build their own code and they have
their own algorithms and they have all the same inputs as us, and they separately as
a team that will build the ranking in and they’ll only sign it off when they’ve replicated
it and they’re able to compare them. So, the actual calculations and the data handling
is all audited. Beyond that, there is a huge element of trust with the sector. We use a
lot of independent data, so we survey academics and that’s very much independent of the universities.
We use bibliometric analysis, so we’re analysing 13 million research papers, 70 plus million
citations to those papers that can’t really be manipulated by the institutions but we
do actually trust the universities to supply basic information, like faculty numbers, income
figures, student numbers. And we do have a whole series of checks and balances in place
there around, you know, manual auditing, automated auditing if the numbers change from one
year to the next. We have a process where universities have to explain and provide evidence,
but fundamentally that part of the ranking is actually based on trust.
And we have sanctions. If we feel that it’s been manipulated or institutions can’t
explain significant change, we are able to remove people. But I hope we’re in a world
where integrity is vital. So, of course, we have checks and balances and measures and
ultimately the audit. But actually, I would like to say to the sector, you know, the only
value of this ranking is if everyone has an honest and clear view of the global picture
and it’s in everyone’s interest to have an honest picture and universities fundamentally
are about pursuit of truth. So that’s the fundamental part of the ranking
process. Now the reputation survey is a key part of the World University Rankings. How
important do you think academic integrity or research integrity are to the reputation
of the university and do you have any opinion as to whether university leaders are putting
enough emphasis on that? I think there’s… a key element of the ranking,
as you say, is the reputation survey. We survey around 10,000 academics every year and we
use United Nations data to make sure it’s geographically representative of scholarship
around the world. So about 20 percent from China, et cetera,
et cetera. And we’re asking subject specialists. So the idea really is if you’re an expert
in astrophysics and we say who’s the greatest universe which are the greatest university
teams around astrophysics, they’ll know based on reading each other’s research and hiring
each other’s grad students and the environment. So I think in that context, absolutely if
a university team has been found to have been lacking in integrity, you know,
there’s upsettingly, there are examples of research fraud. The pressure of academics
to publish, to make breakthroughs, I think does encourage people perhaps sometimes
to cut corners or massage data. Scandals of that nature will resonate dramatically through the communities. And I think we can see that there have been
cases where universities’ reputation scores have diminished at the time of
big academic integrity scandal. So very clearly, universities will be held to account
in our surveys for any lapses of integrity. But I think beyond that, you know, we live
in an information age. There’s a huge amount of information immediately available on
universities around the world. Bad news spreads incredibly rapidly. So, you know, era of social media, you know,
some localised problems of the past, do become national/international scandals these
days and I think university leaders have to take this issue very, very seriously. I
mean fundamentally, they need to take it seriously anyway, because scholarship is nothing but
integrity, but more and more they will be called out and held to account and we’ll suffer
as a result of these lapses of integrity. Okay. Great. And then just finally, you know,
you’ve launched the Impact Rankings now. What do you see as the future of rankings?
Where do you want to take it next? Well, speaking of integrity actually
I think rankings probably will have to be almost turned back on the sector. I think
there’s the era of people like Times Higher Education sort of declaring, you know, this
university is better than that university based on our methodology is kind of moving on.
I think we’re moving away from that. I think from our perspective, it’s really
a question of can we collect the data that genuinely reflects universities own priorities. So rather than some sort of outsider saying,
hey, you’re better than them, it’s almost about creating as many metrics as possible.
And then holding a mirror up and seeing which ones matter to you, which metrics
are relevant to your mission, which ones are relevant to your national priorities, your
international mission? If you’re a research university, we have a wealth of information. If you are a
teaching-led institution we have different metrics. If you’re a university committed
to social justice, we have different metrics. So, I think we get criticise quite a lot for
saying there’s more and more rankings, you know, there’s the Times Higher World Rankings;
the Teaching Rankings; the Impact Rankings, Latin America Rankings; Asia Rankings, Emerging
Economies Ranking. We actually say that’s healthy because there isn’t a right answer.
There is no correct ranking. They’re all about the interpretation and they’re all about the
weights and metrics used. So I think really the era now is to say we
hand the rankings back to the community and say like we’ll collect the data; we’ll present
the data; we’ll analyse the data but more and more we want to give it back to you and allow
you to create your own benchmarks, create your own metrics that suit your
individual mission. Okay. Democratising rankings and also making
them more transparent. So integrity back to the sector. We have all this data, do with
it what you will. Use it to your own strategic needs. Okay. So you see yourselves as,
like you said, it’s more and more data being produced and the difficulty is, you know,
how do you bring all that together and make something meaningful out of it. So you see
your role potentially is assisting in that processing. Okay, this is what
collecting that data, we can create these metrics. We can create these guides
on what’s happening and then it’s up to you to follow whatever you want.
Yeah, we’ll consult with the sectors to correct the ranking, but then we will also allow
the sector to break that all down again. So we’ll create these composite scores and
pillar metrics based on our interpretation of what the sectors tells us they need and
what they’re interested in. But more and more, we’ll turn it back around and say we’ve built
the composites but you break those composites down… break those composites down, and then create
your own profiles and benchmarks against your own peers and your own mission. Okay, great.
Thanks very much. That concludes today’s episode of Integrity Matters.