Few would argue that academia these days is overly dependent on the fraught and shaky pyramid of citation metrics.
There have been numerous ‘grey areas’ (e.g. self-citation and journal self-citation [which affects impact factor]) and outright scandals (e.g. citation cartels) that are related to – some might say driven by – this obsessive and uncontextualised numbers game.
Yet, it persists. And grows.
Research quality assessors and university executives appear to be ignoring the system’s teetering unsustainability in this regard, alongside the ever-more-meaningless achievement of ‘excellence’ that everyone is exhorted to pursue 24/7.
Last week, LSE Impact blog featured a post by Asit Biswas and Julian Kircherr titled Citations are not enough.
After establishing that just about no-one reads academic articles (and pointing out that only 20% of citations are works that have actually been read), they reiterate that scholarship should converse with broader society: “If academics want to have impact on policy makers and practitioners, they must consider popular media.”
Again, we tread on the hoary chestnut of wanting research to influence policy because, after all, everybody wants to rule the world.
Biswas and Kircherr argue that an academic’s popular media presence should count in their promotion folio:
For tenure and promotion considerations, scholars’ impacts on policy formulation and public debates should also be assessed. These publications often showcase the practical relevance and potential application of the research results to solve real world problems.
While I agree with the authors on many points, the call to add another metric to the pile didn’t sit entirely comfortably with me. We’ve seen the consequences of the metrics game. We’ve had warnings about the load and potential ineffectiveness of altmetrics as a performance measure.
I’ll admit a part of me went “hurrah!” at first about having public engagement and broader research communication (including social media) gain more recognition and value within the academy. Having this increasingly expected, but still not rewarded, labour treated as being more than a PR exercise, or part of ‘profile branding’ for institutions, would be very welcome.
But do we want it to be part of how we’ll be judged before a promotions committee?
These were my key concerns:
What if the research you do isn’t solving an immediate problem in society?
Or it doesn’t fit into popular media narratives?
For those who undertake research that isn’t immediately or easily slotted into the ‘applicable to society’ basket, it’s yet another criteria that they’ll have to be creative about. The increasing demand that research be demonstrably part of fixing society’s ills (as we recognise them, at the moment) narrows the field of projects, likely to gain the attention of popular media. The repeated calls for better and more industry partnerships affects research diversity in similar, limiting ways.
I’d venture to say that research that would catch the eye of popular media outlets wouldn’t be a huge swathe of the research that takes place in our institutions. It’s not a failure on the researchers’ part to provide a compelling enough narrative, it’s about the broad applicability and identification that’s necessary for topic to work in popular media. This is the kind of approach that works for mainstream media outlets – and that’s not for every topic or researcher. And, for the record, I don’t think it’s that often that it’s about researchers being wary of having their work ‘dumbed down’. It’s wariness of being misquoted, or having their work misrepresented. It’s a question of trusting who you’re talking to.
Doesn’t naming another promotion criteria just lead to more of the same?
The more I thought about it, the more I wondered whether we’d just be creating another element to be ‘gamed’, and laboured under.
What would public profile performance and promotion matrices require? Two op eds and a Conversation article a year for a Level B academic in Australia? Will Associate Professors need to be interviewed by ABC Radio National or Channel 7’s Sunrise before securing a Professorship?
Not instead of, but as well as.
My final concern that I’ll share here is that shifting recognition of how research travels beyond academia is great, but metricising it defines more necessary work for researchers to do.
Sure, academic journal articles may not be read by many at all, but will research councils and promotion committees be throwing them out as an indicator of quality/productivity/esteem any time soon? As Biswas and Kircherr note at the end of their post, “Change is happening very, very slowly.”
In the meantime, then, will promotions committees be looking for articles and popular media engagement? How does judging of impact from these popular media engagements take place?
Will they look at how often these media items are cited, perhaps?
The Research Whisperer 
Worth reading in relation to this post
Prof, no one is reading you.
An average academic journal article is read in its entirety by about 10 people. To shape policy, professors should start penning commentaries in popular media.
See more at: http://www.straitstimes.com/news/opinion/more-opinion-stories/story/prof-no-one-reading-you-20150411?utm_content=buffere05eb&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer#sthash.XESninWk.16ZdPNnB.dpuf
LikeLike
Thanks, Linda. It was actually that post that spurred me to write this one – because of the emphasis (assumption?) about research shaping policy. The authors’ suggestion for media contribution to be part of promotion folios would leave some non-applied research fields in the cold, but there seemed to be no accounting for this.
LikeLike
Maybe a measure of the ‘value’ of a public comment or social media post would be the number *and quality* of the comments left on a post? (just joking…) Or the number of shares or retweets it garnered? So many questions, so few useful answers.
LikeLike
I know you were being tongue-in-cheek, but it’s true that good quality of discussion on posts and articles is a big indicator of how well the work’s travelling! Good comments generate more good comments, and – I would guess – more serious consideration of issues and controversies. When The Conversation brought a dedicated community manager on board to moderate comments, it became much, much more civilised and discursive. Not as much trollage!
There’s a whole field of altmetrics that’s dedicated to finding good ways to weight various contributions online but, again, we’re going back to ‘easy’ numbers that everyone knows don’t give a very nuanced picture.
LikeLike
[…] Academic promotion by media presence?. […]
LikeLike
We keep talking about ‘quality’ as if we have already defined what it means… social media is about quantity though so I would actively avoid it being used as a criterion for promotion.
LikeLike
I agree that ‘quality’ is a much contested term, Linda! Yet it doesn’t seem to stop it influencing large, expansive (and expensive) frameworks inter/nationally…
LikeLike
There’s a broader question in here, which is the way in which we’ve accepted quite uncritically the value of highly competitive promotion systems, repeat-locked onto the values and assumptions we apply to academic hiring.
This means that from the moment an academic is hired, she’s inducted into a culture of career gaming that is meant to influence every choice she makes, every day. “Will this count towards promotion?” really shouldn’t be the answer to the question “Do I have a contribution to make to this news item/policy working party/conference committee?”
I share your ambivalence. On the one hand, it’s great that there is some stirring recognition that working in public is work. On the other, we know exactly how this calculation has proved to be corrupting in relation to citation.
I wish we could take a harder look at the assumption that competition is a proxy for contribution.
LikeLiked by 1 person
I started writing on a (slight) tangent in a draft of this post about how putting consistent demands on whether researchers create/maintain media profiles ignores both news AND research cycles. Your research won’t always be interesting to the media, and findings can’t / shouldn’t be circulated all the time (b/c when is the research actually being done?).
As always, you’ve cut right through to the bigger framework of assumptions around the logic of ‘gaming’ in academia. It’s why many of these discussions end up interrogating what it is that academics/researchers should be doing vs what the system thinks they should be doing… And the conversation doesn’t seem to move on much from there?
LikeLike
I was just chatting with a researcher last week who smartly pointed out that media mentions really only matter (in terms of “the greater good”) because policy wonks often look to the media, rather than journals, to help them determine what research should underpin their policy recommendations.
And–anecdotally–in digging through articles that have been mentioned in policy documents (using the Altmetric Explorer app), it’s looking more and more like mentions of research in policy documents are more in the vein of “Here’s our recommended course of action, and here are XYZ citations that help give background on the topic” than they are “Here’s a recommended course of action, and here’s the brilliant study whose recommendations underpin the entire thing”. More to come on that, though–plan to write that up and blog about it soon.
LikeLike
Would love to see that post when it’s up, Stacy! Please ping us – quite a few folks I’d like to circulate that to, on- and off-line!
And you make good points about how policy makers/facilitators are more likely to access this kind of information and advocacy for certain types of research or direction through media channels rather than academic ones. I saw this listing from the World Bank blog the other day (http://blogs.worldbank.org/impactevaluations/presenting-policy-vs-academic-audiences-some-thoughts) that was a good primer for this kind of tension!
LikeLike
[…] claim that few scholars read peer-reviewed journal articles (CMM April 10). Whisperer Tseen Khoo is outspokenly ambivalent about their suggestion that research impact should include publications in the […]
LikeLike
I share your concerns about the metricisation of everything. And it occurs to me as I read this post that that approach is based on a false assumption: that the only reason academic do anything at all is to collect points.
People do read academic articles. Every single academic I know not only reads articles but has a list of articles that they feel bad about not having read yet. And most people write because they have somehting to say that they WANT people to read an engage with.
THe issue of WHO reads and engages with that something is a separate one, and while I agree that if one is doing work relevant to audiences beyond your own small corner of academic scholarship then you need to be publishing somewhere that those audiences are likely to come across it (and probably doing more besides), I do not agree that advancing scholarly debate by publishing for other scholars is an illegitimate activity, which is the implication of so much of this “no one reads journal articles, why are you even publishing here” rhetoric. (That sentence got very long. Sorry.)
What would happen if we started from teh assumption that academics are hard working people who care deeply about some very specialized things that matter to some other people, too. That what they most want is to engage in debate and discussion about those things and have their thoughts influence the thoughts and actions of others. That you don’t have to incentivize them or punish them. You need to provide a supportive environment in which they can do that stuff.
Anyway, I’m preaching to the choir here, I suspect but thanks for writing what you wrote. More of us need to say these things.
LikeLike
Thanks for this fab comment, Jo. Really liked your points about how scholars writing for scholars has its value and place in the scholarly system – it shouldn’t be all about whether there’s external engagement (this also has its place in the scholarly system, but it shouldn’t be entirely reified as an ultimate good).
The whole race for citations astounds me – I’m sure there is some research out there (anyone?) on what proportion of citations are nothing that one should be proud of. What is the context of the citation? Was it to dismiss or support your argument? Some of the most highly cited ‘controversial’ pieces that I know of (e.g. Huntington’s ‘clash of civilisations’ work) are cited for good reasons. Does that matter?
And while you may feel you are preaching to the choir, it’s heartening and satisfying to read this. It is rarely something I see as people tend not to spell it out.
LikeLike