Repeat after me:
Submitting a grant application is not a valid Key Performance Indicator (KPI).
If you manage research staff, write out the following line fifty times:
I will not measure ‘dollars won’ as a Key Performance Indicator (KPI).
After Tseen posted her plea for “no more half-baked applications“, one of our twitterati pointed out that sometimes applications are half-baked because submitting the application satisfies a key performance indicator in a work-plan.
I know that there might be a strong temptation to list either ‘grants submitted’ or ‘dollars won’ (or both) as key performance indicators. As a Research Whisperer, this drives me crazy! In the first case, it pushes people to write grant applications that they can’t win. In the second, it adds completely unpredictable factors like government funding policy into the performance review process. In both cases, it measures the wrong thing.
Measuring research performance, like any performance measurement, is fraught with difficulty. In essence, you are trying to measure the quality or value of the research completed. This is a subjective process. The true value of research may not emerge for many years. It would be great to look at the contribution made to a field of knowledge over time in terms of new and powerful concepts, ideas, and methods, but generally that isn’t practical.
‘Indicators’ are meant to stand in for other quality measures. Generally, indicators are objective measures that can be ‘counted’ easily and quickly.
Some indicators that I like include:
- Final reports, which indicate that a research project has been satisfactorily completed.
- Research outputs such as papers, patents, and art works, which indicate that a research project has produced new knowledge.
Indicators that drive me crazy include:
- Applications submitted.
- Number of grants received or total funding received.
This is especially true when a rider is added that the application must be to a ‘top tier’ funding body (usually the largest national funding bodies and research councils in a country).
For a small organisation, an emerging research area, or a harried academic, getting a grant application across the line might be a major achievement. However, it is not an indicator of ‘key performance’, nor is it an indicator of research undertaken.
A grant application is an input, not an output. When people apply for funding, they are describing research that they might like to do in the future.
I understand why some managers might like to count applications as a KPI. In a non-traditional research area, applications might seem like the first step towards research outputs. They certainly represent a lot of hard work on the part of the applicant. However, it is a false measure. All too often it leads to half-baked applications, written in haste to ‘tick the box’.
I can also understand the attraction of counting ‘dollars won’ as a KPI. If someone wins a grant, that is an external measure of their excellence. In addition, many governments provide infrastructure funding to universities based, in part, on the amount of research funding they receive. Dollars are easy to count; there is a whole accounting system charged with counting the number of dollars coming into (and going out of) a university.
As I stated earlier, ‘dollars won’ or ‘grants won’ are false measures of research performance.
Research funding precedes research. Yes, it is an external vote of confidence, but the amount of money available, and the competitiveness of the funding ecosystem is completely beyond the control of the applicant, research manager, and university. Governments can reduce their research funding envelope, or even close whole schemes down. Industry partners change their strategic direction, or eliminate their research spend completely. This means that you don’t have any control over whether the KPI is achieved, which makes for an unrealistic, unpredictable KPI.
And don’t get me started on managers who only want to count applications written or grants won from particular ‘top tier’ funding schemes. Not only are they counting the wrong things, but they are pushing onto their staff a particular vision of how some management thinks the world works. If you get $20,000 of research funding, I don’t care if it comes from an international, Federal, or State research fund, a philanthropic fund, or an industry supporter. It counts as $20,000 in research funding. Go and do $20,000 worth of excellent research with it. Create new knowledge. Solve problems. There will be some outputs from that process. Let’s measure those outputs.
So, if you are a researcher, don’t be submissive. When it comes time to set your work-plan and your key performance indicators, make the case for KPIs that count. Then write the best applications that you can.
If you are a manager, measure the things that matter. Count the outputs, not the inputs. After all, you want your staff to be research active, not research submissive.
The Research Whisperer 
“After Tseen posted her plea for “no more half-baked applications“, one of our twitterati pointed out that sometimes applications are half-baked because submitting the application satisfies a key performance indicator in a work-plan.”
This reminds me of how one will often get a completely absurd job application to an advertised position because people must apply to x jobs per week to keep benefits (in some cases, they are probably applying to jobs as remote from their qualifications as possible to avoid getting them).
“Solve problems. There will be some outputs from that process. Let’s measure those outputs.”
Ultimately, they’re unmeasurable, however.
LikeLike
Similar analogy to bogus job applications, except in the case of researchers they have only one or two major funders to apply to and I doubt many would jeopardise future opportunities by sending in rubbish applications. Furthermore, most university research offices would probably vet their competitive applications before they are sent to the ARC or somewhere else.
I think the idea that many people are sending in really poor applications for competitive grants in order to meet some kind of institutional reward system is exaggerated or unfounded. I’m no expert on this, so I would be very happy to be proven wrong, but I’ve only read about isolated anecdotes so far.
LikeLike
Hi Jonathan,
You raise some genuine problems with the fallacy of treating research funding as a performance output for individual academics, but the problem with “Count the outputs, not the inputs.” is that outputs and inputs become very blurred at the institutional level.
For individual researchers it may be correct that “If you get $20,000 of research funding… It counts as $20,000 in research funding.” and is an input into the research process. But at an institutional level Australian Competitive Grants (ARC and NHMRC) count for much more than other types of funding. ACGs are inputs into the formulae that the Australian Government use to determine the amount of Research Block Grant universities receive, including 100% of the inputs for the Research Infrastructure Block Grant (worth $221 million in 2013). Other types of research funding (government, industry, philanthropic) are also inputs, but they have less effect on the total RBG. In other words, there are multiplier effects for the institution when academics win competitive grants, and these multipliers are greater for certain types of grants (e.g. ARC) than others. Therefore, it would make rational sense for a university manager to encourage ARC funding over other sources.
As you would probably expect, the Go8 dominate the Australian Competitive Grants (they get around 75% of it). Go8 univetrsites get around 75 times more ACG funding than those in the bottom 8. This in turn means that Go8 unis dominate the RBG for the sector and have a much higher proportion of research-only, fixed-term contract academics whose careers are dependent upon more grants coming in.
LikeLike
At a recent job interview (for a job I really wanted) I was asked about what research plans I had and whether I was thinking of applying for research grants. My answer, as part of a strategy of ‘authentic presentation’ went something like this: “I know that academic mangers and those obsessed with matrices really care about ‘grant capture’ but I don’t. Most of the research I care about doing does not require much financial support, indeed a good amount of social scientific research doesn’t. I care about research that matters to those researched and who it is supposed to benefit. Its not there to boost my career and to write articles hardly anybody reads”. Silence. I was happy to have been ‘authentic’ but really wanted the job and was now mentally preparing myself for polite rejection. You know what? I got the job!
LikeLike
I love this story! You’ve made my day. Congratulations on the new job!
LikeLike
There needs to be a balance – people who do win grants should be rewarded too, I think! Maybe put that particular “metric” in amongst other research-related ones – after all, there are some in some disciplines who CAN do research without income. That way, even if you don’t bring in any research income, you can still meet performance criteria based on other things, such as the outputs/outcomes of your research.
LikeLike
Hmmm… I don’t think this post was clear enough on where you stand on ‘grant proposals submitted’ and ‘funding won’ being KPIs. :p I’m going to hazard a guess and say you don’t like it very much. 😀
LikeLike
Let’s strip away the language of inputs/outputs (was the attack o the Westgate Shopping Mall an input or an output – get my drift?). The reality is that when ‘grant capture’ is fetishised by our institutions they are doing one of a number of things: they are responding to government pressure to diversify their income streams as core funding is cut; they are responding to the massive increase in student numbers and increasingly at post-graduate/doctoral level, so impelling them to devolve responsibility for grant capture on research students themselves. This may have something to do with inquiry, knowledge production, etc. but the KPIs often tell the lie to this when it is the sum of monies attracted that is asked for. The social, cultural, (real) economic, ethical value of the proposed research is of marginal value. This is Academic Capitalism. When academics fetishise grant capture they may be: focused on the need for tenure (at best), pure career progression (at worst), and often a mixture of the two – “I have a mortgage, I have kids to put through college”. Some play the game knowingly. They know that at the end of the day it is shit, but hell, you got yo play it, right? Some just have to go for grant capture because the capital infrastructure in areas such as biopharmacology, engineering, etc. is expensive. Of course, this is where the big bucks from the big corps appear attractive. And is ‘government’ funding cleaner – hell no. So, it is an ethical decision. When research students fetishise grant capture it may be driven by the need/desire to secure some security (and why not), first steps on tenure (and why not), blind careerism. But seldom are we seriously asked what the value of this game is beyond income stream. Each one of us has an ethical decision to make and there are no easy options, there is no pure land of knowledge. But we do or should consider what is authentic for us, each one of us. It really doesn’t matter if it is an input or output.
LikeLike
I agree that universities “are responding to government pressure to diversify their income streams as core funding is cut”, but it is not really diversification as ARC and NHMRC are government funded. The risk is that it just wastes a whole lot of time and resources by having academics applying for research council funding, rather than universities receiving a larger block grant and having the autonomy to use the money directly on research (or teaching). Universities are getting more and more funding from state govt and other project work, plus from industry, but the bulk is government block grant+ARC+NHMRC
LikeLike