A quick opening note on terminology: I use ‘assessor’ to refer to experts who read and review research grant applications, then provide comments into the final decision-making process. Terminology varies a bit between funders, and between countries. We’re not talking about journal reviewers, although some of the advice that follows may also apply there.
In 2012, with Adam Golberg of Cash for Questions (UK), we wrote ‘Rational responses to referees’, our advice on preparing your rejoinder or response to comments on your grant application. It was good advice then, and it’s good advice now.
As my applicants are busy writing their responses, this seemed like a good time to build on ‘Rational responses to referees’.
This post provides some advice on the specifics that I want to see in a strong response, and how you might deal with some tricky situations. When your response goes back to funding body, it will be considered along with hundreds or even thousands of other applications.
In such a situation, you want to make it as easy as possible for the reader (the funders) to understand your response.
I’ve seen a draft that was a wall of text, 5,000 characters long. There were no paragraphs breaks and no white space. It was exhausting to look at.
Be kind to your reader – cherish the white space. Put white space between paragraphs. Indent first lines. Use formatting (if the system allows it – the ARC doesn’t).
All the normal rules of civilised writing still apply, even if you have a lot to say and a severe limit on how many characters you can include. Invite your reader to engage with your text.
Make the connections
You are making a case that criticisms of your application are unwarranted. In doing so, you should make your response as specific as possible. Provide an index to your original application. Instead of saying, ‘As described in our application…’, you should say, ‘On page 37…’.
We made this point in Rational responses to referees, but I’m making it here again because it is so easy to do, yet so few first drafts do it.
You can do the same with your publications. Lots of grant applications ask for a numbered list of your publications. You can use that as shorthand in your response. Instead of saying, ‘We have used this technique before…’, you could say, ‘In pub. #25 & #32…’.
Both of these techniques make your response more concrete and help the reader to find the relevant information in your application. Win-win.
Be specific about specifics
If you have feedback from multiple assessors, it is tempting to lump them together. This is a great way to deal with a number of criticisms at once, especially as you probably don’t have a lot of space.
However, the danger in this is that you will lose sight of the specific criticisms leveled at your application. You don’t want to make general comments about general issues – you want to make specific comments about specific issues.
For example, if several different assessors have criticised your methods, it is tempting to write a general reply about your methods. Be careful! If assessor A says that your sample isn’t representative, and assessor B has criticised your data analysis, those are two different criticisms. Both of them are important, so respond to them separately, rather than lumping them together into one general response.
Not this project
One tricky criticism appears when the assessor suggests that you do something different from what you have suggested. This might appear as ‘This project would be stronger if it addressed…’, or ‘I was surprised that the project didn’t address…’
The simplest reply is ‘Not this project’. That is, the assessor has asked you to do something other than the project described. That might be a very interesting project, but it is not this project.
However, be careful that in glibly dismissing the criticism, you aren’t ignoring something deeper. Most valid criticisms point to a gap in your project. What gap is the ‘not this project’ criticism trying to point to? It may be nothing (or nothing important) but don’t dismiss it out of hand.
Be measured in your reply
People often indicate which review they are dealing with by quoting the assessor.
Assessor C said ‘This team needs a statistician…’.
Done well, this can help the reader connect the review to your response clearly and easily. However, I see two issues with this response.
One, it can be wordy. I’ve just read a first draft where my researcher had spent more characters quoting the assessors than they had in rebutting their criticisms. Don’t get the balance wrong. You want to spend most of your words rebutting criticism, not quoting it.
This leads me to my second issue with this structure: By quoting the assessors, I suspect that you are reinforcing their criticisms in the mind of the reader. A more general way to reply, which saves space and avoids this possibility, is to refer back to the criticism in general, once you have answered it. The example above would then become:
Prof Needs-Grant has expertise in statistics, and the Post Doc will have specialist skills in this area [Rev. C].
Some funding agencies provide you with a template for your response. Others don’t. Within the limits of the rules, you should structure your response in a sensible and useful way. I generally see three approaches.
1. The assessor by assessor structure
Some people answer each assessor separately. They deal with all the criticisms of the first assessor, then the second assessor, and so on.
The nice thing about this structure is that it is easy to write. You start with the first review and move through them until you are finished. Then you trim for length and you are done.
I see two issues with this. One is that you place very important criticisms with quite trivial ones. So, for example, if assessor C has issues with your theoretical assumptions and the quote for your airline tickets, they both appear together.
The other, potentially more serious one, is that the rejoinder is not for improving the scores of each assessor. It’s for improving your score against each criteria that the funding agency uses to rank grants. Which leads me to…
2. The criteria by criteria structure
Some people structure their response according to the criteria that the funding agency uses to rank the grants. To my mind, this makes more sense. It is often the way that the funding agency asks assessors to write their response. This approach makes it easier to bring similar criticisms (and your responses) together, so that they can be read at the same time.
If 60% of your final ranking will be for the originality of your work, you want to deal with all of the criticisms about originality to make that score as high as possible, and so on for each of the criteria used.
There are a couple of places where this approach may prove difficult. Assessors don’t always put their criticisms in the right boxes. That is, they might criticise your methodology under the ‘Team’ heading. Does that mean that they have marked you down for the ‘Team’ criteria, or the ‘Methodology’ criteria? Where do you put your reply? It depends on the context.
More importantly, focusing on a ‘score’ that you may never see (depending on the funding agency) moves you into the wrong mindset.
“It’s not a test, it’s a contest” – Mark Bisby, former VP Research, Canadian Institutes of Health Research.
You aren’t aiming for a pass mark. You are aiming to beat all the other applicants across the funding cut-off line. That means that you shouldn’t just be aiming to improve your score – you should be aiming to provide the best rejoinder possible to every important criticism.
Having said that, if this is your first rejoinder, I recommend the criteria by criteria structure.
The final form of rejoinder is…
3. The narrative structure
Done well, the most effective structure is a straight narrative that addresses all the major criticisms together. At its best, this is a retelling of the application, focusing on all the aspects that rebut the assessors’ criticisms. So, for example, imagine that you have received the criticisms listed here:
- Methodology: Assessor A says that your sample isn’t representative.
- Methodology: Assessor B has criticised your data analysis.
- Team: Assessor C said ‘This team needs a statistician’.
A narrative structure might include:
Prof Needs-Grant has expertise in statistics (p22), and the Post Doc will have specialist skills in this area (p56) [C]. Based on publication #35 & #38 (p24), Needs-Grant designed the sample to specifically address research question 3 (pp5, 12) [A]. The Post Doc will use regression (p11) to examine…
However, this is very, very difficult to do well. The main danger with this approach is that you end up with a hot mess of text that doesn’t actually answer the criticisms very well, and is very difficult to understand.
Whatever happens, always submit a rejoinder, and make it as strong as possible. At best, it will get your application across the line. At worst, even if the assessors have massacred you, it provides practice for next time. Good luck!
Thanks so much for writing this. I have a question about conflicting advice. For ARC rejoinders my uni research office said not to group by criteria as it then looks as though there are problems in all areas of your application. I had written in the narrative structure way, grouping together similar criticisms (perhaps not well though!) and was specifically told to use the assessor structure. I’m so confused!
LikeLiked by 1 person
If it helps, I’m in two minds, too.
There are two important points for me here. The first is – follow the advice of your local uni research office whisperer. They know your application, and they know your situation. They also have to actually upload the rejoinder to RMS, so you need them to do that. They’ll also have a better picture of your audience, which brings me to my second point.
Be aware of your audience. You aren’t writing your rejoinder in the abstract. You are writing it for people. I’m acutely aware that different countries and different disciplines have different attitudes to this. Social science and humanities tend to be discursive, compared to the brevity that often appears in science and engineering, for example. Your local research whisperer will have a better idea of who your audience is.
Having said that, there is a lot of folklore in this space. Research funding systems are black boxes, despite the best efforts of major funding bodies to demystify the process. Black boxes promote magical thinking, as they don’t allow much space for evidence-based decision making. That is, your university may have their way of doing something, but if you move across the road to another university, they may advise you differently.
As always, the most important thing is to actually read and understand what the assessors are saying, and provide the committee with the strongest response that you can.
LikeLiked by 1 person