Applicant Perception Survey - Open Window Round 3

Are we providing the right guidelines and information to our applicants? Is our review process perceived as transparent and fair?

To answer these questions 3ie sent a “client satisfaction survey” to all applicants to Open Window 3.

What we've learned

Main findings:

• Most applicants received information about the call for proposal through word of mouth, followed by 3ie news update, email and website.

• Over 88% of applicants thought that having a two stage application process with an initial screening was a good idea.

• 95% were satisfied by their experience in using 3ie’s new online grant management system.

• 92% found the information easily or very easily accessible.

• Over 90% were satisfied by the responses to their queries.

• Most of them found the review process transparent, rigorous and fair.

• 76% found the feedback from the reviewers useful and 56% of unsuccessful applicants said they would use the comments to re-apply.
  
  
• But close to half (47%) said that the reasons for rejection were not clear.

What We Do Well

Overall high satisfaction of applicants. Just under 50 % of the respondents said they were satisfied or extremely satisfied with the grant application process overall, and most the rest (38%) were neutral. That does leave 16% who were dissatisfied or extremely dissatisfied, these all being unsuccessful applicants. As good as, or better than, other grant making agencies: 37% of respondents rated 3ie better than other grant makers, and 48% rated 3ie similar to other agencies, leaving 14% who thought we are worse than others.

• 3ie provides useful guidance. The vast majority, typically over 90 percent, rated the various guideline documents provided for potential applicants as useful or very useful.
  
  
• The 3ie review process is clear and fair but transparency could be improved: Around three quarters of respondents agreed that our review criteria are clear and the process fair, but only 60 percent agreed that it is transparent. The perceived lack of transparency is linked to not being clear on reasons for rejection, which we pick up on again below.

Where We Need to Improve

Dissatisfaction with the feedback reports from reviewers. Some respondents felt that reviewer reports were of poor quality and sometimes contradicted one another. Management response: in response to feedback from the previous survey we do provide all reviews to applicants. We will make it clear to applicants that each full application receives four separate reviews, and the aggregate score is based on the top three scores to avoid adverse influence from a single “rogue review”. People don’t understand why they didn’t get the money. Close to half of rejected applicants did not feel it was clear why they did not get the money.

Management response: we will make it clear that we are left with fundable proposals we cannot fund. That is, we receive more proposals for studies that we believe worth doing than we can have money to fund. So we have to reject perfectly good proposals. However, under both OW2 and OW3 3ie did manage to raise additional resources for additional grants under these windows.

• There was a perception from some respondents that 3ie only funds US based institutions. Rejected applicants complained that only US institutions get grants.
  
  Management response: The review criteria for OW3 put more weight on developing county involvement. As a result, 82% of grants made under OW3 include study teams with at least one PI based in a developing country, and increase from 60% under OW2.

• Some respondents complained that they were rejected as they were in the wrong sector and would not have bothered applying had they known this.
  
  Management response: the Open Window has no sectoral priorities. We will make this point clear to reviewers. However, applications do have to address questions of policy relevance.
  

Steps taken in response to the survey

We have learned from the feedback received from our applicants and have taken a few steps to improve our grant management process:

• Enhancement in online grant application system: Technical problems with the online system mainly happened closing before deadline. The 3ie programme team is making changes to the platform to avoid system overload at the time of deadline.
  
  
• Improved feedback reports from reviewers. The quality of each report will be improved. After the awards are announced 3ie will post the names of all reviewers and the distribution of scores on the OW page. It will be made clear where the proposal was technically fundable and the feedback reviews positive but the proposal did not make the final selection given the limited funding available.
  

We would like to thank the applicants who completed the survey. And we would further like to invite applicants who have ideas about how we can improve —whether they have taken a survey already or not— to send us their thoughts at 3ie@3ieimpact.org