Applicant Perception Survey - Report Summary

Are we providing the right guidelines and information to our applicants? Is our review process perceived as transparent and fair? To help us find out what our applicants think about our grant application process, we conducted a satisfaction survey last February. The survey was sent to the 255 applicants that submitted a proposal under our Open Window Round 2. This report outlines the main findings, what we do well and areas where we need to improve.

What We've learned

• Over 70% satisfied by the responses and timeliness of the response to their queries.

• Over 80%  found the information easily or very easily accessible.

• Most of them found the review process transparent, rigorous and fair.

• 66% found the feedback from the reviewers useful and 77% of unsuccessful applicants said they would use the comments to re-apply.

• A small number of respondents said they did not receive acknowledgement emails to their submission.

• In general, applicants would like more transparency in the scoring, more technical feedback on the proposed evaluation design and guidelines for the budget proposal.

• Applicants also had a perception that 3ie funded more experimental designs as opposed to quasi-experimental designs.

What We Do Well

• Overall high satisfaction of applicants. 50 % of the respondents said they were satisfied or extremely satisfied when asked to rate overall satisfaction with the grant process on a scale of 1 (extremely dissatisfied) to 5 (extremely satisfied). Over a third (37%) were neutral and only 8 out of 63 respondents replied ‘dissatisfied’ or ‘extremely dissatisfied’. There is a correlation between being successful in getting an award and satisfaction.
• Responsiveness and fairness in the review process. When respondents were asked to compare the 3ie grant process with other grant-making institutions, nearly half of the respondents (29) rated 3ie better than other grant makers, and less than 10% (6) rated 3ie as worse than other agencies.

Where We Need to Improve

While responses were overwhelmingly positive, we are taking into consideration the negative feedback received from some applicants:

• Timeliness of query responses. A few respondents did not get a reply to their queries sent either before, during or after the application process and a small number of cases did not get a timely reply (only 8 respondents).

• Perception that 3ie only funds Randomised Control Trials (RCT). 25 percent of respondents perceived the review process as not fair and more specific complaints included that the review process favored RCT designs.

• Dissatisfaction with the reviewers feedback reports.  Respondents asked to receive the individual reviewer reports and breakdown of scores by category. Some also felt that the feedback report did not provide clear justifications as to why the proposal was rejected.

What We're Doing About It

We have learned from the feedback received from our applicants and have taken a few steps to improve our grant management process:

• An online grant application system. Open Window Round 3 will be the first window to run under our new online grant management system, which will provide automatic tracking of proposal status and communications to applicants throughout the application process.  With the new online system, 3ie’s responses to queries will be more systematic and all queries will be answered within a 48 hour period. Applicants will be able to track the status of their own proposal automatically.
• Issue driven not method driven. 3ie has funded both experimental and quasi-experimental designs. Though RCTs represent the majority of the funded studies in the first two Open Window rounds, it is important to note that the majority of the proposals received also proposed an RCT design. However, we expect the share of RCTs to fall for the following two reasons:

Under the Open Window Round 3, we are improving our review process whereby our external technical reviewers will shortlist all the proposals of acceptable technical standard.  A first shortlist will then be screened by a panel of policy makers which will select proposals with high policy relevance and potential policy impact, reducing therefore the bias that technical experts may have to experimental designs in the final selection.

In addition, a new Policy Window has been launched to select priority interventions to be evaluated solely on grounds of policy relevance and potential policy impact.  Some of these may well be ex-post studies, necessitating quasi-experimental designs.

• Improved feedback reports from reviewers. Consolidated review reports including feedback from all 4 reviewers and average scoring for each category will be shared with each applicant who passed the first screening process. The quality of each report will also be improved and will clarify when the proposal was not selected in cases where technical reviews were positive but the ranking of the proposal did not include them in the final shortlist.
• Request for expression of interests. To further streamline the application and review process, 3ie will introduce a preliminary screening step where applicants will be requested to first submit an expression of interest (EOI) form before having to complete a detailed application once the EOI has passed a first review. The EOI review will ensure that the proposed study falls within 3ie’s mandate of rigorous impact evaluation of social and economic development interventions in low and middle income countries.

We would like to thank the applicants who completed the survey. And we would further like to invite applicants who have ideas about how we can improve —whether they have taken a survey already or not— to send us their thoughts at: 3ie@3ieimpact.org