NSF Merit Review Process
It has been a long time. I moved to UCF as of August 2016 and have been re-establishing my research group. We are moving on with more research.
Today's blog topic is somewhat ad hoc. I just submitted my feedback for the National Science Foundation (NSF)'s Merit Review process per NSF's request, and thought I should write to my blog about it. First, let me say that submitting proposals to NSF and participating in the merit review process as a panelist is one of the most enjoyable and rewarding activities you can do as an academic. It is, by far, the most fair and the most thorough system for funding fundamental research. Yet, I have to admit I have not seen the practices of all agencies, e.g., NIH. So, my comments below are meant for making minor improvements to an already excellent system.
If you are not familiar with the NSF proposal merit review process, there are several online resources or experiences you can read to have a good understanding of it, e.g.:
https://www.nsf.gov/bfa/dias/policy/outreach/grantsconf/meritreview_feb16.pdf
http://www.sciencemag.org/careers/2003/04/nsf-grant-reviewer-tells-all
http://www.astrobetter.com/blog/2013/07/08/the-inside-scoop-on-nsf-review-panels
https://web.engr.oregonstate.edu/~grimmc/NSF/TheNSFReviewingProcess.html
Here are some ad hoc thoughts and suggestions on improving the NSF proposal review process:
15-Page Project Description: 15 page limit is too long. As proposers, we naturally tend to fill up all the 15-page space to have a better chance in winning against competitor proposals. But, 15-pages is too long. We end up spending too much time writing proposals, and spend increasingly less time doing research. As it is in the panels, most reviewers don't read the details, which is unfortunate but the reality. So, the proposers end up writing 15 pages of a document which is not fully read by the reviewers. I think 6 pages for the technical content and 2 pages for the broader impact (including the educational component as well as prior NSF support) would be very much enough to lay down a high risk high reward research agenda. The reviewers will then be more inclined to reading everything in the proposal.
Formality (or Balloon) Documents or Content: We have totally unnecessary formality documents or content in the full proposal submissions. For instance, Data Management Plan and Postdoc Mentoring Plan rarely make any difference in the decision making process. If these are needed for regulatory purposes, they can be asked for when a particular proposal is picked for funding -- this would reduce a lot of paperwork considering the fact that 80+% of the proposals don't get funded anyway.
Obsolete Panel Review System at Fastlane: I think an upgrade (or overhaul) of the Panel Review System is way overdue. We, as panelists, are still wasting our time on fixing an apostrophe when copying our reviews into the Panel Review System. Instead of using modern features like tracing of changes (as in MS Word) or suggesting of changes (as in Google doc), we are still fixing the writing errors via sending a comment to the scribe or simply shouting across the table "Hey John, second paragraph third sentence in the panel summary: Please fix the typo 'covfefe' !" Hopefully the scribe is not going to get offended with all this in the mean time. If the typo is in a panelist's review, other panelists just don't bother to avoid any possibility of offending someone who may later be reviewing their proposals! Maybe too fancy, but I wish there was an anonymous commenting/suggesting feature that other panelists can use to give feedback to other panelists. Yes, anonymity is good -- even within a panel. :)
Today's blog topic is somewhat ad hoc. I just submitted my feedback for the National Science Foundation (NSF)'s Merit Review process per NSF's request, and thought I should write to my blog about it. First, let me say that submitting proposals to NSF and participating in the merit review process as a panelist is one of the most enjoyable and rewarding activities you can do as an academic. It is, by far, the most fair and the most thorough system for funding fundamental research. Yet, I have to admit I have not seen the practices of all agencies, e.g., NIH. So, my comments below are meant for making minor improvements to an already excellent system.
If you are not familiar with the NSF proposal merit review process, there are several online resources or experiences you can read to have a good understanding of it, e.g.:
https://www.nsf.gov/bfa/dias/policy/outreach/grantsconf/meritreview_feb16.pdf
http://www.sciencemag.org/careers/2003/04/nsf-grant-reviewer-tells-all
http://www.astrobetter.com/blog/2013/07/08/the-inside-scoop-on-nsf-review-panels
https://web.engr.oregonstate.edu/~grimmc/NSF/TheNSFReviewingProcess.html
15-Page Project Description: 15 page limit is too long. As proposers, we naturally tend to fill up all the 15-page space to have a better chance in winning against competitor proposals. But, 15-pages is too long. We end up spending too much time writing proposals, and spend increasingly less time doing research. As it is in the panels, most reviewers don't read the details, which is unfortunate but the reality. So, the proposers end up writing 15 pages of a document which is not fully read by the reviewers. I think 6 pages for the technical content and 2 pages for the broader impact (including the educational component as well as prior NSF support) would be very much enough to lay down a high risk high reward research agenda. The reviewers will then be more inclined to reading everything in the proposal.
Formality (or Balloon) Documents or Content: We have totally unnecessary formality documents or content in the full proposal submissions. For instance, Data Management Plan and Postdoc Mentoring Plan rarely make any difference in the decision making process. If these are needed for regulatory purposes, they can be asked for when a particular proposal is picked for funding -- this would reduce a lot of paperwork considering the fact that 80+% of the proposals don't get funded anyway.
Obsolete Panel Review System at Fastlane: I think an upgrade (or overhaul) of the Panel Review System is way overdue. We, as panelists, are still wasting our time on fixing an apostrophe when copying our reviews into the Panel Review System. Instead of using modern features like tracing of changes (as in MS Word) or suggesting of changes (as in Google doc), we are still fixing the writing errors via sending a comment to the scribe or simply shouting across the table "Hey John, second paragraph third sentence in the panel summary: Please fix the typo 'covfefe' !" Hopefully the scribe is not going to get offended with all this in the mean time. If the typo is in a panelist's review, other panelists just don't bother to avoid any possibility of offending someone who may later be reviewing their proposals! Maybe too fancy, but I wish there was an anonymous commenting/suggesting feature that other panelists can use to give feedback to other panelists. Yes, anonymity is good -- even within a panel. :)
Panelist Honorarium Amount: Currently, if a panelist is attending in person to a 2-day panel, (s)he gets about $1,200 for covering expenses. Considering all the time needed for reading the proposals and writing the reviews, and the travel hassle one has to go through to attend the panel, I think this is too little. This may be one reason why panelists have increasingly been less careful in writing a quality review. If this amount is not increased to something significant, I think it will be hard to find panelists.
1099 Form for the Panelist Honorarium: After a panel, panelists receive a 1099 form for tax reporting of the honorarium (or whatever it is supposed to be called). The problem is that after entering this form in the tax return, it increases the total taxable income and the panelist ends up paying "self employment" tax. I am not sure, but, depending on your tax bracket, attending a panel may cause a loss rather than additional income! In one case I remember paying "alternative minimum tax" due to the self employment income from the panels. Accounting-wise this may be a stupid question to ask, but can't NSF make the honorarium non-taxable? Let alone the additional tax you have to pay due to attending a panel, having not to deal with entering the 1099 form when filing taxes would be a relief.
1099 Form for the Panelist Honorarium: After a panel, panelists receive a 1099 form for tax reporting of the honorarium (or whatever it is supposed to be called). The problem is that after entering this form in the tax return, it increases the total taxable income and the panelist ends up paying "self employment" tax. I am not sure, but, depending on your tax bracket, attending a panel may cause a loss rather than additional income! In one case I remember paying "alternative minimum tax" due to the self employment income from the panels. Accounting-wise this may be a stupid question to ask, but can't NSF make the honorarium non-taxable? Let alone the additional tax you have to pay due to attending a panel, having not to deal with entering the 1099 form when filing taxes would be a relief.
In my last panel, I learned that I could keep track of all expenses and show them in tax return to avoid paying tax on the honorarium. Definitely more work but avoids paying out of pocket tax on panel expenses.
ReplyDeleteYes, I have been doing that too. But, it still causes more tax to be paid because the expenses you can claim are typically less than the honorarium.
Delete