Thursday, June 27, 2013

Ignoring the demoninator

Grant Reviewer A: "In the past 4 years investigator X has been moderately productive, having  published 5 research papers."
Grant Reviewer B: "Applicant Z has been highly productive, having published 16 papers over the past 4 years."

Which investigator would likely make better use of additional funds? Who should get the better score?  Answer: it makes no sense to try to answer these questions with this data alone (we'll ignore the issue of whether number of papers is a valid measurement of output: it's how the game is all-too often played). I'd be embarrassed if someone from the business world--where there is intense focus on the efficient deployment of capital--were to sit in on one of the study sections I've attended and critically analyze how bioscientists evaluate productivity. In general, we comment on the numerator and pay scant attention to the denominator. Would a manager allocate additional funds for production at a plant that turns out 10,000 light bulbs per day at a cost of $10,000 or a plant that turns out 2,000 at a cost of $600?  Would the the decision be made after looking at the numerators (10,000 and 2,000) alone?

In my experience (service on several different NIH study sections and good knowledge of how reviews for academic promotion proceed at several different institutions), little or no attention is ever given to the denominator: to output per dollar. Is someone who publishes 15 papers using $4,000,000 of public funds and, say, 10 postdocs and grad students (the numbers given here are just for illustration and easy comparison) more meritorious than an investigator who publishes 5 papers using $750,000 and 2 postdocs/grad students? Who is making more efficient use of public or charitable funds? Who is getting the trainees in their labs more first author papers? (Remember, we're ignoring the issue of whether number of papers is a fair measurement of  productivity.)

Renewal applications list all the papers attributable to the grant proposed for renewal, but this listing, in its current incarnation, is of little use. There is no way of knowing how much money from any particular grant was spent on the science in any particular paper. Investigators attribute individual papers to multiple grants. An investigator might make a small contribution to a collaborator's project and associate the resulting  paper with one or all of her grants. Money and lab supplies are fungible. Supplies, personnel and equipment in any one lab get intermixed: there is not a wall separating each project. Unless more detailed accounting is required--and there wouldn't be very many investigators happy about the huge increase in record keeping that would be required--perhaps the best way of measuring how efficiently a lab uses funds is to look carefully at total money in versus total research out. Attention should also be paid to the fact that there seems to be an inflection point (~$750,000 per annum in total funding) beyond which total productivity per dollar begins to decline (the link leads to a good example of an analysis of productivity where attention is paid to funds expended).

Molecular biologists engage in highly complex data analysis in their laboratories. It is interesting that when when we travel to grant review sessions or sit on academic committees, we evaluate productivity in such a haphazard and inconsistent manner. Grant dollars are in short supply these days. It would make sense to evaluate how efficiently they are used in a more sophisticated, business-like manner.

 (The need to distribute funds wisely also applies to grants to New Investigators: see prior post).

No comments:

Post a Comment