Characterizing the File Drawer: Evidence from a Meta-Analysis of Parent-Interventions Around the World
中文摘要
我们对20多个国家的82项随机对照试验进行了元分析，以估计通过短信、电话和应用程序实施的低成本远程家长参与干预的效果。我们估计了一个联合似然函数，其中既纳入已成文的研究，也纳入通过试验注册库、资助方记录、研究实验室、证据信息中心及其他来源识别出的未成文研究。通过同时记录未成文研究的样本量，该模型估计了标准误的分布，识别了以显著性为条件的成文概率，并通过估计已成文与未成文研究的效应分布来刻画文件抽屉。经偏误校正后，考试成绩、课程成绩、出勤率和入学率的效应分别为0.05、0.07、0.05和0.03个标准差。在识别最为充分的考试成绩领域，统计不显著的结果仍有很高的成文率。我们还发现，规模较大的研究往往估计出较小的潜在效应，这可能表明真实效应与研究精度相关，违反了元分析的一项常见假设。在样本量较小的领域，我们的方法通过锚定绝对成文率来帮助识别选择概率。最后，我们估计了新增随机对照试验在为采纳决策提供信息方面的价值。由于家长干预具有较高的公共资金边际价值，任何单项研究的估计结果都不太可能阻止其被采纳。相反，当未来研究能够解释不同情境之间的异质性时，其价值最大。
Abstract
We conduct a meta-analysis of 82 randomized controlled trials across more than 20 countries to estimate the effects of low-cost, remote parental engagement interventions delivered through text messages, phone calls, and apps. We estimate a joint likelihood function that incorporates both written studies and unwritten studies identified through trial registries, funder records, research labs, evidence clearinghouses, and other sources. By also recording sample sizes for unwritten studies, the model estimates the distribution of standard errors, identifies write-up probabilities conditional on significance, and characterizes the file drawer by estimating effect distributions for written and unwritten studies. Bias-corrected effects are 0.05 SD for test scores, 0.07 SD for grades, 0.05 SD for attendance, and 0.03 SD for enrollment. In the best-identified domain, test scores, statistically insignificant results are still written up at high rates. We also find that larger studies tend to estimate smaller latent effects, which could indicate that true effects are correlated with study precision, violating a common meta-analysis assumption. In smaller-sample domains, our approach helps identify selection probabilities by anchoring the absolute write-up rates. Finally, we estimate the value of additional RCTs to inform adoption decisions. Any single study estimate is unlikely to dissuade adoption because parent interventions have high marginal value of public funds. Instead, future research is most valuable when it can explain heterogeneity across settings.