Management Science · 2025 · Maria De-Arteaga、Vincent Jeanselme、Artur Dubrawski、Alexandra Chouldechova
中文摘要
机器学习 (ML) 日益被用于支持高风险决策。然而,其中常常存在构念差距 (construct gap):即决策任务所关注的构念与用作标签来训练 ML 模型的代理变量所捕捉内容之间的差距。由此,ML 模型可能无法捕捉决策标准的重要维度,从而削弱其在决策支持中的效用。因此,在设计用于决策支持的 ML 系统时,一个关键步骤是从可用的代理变量中选择目标标签。在本研究中,我们探索将历史专家决策作为一种丰富却不完美的信息来源,将其与观察到的结果相结合以缩小构念差距。我们认为,当专家彼此表现出一致性时,管理者与系统设计者可能倾向于从专家处学习;而在其他情形下,则应从观察到的结果中学习。我们开发了一套方法,利用组织信息系统中通常可获得的信息来实现这一目标。该方法包含两个核心步骤:第一,我们提出一种基于影响函数的方法,在数据中每个案例仅由单一专家评估时间接估计专家一致性;第二,我们引入一种标签融合 (label amalgamation) 方法,使 ML 模型能够同时从专家决策和观察结果中学习。我们的实证评估——基于临床情境下的模拟以及儿童福利领域的真实数据——表明,所提出的方法能够成功缩小构念差距,其预测性能优于仅从观察结果或仅从专家决策中学习。
Abstract
Machine learning (ML) is increasingly being used to support high-stakes decisions. However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models. As a result, ML models may fail to capture important dimensions of decision criteria, hampering their utility for decision support. Thus, an essential step in the design of ML systems for decision support is selecting a target label among available proxies. In this work, we explore the use of historical expert decisions as a rich—yet also imperfect—source of information that can be combined with observed outcomes to narrow the construct gap. We argue that managers and system designers may be interested in learning from experts in instances where they exhibit consistency with each other while learning from observed outcomes otherwise. We develop a methodology to enable this goal using information that is commonly available in organizational information systems. This involves two core steps. First, we propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data are assessed by a single expert. Second, we introduce a label amalgamation approach that allows ML models to simultaneously learn from expert decisions and observed outcomes. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap, yielding better predictive performance than learning from either observed outcomes or expert decisions alone.