Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies
Methods: We showcase pitfalls of the traditional ML framework and explain how it can be improved with human-inthe-loop techniques. Specifcally, we applied active learning strategies to the automatic scoring of a story recall task and compared the results to a traditional approach.
Results: Human-in-the-loop methodologies supplied a greater understanding of where the model was least confdent or had knowledge gaps during training. As compared to the traditional framework, less than half of the training data were needed to reach a given accuracy.
Conclusions: Human-in-the-loop ML is an approach to data collection and model creation that harnesses active learning to select the most critical data needed to increase a model’s accuracy and generalizability more effciently than classic random sampling would otherwise allow. Such techniques may additionally operate as safeguards from spurious predictions and can aid in decreasing disparities that artifcial intelligence systems otherwise propagate.