Regresar

Applying multimodal learning analytics to naturalistic recordings of clinical simulations: Towards an accurate and scalable pipeline for automated feedback generation

Abstract:

Background: Educational audiovisual recordings, spanning domains from teacher training to clinical simulations, often remain underutilized due to the intensive human labor required for comprehensive analysis and timely feedback. Aims: This study addresses this challenge by developing and evaluating a multimodal learning analytics (MmLA) pipeline that integrates state-of-the-art visual, speech, and language models to automatically capture complex communication skills. Sample: 244 medical and social work students practicing breaking bad news in a team-based simulated clinical scenario. Methods: We developed and evaluated the MmLA pipeline, which integrates speech extraction, gaze tracking, and semantic analysis using a Large Language Model (LLM). We empirically evaluated the pipeline by measuring the accuracy of each feature-extraction stage, including diarized transcript generation, visual-attention pattern tracking, and LLM-based semantic evaluation of discourse, against human assessors, and by comparing the final automated predictions with human-generated scores. Results: The developed MmLA pipeline effectively extracts and fuses features from naturalistic videos of standardized patient simulations, approaching human-level accuracy, though diarization remains challenging with a moderate 29.9 % SER (lower is better). The predictive model using behavioral, verbal, and semantic features achieved 81.82 % accuracy in assessing interaction comfort, with eye contact emerging as the most influential predictor. Conclusions: By leveraging the most common multimodal data sources, such as video and audio, the study provides a scalable methodological blueprint that can be adapted to diverse educational settings to foster communication skills.