Real-Time Blood Glucose Prediction Reveals a Discrepancy Between Performance Metrics and Real-World Evaluations

Abstract

This study evaluates machine learning (ML) algorithms for predicting blood glucose (BG) levels, essential in real-time robotic diabetes control systems that integrate insulin pumps, continuous glucose monitors, and potentially additional sensors. Our objective is to use real-time deployment insights to guide future algorithm design. While existing research presents algorithms with strong performance metrics, these often rely on repetitive datasets, limiting real-world applicability. We compared a Ridge Regressor and a Long-Short-Term Memory deep neural network, focusing on their real-time deployment and evaluation. Initially, we validated our algorithms against a benchmark dataset to ensure consistency with published studies by calculating the root mean square error (RMSE). We then assessed the same models using data from a study participant within a smartphone application, evaluating real-time predictions through a user questionnaire. Our findings revealed a discrepancy between the performance metrics and real-world evaluation, suggesting these metrics might neglect complex transformations within hidden layers and fail to reflect critical situations. This study underscores the need for future research to refine evaluation methods that consider model behavior in critical scenarios and to develop models rooted in domain-specific knowledge, incorporating physiological constraints like insulin effects, to ensure alignment with physical reality.