NLPOps: A Case Study on Automation and Integration of Norwegian Medical Entity Recognition Model in Clinical Environment.

Abstract

Integrating Machine Learning (ML) models into healthcare system, especially by Natural Language Processing (NLP), holds tremendous potential for improving hospital workflows and patient outcomes. This thesis presents the challenges and deployment of an NLP model, Automated Medical Entity Recognition (AMER), designed to extract medical entities from clinical text, within in cloud-based Machine Learning and Operations (MLOps) workflow. The aim of this thesis was to create a scalable, secure and efficient deployment pipeline that can meet the strict data privacy and compliance requirements of clinical environments. The AMER model was already developed as local system, we reconfigured it for deployment on Microsoft Azure by utilizing an automated MLOps pipeline. The pipeline automates crucial stages like disaster management, model testing and deployment, minimizing the deployment from 2-3 hours to 20-30 minutes. Cost analysis showed that an initial configurations cost of $570 and on going monthly operational expenses ranged between $262 during low workload conditions and $342 during high workload periods. The AMER model attained an accuracy of 91.8% locally and 90.4% in the cloud with precision, recall and F1 score above 88% in both environments. Furthermore, optimization techniques were also performed that reduced latency from 450 milliseconds to 180 milliseconds, meeting the target limit of under 200 milliseconds for real-time hospital use. Resource utilization and cost efficiency of AMER were evaluated across different workloads to test dynamic scaling of the system. At low demand (10 request per second), the Central Processing Unit (CPU) usage was 15% with the expense of $1.20 per hour, at medium demand (50 request per second) the usage of CPU increased to 35% costing $3.80 per hour and at the high demand (100 request per second) CPU was utilized at 65% with the cost up to $6.40 per hour. This adaptive scaling provided by the cloud reduces the expenses during the low workload periods and scales resources for high workloads conditions. While the deployment using cloud provides operational advantages, this thesis discuss the challenges of integrating MLOps in clinical environments where on-premises secure data centers are often prioritized. A proposed path for further development includes using hybrid and private cloud system that allow healthcare providers to utilize MLOps based automation while maintaining compliance with local regulatory standards. This thesis contributes to advancements in MLOps in healthcare by showing how cloud-based automation and deployment can improve the efficiency, scalability and reliability of ML models in hospital applications. However, further investigations and enhancements are needed for compliant integrations of ML-driven technologies in healthcare.