Assignment of mental health diagnoses and severity : effectiveness and reliability of online standardized assessment instruments

Abstract

During the last decade, child psychiatry has been a focus of the Norwegian government's plan to improve mental health care. Psychiatric assessment of children and adolescents is more complex than that of adults, and rating scales and diagnostic instruments have become increasingly important tools in both research and clinical practice. This dissertation investigates standardized assessment instruments used in routine clinical practice to assign diagnoses and severity of mental health problems. Both the reliability and the validity of some instruments were examined, and a main focus was the clinical usefulness of these instruments and their potential for more effective use of limited clinical resources. In paper 1 the agreement between diagnoses and severity ratings assigned by clinical specialists who were trained Child and Adolescent Mental Health Service (CAMHS) researchers examined. Information on 100 youths was obtained from multiple informants through a webbased Development and Well-Being Assessment (DAWBA). Based on this information, four experienced clinicians independently diagnosed (according to the International Classification of Diseases Revision 10) and rated the severity of mental health problems according to the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) and the Children’s Global Assessment Scale (C-GAS). Agreement for diagnosis was - =0.69-0.82. Intra-class correlation for single measures was 0.78 for HoNOSCA and 0.74 for C-GAS, and 0.93 and 0.92, respectively for average measures. Information obtained with the online DAWBA may be a sound basis on which to establish reliable clinical diagnoses and severity ratings for common mental health disorders in a clinical setting. A clinical practice that includes systematic, multiple independent assignments of diagnosis and severity, is preferable due to the resulting improved reliability of the severity ratings. In paper 2 the application of specific scoring algorithms for the Strengths and Difficulties Questionnaire (SDQ) was examined. Could available online norms be useful in screening for mental health disorders among children and adolescents in the CAMHS North Study? A total of 286 outpatients, aged 5 to 18 years, were assigned diagnoses based on the DAWBA. The main diagnostic groups (emotional, hyperactivity, conduct and other disorders) were then compared to the SDQ scoring algorithms using two dichotomisation levels: ‘possible’ and ‘probable’ levels. Sensitivity for the diagnostic categories included was 0.47-0.85 (‘probable’ dichotomisation level) and 0.81-1.00 (‘possible’ dichotomisation level). Specificity was 0.52-0.87 (‘probable’ level) and 0.24-0.58 (‘possible’ level). The discriminative ability, as measured by ORD, was in the interval for potentially useful tests for hyperactivity disorders and conduct disorders when dichotomised on the ‘possible’ level, but outside the interval for potentially useful tests for all diagnostic categories when dichotomised on the most common used ‘probable’ level. In conclusion, the ability of the SDQ to detect mental health disorders among patients referred to CAMHS is not sufficient for clinical purposes. When used as a screening instrument to determine whether further evaluation is warranted in a clinical CAMHS sample, the SDQ seems best suited to identify children and adolescents who do not require further psychiatric evaluation, although this also is problematic from a clinical point of view. In paper 3 the agreement between diagnoses and severity assigned by clinical specialists trained as CAMHS researchers, based only on DAWBA information collected online, and the routine clinical assignments by CAMHS clinicians was examined. Routine clinical assignment of diagnoses was compared to online clinical assignment of diagnoses for 286 patients from the CAMHS North study. Chi square analysis, kappa statistics and multinomial logistic regression analyses were performed. Raw agreement for diagnostic categories varied between 74% and 90%, resulting in kappa values of 0.41-0.49. The final multinomial regression models were significant. Agreement on mental health diagnoses can be fair when online clinical assignments and routine clinical assignments of mental health diagnoses are compared. This may be sufficient to replace the routine clinical assignment of diagnoses with an online clinical assignment in order to save time and resources. We also examined factors contributing to agreement or disagreement on the diagnoses. Age, gender and number of informants significantly contributed to the explanation of agreement and disagreement for ‘emotional diagnosis’ and ‘hyperkinetic/conduct diagnosis’. However, the changes in odds were small in magnitude and the factors probably do not consistently contribute to the understanding of agreement or disagreement in any clinically meaningful way. Lastly, implications for further research on reliable and effective assessment methods are discussed.