The development of a tool to assess medical students’ non-technical skills – The Norwegian medical students’ non-technical skills (NorMS-NTS)

Abstract Purpose New physicians need to master non-technical skills (NTS), as high levels of NTS have been shown to increase patient safety. It has also been shown that NTS can be improved through training. This study aimed to establish the necessary NTS for Norwegian medical students to create a tool for formative and summative assessments. Methods Focus group interviews were conducted with colleagues and patients of newly graduated physicians. Interviews were then analyzed using card sort methods, and the identified NTS were used to establish a framework. Focus groups commented on a prototype of an NTS assessment tool. Finally, we conducted a search of existing tools and literature. The final tool was developed based on the combined inputs. Results We created Norwegian medical students’ non-technical skills (NorMS-NTS) assessment tool containing four main categories; together comprising 13 elements and a rating scale for the NTS of the person observed. Conclusions The NorMS-NTS represents a purpose-made tool for assessing newly graduated physicians’ NTS. It is similar to existing assessment tools but based on domain-specific user perspectives obtained through focus group interviews and feedback, integrated with results from a literature search, and with consideration of existing NTS tools.


Introduction
It is widely accepted that physicians need to master not only technical skills but also several non-technical skills (NTS) (Baldwin et al. 1999;Fletcher et al. 2002;Odell 2011;Donaldson et al. 2021). Flin and O'Connor (2008) defined NTS as 'The cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance.' That is, skills in communication, teamwork, situation awareness, and decision-making (Donaldson et al. 2021). The terminology to describes these skills is a matter of some debate (Murphy et al. 2019). We chose to use the term 'non-technical skills' in this work since most existing tools use this nomenclature.
Evidence suggests that poor NTS performance puts patients at risk (Manser 2009;Donaldson et al. 2021). Studies have found that in 70% of in-hospital adverse events, inadequate performance of NTS is a contributing factor (Fletcher et al. 2002). It has also been shown that it is possible to improve health professionals' levels of NTS by training (Yeung et al. 2015;Hagemann et al. 2017;Somasundram et al. 2018;Nicolaides et al. 2021). Evidence has shown that if health professionals improve these skills, this has positive implications for patient mortality and morbidity (Neily et al. 2010;Donaldson et al. 2021). High-level NTS are considered critical for patient safety (Flin and O'Connor 2008;Donaldson et al. 2021).
In Norway, medical school is a homogenous, general medical education with the same learning objectives applied throughout the country (Regulations on National Guidelines for Medical Education 2022). The education qualifies students to work as a physician in general, e.g. in hospitals of different sizes, general practice, or the emergency room (ER). Medical specialization begins when a graduated physician starts a common 18-month internship (Regulations on Specialist Education and Specialist Approval for Doctors and Dentists 2017). New graduates are, therefore, expected to master the role of independent caregivers and team leaders in emergencies immediately after the final licensing exam.

Practice points
NorMS-NTS is a tool for assessment of medical students' NTS. High proficiencies in NTS increase patient safety. We identified the necessary non-technical skills for newly graduated physicians. We developed an assessment tool to assess NTS during medical school. This implies that students should learn NTS during medical school, and that the goal is to reach the necessary level of these skills by graduation. Most medical school curricula have been found to contain limited training for this role (Stergiopoulos et al. 2010). Furthermore, newly graduated physicians have been shown to perceive a self-reported lack of NTS skills (Stergiopoulos et al. 2010). The need for training in NTS in pre-graduate education was, among others, required by the British Parliament in 2009 (Flin and Patey 2009;Health Committee HoC UK 2009). Several initiatives have been launched to address this need (Frank and Danoff 2007;Gregory et al. 2015).
Over the last two decades, there has been an ongoing change in medical school curricula from time-based to competency-based teaching . New curricula, focusing on medical students' NTS performance, require assessment tools to ascertain that students achieve these skills. These tools can be used during the learning process to provide formative feedback or in a summative format to check learners' abilities. Currently, no instruments exist for evaluating medical students' NTS in Norway. We found one tool for assessing medical students' NTS in simulated acute events, developed in Scotland: the Medi-StuNTS (Phillips et al. 2021). Several tools for other health professions also exist (Fletcher et al. 2004;Yule et al. 2008). Studies have found that different cultures and health professions require different NTS; hence, different NTS assessment tools (Lene Spanager et al. 2012;Jepsen et al. 2015). These studies exposed differences in tasks, responsibilities, and culture between Denmark and Scotland and the need to customize NTS tools to adapt the tool to the Danish setting. Based on Hofstede's work on cultural dimensions (Hofstede 1984), we theorized that there would be differences between Scotland and Norway as well. Hofstede uses six dimensions to describe differences between countries. Norway and Scotland especially differ in the 'masculinity' dimension, indicating that, in general Norway would have a more collaborative approach than would Scotland. This implies that a customization is needed. We wanted a tool that was easy to use, that fit the Norwegian setting, and that was usable in various situations. Existing tools are commonly adapted by customizing them in different ways (Lene Spanager et al. 2012;Jepsen et al. 2015;Yule et al. 2021). We chose a broader approach to prevent overlooking important NTS that may be particular to this setting, population, and cultural context.
We aimed to identify the necessary NTS in newly graduated Norwegian physicians to create a framework of learning objectives and develop an assessment tool to see if students achieve this during medical school.

Overview
In this study, we aimed to find the necessary NTS as informed by collaborative health professionals and patient representatives. The first part of the study was exploratory and qualitative. We first conducted focus group interviews (Stalmeijer et al. 2014) to explore which NTS general practitioners (GPs), paramedics, ER nurses, and patient representatives see as necessary for newly graduated physicians. The interviews were transcribed and analyzed using NVivo (alfasoft.com). The NTS found in the analysis formed the framework of learning objectives.
After completing the above, we began the second part of this studythe development of the Norwegian medical students' non-technical skills (NorMS-NTS) assessment tool. We performed a modified version of an open-card sort analysis (Fincher and Tenenberg 2005) of interview transcriptions. The first prototype of the assessment tool was returned to participants in the focus groups. Participants were then asked to give feedback on whether the tool reflected their opinions and input. After collecting feedback from informants, a literature search was conducted to explore existing tools and literature.
The final proposed assessment tool was therefore based on inputs from informants, existing tools, and reviewed literature (see Figure 1).

Ethics
Norwegian law exempts educational studies from medical ethical approval if studies do not involve patients. However, we sent an application including the research protocol to the Regional Committee of North Norway for Medical and Health Research. They waived the need for a formal review of the study (ref: 2016/1539/ REK nord).
The Data Protection Official for Research for Finnmarkssykehuset, the Norwegian Centre for Research Data approved the interview guide for the focus group interviews (NSD ref: 57474/2017). NSD provides data protection services to all Norwegian universities. Their purpose is 'to ensure legal access to necessary personal data for research' (Norwegian Centre for Research Data 2022).
Written informed consent was obtained from all participants after obtaining oral and written information on the purpose and objectives of the study Setting After graduating from medical school, all Norwegian physicians start their medical or surgical specialization with a common 18-month internship. The first 12 months of internship are spent at a hospital and the following 6 months as a GP intern in general practice. In Norway, emergency medicine is not recognized as a medical specialty. Most patients with an emergency condition are initially diagnosed by GPs. Out-of-hours emergency primary health care is run by GPs and GP interns. They see patients at emergency care centers and are called out for acute events. Physicians most often collaborate with nurses at emergency care centers. When working during acute events out-of-office, physicians usually team up with paramedics.

Participants
The participants for this study were recruited using purposive sampling. We chose the participants based on our objective, to gain a broad perspective on necessary NTS from colleagues and patients. All participants were either working with or being treated by new physicians. Seven focus groups, comprising 36 professionals and 15 patient representatives, were assembled. On average, there were seven participants in each group (range 4-8). There were men and women of different ages and experiences in all groups.
The groups consisted of three groups of experienced GPs, one group of paramedics, one group of ER nurses, and two groups of patient representatives from the user committee at the Finnmark Health Trust, Hammerfest Hospital in Hammerfest, Norway. All groups were connected to this local hospital in Hammerfest. The ER nurses worked in the hospital's ER, and the paramedics picked up and delivered patients at the hospital. The three groups of doctors were from three different locations that were all within the referral network for this hospital and located within a distance of 180 km. The population in these locations varies from approximately 3000 to 20,000 people.
The user committee consists of patients and patient representatives. Men and women of different ages and experiences were considered to represent all patients and relatives in the area of health trust (Northern Norway Regional Health Authority 2022).

Interviews
The main purpose of the interviews was to explore the groups' opinions on which NTS newly graduated physicians should hold. Participants were informed about the purpose, written and orally, at the beginning of each interview. The first author conducted all the interviews mono-professionally and in a semi-structured format (Malterud 2012). The interview guide was designed as a topic guide. Each interview started with a vignette describing a scenario where an acute sick patient arrives in the ER. A team consisting of nurses, paramedics, and a newly graduated physician is taking care of the patient. Followed by six questions. The participants were encouraged to speak freely. The questions were used when group discussions were paused. The interviewer asked further questions when it was deemed that specific opinions needed to be elaborated. Each interview NorMS-NTS prototype -4 Categories (CommunicaƟon, SituaƟon Awareness, CooperaƟon Skills, Decision Making) -13 elements Refinement of the NorMS-NTS prototype -The tool was compared to the literature and exisƟng tools.
-Based on the feedback from the focus group parƟcipants, face validity was achieved.
Literature search -ExisƟng tools and literature were idenƟfied and reviewed -The research team found ANTS, ANTSdk, NOTSS and Medi-StuNTS most relevant Feedback from focus group parƟcipants -The parƟcipants in the focus groups were asked to provide feedback about the NorMS-NTS prototype.

NorMS-NTS
-4 categories (CommunicaƟon, SituaƟon Awareness, Teamwork, Decision Making) -13 elements IdenƟfying desired and necessary non-technical skills -7 focus group interviews with paƟents and professionals (n=36) -Interviews transcribed and analyzed using card sort analysis in NVivo by two of the researchers independently -The two researchers discussed their sorƟng unƟl consensus was reached was recorded and transcribed verbatim. Interviews were conducted until saturation of the information was reached.

Interview analysis
The transcribed interviews were then transferred to NVivo. Two of the researchers, KP and TW, used the program independently to analyze the interviews. First, we marked all units of meaning related to medical students' NTS. We then conducted a modified version of an open card sorting analysis in NVivo. All units of meaning were sorted into groups using an iterative process where each group consisted of units with similar meanings. Thereafter, groups were combined with other groups based on the same theme. After the first separate analysis, the two researchers met and discussed their sorting results until a consensus was reached. Sorted groups were defined as elements, and themes were defined as categories. This process resulted in a prototype of the assessment tool. The resulting prototype tool was sent to each of the interviewed groups via email. Groups were then asked to provide feedback on whether the tool reflected their opinions and input.

Literature and existing tools
We chose to search for existing literature and tools after the analysis of the interviews. The searches were conducted in August 2021, and we had two different search strategies.
In the first search, we searched the following electronic databases: Medline (Ovid) and Embase (Ovid). The search combined the keywords 'medical student Ã ' and 'non-technical skill Ã .' We read the abstracts of the articles, and the most relevant articles were read in the full version.
The second search was a broad search for existing assessment tools for medical students' and physicians' NTS was also conducted. Medline (Ovid), Embase (Ovid), and Web of Science (Clarivate) databases were searched. All relevant tools were collected, and the research team discussed their relevance. The most relevant tools were used for the comparison with the prototype.

Final revision
Feedback from focus groups was used to clarify and refine the tool. The tool was also compared with findings from the review of literature and existing tools to ensure that all relevant learning objectives were included.

Interviews and analysis
The first author conducted all interviews. After the first five interviews were transcribed, it was felt that data saturation was reached. Two further interviews were already planned, and they confirmed this finding of saturation. In total, seven focus-group interviews were conducted.  Table 1). We considered these 56 examples as the framework of what NTS newly graduated physicians should hold. That is, what we should be able to assess with the NorMS-NTS. The necessary NTS was further analyzed, and it emerged into the first prototype of NorMS-NTS with 13 elements and four main categories. The four main categories were communication, situational awareness, cooperation skills, and decision-making.
Four out of seven focus groups replied to the email requesting feedback on the prototype. Their feedback indicated that the assessment tool reflected their opinions overall and was understandable and coherent. They pointed out two unclear elements, and minor changes were made to the nomenclature.

Literature and existing tools
The literature searches identified 131 articles in Medline and 69 articles in Embase. Fifteen articles were read in the full version. The research team chose to use ANTS (Fletcher et al. 2004), ANTSdk (Jepsen et al. 2015), and NOTSS (Yule et al. 2006) because they are widely used and based on comprehensive scientific work (Fletcher et al. 2004;Yule et al. 2008;Lene Spanager et al. 2012). Additionally, Medi-StuNTS (Hamilton et al. 2019) was chosen as it is the only complete tool we found for medical students' NTS.
We compared NorMS-NTS to the existing tools ANTS, ANTSdk, NOTSS, and Medi-StuNTS (see Table 2) (Fletcher et al. 2004;Yule et al. 2008;Jepsen et al. 2015;Hamilton et al. 2019), three of the categories are quite similar in all five tools. That is 'Situation awareness,' 'Decision making,' and 'Teamwork.' The fourth category is 'Leadership' in two of the tools (ANTSdk and NOTSS), 'Task management' in one (ANTS), and the fourth category in Medi-StuNTS is 'Self-awareness.' In NorMS-NTS, the fourth category is 'Communication.' The Medi-StuNTS also has an extra category that corresponds to the other four. This skill category is called 'Escalating care' and comprises the importance and complexity of escalating care. The NorMS-NTS and the Medi-StuNTS have both 'uncertainty' as part of their elements, and none of the others have that.
The scales used for the various tools identified are somewhat different (see Table 2). The ANTSdk uses a fivepoint Likert scale to rate elements and categories. The authors of ANTSdk (Jepsen et al. 2015) state that this was chosen to prevent the ceiling effect and allow raters to differentiate behaviors. In addition, this tool uses a sevenpoint Likert scale to rate global overall performance to enable raters to evaluate the overall level of NTS. The wide range in this scale was chosen to prevent averaging the scores from the elements and categories. The ANTSdk tool also provides space for free text notes to provide more specific feedback to students. We found that the Likert scale in ANTSdk most closely met our requirements and we have therefore adopted a similar scale.

Final revision
Based on the findings in the existing tools and from a review of the literature, one of the preliminary categories within our tool was changed from 'cooperation skills' to 'teamwork. ' We saw that all the other tools had a teamwork category, and we discussed among the research team if teamwork better described the NTS in this category. The category comprised cooperation on a team, where team members have clear roles and the team members must trust and rely on their team members correctly conducting their tasks. In consultation with the research team, it was agreed that 'teamwork' better described the NTS in this category than 'cooperation skills.' Minor nomenclature changes in the elements were also incorporated.
After the final revision, the NorMS-NTS tool contains four main categories; 'Communication,' 'Situation awareness,' 'Teamwork skills,' and 'Decision making' (see Table 3). The four categories consisted of 13 elements. We chose to use a seven-point Likert scale for the overall global score and a five-point Likert scale for the categories and elements.

Discussion
Instead of customizing existing tools, we chose a more generalized approach. Focus group interviews were conducted as openly as possible to examine participants' experiences, opinions, desires, and concerns. The research team assumed that the persons working with or treated by newly graduated physicians had a qualified opinion on which NTS they should possess. Even though these colleagues and patients were novices in the field of NTS, they were still considered to have a great amount of implicit knowledge about the provision of care in this setting. By using focus groups, group dynamics were used to explore and extract this knowledge. The groups were purposively sampled. The groups were not chosen to be representative but to bring as much knowledge as possible. We chose to have patient representatives and health professionals of different ages, experiences, cities, and professions (men and women). We also purposively chose to include informants from both rural and urban areas to see if there were differences in the required NTS, but no difference was found. Despite this broad approach, saturation was deemed to have been reached after five interviews were conducted.
The research team found that the NorMS-NTS developed in this project were similar to existing tools. One important finding was that 'uncertainty' was part of the elements in both NorMS-NTS and Medi-StuNTS but in none of the other examined tools. Managing uncertainty is an integral part of health care work (Mackintosh and Armstrong 2020) and, especially important when managing the role of a new physician and independent caregiver. Another important result was that 'leadership' was not kept as its own 'And that you ensure closed loop communication, is a two way thing. It is not just the doctors responsibility. But it characterises a good team.' (1) 'You need to feel that the doctor is present.
The doctor should be aware, and not all into his computer or somewhere else. That is the most important.' (2) Situation awareness It is important that the doctor is present and aware.

Attentiveness
Aware of the situation, the patient and team members.
'But i think that the doctor should be so confident in his profession that his or her attention, focus, goes to the signals that the patient gives in the moment. That is not just the words, but the whole facial expression, looks, everything. To the extent that it is possible to receive advice. We are all beginners once … … Also there are some who will hide their insecurity for that job, so it is to enter the role with an awareness of what it entails, but also own limitations, it is important.' (4) category but was sorted under decision making. The reason for that was that, in the focus groups, leadership was described as the medical responsibility. The participants wanted the physician to decide the treatment plan. Teamwork more aptly described cooperation, modesty, and consensus. This is in line with Hofstede's cultural dimensions theory and characterization of Norwegian cultural norms, in particular, lower national scores on his dimension of masculinity (Hofstede 1984). Countries with a higher score on masculinity, such as the UK, are more likely to endorse attributes such as achievement, heroism, assertiveness, and competitiveness. In those countries, leadership would likely be associated with a different role and perceptions. The broad approach to define the necessary NTS for newly graduated physicians resulted in a tool that is similar to existing tools. We recognize that our previous knowledge of existing tools may have influenced this result, both when conducting interviews and in the analysis of interview data. Before developing a new NTS tool, it was important for us to understand the general NTS concept in some detail and to clarify whether such tool already existed. We also recognize that this prior knowledge may influence how we interpreted our data. Without this knowledge, it would be difficult to perform this process. However, we did not define the numbers of categories or the category titles before the analysis. We consider that the consistent findings despite the different approaches support the face validity of both the NorMS-NTS and existing tools. It also indicates that there is minimal difference between adapting an existing tool versus creating a completely new one. In future research, one may consider adapting tools rather than creating new ones, as this is a shorter process that requires fewer resources.

Further validation
Face validity does not validate a tool (Cook and Hatala 2016). It is helpful in the development process to confirm that the analysis captured the participants' opinions. Furthermore, validating assessment tools are not simply answering questions with yes or no. Validity consists of scores, interpretation, and use in actual settings, not only an assessment of the tool's properties. Validation is a continuous process of collecting evidence over time. Full validation is, therefore, out of scope for the development of an initial prototype tool. We plan to examine the validation in subsequent work. When the tool is validated for different settings, it will be published with an introduction video and a user manual, and observable behaviors will be listed.

Limitations
NorMS-NTS was developed in a country where physicians play an independent role as caregivers soon after graduation. The physicians are set to lead teams in emergencies within a year of starting their internships. The role of new doctors differs around the world; therefore, the tool will not be entirely transferable to all countries. We consider this tool appropriate for different parts of Norway as Norwegian Medical Schools have the same curriculum. All admissions are based on grades and go through the national admission services. Therefore, we assume that the student populations are similar. The tool was developed in northern Norway, where rural areas dominate. We took this into account by conducting interviews with focus groups situated both in the same city as the hospital and hours away. We recognize that there may have been some differences if we had chosen other groups of physicians, patients, or caregivers to participate in the focus groups.

Implications
The NorMS-NTS tool provides a tool for rating Norwegian medical students' NTS. The rating can be used in different settings, for example, to provide specific feedback to students, evaluate the education delivered, or compare NTS levels across campuses. The tool developed in this study has the potential to be used to improve medical students' NTS, thereby improving new doctors' performance of NTS. From a broader perspective, this may hopefully decrease adverse events in hospitals and improve the quality of care delivered.

Conclusions
NorMS-NTS is a unique tool for assessing the Norwegian medical students NTS. It was created by identifying the necessary NTS based on user perspective through focus group interviews and refined by feedback from the groups, a comprehensive literature search, and an examination of existing NTS assessment tools. The tool addresses four skill categories and 13 elements within those categories.
Disclosure statement