We see this inconsistency amongst many clients using Qpercom’s bespoke OSCE Management Information System. One of the reasons for this inconsistency might be due to the fact that in the default setting of the performance scoresheet, the examiners mark the students ‘blind’, on a Likert scale from 0 to 4. In general, ‘0’ stands for ‘not done’ and ‘4’ stands for ‘done and complete’, while observing and marking a specific clinical skill. All ‘done and complete’ scores should result in an Excellent score on the GRS, whereas ‘not done’ scores should result in a Fail or at most a Borderline. Other possible scores are Pass and Good.
The written feedback added to the scores and GRS(examiners’ overall impression of a student’s performance) often doesn’t offer anything but confusion for the student receiving these kinds of results. We’re currently awaiting the outcome of a PhD study analysing whether written feedback matches either the scores and/or the GRS of the examiner assessing the students.
Enthusiastic about the arrival of ChatGPT which surfaced around November 2022, we have experimented with adding item scores, global rating scores and written feedback into ChatGPT to see what we would get back from this Artificial Intelligence language model chatbot.
For the highest performing student, ‘Mark’, this is what the chatbot came back with (500 words only):
“Mark performed very well in his year 5 clinical skills exams. In Clinical station 1, he scored 90.0%, in Clinical station 2, he scored 80.0%, in Clinical station 3, he scored 100.0%, and in Clinical station 4, he scored 86.0%. These scores indicate that Mark has a strong understanding of clinical skills and is able to apply them effectively.
In the patient-related stations, which focus on physical examination, Mark scored 3, 2.4, 3, and 2.4 out of 5, respectively. In terms of his ability to interpret physical signs and provide a differential diagnosis, Mark scored 2.4, 2.4, 3, and 3 out of 3. These scores suggest that Mark has a good understanding of physical examination techniques and is able to apply them in the diagnosis of patients.
For case discussion, Mark scored 2, 1.6, 2, and 1.6 out of 3. These scores indicate that Mark is able to discuss cases effectively and provide appropriate management plans.
In terms of professionalism, communication, and maintaining patient safety, Mark scored 1.6, 2, 1.6, and 1.6 out of 3. These scores suggest that Mark has a good understanding of the importance of professionalism, communication, and patient safety, but there may be room for improvement in these areas.
Overall, Mark’s performance in the exams was rated as “Excellent” by the examiners. One of the examiners provided positive written feedback, describing Mark as a “really strong candidate” who is “very confident and appropriate” and “progressed very well”. The examiner also noted Mark’s “excellent differential and management plan”.
In conclusion, Mark has demonstrated a strong understanding of clinical skills and is able to apply them effectively in the diagnosis and management of patients. While there may be room for improvement in areas such as professionalism, communication, and patient safety, Mark’s overall performance was rated as “Excellent” and he received positive feedback from one of the examiners”.
“Based on the scores provided, Lea’s performance in the clinical skills exams is below average, with scores ranging from 23% to 51%. It is important to note that the scores are not the only indicator of performance, but they are a significant factor in evaluating a student’s ability to apply clinical skills.
Looking at the patient-related stations, Lea’s ability to approach and identify physical signs scored below average, with scores ranging from 50% to 55%. Additionally, Lea’s ability to interpret and provide differential diagnosis of physical signs scored slightly better, but still below average, with scores ranging from 45% to 50%. Case discussion and Professionalism scores are also below average, with scores ranging from 30% to 37%.
Given the above scores, it is likely that Lea will receive a fail or borderline result. It is important to note that there may be mitigating factors that could have affected Lea’s performance, such as test anxiety, poor time management, or inadequate preparation. It may be helpful to discuss these factors with Lea to identify any areas for improvement.
In terms of feedback, it would be helpful to provide specific recommendations for improvement in each area where Lea scored below average (as provided by some of the examiners). For example, in the patient-related stations, Lea may benefit from additional training in physical examination techniques or differential diagnosis. In the case discussion and professionalism areas, Lea may benefit from additional training in communication skills, patient safety, or ethical considerations.
Overall, it is important to emphasise that these scores do not define Lea’s potential as a medical student or future physician. With dedication, hard work, and targeted training, Lea can improve and ultimately excel in the clinical skills required for a successful career in medicine.”