Clinical skills examinations planned by University College Dublin (UCD) in 2020 were disrupted by the arrival of the COVID-19 pandemic which first hit Ireland in March. Around 200 students in the university’s Department of Psychiatry would not be able to graduate as normal if their final exams scheduled for September, October, November and December fell victim to countrywide restrictions imposed by the Irish government to suppress the virus.
The Department of Psychiatry, already long-term users of Qpercom’s OSCE management system Observe, approached the company to discuss a solution that would help facilitate their St Vincent’s (SVUH) OSCE while adhering to the lockdown restrictions.
According to The Irish Times in 2016, Qpercom dragged assessment out of the dark ages but 2020 provided a whole new challenge. Could assessment be dragged into the pandemic age?
The problem
When global pandemic restrictions meant face-to-face exams were no longer an option, many universities and institutions turned to Zoom, MS Teams and Blackboard Collaborate to facilitate remote exams during the early stages of the pandemic. These standard communication platforms became part of everyday life for many but trying to use them as an assessment platform in a high stakes exam situation compromises assessment methodology.
Thinking outside of the box while searching for a solution that would integrate video communication with our bespoke OSCE assessment logistics, in early May 2020 Qpercom’s development team challenged the existing standard platforms (Zoom, MS Teams) and started to develop a remote assessment platform that offered video as standard.
Step forward Observe VI 2020.
Video Integration and Assessment
VI stands for Video Integration and 2020 characterises the year that the pandemic took hold. Observe facilitates time-constrained station based assessment Objective Structured Clinical Examinations (OSCEs) and Multi-Mini-Interviews (MMIs). It was originally built to be used by high stake educational institutions worldwide who wanted to replace paper-based exams with logistically enhanced student streams that could store and analyse OSCEs online.
Video technology was introduced to complement Qpercom’s OSCE logistics and in early August, we ran our first internal tests. Notwithstanding the normal teething experiences, we coped with the workload and stress levels juggling our own survival and associated financial pressures that came with the pandemic.
With clients growing frustrated after moving to free communication platforms like Zoom, and ultimately judging them as not fit for purpose for assessment, the race to develop and roll-out video integration for Observe became Qpercom’s vaccine programme for e-assessment in medical education.
The UCD experiment
Could Qpercom support a multi-station OSCE fully remotely?
By mid-August when requests started to come in referring to already postponed OSCEs and MMis, we were in our early trial stages, enhancing OSCE stations as video rooms manned with assigned examiners (working from home) and actors (working from home) per station, populated logistically with rotating students (from homes) using the standard OSCE logistics. We couldn’t confirm nor guarantee an error-free staging of a 7-station OSCE due to software and testing limitations. Additionally, we simply didn’t have the people for testing at capacity.
The visionist Dr Allys Guerandel, together with Dr Katherine Stroughair and Prof Kevin Malone from SVUH, Prof Anne Doherty, Consultant Liaison Psychiatrist, Mater Hospital and Chris Carr, UCD Psychiatry Coordinator adopted the option of a trial run and were willing to take the risk. The ‘remote OSCE’ would be administered on-campus and students, actors/simulated patients and examiners would operate from different UCD locations, all with equal WIFI access. For a challenging but doable scenario, we mobilised all of our available resources during 4 separate full-blown exams to make this happen.
In the meantime, the University of Nicosia, Cyprus enquired about joining the early stages of trialling as they had OSCEs and International Licencing Exams upcoming in early 2021. With supporting our client in Nicosia came one important difference. The ongoing pandemic restrictions meant we couldn’t fly abroad to support the exam in Cyprus so all had to be done remotely – a high bar to set at the early stages of trialling but one we didn’t shy away from.
One of the uncertainties surrounding the trials was accessibility and connectivity issues from remote home locations. Internal trials incorporating development and testing teams in Ukraine and India went smoothly. Bringing everybody within the walls of UCD created a scenario where participants had equal access to the Internet thus tackling at least one ‘unknown outcome’ for that trial. Bringing Qpercom staff from the West of Ireland to Dublin meant examiners, actor/patients and students could be assisted (using PPE and adhering to social distancing) during the exam to investigate and solve any software related problems on the spot.
The outcome
Table 1. University College Dublin’s illustrated progress of remote (video integrated) assessment of final year OSCEs at UCD/Department of Psychiatry
We had a cohort of 60 final year students each month, 8 examiners, 8 actors (plus spares) and 2 administrators managing Qpercom’s online monitoring tool, 4 staff members of Qpercom on-site and an extra 4 software engineers remotely for technical backup. Added to that was Dr Allys Guerandel and her organising team of 3 others; plus 6 invigilators meaning a total of 96 people were involved in the trial. Repeating the same trial 4 times with a new group of students (September, October, November and December) and time in-between to work on ‘fixes’ helped us to work towards user acceptance.
User Acceptance
On day 4 of the UCD exams, in December 2020, we got User Acceptance from UCD. A full-blown video-integrated solution managed 100% of data retrieval, storage and analysis through logistically-managed video streaming within the time frame schedule (09.00 – 15.30).
What did we use to deliver the video-integrated remote OSCE?
- Qpercom Observe, our bespoke platform for OSCEs and MMIs
- We embedded Video Technology alongside the OSCE logistics in Observe
- We added a monitoring, control and view tab to help administrators control the stages in the OSCE process while this new watch tab allows invigilation and two ways video communication
- A Preview stage to check video/cam settings
- A Preparation stage to rendezvous Examiners and actors in video stations. It then moves to:
-
- 1. Reading time stage (student reads station instructions);
- 2. Admit and confirm student stage (examiners admit students to the room and confirm ID);
- 3. Assessment stage, station score sheets available to examiners
- A Pause stage, in case of any technical issues in one or more circuits
- A Finishing stage
- The staging process during the on-going OSCE is:
-
- Reading time;
- Admit student;
- Assess student and back to Reading time. In ‘back to reading time’ all students in the rotation are moved automatically from their initial station to the next station (one tick of the button ‘Reading time’)
Figure 1 The ‘Qpercom watch tab’’ where the OSCE coordinator can directly interact with the video stations.