No more Zoom: how we built a bespoke software platform for virtual OSCEs

by | Nov 17, 2020

In 2020 as the COVID-19 pandemic forced colleges and universities across the world to shut, Qpercom’s clients as others were forced to find a new way to stage high-stakes clinical exams and admissions interviews. Off-the-shelf products like Zoom, MS Teams or Google Meet were appealing and had the potential to facilitate a remote exam or assessment interview virtually. While these video conferencing platforms could allow students, examiners and actors to participate in the exam from home, they are not designed to support time constrained practical assessments. The remote exam was now fragmented. It relied on Zoom, MS Teams or Google Meet to stage the exam and Qpercom Observe to assess the performance of participants in real time. A better alternative was not available. 

Apart from the compromised time constrained practical exams like OSCE or MMI, validity and reliability will mostly be reduced due to a reduced amount of stations. Furthermore, an increased amount of (technical support) staff are required to execute these time restricted consecutive station exams remotely.

To eliminate the need to use two separate platforms (Observe and Zoom, MS Teams, Google Meet or similar), we took action and added video to our assessment platform so that the entire remote exam, including its logistics and assessment, could be delivered with one bespoke solution. 


The global impact of COVID-19 on practical assessments


Zoom, MS Teams and Google Meet changed the world of telecommunication during the COVID-19 pandemic.  Teleconference software was already used for Objective Structured Clinical Examinations (OSCE)
two decades ago but performing an OSCE (Objective Structured Clinical Examination) or MMI (Multiple Mini Interviews) on a flat communication platform is not easy according to  mid-pandemic published literature.

In early March 2020, Duke-National University Singapore Medical School embraced the challenge of ensuring competent final year medical students could complete their final year of studies to enter the medical workforce in Singapore without delay. Using Zoom facilitated briefings, they ensured the validity and reliability of such high-stakes assessments whilst protecting the safety of all participants, minimising risk and maintaining defensibility to key stakeholders.

With more strict lockdown conditions enforced globally, a few other groups published their experiences in using video communication technology platforms for clinical skills assessments.
The University of Melbourne Doctor of Medicine (MD) program outlined their successful implementation of a 2-station OSCE involving 361 candidates over 2 days using 5 different Zoom video rooms and other online collaboration tools.

In March 2020, the New York University Grossman School of Medicine Primary Care program adapted its annual comprehensive OSCE to a telemedicine-based platform, to comply with distance learning and social distancing policies during the COVID-19 pandemic. Five of the originally planned 11 OSCE cases were adapted to a “virtual visit” telemedicine format, with learners and SPs interacting remotely over a web-based platform installed in the simulation space. Three areas of telemedicine competency needs were identified in this case study: technical proficiency; virtual information gathering, including history, collateral information collection, and physical exam; and interpersonal communication skills, both verbal and nonverbal.

Harvard School of Dental Medicine, Boston, Massachusetts, USA successfully conducted a 21 station dental OSCE using Zoom. In their feedback most students in this study thought the online OSCE was comparable to traditional OSCEs, and all student respondents felt they were able to fully showcase their knowledge. Half of the student respondents thought that all future OSCEs should be performed online.

 

Impact of COVID‐19 on dental education in the United States

Table 1: From ‘Impact of COVID‐19 on dental education in the United States’ (J Dent Educ. 2020;1–3.).


Nevertheless, despite the aforementioned positive experiences, 440 students from 32 UK medical schools responded to a
National Survey in the UK and 38.4% (n = 169) of respondents had their final OSCEs cancelled while 43.0% (n = 189) had already completed their final OSCEs before restrictions. 43.0% (n = 189) of assistantship placements were postponed while 77.3% (n = 340) had electives cancelled. The impact of COVID-19 on OSCEs, written examinations and student assistantships significantly affected students’ preparedness for medical practice.

Moreover, in 2011, long before the pandemic, Divid Tiller and team from the University of Sydney (2013) compared the outcome of Skype (Microsoft) based MMIs with those facilitated in a face-to-face setting in 2011 and in 2009. Apart from comparable descriptive results, multivariate G-theory analysis over 9 stations/questions showed similar reproducible results (G-coeff. ~ 0.735). By assessing online, they achieved an 84% cost saving compared to the physical alternative (AU$ 50.000) while travelling to an assessment centre was no longer required.

 

Introducing video technology to e-assessment software


Qpercom Observe is Qpercom’s advanced assessment solution for physical station type examinations and admission interviews and is used by numerous clients in higher education worldwide. It was developed in 2008 to reduce error in paper based assessments and
we saved about 70% of administration time by automating the assessment logistics and Quality Assurance (QA) procedures. The order of students appearing at each assessment station is determined by the logistics of the management system. Time management, creation of assessment forms and even more important instant QA analysis (internal consistency analysis, R square, Borderline Regression Analysis and Standard Error of Measurement (applying a 68% and 95% Confidence Interval around the ‘observed score’)) are automated. Instant feedback to students and examiners can be released once QA is approved.

Hearing about the trouble clients were going through while moving towards communication platforms while trying to continue their assessments remotely during the pandemic was concerning. We envisaged potential solutions. We knew that embedding video into our assessment solution was the answer and at the same time a major challenge as it had never been done before. We also knew that competitors wouldn’t sit still and wait to see what solutions the market would come up with. We are the market and we might get compromised if clinical skills assessments and admission interviews can’t be done remotely the way they were originally designed by Ronald Harden in 1975. We challenged our team of seven with an ambitious plan: add an option to conduct a remote station type assessment using video from home.

To make a long story short, merging independent video software into our logistics would allow clients to organise, analyse and reflect on their station performances as before. While adding a student/actor console; students, simulated patients (SPs)/Actors and the mechanism that allows examiners to participate from home. couldn’t be simpler, a major technical and financial challenge was born in a time frame between June and September 2020.

Our proof of concept was tested in trustful collaboration and reported by University College Dublin’s Department of Psychiatry. Our mutual experiences during these high stake remote clinical exams are similar to those reported by Harvard Dental School (see Table 1). Having video integrated in our bespoke assessment platform allows instant access to results by students and examiners while the remote assessment is ongoing. External examiners and invigilators can access video rooms (stations) to assure correct procedures or proctoring. Furthermore, quality assurance analysis and our student feedback email system allows instant analysis and internal reporting. All in the one platform.

 

Observe on all devices

Zoom, MS Teams and Google Meet might be Goliath, we’re happy to be David delivering a tailor made staged assessment and interviewing video system for higher education.

Interested? Click below to find out more about how you can run and assess virtual OSCEs running our platform:

Observe Demo

Free eBook:

The Use of Technology in Clinical Skills Assessment
An essential read for those introducing technology into clinical skills assessment.

Technology can:

  • Reduce error rates
  • Decrease administration time
  • Increase quality standards