Captures data in real time and allows for online reporting and analysis
The summary of results for the 2012-2013 cohort (n=213), as produced by the OMIS, demonstrated an overall internal consistency (how well predicts the OSCE the overall outcome) of 0.696, where the Cronbach’s Alpha per individual station varied between 0.486 for the Basic Life Support station and 0.769 for the Vital Signs station. In classical psychometric terms internal consistency was moderate . The overall average student performance for the clinical stations was 80.5%, with a minimum score of 16 out of 60 and a maximum of 100% (60 out of 60). The overall average student performance for the Basic Life Support station was 93.7% , with a minimum score of 21 out of 60, and a maximum score of 60 out 60. (See below).
The summary results for the 2013-2014 cohort (n=170), as produced by OMIS, demonstrated an overall internal consistency of 0.666, whereby the Cronbach’s Alpha per station varied between 0.426 for the Basic Life Support station and 0.842 for the Vital Signs station. In classical psychometrics terms internal consistency was moderate. The overall average performance (SD) of the students was 84.8% for the clinical stations, with a minimum score of 20 out of 60 and a maximum score of 100% (60 out of 60). The overall average performance of students was 88.5% for the Basic Life Support station, with a minimum score of 28 out of 60 and a maximum score 100% (60 out of 60). In classical psychometric terms internal consistency was moderate (See below).
Borderline regression analysis
Borderline Group Analysis is a simple way of calculating the average cut-off score of those students that were addressed as ‘borderline performers’ (i.e. examiners not being sure whether the student performance should be marked as fail or pass). Where there are a small number of students in this category, then Borderline Group Average estimates may be very unreliable as shown in the 2012-2013 cohort, where in some stations only 1 or 6 student performances were ‘marked as ‘borderline’ (i.e. 1 borderline scores for the Vital Signs station and 6 for the Basic Life Support station). The ‘average’ score of these students was 60.0% (1 student) for the Vital Signs station and 70.3% (6 students) in the Basic Life Support station. Using Borderline Group Average in this cohort would not provide similar information for those stations where no students were marked as borderline performers. A fully fledged Borderline Regression Analysis would however provide this information due to the inclusion of all Global Ratings from Fail to Excellent.
A similar situation arises with the Borderline Group Analysis of the 2013-2014 cohort, within which 42 student performances were regarded as borderline (i.e. 9 for urine analysis, 0 for the Chest X-ray station, 4 for the BMI station, 11 for the Vital Sign station and 18 for the Basic Life Support station). Due to the small numbers of students, cut-off scores of 67.8%, N/A, 63.8%, 69.2 and 81.5% may be unreliable but provide an indication about the difficulty of each station (BGS>50 means easy station; BGS<50 means a difficult station).
A fully fledged Borderline Regression Analysis is embedded into the OSCE Management Information System software whereby the forecast method is used to calculate new cut-off scores for each station taking into account the ‘difficulty of the station’ and the ‘hawk and dove effect’ of different examiners involved in the OSCE.
Raw scores and adjusted raw scores according to the ‘dynamic cut-off score’ after Borderline Regression Analysis adjusted after John Patterson, honorary senior lecturer at the Centre for Medical Education of the Barts and London School of Medicine and Dentistry and Assessment Consultant.