Schmitz, highlights a number of advantages to use an electronic handheld devices to assess OSCEs including, speed of data gathering, simplicity of data evaluation and fast automatic feedback. Segall et al support computer based assessment suggesting that grading is more accurate, feedback is immediate, security is enhanced and less time is spent by instructors on grading and data entry.
OMIS produces an online analysis of items and overall total (raw) scores and adjusted (raw) scores using standard setting of student performance after regression analysis. The mean result, standard deviation (SD), minimum and maximum and range and mid range are produced instantly, in real time, during the examination. Internal consistency of OSCE station item forms (Cronbach’s Alpha) is used to provide insight into the consistency of items in each station predicting the overall score of the student of that specific station. Borderline Regression Analysis (Borderline Group Average versus Borderline Regression Method) calculates a ‘flexible cut-off score’ complementary to the general static ‘standard’ cut score for each individual station. The overall average regression cut-score is used to adjust the average overall raw score of the students. Borderline Group Average, which is based upon calculation of the average mark of those students that were globally rated by their examiners as ‘borderline’, is the most simplistic method to use. A complete Borderline Regression Analysis, which is performed over all item marks matched with all of the global ratings (from fail to excellent), can also be used. The flexible cut-off score is calculated using the BRM Cut score (Intercept+1 × Slope)- since borderline=1 using FORECAST method, (Figure 1).
All analysis reports and data can be exported to Excel to facilitate further detailed analysis.
Data can be exported to perform a Generalizability Coefficient analysis using a G- and D-study with EduG software. We use the Standard Error of Measurement (SEM) for all of our supported assessments. The G-study generates information about whether the outcome can be generalised to other medicine OSCEs. The D-study provides information on how the generalizability of results can be improved.