By Dr Thomas Kropmans
Statistical software R is a free open source statistical software development (the R project) and embraced by a wide community of users and developers and funded by the R Foundation. The R Foundation (@_R_Foundation) is a not for profit organization founded by the R Core Team to provide support for the R project.
Since December 2008, Qpercom clients are familiar with hardcoded basic descriptive statistics and downloadable Excel templates in Qpercom’s Advance Assessment Solution. We are very proud of our analysis showing continuous improvements of assessment analysis by clients. We tried in the past few years to catch up with Quality Assurance literature as published in various AMEE Guide No 81 part 1 , part 2 and AMEE Guide No 49. We are concerned that this would not be enough.
The referenced guides are outdated and more and more evidence is produced that Classical Test Theory (CTT) statistics are not enough to provide insight in assessment analysis or big data. Furthermore, many of our clients are performing their own analysis on top of our summary table with Cronbach’s Alpha, examiners comparison or Borderline Regression Analysis. We recently launched ‘Dive into Data’ and ‘Compute your Assessment’ if you want to be trained in these analysis. Moreover, once Qpercom data is downloaded into the Excel templates at the time generously offered for use by Dr. J.A Patterson BSc PhD Honorary Senior Lecturer at the Centre for Medical Education Barts and the London School of Medicine and Dentistry the link with the original data is broken and online analysis don’t match the offline analysis and that worried us.
As Qpercom is considering student access to data as being owner of their assessment outcome plus the legacy ‘old’ and for various clients adjusted ‘Excel templates’, we needed to think about a fully fledged statistical analysis package that could run on Qpercom’s AWS cloud computing solution. Furthermore, local statisticians are all more and more adopting ‘R’ for psychometric within their own institutions.
The idea of Qpercom Analyse is developed with the National University of Singapore embedding Borderline Regression Analysis (Method 1 and 2) into their analysis and wanting to be able to compare the outcome and the amount of students failing while using Fail, Borderline, Pass criteria (1) vs Borderline Fail/Borderline Pass methods (2). As the Excel downloadable templates in Qpercom’s software can only deal with one of the 4 regression options a Request for Change and embedding of R on our AWS Cloud solution was born. The delivery after all as ‘microservices’ appeared to be quite a challenging story!
Where the analysis in OMIS/Observe were all hard coded and statistical outcome was produced ‘dynamically’ and ‘on the fly’ the initial exam analysis online was very accurate in terms of ‘summary tables’, ‘examiners analyse’, and ‘Borderline Group Analysis’.
Qpercom’s Analyse allows you to merge MCQ’s, EMQ’s from Qpercom Choice with OSCEs from Qpercom Observe or just different OSCEs from different modules or years. Furthermore you can compare outcome while applying various Borderline Regression methods with/without utilizing the Standard Error of Measurement (SEM). Utilizing 1 SEM represents a 68% Confidence Interval (CI) around the Observed score whereas 2 SEM represents a 95% CI.
In the previous versions of Observe (or Choice) assessment analysis was partly hardcoded in the software and results were shown ‘on the fly’.
Where previous analysis was hard coded, we now embed ‘R’ pipelines for various analysis. Due course of the developments on more ‘R’ pipelines leads to fully fledged Quality Assurance guidelines as they are presented in the literature. All types of exams (OSCE and MCQ type) will be included in the software. Moreover, your own statistician will be able to enter his own ‘R’ scripts.
Results in Analyse will be presented in ag-Grid, a highly dynamic software package. The performance, feature set and quality of ag-Grid has not been seen before in a JavaScript datagrid and therefore Qpercom purchased a company wide licence to use ag-Grid. Many features in ag-Grid are unique to ag-Grid, and simply put ag-Grid is in a class of its own, without compromising on quality or performance.
The highly interactive ‘Grid’ from ag-Grid. Watch the ag-Grid video to see it’s full potential.
So, R is a very popular language in academia and finds more and more adopters. Many researchers and scholars use R for experimenting with data science. Many popular books and learning resources on data science use R for statistical analysis as well. Since it is a language preferred by academicians, this creates a large pool of people who have a good working knowledge of R programming. Putting it differently, if many people study R programming in their academic years then this will create a large pool of skilled statisticians who can use this knowledge when they move to the assessment analysis in Medical Education or Volume Recruitment. Thus, leading increased traction towards this language and we choose R to embed into our software to be more flexible and staying ahead of our competitors.