VISROC 2.0: Updated Software for the Visualization of the significance of Receiver Operating Characteristics based on confidence ellipses

The Receiver Operating Characteristics (ROC) method is used to evaluate the diagnostic accuracy of binary quantitative tests in a broad spectrum of disciplines, including medicine, physics of complex systems, geophysics, meteorology, etc. The estimation of the significance of the examined prediction method is of high importance and it's usually approximated by Monte Carlo calculations. To simplify this problem, a FORTRAN code called VISROC was submitted to the CPC Program Library in 2014. VISROC evaluates the significance of binary diagnostic and prognostic tools for a family of k-ellipses which are based on confidence ellipses and cover the whole ROC space. Since that time, the code has been significantly improved and several new capabilities have been added. Most importantly, a Graphical User Interface (GUI) has been implemented, which can be invoked using either the R shiny web application or the Python application available for Windows, Mac, and Linux operating systems, both of which are described here. New version program summary: Program Title: VISROC 2.0 CPC Library link to program files: https://doi.org/10.17632/dktybrydvf.1 Licensing provisions: AGPLv3 Programming language: R and Python Journal reference of previous version: N.V. Sarlis, S.-R. G. Christopoulos, Comput. Phys. Commun. 185 (2014) 1172. Does the new version supersede the previous version?: Yes Reasons for the new version: Overcome previous version's limitations and implement GUI. Summary of revisions: R and Python applications, GUI, ability to input files containing ROC data. Nature of problem: The Receiver Operating Characteristics (ROC) [1] is a method used to evaluate the diagnostic ability of binary tests and prediction methods in a broad spectrum of disciplines. Apart from the sensitivity (or True Positive rate, TPr) and specificity (which is complementary to False Positive rate, FPr, i.e., specificity=1-FPr) measures, the estimation of the statistical significance of the examined method is of high importance. VISROC evaluates the significance of binary diagnostic and prognostic tools for a family of k-ellipses which are based on confidence ellipses and cover the whole ROC space. Solution method: Using the statistics of random binary predictions for a given value of the predictor threshold ϵ_t, one can construct the corresponding confidence ellipses. The envelope of these confidence ellipses is estimated by varying ϵ_t in the interval [0,1] and one obtains a new family of ellipses, called k-ellipses [2]. They cover the whole ROC space and lead to a well defined Area Under the Curve (AUC). Mason and Graham [3] have shown that AUC follows the Mann-Whitney U-statistics [4] which can be used [5] to estimate the statistical significance of each k-ellipse. As the transformation is invertible, any point on the ROC plane corresponds to a unique value of k, hence it belongs to a unique k-ellipse that allows the estimation of the probability (p-value) to obtain this point by chance. The present GUI applications provide the p-value on the ROC plane as well as the k-ellipses corresponding to the (p=)10%, 5% and 1% significance levels using as input the number of the positive (P) and negative (Q) cases to be predicted. References: [1] T. Fawcett, Pattern Recognit. Lett. 27 (8) (2006) 861–874, 1172. [2] N.V. Sarlis, S.-R.G. Christopoulos, Comput. Phys. Commun. 185 (2014) 1172. [3] S.J. Mason, N.E. Graham, Q. J. R. Meteorol. Soc. 128 (2002) 2145. [4] H.B. Mann, D.R. Whitney, Ann. Math. Stat. 18 (1947) 50. Program Summary section. [5] L.C. Dinneen, B.C. Blakesley, J. R. Stat. Soc., Ser. C, Appl. Stat. 22 (1973) 269.