Improved DBM ROC Methods in Diagnostic Radiology

This grant addressed a fundamental methodological problem in diagnostic radiology research: how to evaluate the diagnostic performance of imaging modalities. Receiver operating characteristic (ROC) analysis is an accepted procedure for measuring how well imaging modalities allow physicians to detect disease. Traditional statistical comparison of different imaging modalities only analyzes ROC measures of a single radiologist from each modality. Dorfman, Berbaum and Metz (1992) developed a statistical methodology that overcomes this limitation and provides a comprehensive framework within which to analyze ROC studies with multiple readers, taking into account both reader-sample and case-sample variation. Over time, the Obuchowski-Rockette (1995) methodology of estimating variances and covariances was incorporated was added as an option for the statistical analyses. The Obuchowski-Rockette (OR) and Dorfman-Berbaum-Metz (DBM) methodologies allow greater statistical power to detect clinically meaningful differences in diagnostic performance than single-radiologist approaches. Its validity has been tested extensively using computer simulations and has wide acceptance in diagnostic radiology experiments. Because OR-DBM methodology relies on ROC analysis for its dependent measures, the robustness and accuracy of the ROC analysis affect the statistical precision and power of OR-DBM methodology. The software now includes options for “proper” ROC analysis modules that provide more interpretable ROC results than traditional "improper" ROC analysis. The software is now referred to as the "OR-DBM MRMC" software rather than the "DBM MRMC" software.  

OR-DBM MRMC References

Dorfman, D.D., Berbaum, K.S., & Metz, C.E. (1992). Receiver operating characteristic rating analysis: Generalization to the population of  readers and patients with the jackknife method. Investigative Radiology, 27, 723-731.

Dorfman, D.D., Berbaum, K.S., Lenth, R.V., Chen, Y.F., & Donaghy, B.A. (1998).  Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design. Academic Radiology, 5, 591-602.

Hillis, S.L., & Berbaum, K.S. (2004). Power estimation for the Dorfman-Berbaum-Metz method. Academic Radiology, 11, 1260-1273.

Hillis, S.L., Obuchowski, N.A., Schartz, K.M., & Berbaum, K.S. (2005). A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Statistics in Medicine, 24, 1579-1607.  DOI:10.1002/sim.2024.

Hillis, S.L. (2005). Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Academic Radiology, 12, 1534-1541  DOI:10.1016/j.acra.2005.07.012.

Hillis, S.L. (2007). A comparison of denominator degrees of freedom for multiple observer ROC analysis. Statistics in Medicine, 26, 596-619. DOI: 10.1002/sim.2532.
 
Hillis, S.L., Berbaum, K.S., Metz, C.E. (2008).  Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Academic Radiology, 15, 647-661. DOI:10.1016/j.acra.2007.12.015

Obuchowski NA, Rockette HE. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations. Communications in Statistics-Simulation and Computation 1995; 24(2), 285-308.