The Pooled ROC Curve and Jackknifing Readers
Analysis of ratingmethod ROC data in which data from patients and readers is pooled has a number of advantages over analysis of the data from individual readers: (1) it measures the performance of a diagnostic system rather than an average reader, (2) it is more appropriate for cost/benefit analysis, (3) it matches the commonly used visual display of pooled results.
Choice of design and statistical technique for an experiment depends upon whether statistical generalization to readers or patients is more fundamental, which in turn depends upon the nature of the experimental question. In studies of physical image characteristics, investigators often assume that there are no important differences among readers. Such studies are concerned with whether a physical manipulation influences image interpretability of a population of patients; thus the appropriate error term for testing differences is based on variation among patients. On the other hand, research designs in psychology tend to treat images within an experimental condition as a fixed factor with readers as a random variable. An experimental factor that might affect the reader’s perception of a radiograph, but does not change the radiograph, is a psychological factor. The basic problem in investigations of psychological variables is to study consistency of effect across readers so as to be able to generalize to new readers. The appropriate error term for testing differences related to psychological variables is based on reader variation. Because our research involves manipulation of the perception and/ or cognitive behavior of the reader, our application of maximumlikelihood and jackknife methods is designed to allow experimental results to be generalized to the population of radiologists from which the sample was selected.
Perceptual accuracy of the pooled ROC curve can be analyzed by the jackknife method. In previous investigations using a relatively small number of patients, conclusions derived from pseudovalues of the jackknife method agreed with conclusions derived from estimates of the maximum likelihood method (Berbaum et al. 1986; Berbaum, Franken, et al. 1988; Berbaum, El Khoury et al., 1988). When the maximumlikelihood method does not converge to a solution for individualreader ROC curves, the most conservative approach is simply to exclude that reader’s data from further analysis. With smaller patient samples, conclusions from pseudovalues vs. maximumlikelihood estimates might diverge because of possible degenerate individual ROC data or strong statistical bias in maximumlikelihood estimates. Maximumlikelihood estimates may be biased in small samples, whereas the jackknife is a biasreducing method of estimation.
RSCOREJ Description
RSCOREJ: Computation of Estimates and Standard Errors for Parameters of Signal Detection Theory Using Jackknife Method.
RSCOREJ is a pooled data version of RSCORE II. This program incorporates many parts of the RSCORE II program for rating method data. RSCORE II is a modified version of a program developed by Dorfman and Alf (1969).
RSCORE II employs a variant of the NewtonRaphson method called the method of scoring, to obtain maximumlikelihood estimates of the parameters of signal detection theory for ratingmethod data. In the method of scoring the expected secondpartial derivatives replace the observed secondpartial derivatives used in the NewtonRaphson method. The method of scoring requires a set of initial guesses or preliminary estimates of the parameters. RSCORE II does not require a set of initial estimates. RSCORE II calculates the leastsquares solution for the parameter estimates from the data and uses these leastsquares estimates as the initial values for the method of scoring.
With pooled data of a group of observers, it has been assumed that the pooled observers are homogeneous with regard to sensitivity and decision criteria. However, statistical inferences may not be robust with regard to mild violations of this assumption. RSCOREJ applies the jackknife technique to rating data pooled from a group of observers to obtain estimates and standard errors for parameters of a ratingmethod roc curve of pooled data, and to reduce statistical bias of such estimates so as to permit robust statistical inference.
RSCOREJ References
Berbaum KS, Franken Jr EA, Dorfman DD, Barloon TJ, Ell SR, Lu CH, Smith WL, AbuYousef MM. Tentative diagnoses facilitate the detection of diverse lesions in chest radiographs. Investigative Radiology 1986;21:532539. Berbaum KS, Franken EA, Dorfman DD, Barloon TJ. Influence of clinical history upon detection of nodules and other lesions. Radiology 1988;23:4855. Berbaum KS, El Khoury GE, Franken EA, Kathol MC, Montgomery WJ, Hesson W. The impact of clinical history on detection of fractures. Radiology 1988;168:507511. Dorfman DD. RSCORE II. In J. A. Swets & R. M. Pickett, Evaluation of diagnostic systems: Methods from signal detection theory. New York: Academic Press, 1982:212232. Dorfman DD, Alf E Jr. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervalsRating method data. Journal of Mathematical Psychology 1969;6:487496. Dorfman DD, Berbaum KS. RSCOREJ: pooled ratingmethod data: a computer program for analyzing pooled ROC curves. Behavior Research Methods, Instruments, and Computers 1986;18:452462.

RSCOREJ (2006 Version) Download
Title  Size  Description  
RSCOREJ Version 2006  189 kb  Program, manual, and sample data  Download 