ROC (receiver operating characteristic) curve

This macro performs three functions as a subsequent analysis to a binary logistic (BLR) regression analysis to evaluate how well the model performs:
  1. Generates a classification table
  2. Generates an ROC (Receiver Operating Characteristic) curve
  3. Given the event probabilities, stores an event probability for each row, not just the first instance of a unique set of predictor values

Download the Macro

Be sure that Minitab knows where to find your downloaded macro. Choose Tools > Options > General. Under Macro location browse to the location where you save macro files.

Important

If you use an older web browser, when you click the Download button, the file may open in Quicktime, which shares the .mac file extension with Minitab macros. To save the macro, right-click the Download button and choose Save target as.

Required Inputs

  • A binary response variable, in the response/frequency format
  • One or more columns of predictor variables
  • A column of event probabilities calculated during the binary logistic regression analysis already done on the data
Note

To store the event probabilities from a binary logistic regression, click Storage from the main dialog box and select Event probability.

Optional Inputs

FREQ C
Use if you specified a frequency column when you ran the binary logistic regression. For example, if you specified C4 as the frequency column, you must submit FREQ C4.
REFEVENT "text"
Use if you specified a reference event when running binary logistic regression. For example, if your binary response variable includes the values, "Pass" and "Fail" and you set the reference event to be "Pass", you must submit REFEVENT "Pass".
FITSTORE C
Use to specify a column to store the fitted values, which are used to generate the classification table. In binary logistic regression, an observation's fitted value is equal to the reference event if that observation's event probability is greater than or equal to 0.5.
ROCSTORE C C C
Use to store the data used to generate the ROC curve. Specify three columns: one to store the values of p, and two to store the sensitivity and specificity values for each value of p. (Note that the ROC curve plots sensitivity versus 1 - specificity).

Running the Macro

Suppose your binary response variable is in column C3, and you have two predictors in columns C1 and C2. The event probabilities are stored in C5. To run this macro, choose Edit > Command Line Editor and type:

%ROCBLR C3;
MODEL C1 C2;
EPRO C5.

Click Submit Commands.

Additional Information

Classification Tables and ROC Curves

One measure of the adequacy of a binary logistic regression model is its ability to accurately predict a binary response, given the values of the terms (predictors) in the model. This ability can be summarized in a classification table which tabulates the number of times the fitted model classifies the response correctly for each of the two response values.

For an example, refer to the topic Example of binary logistic regression in Minitab Help. This example fits a binary logistic regression model where the response is RestingPulse (High or Low) and the predictors are Weight (a continuous variable) and Smokes (a categorical variable with two values, Yes or No).

In the dataset, one person is a non-smoker and has a weight of 175 pounds. To predict whether this person will have a high resting pulse or low resting pulse, the person's weight and whether or not they smoke (1 if they do, 0 if they do not) are "plugged into" the regression equation produced in the analysis. The result is an estimated probability of having a low resting pulse (since "Low" is the reference event in this example):

In other words, given the estimated coefficients, the non-smoker with a weight of 175 pounds has an estimated 92% chance of having a low resting pulse. This is the event probability. Because this probability is so high, we might classify this observation as "Low". We are taking an estimated probability and classifying it as one of two values. As well, this particular person's actual, observed resting pulse was low. So, the model was accurate in predicting the resting pulse for this individual.

We can do the same thing for every other observation in the dataset, but we must first decide where to "cut off" the event probability. The most obvious cutoff is 0.5. We can say, for every observation in the dataset, if the event probability is greater than 0.5 we will classify the resting pulse as "Low" and if the event probability is less than 0.5, we will classify the resting pulse as "High".

Finally, we can compare the predicted resting pulse with the actual resting pulse and tabulate these results in a classification table (not shown).

The table would show that there were 68 individuals whose resting pulse was low and the event probability was greater than 0.5; the classification was accurate. Also, there were two individuals whose resting pulse was low, but the event probability was less than 0.5; the classification was inaccurate. Similarly, there were two individuals whose resting pulse was high and the model classified the predicted resting pulse correctly. However, there were also 20 individuals whose resting pulse was high, but the model incorrectly classified the predicted resting pulse as low. So, you might conclude that the model leans toward predicting resting pulse as low.

Two other statistics that help summarize the accuracy of the model are sensitivity and specificity. The sensitivity measures how well the model predicts a low resting pulse, given that an individual's actual resting pulse is low. The specificity measures how well the model predicts a high resting pulse, given that an individual's actual resting pulse is high. For both statistics, the higher the value, the better the model's ability to classify observations.

In this example, sensitivity = 68 /70 ≅ 97% and specificity = 2/22 ≅ 9%.

However, classification tables are very dependent on the distribution of observations in the classification table and can be very misleading. A thorough and clear discussion of the problems inherent to classification tables is included in Hosmer and Lemeshow's Applied Logistic Regression.

A better means of assessing a binary logistic regression model's ability to accurately classify observations is a receiver operating characteristic (ROC) curve. An ROC curve is constructed by generating several classification tables, for cutoff values ranging from 0 to 1 and calculating the sensitivity and specificity for each value. Sensitivity is plotted against 1 - specificity to construct an ROC curve.

The area under the ROC curve (AUC) is a measure of discrimination; a model with a high area under the ROC curve suggests that the model is able to accurately predict the value of an observation's response.

Hosmer and Lemeshow provide general rules for interpreting AUC values. Paraphrasing their rules gives the general guidelines below:

AUC = 0.5 No discrimination (i.e., might as well flip a coin)
0.7 ≤ AUC < 0.8 Acceptable discrimination
0.8 ≤ AUC < 0.9 Excellent discrimination
AUC ≥ 0.9 Outstanding discrimination (but extremely rare)

Because the AUC for this model is less than 0.7, you might conclude that the model does not provide adequate discrimination.

References:

D.W. Hosmer and S. Lemeshow (2000). Applied Logistic Regression. 2nd ed. John Wiley & Sons, Inc. Pp. 156-164.

A. Agresti (2002). Categorical Data Analysis. 2nd ed. John Wiley & Sons, Inc. Pp.228-230.

By using this site you agree to the use of cookies for analytics and personalized content.  Read our policy