Interpret all statistics for Best Subsets Regression

Interpretation

Use R² to determine how well the model fits your data. The higher the R² value, the better the model fits your data. R² is always between 0% and 100%.

You can use a fitted line plot to graphically illustrate different R² values. The first plot illustrates a simple regression model that explains 85.5% of the variation in the response. The second plot illustrates a model that explains 22.6% of the variation in the response. The more variation that is explained by the model, the closer the data points fall to the fitted regression line. Theoretically, if a model could explain 100% of the variation, the fitted values would always equal the observed values and all of the data points would fall on the fitted line. However, even if R² is 100%, the model does not necessarily predict new observations well.

Interpretation

Use adjusted R² when you want to compare models that have different numbers of predictors. R² always increases when you add a predictor to the model, even when there is no real improvement to the model. The adjusted R² value incorporates the number of predictors in the model to help you choose the correct model.

The first model yields an R² of more than 50%. The second model adds cooling rate to the model. Adjusted R² increases, which indicates that cooling rate improves the model. The third model, which adds cooking temperature, increases the R² but not the adjusted R². These results indicate that cooking temperature does not improve the model. Based on these results, you consider removing cooking temperature from the model.

Interpretation

Use PRESS to assess your model's predictive ability. Usually, the smaller the PRESS value, the better the model's predictive ability. Minitab uses PRESS to calculate the predicted R², which is usually more intuitive to interpret. Together, these statistics can prevent over-fitting the model. An over-fit model occurs when you add terms for effects that are not important in the population, although they may appear important in the sample data. The model becomes tailored to the sample data and therefore, may not be useful for making predictions about the population.

Interpretation

Use predicted R² to determine how well your model predicts the response for new observations. Models that have larger predicted R² values have better predictive ability.

A predicted R² that is substantially less than R² may indicate that the model is over-fit. An over-fit model occurs when you add terms for effects that are not important in the population. The model becomes tailored to the sample data and, therefore, may not be useful for making predictions about the population.

Predicted R² can also be more useful than adjusted R² for comparing models because it is calculated with observations that are not included in the model calculation.

For example, an analyst at a financial consulting company develops a model to predict future market conditions. The model looks promising because it has an R² of 87%. However, the predicted R² is 52%, which indicates that the model may be over-fit.

Interpretation

A Mallows' Cp value that is close to the number of predictors plus the constant indicates that the model produces relatively precise and unbiased estimates.

A Mallows' Cp value that is greater than the number of predictors plus the constant indicates that the model is biased and does not fit the data well.

Interpretation

Use S to assess how well the model describes the response. S is measured in the units of the response variable and represents how far the data values fall from the fitted values. The lower the value of S, the better the model describes the response. However, a low S value by itself does not indicate that the model meets the model assumptions. You should check the residual plots to verify the assumptions.

For example, you work for a potato chip company that examines the factors that affect the percentage of crumbled potato chips per container. You reduce the model to the significant predictors, and S is calculated as 1.79. This result indicates that the standard deviation of the data points around the fitted values is 1.79. If you are comparing models, values that are lower than 1.79 indicate a better fit, and higher values indicate a worse fit.

Interpretation

Use AICc and BIC to compare different models. Smaller values are desirable. However, the model with the least value for a set of predictors does not necessarily fit the data well. Also use tests and residual plots to assess how well the model fits the data.

Both AICc and BIC assess the likelihood of the model and then apply a penalty for adding terms to the model. The penalty reduces the tendency to overfit the model to the sample data. This reduction can yield a model that performs better in general.

As a general guideline, when the number of parameters is small relative to the sample size, BIC has a larger penalty for the addition of each parameter than AICc. In these cases, the model that minimizes BIC tends to be smaller than the model that minimizes AICc.

In some common cases, such as screening designs, the number of parameters is usually large relative to the sample size. In these cases, the model that minimizes AICc tends to be smaller than the model that minimizes BIC. For example, for a 13-run definitive screening design, the model that minimizes AICc will tend to be smaller than the model that minimizes BIC among the set of models with 6 or more parameters.

For more information on AICc and BIC, see Burnham and Anderson.¹

Interpretation

Use the condition number to compare models with different terms. A condition number of 1 indicates that the model terms are uncorrelated. Larger values indicate more multicollinearity.

While the consequences of a certain condition number depend on several conditions, values greater than 100 usually indicate the need for investigation. When terms in the model are multicollinear, the interpretation of the model is less straightforward than the interpretation of a model with uncorrelated terms. For more information, go to Multicollinearity in regression.

In these results, the condition number is 1 when the model has only 1 term. (The condition number is always 1 when the model has 1 continuous predictor.) None of the models have a condition number greater than 100, so the multicollinearity among the predictors is unlikely to have a large effect on the results.