Using stepwise regression and best subsets regression

Problems with stepwise regression

Exercise caution when using variable selection procedures such as best subsets and stepwise regression. One problem is that these procedures cannot consider special knowledge the analyst might have about the data. The procedure cannot consider the practical importance of any of the predictors.

A related problem to the procedure's inability to consider special knowledge is that when two predictors are highly correlated, the procedure can select only one of the two predictors even though either can be important. For example, the procedure can remove a predictor that is cheap and easy to measure in favor of a correlated predictor that is difficult and expensive to measure. The analyst would have to use their knowledge of the data to make judgements about criteria that the procedure cannot consider.

Another problem with stepwise procedures is that the different models can optimize different criteria. For example, the model with the highest adjusted R² value will not necessarily be the model with the highest test R² value. The analyst has to consider the different criteria to select a final model.

To ensure that your model doesn't just fit one specific data set, you should verify the model found by the selection procedure on a new set of data. You can also take the original data set, randomly divide it into two parts, use one part to select a model, and then verify the fit on the second part. This procedure helps ensure that the model you select will apply to other data sets. Go to the section on stepwise procedures with automatic validation to learn about commands that can partition your data automatically and calculate validation statistics.

Stepwise procedures

All of the analyses that include automatic stepwise procedures in Minitab include the following procedures. The following methods let you quickly evaluate a high number of different models in terms of their model summary statistics for the data that you use to build the model.

• Standard stepwise regression adds or removes a predictor for each step. Minitab stops when all variables not in the model have p-values that are greater than the specified alpha-to-enter value and when all variables in the model have p-values that are less than or equal to the specified alpha-to-remove value.
• The forward information criteria procedure adds the term with the lowest p-value to the model at each step. Additional terms can enter the model in 1 step if the settings for the analysis allow consideration of non-hierarchical terms but require each model to be hierarchical. Minitab calculates the information criteria for each step. In most cases, the procedure continues until one of the following conditions occurs:
  • The procedure does not find a new minimum of the criterion for 8 consecutive steps.
  • The procedure fits the full model.
  • The procedure fits a model that leaves 1 degree of freedom for error.
  If you specify settings for the procedure that require a hierarchical model at each step and allow only one term to enter at a time, then the procedure continues until it either fits the full model or fits a model that leaves 1 degree of freedom for error. Minitab displays the results of the analysis for the model with the minimum value of the selected information criterion, either AICc or BIC.
• Forward selection starts with an empty model or a model with terms that you specify. Then, Minitab adds the most significant term for each step. Minitab stops when all variables not in the model have p-values that are greater than the specified alpha-to-enter value.
• Backward elimination starts with all predictors in the model and Minitab removes the least significant variable for each step. Minitab stops when all variables in the model have p-values that are less than or equal to the specified alpha-to-remove value.

Stepwise regression procedures with automatic validation

The stepwise procedure that Minitab can automatically perform with a test data set is called forward selection with validation with a test data set. In this procedure, the initial model is empty or includes model terms that you specifically select. Then, Minitab adds the next potential term with the smallest p-value at each step. Minitab calculates the test R² for the model at each step as the R² value for the model on the test data set. The model results that Minitab presents are for the model with the maximum value of the test R² value.

For Fit Regression Model, you can choose a second validation technique to perform with stepwise selection called forward selection with k-fold cross-validation. In k-fold cross-validation, Minitab divides the dataset into k subsets. These subsets are called folds. Most often, validation uses 10 folds, but other numbers are possible. The folds have as close to equal numbers of observations as possible. Minitab performs forward selection k times. For each forward selection, k–1 folds are the training data set and the last fold is the test data set. As in other forward selection procedures, the initial model is empty or includes model terms that you specifically select. Then, Minitab adds the next potential term with the smallest p-value at each step. For each step, Minitab calculates the k-fold stepwise R² value by combining the information from the different stepwise selection procedures.

Hierarchy

A hierarchical model is a model where for each term in the model, all lower order terms contained in it must also be in the model. For example, suppose there is a model with four factors: A, B, C, and D. If the term A * B * C is in the model then the terms A, B, C, A*B, A*C, and B*C must also be in the model, though any terms with D do not have to be in the model.

The terms that enter or leave a model at a step depend on the specifications for hierarchy. By default, Minitab Statistical Software requires a hierarchical model at each step, requires hierarchy for all terms, and allows only one term to enter the model at each step. These settings limit the terms that Minitab considers at each step. For example, a two-way interaction cannot enter the model unless both of the lower-order terms in the interaction are already in the model. You can adjust these settings by clicking Hierarchy when you select a stepwise method.

Comparison of best subsets regression and stepwise regression

Best subsets is an analysis in Minitab Statistical Software. Stepwise regression is an option in several analyses. Both of these automated model selection techniques provide information about the fit of several different models. From the different models, you can identify any models that deserve further exploration.