The champion/challenger framework in Minitab Model Ops allows you to use the same production data to compare competing models. Each deployment has a single champion model that is used to produce the predictions for the production environment. The deployment also allows up to two challenger models that can run the same data and provide performance metrics to your team. If and when a challenger model becomes the best model, you can easily promote it to be the new champion model.

Champion and challenger details

Champion model
The champion model is the best predictive model chosen from a group of models that meets your champion criteria. Upon deployment, the engineer selects the initial champion and sets the response variable and initial predictor variables.
Challenger models
Challenger models are competing models. Each challenger model must have the same response variable as the champion model. However, challenger models can have different predictors. The challenger model that generates the best results can become the new champion model if you decide to promote it.
All models in the deployment must use the same response variable. Challenger models may share predictors and/or have different predictors. A shared predictor must have the same name and be the same type (numeric or text) across all models. Use the following information for the response and predictor variables from the Models tab in the deployment details area to verify that the correct variables are in any data sets you prepare to make predictions, monitor drift, and monitor stability.
Name
Displays the name of the variable. The production and baseline data sets must use identical names as the variables in the deployment.
Type
Displays the variable type, either continuous or categorical. The variables in the production and baseline data sets must use the same variable type as the model.
Data Type
Displays the data type, either text or numeric. Continuous variables will always have a numeric data type and categorical variables can use text or numeric values. The variables in the production and baseline data sets must use the same data type as the model.
Classes
Displays the class values of the variable. Only categorical variables have classes. Use the class values to make predictions.
Models
Displays which models contain the predictor variables.

Add challenger models to a deployment

The Models tab contains a list of the deployment’s competing models. Use the following steps to add up to two challenger models.

  1. Open the Deployments page.
  2. Select a deployment name to open the deployment details area.
  3. Open the Models tab.
  4. Select Add Model.
  5. Choose an available model from the repository. Challengers must have the same response variable and cannot be a challenger in any other deployment.

Upload baseline data for a challenger model

For best results, it is good practice to upload the training data as the initial baseline data. Minitab Model Ops monitors drift for the champion model of the deployment. After a challenger is promoted to champion, Minitab Model Ops uses the baseline data for the Drift report.
  1. Open the Deployments page.
  2. Select a deployment name to open the deployment details area.
  3. Open the Models tab.
  4. Open the settings of the challenger model .
  5. Select Upload baseline data or Replace baseline data.

Evaluate the performance of challenger models

Use the model performance metrics and graphs on the Performance tab to evaluate the champion and challenger models. Only the champion model is live at any given time. All the prediction requests are used by the current champion model in real time. Once a day, the same prediction requests are replayed against the challengers for analytical purposes.

By assessing the challenger models with the same production data as the champion model, you can easily compare the available models in the deployment.

Promote a challenger model

Review the performance metrics to determine whether one of the challenger models has better performance than the champion model. If a challenger model has better performance than the champion model, you can promote it to be the new champion model.

  1. Open the Deployments page.
  2. Select a deployment name to open the deployment details area.
  3. Open the Models tab.
  4. Open the settings of the challenger model .
  5. Select Promote to Champion.