Specify Hyperparameters Manually

It’s natural that you have some specific sets of hyperparameters to try first such as initial learning rate values and the number of leaves. Also, it’s also possible that you’ve already tried those sets before having Optuna find better sets of hyperparameters.

Optuna provides two APIs to support such cases:

  1. Passing those sets of hyperparameters and let Optuna evaluate them - enqueue_trial()

  2. Adding the results of those sets as completed Trials - add_trial()

First Scenario: Have Optuna evaluate your hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters but have not evaluated them yet and decided to use Optuna to find better sets of hyperparameters.

Optuna has optuna.study.Study.enqueue_trial() which lets you pass those sets of hyperparameters to Optuna and Optuna will evaluate them.

This section walks you through how to use this lit API with LightGBM.

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna

Define the objective function.

def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1),
        "bagging_freq": trial.suggest_int("bagging_freq", 0, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    # Add a callback for pruning.
    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
    gbm = lgb.train(
        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
    )

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

Then, construct Study for hyperparameter optimization.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())

Here, we get Optuna evaluate some sets with larger "bagging_fraq" value and the default values.

study.enqueue_trial(
    {
        "bagging_fraction": 1.0,
        "bagging_freq": 0,
        "min_child_samples": 20,
    }
)

study.enqueue_trial(
    {
        "bagging_fraction": 0.75,
        "bagging_freq": 5,
        "min_child_samples": 20,
    }
)

import logging
import sys

# Add stream handler of stdout to show the messages to see Optuna works expectedly.
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study.optimize(objective, n_trials=100, timeout=600)

Out:

/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:77: ExperimentalWarning:

enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/envs/stable/lib/python3.8/site-packages/optuna/study.py:857: ExperimentalWarning:

create_trial is experimental (supported from v2.0.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/envs/stable/lib/python3.8/site-packages/optuna/study.py:856: ExperimentalWarning:

add_trial is experimental (supported from v2.0.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:85: ExperimentalWarning:

enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.

Trial 0 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 1.0, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 0 with value: 0.972027972027972.
Trial 1 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.75, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 0 with value: 0.972027972027972.
Trial 2 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5324977509988726, 'bagging_freq': 2, 'min_child_samples': 81}. Best is trial 0 with value: 0.972027972027972.
Trial 3 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.42433189809225264, 'bagging_freq': 3, 'min_child_samples': 80}. Best is trial 0 with value: 0.972027972027972.
Trial 4 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6988023430080877, 'bagging_freq': 3, 'min_child_samples': 43}. Best is trial 0 with value: 0.972027972027972.
Trial 5 pruned. Trial was pruned at iteration 9.
Trial 6 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.4177009649308475, 'bagging_freq': 1, 'min_child_samples': 22}. Best is trial 6 with value: 0.9790209790209791.
Trial 7 pruned. Trial was pruned at iteration 9.
Trial 8 pruned. Trial was pruned at iteration 9.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 1.
Trial 11 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.9933180831306018, 'bagging_freq': 0, 'min_child_samples': 5}. Best is trial 6 with value: 0.9790209790209791.
Trial 12 pruned. Trial was pruned at iteration 1.
Trial 13 pruned. Trial was pruned at iteration 1.
Trial 14 pruned. Trial was pruned at iteration 1.
Trial 15 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7852752029178682, 'bagging_freq': 0, 'min_child_samples': 62}. Best is trial 6 with value: 0.9790209790209791.
Trial 16 pruned. Trial was pruned at iteration 1.
Trial 17 pruned. Trial was pruned at iteration 0.
Trial 18 pruned. Trial was pruned at iteration 0.
Trial 19 pruned. Trial was pruned at iteration 1.
Trial 20 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8622111164136266, 'bagging_freq': 2, 'min_child_samples': 30}. Best is trial 20 with value: 0.986013986013986.
Trial 21 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.8547973473979581, 'bagging_freq': 2, 'min_child_samples': 27}. Best is trial 21 with value: 0.993006993006993.
Trial 22 pruned. Trial was pruned at iteration 0.
Trial 23 pruned. Trial was pruned at iteration 6.
Trial 24 pruned. Trial was pruned at iteration 0.
Trial 25 pruned. Trial was pruned at iteration 0.
Trial 26 pruned. Trial was pruned at iteration 1.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 1.
Trial 30 pruned. Trial was pruned at iteration 84.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 pruned. Trial was pruned at iteration 0.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 1.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.47365653716017553, 'bagging_freq': 2, 'min_child_samples': 21}. Best is trial 21 with value: 0.993006993006993.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 0.
Trial 42 pruned. Trial was pruned at iteration 0.
Trial 43 pruned. Trial was pruned at iteration 0.
Trial 44 pruned. Trial was pruned at iteration 0.
Trial 45 pruned. Trial was pruned at iteration 0.
Trial 46 pruned. Trial was pruned at iteration 0.
Trial 47 pruned. Trial was pruned at iteration 82.
Trial 48 pruned. Trial was pruned at iteration 0.
Trial 49 pruned. Trial was pruned at iteration 8.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 0.
Trial 53 pruned. Trial was pruned at iteration 0.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 pruned. Trial was pruned at iteration 0.
Trial 56 pruned. Trial was pruned at iteration 0.
Trial 57 pruned. Trial was pruned at iteration 0.
Trial 58 pruned. Trial was pruned at iteration 0.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 0.
Trial 63 pruned. Trial was pruned at iteration 0.
Trial 64 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.500859328136199, 'bagging_freq': 4, 'min_child_samples': 14}. Best is trial 21 with value: 0.993006993006993.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 0.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5371949947821498, 'bagging_freq': 3, 'min_child_samples': 19}. Best is trial 21 with value: 0.993006993006993.
Trial 72 pruned. Trial was pruned at iteration 0.
Trial 73 pruned. Trial was pruned at iteration 0.
Trial 74 pruned. Trial was pruned at iteration 0.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 1.
Trial 77 pruned. Trial was pruned at iteration 0.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 0.
Trial 81 pruned. Trial was pruned at iteration 0.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 pruned. Trial was pruned at iteration 0.
Trial 84 pruned. Trial was pruned at iteration 3.
Trial 85 pruned. Trial was pruned at iteration 0.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 0.
Trial 93 pruned. Trial was pruned at iteration 0.
Trial 94 pruned. Trial was pruned at iteration 0.
Trial 95 pruned. Trial was pruned at iteration 4.
Trial 96 pruned. Trial was pruned at iteration 0.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 0.
Trial 99 pruned. Trial was pruned at iteration 0.

Second scenario: Have Optuna utilize already evaluated hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters and you have already evaluated them but the results are not desirable so that you are thinking of using Optuna.

Optuna has optuna.study.Study.add_trial() which lets you register those results to Optuna and then Optuna will sample hyperparameters taking them into account.

In this section, the objective is the same as the first scenario.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 1.0,
            "bagging_freq": 0,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntUniformDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntUniformDistribution(5, 100),
        },
        value=0.94,
    )
)
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 0.75,
            "bagging_freq": 5,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntUniformDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntUniformDistribution(5, 100),
        },
        value=0.95,
    )
)
study.optimize(objective, n_trials=100, timeout=600)

Out:

A new study created in memory with name: no-name-12b01560-f613-4c54-83bc-9a28be285201
/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:115: ExperimentalWarning:

create_trial is experimental (supported from v2.0.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:114: ExperimentalWarning:

add_trial is experimental (supported from v2.0.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:130: ExperimentalWarning:

create_trial is experimental (supported from v2.0.0). The interface can change in the future.

/home/docs/checkouts/readthedocs.org/user_builds/optuna/checkouts/stable/tutorial/20_recipes/008_specify_params.py:129: ExperimentalWarning:

add_trial is experimental (supported from v2.0.0). The interface can change in the future.

Trial 2 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7342730126166237, 'bagging_freq': 2, 'min_child_samples': 20}. Best is trial 2 with value: 0.972027972027972.
Trial 3 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8608096968406904, 'bagging_freq': 6, 'min_child_samples': 47}. Best is trial 3 with value: 0.986013986013986.
Trial 4 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6369854556903296, 'bagging_freq': 7, 'min_child_samples': 76}. Best is trial 3 with value: 0.986013986013986.
Trial 5 pruned. Trial was pruned at iteration 0.
Trial 6 pruned. Trial was pruned at iteration 0.
Trial 7 pruned. Trial was pruned at iteration 0.
Trial 8 pruned. Trial was pruned at iteration 0.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 0.
Trial 11 pruned. Trial was pruned at iteration 0.
Trial 12 pruned. Trial was pruned at iteration 0.
Trial 13 pruned. Trial was pruned at iteration 0.
Trial 14 pruned. Trial was pruned at iteration 0.
Trial 15 pruned. Trial was pruned at iteration 0.
Trial 16 pruned. Trial was pruned at iteration 0.
Trial 17 pruned. Trial was pruned at iteration 2.
Trial 18 pruned. Trial was pruned at iteration 0.
Trial 19 pruned. Trial was pruned at iteration 0.
Trial 20 pruned. Trial was pruned at iteration 0.
Trial 21 pruned. Trial was pruned at iteration 0.
Trial 22 pruned. Trial was pruned at iteration 0.
Trial 23 pruned. Trial was pruned at iteration 0.
Trial 24 pruned. Trial was pruned at iteration 0.
Trial 25 pruned. Trial was pruned at iteration 0.
Trial 26 pruned. Trial was pruned at iteration 0.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 0.
Trial 30 pruned. Trial was pruned at iteration 0.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7199545856474246, 'bagging_freq': 7, 'min_child_samples': 34}. Best is trial 3 with value: 0.986013986013986.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 0.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 pruned. Trial was pruned at iteration 0.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 0.
Trial 42 pruned. Trial was pruned at iteration 1.
Trial 43 pruned. Trial was pruned at iteration 0.
Trial 44 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7734034642443408, 'bagging_freq': 4, 'min_child_samples': 34}. Best is trial 3 with value: 0.986013986013986.
Trial 45 pruned. Trial was pruned at iteration 0.
Trial 46 pruned. Trial was pruned at iteration 0.
Trial 47 pruned. Trial was pruned at iteration 0.
Trial 48 pruned. Trial was pruned at iteration 0.
Trial 49 pruned. Trial was pruned at iteration 0.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 0.
Trial 53 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.8594247677154321, 'bagging_freq': 3, 'min_child_samples': 15}. Best is trial 53 with value: 0.993006993006993.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 pruned. Trial was pruned at iteration 0.
Trial 56 pruned. Trial was pruned at iteration 0.
Trial 57 pruned. Trial was pruned at iteration 0.
Trial 58 pruned. Trial was pruned at iteration 4.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 3.
Trial 63 pruned. Trial was pruned at iteration 0.
Trial 64 pruned. Trial was pruned at iteration 0.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 0.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 pruned. Trial was pruned at iteration 34.
Trial 72 pruned. Trial was pruned at iteration 0.
Trial 73 pruned. Trial was pruned at iteration 0.
Trial 74 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.77300206452511, 'bagging_freq': 6, 'min_child_samples': 37}. Best is trial 53 with value: 0.993006993006993.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 0.
Trial 77 pruned. Trial was pruned at iteration 0.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 0.
Trial 81 pruned. Trial was pruned at iteration 0.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6876631836859819, 'bagging_freq': 7, 'min_child_samples': 37}. Best is trial 53 with value: 0.993006993006993.
Trial 84 pruned. Trial was pruned at iteration 0.
Trial 85 pruned. Trial was pruned at iteration 0.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 0.
Trial 93 finished with value: 1.0 and parameters: {'bagging_fraction': 0.8145518073946173, 'bagging_freq': 5, 'min_child_samples': 24}. Best is trial 93 with value: 1.0.
Trial 94 pruned. Trial was pruned at iteration 0.
Trial 95 pruned. Trial was pruned at iteration 0.
Trial 96 pruned. Trial was pruned at iteration 0.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 0.
Trial 99 pruned. Trial was pruned at iteration 0.
Trial 100 pruned. Trial was pruned at iteration 1.
Trial 101 pruned. Trial was pruned at iteration 0.

Total running time of the script: ( 0 minutes 5.830 seconds)

Gallery generated by Sphinx-Gallery