Specify Hyperparameters Manually

It’s natural that you have some specific sets of hyperparameters to try first such as initial learning rate values and the number of leaves. Also, it’s possible that you’ve already tried those sets before having Optuna find better sets of hyperparameters.

Optuna provides two APIs to support such cases:

  1. Passing those sets of hyperparameters and let Optuna evaluate them - enqueue_trial()

  2. Adding the results of those sets as completed Trials - add_trial()

First Scenario: Have Optuna evaluate your hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters but have not evaluated them yet and decided to use Optuna to find better sets of hyperparameters.

Optuna has optuna.study.Study.enqueue_trial() which lets you pass those sets of hyperparameters to Optuna and Optuna will evaluate them.

This section walks you through how to use this lit API with LightGBM.

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna

Define the objective function.

def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1),
        "bagging_freq": trial.suggest_int("bagging_freq", 0, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    gbm = lgb.train(param, dtrain, valid_sets=[dvalid])

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

Then, construct Study for hyperparameter optimization.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())

Here, we get Optuna evaluate some sets with larger "bagging_fraq" value and the default values.

study.enqueue_trial(
    {
        "bagging_fraction": 1.0,
        "bagging_freq": 0,
        "min_child_samples": 20,
    }
)

study.enqueue_trial(
    {
        "bagging_fraction": 0.75,
        "bagging_freq": 5,
        "min_child_samples": 20,
    }
)

import logging
import sys

# Add stream handler of stdout to show the messages to see Optuna works expectedly.
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study.optimize(objective, n_trials=100, timeout=600)
Trial 0 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 1.0, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 0 with value: 0.958041958041958.
Trial 1 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.75, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 0 with value: 0.958041958041958.
Trial 2 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5098562293124154, 'bagging_freq': 4, 'min_child_samples': 6}. Best is trial 2 with value: 0.9790209790209791.
Trial 3 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6289140664272095, 'bagging_freq': 7, 'min_child_samples': 18}. Best is trial 2 with value: 0.9790209790209791.
Trial 4 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9826997990189604, 'bagging_freq': 5, 'min_child_samples': 46}. Best is trial 2 with value: 0.9790209790209791.
Trial 5 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5007834034057723, 'bagging_freq': 7, 'min_child_samples': 93}. Best is trial 2 with value: 0.9790209790209791.
Trial 6 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7212013928065674, 'bagging_freq': 4, 'min_child_samples': 89}. Best is trial 2 with value: 0.9790209790209791.
Trial 7 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5331635655799252, 'bagging_freq': 6, 'min_child_samples': 75}. Best is trial 2 with value: 0.9790209790209791.
Trial 8 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8842287762306096, 'bagging_freq': 5, 'min_child_samples': 99}. Best is trial 2 with value: 0.9790209790209791.
Trial 9 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7377943542672724, 'bagging_freq': 4, 'min_child_samples': 15}. Best is trial 2 with value: 0.9790209790209791.
Trial 10 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.4394301371420696, 'bagging_freq': 1, 'min_child_samples': 45}. Best is trial 2 with value: 0.9790209790209791.
Trial 11 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6366275848962873, 'bagging_freq': 2, 'min_child_samples': 5}. Best is trial 2 with value: 0.9790209790209791.
Trial 12 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.8003939004596, 'bagging_freq': 3, 'min_child_samples': 32}. Best is trial 2 with value: 0.9790209790209791.
Trial 13 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6079453990987993, 'bagging_freq': 3, 'min_child_samples': 9}. Best is trial 2 with value: 0.9790209790209791.
Trial 14 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8493119782590492, 'bagging_freq': 4, 'min_child_samples': 63}. Best is trial 2 with value: 0.9790209790209791.
Trial 15 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5609375885252096, 'bagging_freq': 2, 'min_child_samples': 33}. Best is trial 2 with value: 0.9790209790209791.
Trial 16 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4040195223280127, 'bagging_freq': 4, 'min_child_samples': 33}. Best is trial 2 with value: 0.9790209790209791.
Trial 17 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6746369168923052, 'bagging_freq': 2, 'min_child_samples': 59}. Best is trial 2 with value: 0.9790209790209791.
Trial 18 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.783441194483997, 'bagging_freq': 6, 'min_child_samples': 12}. Best is trial 2 with value: 0.9790209790209791.
Trial 19 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4764632232916598, 'bagging_freq': 3, 'min_child_samples': 29}. Best is trial 2 with value: 0.9790209790209791.
Trial 20 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5731481007160912, 'bagging_freq': 6, 'min_child_samples': 44}. Best is trial 2 with value: 0.9790209790209791.
Trial 21 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5619609659539333, 'bagging_freq': 2, 'min_child_samples': 24}. Best is trial 2 with value: 0.9790209790209791.
Trial 22 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6681504492798565, 'bagging_freq': 1, 'min_child_samples': 12}. Best is trial 2 with value: 0.9790209790209791.
Trial 23 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.5107408198898729, 'bagging_freq': 3, 'min_child_samples': 5}. Best is trial 2 with value: 0.9790209790209791.
Trial 24 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5819312853138565, 'bagging_freq': 1, 'min_child_samples': 37}. Best is trial 2 with value: 0.9790209790209791.
Trial 25 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.721706635709866, 'bagging_freq': 4, 'min_child_samples': 26}. Best is trial 2 with value: 0.9790209790209791.
Trial 26 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.443588953789973, 'bagging_freq': 2, 'min_child_samples': 14}. Best is trial 2 with value: 0.9790209790209791.
Trial 27 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.9102908730171952, 'bagging_freq': 5, 'min_child_samples': 40}. Best is trial 2 with value: 0.9790209790209791.
Trial 28 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5432250039100909, 'bagging_freq': 0, 'min_child_samples': 24}. Best is trial 2 with value: 0.9790209790209791.
Trial 29 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6744111346367622, 'bagging_freq': 0, 'min_child_samples': 17}. Best is trial 29 with value: 0.986013986013986.
Trial 30 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7982771478431976, 'bagging_freq': 0, 'min_child_samples': 16}. Best is trial 29 with value: 0.986013986013986.
Trial 31 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6774854988969374, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 29 with value: 0.986013986013986.
Trial 32 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.741852990268124, 'bagging_freq': 1, 'min_child_samples': 20}. Best is trial 29 with value: 0.986013986013986.
Trial 33 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.6082546418686429, 'bagging_freq': 4, 'min_child_samples': 8}. Best is trial 29 with value: 0.986013986013986.
Trial 34 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6497171513310891, 'bagging_freq': 5, 'min_child_samples': 52}. Best is trial 29 with value: 0.986013986013986.
Trial 35 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6999603461506562, 'bagging_freq': 3, 'min_child_samples': 16}. Best is trial 29 with value: 0.986013986013986.
Trial 36 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.4854727933461422, 'bagging_freq': 2, 'min_child_samples': 25}. Best is trial 36 with value: 0.993006993006993.
Trial 37 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.48174704327087664, 'bagging_freq': 4, 'min_child_samples': 24}. Best is trial 36 with value: 0.993006993006993.
Trial 38 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4709439059109406, 'bagging_freq': 5, 'min_child_samples': 21}. Best is trial 36 with value: 0.993006993006993.
Trial 39 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5045254890398418, 'bagging_freq': 0, 'min_child_samples': 78}. Best is trial 36 with value: 0.993006993006993.
Trial 40 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.40261099391630106, 'bagging_freq': 1, 'min_child_samples': 27}. Best is trial 36 with value: 0.993006993006993.
Trial 41 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.45430232686310335, 'bagging_freq': 4, 'min_child_samples': 13}. Best is trial 36 with value: 0.993006993006993.
Trial 42 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.5232642744838818, 'bagging_freq': 5, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 43 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5270986707791848, 'bagging_freq': 6, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 44 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4827589267766077, 'bagging_freq': 5, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 45 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5967618114153374, 'bagging_freq': 4, 'min_child_samples': 19}. Best is trial 36 with value: 0.993006993006993.
Trial 46 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.43188800325937926, 'bagging_freq': 5, 'min_child_samples': 10}. Best is trial 36 with value: 0.993006993006993.
Trial 47 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5311032653959807, 'bagging_freq': 7, 'min_child_samples': 30}. Best is trial 36 with value: 0.993006993006993.
Trial 48 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.48613547141818647, 'bagging_freq': 3, 'min_child_samples': 36}. Best is trial 36 with value: 0.993006993006993.
Trial 49 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.424990671883119, 'bagging_freq': 6, 'min_child_samples': 22}. Best is trial 36 with value: 0.993006993006993.
Trial 50 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5501396776825149, 'bagging_freq': 4, 'min_child_samples': 17}. Best is trial 36 with value: 0.993006993006993.
Trial 51 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8374248121191905, 'bagging_freq': 4, 'min_child_samples': 9}. Best is trial 36 with value: 0.993006993006993.
Trial 52 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7624863838986972, 'bagging_freq': 5, 'min_child_samples': 13}. Best is trial 36 with value: 0.993006993006993.
Trial 53 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.628240250094687, 'bagging_freq': 3, 'min_child_samples': 17}. Best is trial 36 with value: 0.993006993006993.
Trial 54 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9573308925770683, 'bagging_freq': 4, 'min_child_samples': 27}. Best is trial 36 with value: 0.993006993006993.
Trial 55 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5131050752593587, 'bagging_freq': 2, 'min_child_samples': 68}. Best is trial 36 with value: 0.993006993006993.
Trial 56 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.47719053796384775, 'bagging_freq': 4, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 57 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.4569195948954877, 'bagging_freq': 5, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 58 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4962396208841944, 'bagging_freq': 3, 'min_child_samples': 11}. Best is trial 36 with value: 0.993006993006993.
Trial 59 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.41680590463150036, 'bagging_freq': 4, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 60 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4228449500206325, 'bagging_freq': 5, 'min_child_samples': 23}. Best is trial 36 with value: 0.993006993006993.
Trial 61 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.4608364813015804, 'bagging_freq': 4, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 62 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4646569363731024, 'bagging_freq': 4, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 63 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.4469450293096247, 'bagging_freq': 4, 'min_child_samples': 13}. Best is trial 36 with value: 0.993006993006993.
Trial 64 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.40779031832599727, 'bagging_freq': 3, 'min_child_samples': 10}. Best is trial 36 with value: 0.993006993006993.
Trial 65 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.5166760583168062, 'bagging_freq': 4, 'min_child_samples': 15}. Best is trial 36 with value: 0.993006993006993.
Trial 66 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.4887583250263905, 'bagging_freq': 3, 'min_child_samples': 7}. Best is trial 36 with value: 0.993006993006993.
Trial 67 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.42065158143606224, 'bagging_freq': 5, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 68 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4652322665272651, 'bagging_freq': 2, 'min_child_samples': 18}. Best is trial 36 with value: 0.993006993006993.
Trial 69 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4436472843178725, 'bagging_freq': 4, 'min_child_samples': 25}. Best is trial 36 with value: 0.993006993006993.
Trial 70 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.570442722930325, 'bagging_freq': 6, 'min_child_samples': 50}. Best is trial 36 with value: 0.993006993006993.
Trial 71 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5251943478487521, 'bagging_freq': 4, 'min_child_samples': 15}. Best is trial 36 with value: 0.993006993006993.
Trial 72 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.506850683824722, 'bagging_freq': 4, 'min_child_samples': 11}. Best is trial 36 with value: 0.993006993006993.
Trial 73 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4745094697385822, 'bagging_freq': 3, 'min_child_samples': 16}. Best is trial 36 with value: 0.993006993006993.
Trial 74 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5481763252757128, 'bagging_freq': 4, 'min_child_samples': 31}. Best is trial 36 with value: 0.993006993006993.
Trial 75 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.49835637380740194, 'bagging_freq': 5, 'min_child_samples': 21}. Best is trial 36 with value: 0.993006993006993.
Trial 76 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5188883545372349, 'bagging_freq': 4, 'min_child_samples': 14}. Best is trial 36 with value: 0.993006993006993.
Trial 77 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4428700361715301, 'bagging_freq': 1, 'min_child_samples': 8}. Best is trial 36 with value: 0.993006993006993.
Trial 78 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5876237736491057, 'bagging_freq': 3, 'min_child_samples': 19}. Best is trial 36 with value: 0.993006993006993.
Trial 79 finished with value: 0.9230769230769231 and parameters: {'bagging_fraction': 0.4167411355663741, 'bagging_freq': 5, 'min_child_samples': 99}. Best is trial 36 with value: 0.993006993006993.
Trial 80 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5452323259485715, 'bagging_freq': 0, 'min_child_samples': 34}. Best is trial 36 with value: 0.993006993006993.
Trial 81 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.42105440925901755, 'bagging_freq': 5, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 82 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.47937207950318167, 'bagging_freq': 4, 'min_child_samples': 11}. Best is trial 36 with value: 0.993006993006993.
Trial 83 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.43181679091741054, 'bagging_freq': 5, 'min_child_samples': 7}. Best is trial 36 with value: 0.993006993006993.
Trial 84 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.40115712786214835, 'bagging_freq': 4, 'min_child_samples': 13}. Best is trial 36 with value: 0.993006993006993.
Trial 85 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.46363025325361196, 'bagging_freq': 5, 'min_child_samples': 10}. Best is trial 36 with value: 0.993006993006993.
Trial 86 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4554613203689963, 'bagging_freq': 4, 'min_child_samples': 5}. Best is trial 36 with value: 0.993006993006993.
Trial 87 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.49153932210312495, 'bagging_freq': 5, 'min_child_samples': 15}. Best is trial 36 with value: 0.993006993006993.
Trial 88 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.4369536642746895, 'bagging_freq': 4, 'min_child_samples': 7}. Best is trial 36 with value: 0.993006993006993.
Trial 89 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.5616834274473028, 'bagging_freq': 6, 'min_child_samples': 90}. Best is trial 36 with value: 0.993006993006993.
Trial 90 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7011436890514136, 'bagging_freq': 3, 'min_child_samples': 39}. Best is trial 36 with value: 0.993006993006993.
Trial 91 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5756252247400268, 'bagging_freq': 6, 'min_child_samples': 51}. Best is trial 36 with value: 0.993006993006993.
Trial 92 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6127945779565162, 'bagging_freq': 6, 'min_child_samples': 44}. Best is trial 36 with value: 0.993006993006993.
Trial 93 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5325495504375728, 'bagging_freq': 7, 'min_child_samples': 57}. Best is trial 36 with value: 0.993006993006993.
Trial 94 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5173282665215031, 'bagging_freq': 6, 'min_child_samples': 81}. Best is trial 36 with value: 0.993006993006993.
Trial 95 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.4783236238694384, 'bagging_freq': 1, 'min_child_samples': 47}. Best is trial 36 with value: 0.993006993006993.
Trial 96 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.656327626055789, 'bagging_freq': 7, 'min_child_samples': 28}. Best is trial 36 with value: 0.993006993006993.
Trial 97 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5009898787338668, 'bagging_freq': 6, 'min_child_samples': 70}. Best is trial 36 with value: 0.993006993006993.
Trial 98 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.44765923395814883, 'bagging_freq': 5, 'min_child_samples': 9}. Best is trial 36 with value: 0.993006993006993.
Trial 99 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5582674177295864, 'bagging_freq': 4, 'min_child_samples': 23}. Best is trial 36 with value: 0.993006993006993.

Second scenario: Have Optuna utilize already evaluated hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters and you have already evaluated them but the results are not desirable so that you are thinking of using Optuna.

Optuna has optuna.study.Study.add_trial() which lets you register those results to Optuna and then Optuna will sample hyperparameters taking them into account.

In this section, the objective is the same as the first scenario.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 1.0,
            "bagging_freq": 0,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.94,
    )
)
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 0.75,
            "bagging_freq": 5,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.95,
    )
)
study.optimize(objective, n_trials=100, timeout=600)
A new study created in memory with name: no-name-025e54bd-192f-4529-b6cc-ccb0b530aada
Trial 2 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5084764311156726, 'bagging_freq': 5, 'min_child_samples': 61}. Best is trial 2 with value: 0.9790209790209791.
Trial 3 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7643612368681749, 'bagging_freq': 3, 'min_child_samples': 6}. Best is trial 2 with value: 0.9790209790209791.
Trial 4 finished with value: 0.9300699300699301 and parameters: {'bagging_fraction': 0.9704349619059731, 'bagging_freq': 0, 'min_child_samples': 95}. Best is trial 2 with value: 0.9790209790209791.
Trial 5 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8949122643572791, 'bagging_freq': 5, 'min_child_samples': 57}. Best is trial 2 with value: 0.9790209790209791.
Trial 6 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7510658102384844, 'bagging_freq': 5, 'min_child_samples': 79}. Best is trial 6 with value: 0.986013986013986.
Trial 7 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7428477264596351, 'bagging_freq': 7, 'min_child_samples': 62}. Best is trial 6 with value: 0.986013986013986.
Trial 8 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9893319900029593, 'bagging_freq': 2, 'min_child_samples': 33}. Best is trial 6 with value: 0.986013986013986.
Trial 9 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6149103674256602, 'bagging_freq': 3, 'min_child_samples': 42}. Best is trial 6 with value: 0.986013986013986.
Trial 10 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4524941105002545, 'bagging_freq': 7, 'min_child_samples': 88}. Best is trial 6 with value: 0.986013986013986.
Trial 11 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.561167417156565, 'bagging_freq': 5, 'min_child_samples': 75}. Best is trial 6 with value: 0.986013986013986.
Trial 12 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.41110000321589873, 'bagging_freq': 6, 'min_child_samples': 73}. Best is trial 6 with value: 0.986013986013986.
Trial 13 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.616016686975278, 'bagging_freq': 4, 'min_child_samples': 75}. Best is trial 6 with value: 0.986013986013986.
Trial 14 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8439281522640828, 'bagging_freq': 4, 'min_child_samples': 47}. Best is trial 6 with value: 0.986013986013986.
Trial 15 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5106121834664704, 'bagging_freq': 6, 'min_child_samples': 65}. Best is trial 6 with value: 0.986013986013986.
Trial 16 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6556942210205733, 'bagging_freq': 2, 'min_child_samples': 86}. Best is trial 6 with value: 0.986013986013986.
Trial 17 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8314828591535726, 'bagging_freq': 6, 'min_child_samples': 97}. Best is trial 6 with value: 0.986013986013986.
Trial 18 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.5257910936375729, 'bagging_freq': 4, 'min_child_samples': 83}. Best is trial 6 with value: 0.986013986013986.
Trial 19 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.679054397745491, 'bagging_freq': 2, 'min_child_samples': 52}. Best is trial 6 with value: 0.986013986013986.
Trial 20 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8085352152437245, 'bagging_freq': 5, 'min_child_samples': 67}. Best is trial 6 with value: 0.986013986013986.
Trial 21 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7968815627469007, 'bagging_freq': 5, 'min_child_samples': 67}. Best is trial 6 with value: 0.986013986013986.
Trial 22 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8196655643291503, 'bagging_freq': 6, 'min_child_samples': 70}. Best is trial 6 with value: 0.986013986013986.
Trial 23 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.8770834371969851, 'bagging_freq': 5, 'min_child_samples': 80}. Best is trial 6 with value: 0.986013986013986.
Trial 24 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7803654706841054, 'bagging_freq': 4, 'min_child_samples': 66}. Best is trial 6 with value: 0.986013986013986.
Trial 25 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.9222631273292298, 'bagging_freq': 7, 'min_child_samples': 53}. Best is trial 6 with value: 0.986013986013986.
Trial 26 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7185480151096512, 'bagging_freq': 3, 'min_child_samples': 90}. Best is trial 6 with value: 0.986013986013986.
Trial 27 finished with value: 1.0 and parameters: {'bagging_fraction': 0.8074468041608422, 'bagging_freq': 6, 'min_child_samples': 79}. Best is trial 27 with value: 1.0.
Trial 28 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9301622427058104, 'bagging_freq': 6, 'min_child_samples': 92}. Best is trial 27 with value: 1.0.
Trial 29 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6994911713143596, 'bagging_freq': 7, 'min_child_samples': 81}. Best is trial 27 with value: 1.0.
Trial 30 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8628712320551624, 'bagging_freq': 6, 'min_child_samples': 78}. Best is trial 27 with value: 1.0.
Trial 31 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7968364718048139, 'bagging_freq': 5, 'min_child_samples': 69}. Best is trial 27 with value: 1.0.
Trial 32 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8098149405749168, 'bagging_freq': 5, 'min_child_samples': 60}. Best is trial 27 with value: 1.0.
Trial 33 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7355370130090737, 'bagging_freq': 5, 'min_child_samples': 39}. Best is trial 27 with value: 1.0.
Trial 34 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.768132301136963, 'bagging_freq': 4, 'min_child_samples': 70}. Best is trial 27 with value: 1.0.
Trial 35 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7790584771338693, 'bagging_freq': 6, 'min_child_samples': 28}. Best is trial 27 with value: 1.0.
Trial 36 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9204095644565407, 'bagging_freq': 5, 'min_child_samples': 99}. Best is trial 27 with value: 1.0.
Trial 37 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6674951509711528, 'bagging_freq': 0, 'min_child_samples': 58}. Best is trial 27 with value: 1.0.
Trial 38 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.847096872730974, 'bagging_freq': 4, 'min_child_samples': 84}. Best is trial 27 with value: 1.0.
Trial 39 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7386761498989438, 'bagging_freq': 5, 'min_child_samples': 14}. Best is trial 27 with value: 1.0.
Trial 40 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.8920898132760461, 'bagging_freq': 7, 'min_child_samples': 65}. Best is trial 27 with value: 1.0.
Trial 41 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9541689319386009, 'bagging_freq': 7, 'min_child_samples': 54}. Best is trial 27 with value: 1.0.
Trial 42 finished with value: 0.9300699300699301 and parameters: {'bagging_fraction': 0.9316643464823351, 'bagging_freq': 7, 'min_child_samples': 55}. Best is trial 27 with value: 1.0.
Trial 43 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8109290688349264, 'bagging_freq': 6, 'min_child_samples': 49}. Best is trial 27 with value: 1.0.
Trial 44 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.9972703406397875, 'bagging_freq': 6, 'min_child_samples': 76}. Best is trial 27 with value: 1.0.
Trial 45 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7583235105701099, 'bagging_freq': 7, 'min_child_samples': 63}. Best is trial 27 with value: 1.0.
Trial 46 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.9062825067294289, 'bagging_freq': 5, 'min_child_samples': 44}. Best is trial 27 with value: 1.0.
Trial 47 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8680837076520928, 'bagging_freq': 3, 'min_child_samples': 70}. Best is trial 27 with value: 1.0.
Trial 48 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7135133256595502, 'bagging_freq': 6, 'min_child_samples': 37}. Best is trial 27 with value: 1.0.
Trial 49 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6285878679324521, 'bagging_freq': 5, 'min_child_samples': 93}. Best is trial 27 with value: 1.0.
Trial 50 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9576213809255724, 'bagging_freq': 7, 'min_child_samples': 74}. Best is trial 27 with value: 1.0.
Trial 51 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6810225877347517, 'bagging_freq': 7, 'min_child_samples': 81}. Best is trial 27 with value: 1.0.
Trial 52 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.6993767875634771, 'bagging_freq': 7, 'min_child_samples': 85}. Best is trial 27 with value: 1.0.
Trial 53 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6407011565906021, 'bagging_freq': 6, 'min_child_samples': 78}. Best is trial 27 with value: 1.0.
Trial 54 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.5761770700646891, 'bagging_freq': 6, 'min_child_samples': 88}. Best is trial 27 with value: 1.0.
Trial 55 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.7914234559938332, 'bagging_freq': 7, 'min_child_samples': 67}. Best is trial 27 with value: 1.0.
Trial 56 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7952473626592433, 'bagging_freq': 7, 'min_child_samples': 60}. Best is trial 27 with value: 1.0.
Trial 57 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8422488788759421, 'bagging_freq': 4, 'min_child_samples': 67}. Best is trial 27 with value: 1.0.
Trial 58 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8241504213512579, 'bagging_freq': 6, 'min_child_samples': 51}. Best is trial 27 with value: 1.0.
Trial 59 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7608773433454726, 'bagging_freq': 5, 'min_child_samples': 72}. Best is trial 27 with value: 1.0.
Trial 60 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7884741835181668, 'bagging_freq': 7, 'min_child_samples': 63}. Best is trial 27 with value: 1.0.
Trial 61 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7187444577972363, 'bagging_freq': 7, 'min_child_samples': 78}. Best is trial 27 with value: 1.0.
Trial 62 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7416650712438764, 'bagging_freq': 6, 'min_child_samples': 73}. Best is trial 27 with value: 1.0.
Trial 63 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6950335355953073, 'bagging_freq': 7, 'min_child_samples': 82}. Best is trial 27 with value: 1.0.
Trial 64 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8556640085700947, 'bagging_freq': 6, 'min_child_samples': 88}. Best is trial 27 with value: 1.0.
Trial 65 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8062737258393243, 'bagging_freq': 1, 'min_child_samples': 66}. Best is trial 27 with value: 1.0.
Trial 66 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7780062961063208, 'bagging_freq': 6, 'min_child_samples': 56}. Best is trial 27 with value: 1.0.
Trial 67 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.8328448223395156, 'bagging_freq': 4, 'min_child_samples': 76}. Best is trial 27 with value: 1.0.
Trial 68 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8769674354740403, 'bagging_freq': 5, 'min_child_samples': 68}. Best is trial 27 with value: 1.0.
Trial 69 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7464264598168607, 'bagging_freq': 7, 'min_child_samples': 58}. Best is trial 27 with value: 1.0.
Trial 70 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7238457212221101, 'bagging_freq': 5, 'min_child_samples': 79}. Best is trial 27 with value: 1.0.
Trial 71 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7339329512603755, 'bagging_freq': 5, 'min_child_samples': 18}. Best is trial 27 with value: 1.0.
Trial 72 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7702862536553111, 'bagging_freq': 5, 'min_child_samples': 72}. Best is trial 27 with value: 1.0.
Trial 73 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.751674637917608, 'bagging_freq': 6, 'min_child_samples': 17}. Best is trial 27 with value: 1.0.
Trial 74 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7941503812574684, 'bagging_freq': 4, 'min_child_samples': 9}. Best is trial 27 with value: 1.0.
Trial 75 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6907565027853492, 'bagging_freq': 7, 'min_child_samples': 30}. Best is trial 27 with value: 1.0.
Trial 76 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.6526047180653336, 'bagging_freq': 5, 'min_child_samples': 46}. Best is trial 27 with value: 1.0.
Trial 77 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.8168060108889694, 'bagging_freq': 4, 'min_child_samples': 62}. Best is trial 27 with value: 1.0.
Trial 78 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.46933503224067435, 'bagging_freq': 7, 'min_child_samples': 8}. Best is trial 27 with value: 1.0.
Trial 79 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8907538107437984, 'bagging_freq': 6, 'min_child_samples': 25}. Best is trial 27 with value: 1.0.
Trial 80 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9793703764816245, 'bagging_freq': 3, 'min_child_samples': 95}. Best is trial 27 with value: 1.0.
Trial 81 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6689731633111342, 'bagging_freq': 7, 'min_child_samples': 82}. Best is trial 27 with value: 1.0.
Trial 82 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.7093893953383731, 'bagging_freq': 7, 'min_child_samples': 80}. Best is trial 27 with value: 1.0.
Trial 83 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7287908723928403, 'bagging_freq': 7, 'min_child_samples': 90}. Best is trial 27 with value: 1.0.
Trial 84 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6842202639742547, 'bagging_freq': 7, 'min_child_samples': 85}. Best is trial 27 with value: 1.0.
Trial 85 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.7660080138305918, 'bagging_freq': 6, 'min_child_samples': 82}. Best is trial 27 with value: 1.0.
Trial 86 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5993728666397979, 'bagging_freq': 5, 'min_child_samples': 76}. Best is trial 27 with value: 1.0.
Trial 87 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7069941048303439, 'bagging_freq': 5, 'min_child_samples': 40}. Best is trial 27 with value: 1.0.
Trial 88 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6591205363242839, 'bagging_freq': 6, 'min_child_samples': 35}. Best is trial 27 with value: 1.0.
Trial 89 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8318897349032944, 'bagging_freq': 7, 'min_child_samples': 71}. Best is trial 27 with value: 1.0.
Trial 90 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8020641458370406, 'bagging_freq': 7, 'min_child_samples': 51}. Best is trial 27 with value: 1.0.
Trial 91 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8436417873167251, 'bagging_freq': 4, 'min_child_samples': 60}. Best is trial 27 with value: 1.0.
Trial 92 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.781478005443583, 'bagging_freq': 4, 'min_child_samples': 68}. Best is trial 27 with value: 1.0.
Trial 93 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7550522877105561, 'bagging_freq': 4, 'min_child_samples': 65}. Best is trial 27 with value: 1.0.
Trial 94 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.851535001498836, 'bagging_freq': 5, 'min_child_samples': 75}. Best is trial 27 with value: 1.0.
Trial 95 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.9134561788276594, 'bagging_freq': 6, 'min_child_samples': 68}. Best is trial 27 with value: 1.0.
Trial 96 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9389334749133473, 'bagging_freq': 5, 'min_child_samples': 54}. Best is trial 27 with value: 1.0.
Trial 97 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8209357659674256, 'bagging_freq': 4, 'min_child_samples': 81}. Best is trial 27 with value: 1.0.
Trial 98 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6757926413643746, 'bagging_freq': 3, 'min_child_samples': 78}. Best is trial 27 with value: 1.0.
Trial 99 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8719819190584226, 'bagging_freq': 6, 'min_child_samples': 66}. Best is trial 27 with value: 1.0.
Trial 100 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.8410450069105995, 'bagging_freq': 5, 'min_child_samples': 88}. Best is trial 27 with value: 1.0.
Trial 101 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7415196180876936, 'bagging_freq': 6, 'min_child_samples': 73}. Best is trial 27 with value: 1.0.

Total running time of the script: (0 minutes 15.782 seconds)

Gallery generated by Sphinx-Gallery