Optuna: A hyperparameter optimization framework

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.

Key Features

Optuna has modern functionalities as follows:

Lightweight, versatile, and platform agnostic architecture
- Handle a wide variety of tasks with a simple installation that has few requirements.
Pythonic search spaces
- Define search spaces using familiar Python syntax including conditionals and loops.
Efficient optimization algorithms
- Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials.
Easy parallelization
- Scale studies to tens or hundreds or workers with little or no changes to the code.
Quick visualization
- Inspect optimization histories from a variety of plotting functions.

Basic Concepts

We use the terms study and trial as follows:

Study: optimization based on an objective function
Trial: a single execution of the objective function

Please refer to sample code below. The goal of a study is to find out the optimal set of hyperparameter values (e.g., classifier and svm_c) through multiple trials (e.g., n_trials=100). Optuna is a framework designed for the automation and the acceleration of the optimization studies.

import ...

# Define an objective function to be minimized.
def objective(trial):

    # Invoke suggest methods of a Trial object to generate hyperparameters.
    regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
    if regressor_name == 'SVR':
        svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
        regressor_obj = sklearn.svm.SVR(C=svr_c)
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
        regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)

    X, y = sklearn.datasets.fetch_california_housing(return_X_y=True)
    X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)

    regressor_obj.fit(X_train, y_train)
    y_pred = regressor_obj.predict(X_val)

    error = sklearn.metrics.mean_squared_error(y_val, y_pred)

    return error  # An objective value linked with the Trial object.

study = optuna.create_study()  # Create a new study.
study.optimize(objective, n_trials=100)  # Invoke optimization of the objective function.

Communication

GitHub Issues for bug reports, feature requests and questions.
Gitter for interactive chat with developers.
Stack Overflow for questions.

Contribution

Any contributions to Optuna are welcome! When you send a pull request, please follow the contribution guide.

License

MIT License (see LICENSE).

Reference

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).

Installation

Optuna supports Python 3.6 or newer.

We recommend to install Optuna via pip:

$ pip install optuna

You can also install the development version of Optuna from master branch of Git repository:

$ pip install git+https://github.com/optuna/optuna.git

You can also install Optuna via conda:

$ conda install -c conda-forge optuna

Tutorial

If you are new to Optuna or want a general introduction, we highly recommend the below video.

Key Features

Showcases Optuna’s Key Features.

1. Lightweight, versatile, and platform agnostic architecture

2. Pythonic Search Space

3. Efficient Optimization Algorithms

4. Easy Parallelization

5. Quick Visualization for Hyperparameter Optimization Analysis

Recipes

Showcases the recipes that might help you using Optuna with comfort.

Saving/Resuming Study with RDB Backend

Multi-objective Optimization with Optuna

User Attributes

Command-Line Interface

User-Defined Sampler

User-Defined Pruner

Callback for Study.optimize

Specify Hyperparameters Manually

Ask-and-Tell Interface

Re-use the best trial

Key Features

Showcases Optuna’s Key Features.

1. Lightweight, versatile, and platform agnostic architecture

2. Pythonic Search Space

3. Efficient Optimization Algorithms

4. Easy Parallelization

5. Quick Visualization for Hyperparameter Optimization Analysis

Note

Click here to download the full example code

1. Lightweight, versatile, and platform agnostic architecture

Optuna is entirely written in Python and has few dependencies. This means that we can quickly move to the real example once you get interested in Optuna.

Quadratic Function Example

Usually, Optuna is used to optimize hyperparameters, but as an example, let’s optimize a simple quadratic function: \((x - 2)^2\).

First of all, import optuna.

import optuna

In optuna, conventionally functions to be optimized are named objective.

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2

This function returns the value of \((x - 2)^2\). Our goal is to find the value of x that minimizes the output of the objective function. This is the “optimization.” During the optimization, Optuna repeatedly calls and evaluates the objective function with different values of x.

A Trial object corresponds to a single execution of the objective function and is internally instantiated upon each invocation of the function.

The suggest APIs (for example, suggest_float()) are called inside the objective function to obtain parameters for a trial. suggest_float() selects parameters uniformly within the range provided. In our example, from \(-10\) to \(10\).

To start the optimization, we create a study object and pass the objective function to method optimize() as follows.

study = optuna.create_study()
study.optimize(objective, n_trials=100)

You can get the best parameter as follows.

best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))

Found x: 1.9997544157360778, (x - 2)^2: 6.031163068620034e-08

We can see that the x value found by Optuna is close to the optimal value of 2.

Note

When used to search for hyperparameters in machine learning, usually the objective function would return the loss or accuracy of the model.

Study Object

Let us clarify the terminology in Optuna as follows:

Trial: A single call of the objective function
Study: An optimization session, which is a set of trials
Parameter: A variable whose value is to be optimized, such as x in the above example

In Optuna, we use the study object to manage optimization. Method create_study() returns a study object. A study object has useful properties for analyzing the optimization outcome.

To get the dictionary of parameter name and parameter values:

study.best_params

{'x': 1.9997544157360778}

To get the best observed value of the objective function:

study.best_value

6.031163068620034e-08

To get the best trial:

study.best_trial

FrozenTrial(number=92, values=[6.031163068620034e-08], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 795382), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 798824), params={'x': 1.9997544157360778}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=92, state=TrialState.COMPLETE, value=None)

To get all trials:

study.trials

[FrozenTrial(number=0, values=[125.2764996942231], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 483308), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 483747), params={'x': -9.192698499210238}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE, value=None), FrozenTrial(number=1, values=[87.32836840494411], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 484206), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 484508), params={'x': -7.344964869112356}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE, value=None), FrozenTrial(number=2, values=[88.03209402007853], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 484887), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 485173), params={'x': -7.382541980725614}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=2, state=TrialState.COMPLETE, value=None), FrozenTrial(number=3, values=[47.72259731109812], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 485543), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 485822), params={'x': -4.908154407010466}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE, value=None), FrozenTrial(number=4, values=[51.67985470730038], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 486220), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 486507), params={'x': 9.188870196859892}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=4, state=TrialState.COMPLETE, value=None), FrozenTrial(number=5, values=[52.41590877752263], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 486876), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 487163), params={'x': 9.239883201925473}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=5, state=TrialState.COMPLETE, value=None), FrozenTrial(number=6, values=[12.124612704865578], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 487538), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 487828), params={'x': 5.482041456511622}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=6, state=TrialState.COMPLETE, value=None), FrozenTrial(number=7, values=[2.108152123566998], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 488203), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 488493), params={'x': 3.451947700010919}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=7, state=TrialState.COMPLETE, value=None), FrozenTrial(number=8, values=[26.495080960280028], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 488859), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 489154), params={'x': -3.1473372689459573}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=8, state=TrialState.COMPLETE, value=None), FrozenTrial(number=9, values=[41.256534222834496], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 489525), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 489817), params={'x': 8.423124957747163}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=9, state=TrialState.COMPLETE, value=None), FrozenTrial(number=10, values=[0.12195517818365643], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 490218), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 493912), params={'x': 1.650779184206244}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=10, state=TrialState.COMPLETE, value=None), FrozenTrial(number=11, values=[0.36872842420131796], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 494398), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 497520), params={'x': 2.6072301245831913}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=11, state=TrialState.COMPLETE, value=None), FrozenTrial(number=12, values=[1.9961649345895727], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 497951), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 501119), params={'x': 0.5871429886256809}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=12, state=TrialState.COMPLETE, value=None), FrozenTrial(number=13, values=[1.812931939942966], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 501545), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 504635), params={'x': 0.6535483893050721}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=13, state=TrialState.COMPLETE, value=None), FrozenTrial(number=14, values=[3.348981463730014], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 505062), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 508180), params={'x': 3.830022257714374}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=14, state=TrialState.COMPLETE, value=None), FrozenTrial(number=15, values=[16.717615335093367], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 508607), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 511691), params={'x': -2.0887180552213875}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=15, state=TrialState.COMPLETE, value=None), FrozenTrial(number=16, values=[0.9230294488516524], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 512124), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 515209), params={'x': 2.9607442161427007}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=16, state=TrialState.COMPLETE, value=None), FrozenTrial(number=17, values=[20.61454653336603], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 515641), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 518792), params={'x': 6.540324496483267}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=17, state=TrialState.COMPLETE, value=None), FrozenTrial(number=18, values=[9.477984637383852], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 519215), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 522310), params={'x': -1.0786335665979885}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=18, state=TrialState.COMPLETE, value=None), FrozenTrial(number=19, values=[0.04004706362394386], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 522740), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 525795), params={'x': 1.7998823755289308}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=19, state=TrialState.COMPLETE, value=None), FrozenTrial(number=20, values=[0.26517827556430656], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 526245), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 529351), params={'x': 1.485045365527888}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=20, state=TrialState.COMPLETE, value=None), FrozenTrial(number=21, values=[1.878824984623896], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 529778), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 532918), params={'x': 0.6292976309118393}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=21, state=TrialState.COMPLETE, value=None), FrozenTrial(number=22, values=[10.724245547788573], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 533352), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 536491), params={'x': 5.274789389836936}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=22, state=TrialState.COMPLETE, value=None), FrozenTrial(number=23, values=[0.047961378875452564], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 536958), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 540794), params={'x': 1.7809991349892595}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=23, state=TrialState.COMPLETE, value=None), FrozenTrial(number=24, values=[13.881065213110569], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 541224), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 544371), params={'x': -1.7257301583864828}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=24, state=TrialState.COMPLETE, value=None), FrozenTrial(number=25, values=[33.564102517600865], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 544799), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 547909), params={'x': -3.7934534189549556}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=25, state=TrialState.COMPLETE, value=None), FrozenTrial(number=26, values=[6.468566530649236], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 548355), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 551467), params={'x': -0.5433376753095991}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=26, state=TrialState.COMPLETE, value=None), FrozenTrial(number=27, values=[6.413088330648913], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 551894), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 555086), params={'x': 4.5324076154223105}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=27, state=TrialState.COMPLETE, value=None), FrozenTrial(number=28, values=[26.039899952278358], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 555514), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 558706), params={'x': 7.102930525911396}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=28, state=TrialState.COMPLETE, value=None), FrozenTrial(number=29, values=[0.005471465454286689], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 559134), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 562330), params={'x': 1.9260306451678353}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=29, state=TrialState.COMPLETE, value=None), FrozenTrial(number=30, values=[0.006695131281881252], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 562766), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 565933), params={'x': 2.081823781884494}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=30, state=TrialState.COMPLETE, value=None), FrozenTrial(number=31, values=[0.04857718212621632], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 566378), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 569574), params={'x': 2.2204023187859336}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=31, state=TrialState.COMPLETE, value=None), FrozenTrial(number=32, values=[5.283125803588961], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 570014), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 573208), params={'x': 4.298505123681251}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=32, state=TrialState.COMPLETE, value=None), FrozenTrial(number=33, values=[131.98925863282048], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 573635), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 576832), params={'x': -9.488657825560846}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=33, state=TrialState.COMPLETE, value=None), FrozenTrial(number=34, values=[6.638572809214283], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 577267), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 580489), params={'x': -0.5765428017431193}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=34, state=TrialState.COMPLETE, value=None), FrozenTrial(number=35, values=[76.01313594164668], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 580919), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 584129), params={'x': -6.718551252452823}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=35, state=TrialState.COMPLETE, value=None), FrozenTrial(number=36, values=[0.2197197490949997], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 584560), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 587767), params={'x': 1.531257267688341}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=36, state=TrialState.COMPLETE, value=None), FrozenTrial(number=37, values=[17.831882167370537], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 588212), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 591419), params={'x': 6.222781330754712}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=37, state=TrialState.COMPLETE, value=None), FrozenTrial(number=38, values=[0.6418624059210738], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 591852), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 595100), params={'x': 2.8011631581151706}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=38, state=TrialState.COMPLETE, value=None), FrozenTrial(number=39, values=[22.611566671825692], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 595555), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 598786), params={'x': -2.755162107838774}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=39, state=TrialState.COMPLETE, value=None), FrozenTrial(number=40, values=[49.45210712916966], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 599220), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 602475), params={'x': -5.032219217940355}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=40, state=TrialState.COMPLETE, value=None), FrozenTrial(number=41, values=[0.0029888813619107233], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 602931), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 606160), params={'x': 1.945329337283048}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=41, state=TrialState.COMPLETE, value=None), FrozenTrial(number=42, values=[2.2933117605973923], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 606614), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 609889), params={'x': 0.485631563787269}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=42, state=TrialState.COMPLETE, value=None), FrozenTrial(number=43, values=[0.04524770102398259], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 610372), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 613554), params={'x': 2.212715070044373}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=43, state=TrialState.COMPLETE, value=None), FrozenTrial(number=44, values=[2.4800991798026537], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 614010), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 617246), params={'x': 3.5748330641063686}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=44, state=TrialState.COMPLETE, value=None), FrozenTrial(number=45, values=[9.087723262528751], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 617712), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 621004), params={'x': 5.014585089614946}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=45, state=TrialState.COMPLETE, value=None), FrozenTrial(number=46, values=[4.301309389587277], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 621441), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 624698), params={'x': -0.0739598331663216}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=46, state=TrialState.COMPLETE, value=None), FrozenTrial(number=47, values=[1.9607204673773964], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 625148), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 628408), params={'x': 3.4002572861361573}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=47, state=TrialState.COMPLETE, value=None), FrozenTrial(number=48, values=[0.6059974521087018], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 628846), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 632101), params={'x': 1.22154161825522}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=48, state=TrialState.COMPLETE, value=None), FrozenTrial(number=49, values=[0.15089389370913875], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 632536), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 635773), params={'x': 2.388450632267652}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=49, state=TrialState.COMPLETE, value=None), FrozenTrial(number=50, values=[12.232221334341903], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 636242), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 639484), params={'x': -1.497459268432143}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=50, state=TrialState.COMPLETE, value=None), FrozenTrial(number=51, values=[0.02003270687266356], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 639937), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 643177), params={'x': 2.1415369452569313}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=51, state=TrialState.COMPLETE, value=None), FrozenTrial(number=52, values=[0.20479637327174136], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 643635), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 647137), params={'x': 2.4525443329351737}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=52, state=TrialState.COMPLETE, value=None), FrozenTrial(number=53, values=[4.752632337746037], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 647598), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 650882), params={'x': 4.180053287822579}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=53, state=TrialState.COMPLETE, value=None), FrozenTrial(number=54, values=[4.141113591574467], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 651341), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 654549), params={'x': -0.03497262673837098}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=54, state=TrialState.COMPLETE, value=None), FrozenTrial(number=55, values=[0.28631012732783057], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 655008), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 658230), params={'x': 1.4649204476642463}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=55, state=TrialState.COMPLETE, value=None), FrozenTrial(number=56, values=[1.0847344203844098], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 658687), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 661905), params={'x': 3.0415058427029633}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=56, state=TrialState.COMPLETE, value=None), FrozenTrial(number=57, values=[1.892860395955644], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 662386), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 665624), params={'x': 0.624187368877708}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=57, state=TrialState.COMPLETE, value=None), FrozenTrial(number=58, values=[16.405151549162778], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 666080), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 669307), params={'x': 6.050327338520034}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=58, state=TrialState.COMPLETE, value=None), FrozenTrial(number=59, values=[35.47009363096232], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 669771), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 672996), params={'x': 7.9556774283839715}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=59, state=TrialState.COMPLETE, value=None), FrozenTrial(number=60, values=[0.002279646772341111], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 673468), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 676707), params={'x': 1.9522543533676513}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=60, state=TrialState.COMPLETE, value=None), FrozenTrial(number=61, values=[0.06791689656686103], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 677171), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 680669), params={'x': 2.2606087039353464}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=61, state=TrialState.COMPLETE, value=None), FrozenTrial(number=62, values=[1.618549886036281], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 681146), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 684409), params={'x': 3.2722224200336516}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=62, state=TrialState.COMPLETE, value=None), FrozenTrial(number=63, values=[0.5922907438162334], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 684883), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 688140), params={'x': 1.2303957225845004}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=63, state=TrialState.COMPLETE, value=None), FrozenTrial(number=64, values=[0.011078630928785041], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 688621), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 691866), params={'x': 1.8947449244511931}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=64, state=TrialState.COMPLETE, value=None), FrozenTrial(number=65, values=[7.651415399550495], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 692355), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 695644), params={'x': -0.7661191947474886}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=65, state=TrialState.COMPLETE, value=None), FrozenTrial(number=66, values=[8.545299232745672], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 696126), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 699415), params={'x': 4.923234378688385}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=66, state=TrialState.COMPLETE, value=None), FrozenTrial(number=67, values=[3.7302876298504146], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 699881), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 703139), params={'x': 3.931395254692942}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=67, state=TrialState.COMPLETE, value=None), FrozenTrial(number=68, values=[0.7962690293205721], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 703579), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 707160), params={'x': 1.10766092245124}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=68, state=TrialState.COMPLETE, value=None), FrozenTrial(number=69, values=[0.013452958927682659], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 707600), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 710909), params={'x': 1.884013108811027}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=69, state=TrialState.COMPLETE, value=None), FrozenTrial(number=70, values=[3.553206907873867], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 711348), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 714648), params={'x': 0.11500479897855787}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=70, state=TrialState.COMPLETE, value=None), FrozenTrial(number=71, values=[0.0064698253926434], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 715097), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 718409), params={'x': 1.9195647751750304}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=71, state=TrialState.COMPLETE, value=None), FrozenTrial(number=72, values=[0.012643042954931254], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 718891), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 722521), params={'x': 1.887558713299201}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=72, state=TrialState.COMPLETE, value=None), FrozenTrial(number=73, values=[1.2954215591502392], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 723177), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 726445), params={'x': 0.8618341249403763}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=73, state=TrialState.COMPLETE, value=None), FrozenTrial(number=74, values=[0.6035551715027532], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 726890), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 730219), params={'x': 2.776888133197279}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=74, state=TrialState.COMPLETE, value=None), FrozenTrial(number=75, values=[0.05937768108060019], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 730670), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 734057), params={'x': 1.7563246399805672}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=75, state=TrialState.COMPLETE, value=None), FrozenTrial(number=76, values=[3.4283725156899223], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 734534), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 737847), params={'x': 0.14841351385091328}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=76, state=TrialState.COMPLETE, value=None), FrozenTrial(number=77, values=[2.2301188498021336], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 738326), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 741610), params={'x': 3.4933582456336905}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=77, state=TrialState.COMPLETE, value=None), FrozenTrial(number=78, values=[6.321335196634581], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 742057), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 745372), params={'x': 4.51422656032319}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=78, state=TrialState.COMPLETE, value=None), FrozenTrial(number=79, values=[0.017762368809936053], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 745817), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 749164), params={'x': 1.8667244628225568}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=79, state=TrialState.COMPLETE, value=None), FrozenTrial(number=80, values=[0.7605476959102747], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 749611), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 752958), params={'x': 2.8720938572827324}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=80, state=TrialState.COMPLETE, value=None), FrozenTrial(number=81, values=[0.002403308469114219], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 753409), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 756746), params={'x': 1.9509764498519924}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=81, state=TrialState.COMPLETE, value=None), FrozenTrial(number=82, values=[1.1884072483766088], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 757195), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 760549), params={'x': 0.9098590694884405}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=82, state=TrialState.COMPLETE, value=None), FrozenTrial(number=83, values=[60.93687862798425], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 761003), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 764372), params={'x': 9.806207698235056}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=83, state=TrialState.COMPLETE, value=None), FrozenTrial(number=84, values=[109.96946641054473], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 764824), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 768181), params={'x': -8.486632748911575}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=84, state=TrialState.COMPLETE, value=None), FrozenTrial(number=85, values=[0.05635262200135926], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 768632), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 771975), params={'x': 1.7626129278975806}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=85, state=TrialState.COMPLETE, value=None), FrozenTrial(number=86, values=[3.4369440067582695], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 772438), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 775823), params={'x': 3.8538996754836194}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=86, state=TrialState.COMPLETE, value=None), FrozenTrial(number=87, values=[6.0660603454159245], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 776282), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 779646), params={'x': -0.46293734094392347}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=87, state=TrialState.COMPLETE, value=None), FrozenTrial(number=88, values=[0.36974174299551266], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 780129), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 783454), params={'x': 2.6080639300234085}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=88, state=TrialState.COMPLETE, value=None), FrozenTrial(number=89, values=[2.763743189947601], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 783900), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 787296), params={'x': 0.33754904134058705}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=89, state=TrialState.COMPLETE, value=None), FrozenTrial(number=90, values=[0.5159875911475527], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 787743), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 791090), params={'x': 1.2816772374847414}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=90, state=TrialState.COMPLETE, value=None), FrozenTrial(number=91, values=[0.005674762512575395], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 791539), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 794932), params={'x': 1.9246689804092936}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=91, state=TrialState.COMPLETE, value=None), FrozenTrial(number=92, values=[6.031163068620034e-08], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 795382), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 798824), params={'x': 1.9997544157360778}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=92, state=TrialState.COMPLETE, value=None), FrozenTrial(number=93, values=[1.2222553027394607], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 799282), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 802656), params={'x': 3.105556557910748}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=93, state=TrialState.COMPLETE, value=None), FrozenTrial(number=94, values=[0.011518039445835094], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 803111), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 806488), params={'x': 2.1073221293388977}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=94, state=TrialState.COMPLETE, value=None), FrozenTrial(number=95, values=[1.2800313000799999], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 806941), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 810321), params={'x': 0.8686153173743247}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=95, state=TrialState.COMPLETE, value=None), FrozenTrial(number=96, values=[0.17780176049308474], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 810776), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 814214), params={'x': 2.421665460398507}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=96, state=TrialState.COMPLETE, value=None), FrozenTrial(number=97, values=[9.82187924060899], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 814669), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 818308), params={'x': -1.1339877537426644}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=97, state=TrialState.COMPLETE, value=None), FrozenTrial(number=98, values=[0.46170645184048204], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 818784), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 822160), params={'x': 1.320510153246951}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=98, state=TrialState.COMPLETE, value=None), FrozenTrial(number=99, values=[1.869866048088801], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 45, 822614), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 45, 826264), params={'x': 3.367430454571201}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=99, state=TrialState.COMPLETE, value=None)]

To get the number of trials:

len(study.trials)

By executing optimize() again, we can continue the optimization.

study.optimize(objective, n_trials=100)

To get the updated number of trials:

len(study.trials)

As the objective function is so easy that the last 100 trials don’t improve the result. However, we can check the result again:

best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))

Found x: 1.9997544157360778, (x - 2)^2: 6.031163068620034e-08

Total running time of the script: ( 0 minutes 0.779 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

2. Pythonic Search Space

For hyperparameter sampling, Optuna provides the following features:

optuna.trial.Trial.suggest_categorical() for categorical parameters
optuna.trial.Trial.suggest_int() for integer parameters
optuna.trial.Trial.suggest_float() for floating point parameters

With optional arguments of step and log, we can discretize or take the logarithm of integer and floating point parameters.

import optuna


def objective(trial):
    # Categorical parameter
    optimizer = trial.suggest_categorical("optimizer", ["MomentumSGD", "Adam"])

    # Integer parameter
    num_layers = trial.suggest_int("num_layers", 1, 3)

    # Integer parameter (log)
    num_channels = trial.suggest_int("num_channels", 32, 512, log=True)

    # Integer parameter (discretized)
    num_units = trial.suggest_int("num_units", 10, 100, step=5)

    # Floating point parameter
    dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0)

    # Floating point parameter (log)
    learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True)

    # Floating point parameter (discretized)
    drop_path_rate = trial.suggest_float("drop_path_rate", 0.0, 1.0, step=0.1)

Defining Parameter Spaces

In Optuna, we define search spaces using familiar Python syntax including conditionals and loops.

Also, you can use branches or loops depending on the parameter values.

For more various use, see examples.

Branches:

import sklearn.ensemble
import sklearn.svm


def objective(trial):
    classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"])
    if classifier_name == "SVC":
        svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
        classifier_obj = sklearn.svm.SVC(C=svc_c)
    else:
        rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True)
        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)

Loops:

import torch
import torch.nn as nn


def create_model(trial, in_size):
    n_layers = trial.suggest_int("n_layers", 1, 3)

    layers = []
    for i in range(n_layers):
        n_units = trial.suggest_int("n_units_l{}".format(i), 4, 128, log=True)
        layers.append(nn.Linear(in_size, n_units))
        layers.append(nn.ReLU())
        in_size = n_units
    layers.append(nn.Linear(in_size, 10))

    return nn.Sequential(*layers)

Note on the Number of Parameters

The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.

Total running time of the script: ( 0 minutes 0.001 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

3. Efficient Optimization Algorithms

Optuna enables efficient hyperparameter optimization by adopting state-of-the-art algorithms for sampling hyperparameters and pruning efficiently unpromising trials.

Sampling Algorithms

Samplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values, leading to an optimal search space which giving off parameters leading to better objective values. More detailed explanation of how samplers suggest parameters is in BaseSampler.

Optuna provides the following sampling algorithms:

Grid Search implemented in GridSampler
Random Search implemented in RandomSampler
Tree-structured Parzen Estimator algorithm implemented in TPESampler
CMA-ES based algorithm implemented in CmaEsSampler
Algorithm to enable partial fixed parameters implemented in PartialFixedSampler
Nondominated Sorting Genetic Algorithm II implemented in NSGAIISampler
A Quasi Monte Carlo sampling algorithm implemented in QMCSampler

The default sampler is TPESampler.

Switching Samplers

import optuna

By default, Optuna uses TPESampler as follows.

study = optuna.create_study()
print(f"Sampler is {study.sampler.__class__.__name__}")

Sampler is TPESampler

If you want to use different samplers for example RandomSampler and CmaEsSampler,

study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")

study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")

Sampler is RandomSampler
Sampler is CmaEsSampler

Pruning Algorithms

Pruners automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).

Optuna provides the following pruning algorithms:

Median pruning algorithm implemented in MedianPruner
Non-pruning algorithm implemented in NopPruner
Algorithm to operate pruner with tolerance implemented in PatientPruner
Algorithm to prune specified percentile of trials implemented in PercentilePruner
Asynchronous Successive Halving algorithm implemented in SuccessiveHalvingPruner
Hyperband algorithm implemented in HyperbandPruner
Threshold pruning algorithm implemented in ThresholdPruner

We use MedianPruner in most examples, though basically it is outperformed by SuccessiveHalvingPruner and HyperbandPruner as in this benchmark result.

Activating Pruners

To turn on the pruning feature, you need to call report() and should_prune() after each step of the iterative training. report() periodically monitors the intermediate objective values. should_prune() decides termination of the trial that does not meet a predefined condition.

We would recommend using integration modules for major machine learning frameworks. Exclusive list is integration and usecases are available in ~optuna/examples.

import logging
import sys

import sklearn.datasets
import sklearn.linear_model
import sklearn.model_selection


def objective(trial):
    iris = sklearn.datasets.load_iris()
    classes = list(set(iris.target))
    train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(
        iris.data, iris.target, test_size=0.25, random_state=0
    )

    alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True)
    clf = sklearn.linear_model.SGDClassifier(alpha=alpha)

    for step in range(100):
        clf.partial_fit(train_x, train_y, classes=classes)

        # Report intermediate objective value.
        intermediate_value = 1.0 - clf.score(valid_x, valid_y)
        trial.report(intermediate_value, step)

        # Handle pruning based on the intermediate value.
        if trial.should_prune():
            raise optuna.TrialPruned()

    return 1.0 - clf.score(valid_x, valid_y)

Set up the median stopping rule as the pruning condition.

# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
study.optimize(objective, n_trials=20)

A new study created in memory with name: no-name-456bce7c-12a0-4529-9918-1ffc1c3f0c0e
Trial 0 finished with value: 0.02631578947368418 and parameters: {'alpha': 0.028276729459299658}. Best is trial 0 with value: 0.02631578947368418.
Trial 1 finished with value: 0.052631578947368474 and parameters: {'alpha': 0.007548339410814463}. Best is trial 0 with value: 0.02631578947368418.
Trial 2 finished with value: 0.368421052631579 and parameters: {'alpha': 3.9628705911409236e-05}. Best is trial 0 with value: 0.02631578947368418.
Trial 3 finished with value: 0.07894736842105265 and parameters: {'alpha': 0.00010175120082817493}. Best is trial 0 with value: 0.02631578947368418.
Trial 4 finished with value: 0.07894736842105265 and parameters: {'alpha': 0.017243285560519364}. Best is trial 0 with value: 0.02631578947368418.
Trial 5 finished with value: 0.23684210526315785 and parameters: {'alpha': 0.015020081290575995}. Best is trial 0 with value: 0.02631578947368418.
Trial 6 finished with value: 0.21052631578947367 and parameters: {'alpha': 0.05378928430363982}. Best is trial 0 with value: 0.02631578947368418.
Trial 7 finished with value: 0.3157894736842105 and parameters: {'alpha': 2.166400433272187e-05}. Best is trial 0 with value: 0.02631578947368418.
Trial 8 pruned.
Trial 9 finished with value: 0.21052631578947367 and parameters: {'alpha': 1.0741993481286064e-05}. Best is trial 0 with value: 0.02631578947368418.
Trial 10 finished with value: 0.02631578947368418 and parameters: {'alpha': 0.001405420283855543}. Best is trial 0 with value: 0.02631578947368418.
Trial 11 pruned.
Trial 12 pruned.
Trial 13 pruned.
Trial 14 finished with value: 0.2894736842105263 and parameters: {'alpha': 0.004640376715552643}. Best is trial 0 with value: 0.02631578947368418.
Trial 15 finished with value: 0.07894736842105265 and parameters: {'alpha': 0.0003424006621233019}. Best is trial 0 with value: 0.02631578947368418.
Trial 16 pruned.
Trial 17 finished with value: 0.23684210526315785 and parameters: {'alpha': 0.0031464124629097426}. Best is trial 0 with value: 0.02631578947368418.
Trial 18 pruned.
Trial 19 finished with value: 0.1578947368421053 and parameters: {'alpha': 0.043211300659571725}. Best is trial 0 with value: 0.02631578947368418.

As you can see, several trials were pruned (stopped) before they finished all of the iterations. The format of message is "Trial <Trial Number> pruned.".

Which Sampler and Pruner Should be Used?

From the benchmark results which are available at optuna/optuna - wiki “Benchmarks with Kurobako”, at least for not deep learning tasks, we would say that

For RandomSampler, MedianPruner is the best.
For TPESampler, Hyperband is the best.

However, note that the benchmark is not deep learning. For deep learning tasks, consult the below table. This table is from the Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 paper, which is written in Japanese.

Parallel Compute Resource	Categorical/Conditional Hyperparameters	Recommended Algorithms
Limited	No	TPE. GP-EI if search space is low-dimensional and continuous.
Limited	Yes	TPE. GP-EI if search space is low-dimensional and continuous
Sufficient	No	CMA-ES, Random Search
Sufficient	Yes	Random Search or Genetic Algorithm

Integration Modules for Pruning

To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.

For the complete list of Optuna’s integration modules, see integration.

For example, XGBoostPruningCallback introduces pruning without directly changing the logic of training iteration. (See also example for the entire script.)

pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])

Total running time of the script: ( 0 minutes 3.162 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

4. Easy Parallelization

It’s straightforward to parallelize optuna.study.Study.optimize().

If you want to manually execute Optuna optimization:

start an RDB server (this example uses MySQL)

create a study with –storage argument

share the study among multiple nodes and processes

Of course, you can use Kubernetes as in the kubernetes examples.

To just see how parallel optimization works in Optuna, check the below video.

Create a Study

You can create a study using optuna create-study command. Alternatively, in Python script you can use optuna.create_study().

$ mysql -u root -e "CREATE DATABASE IF NOT EXISTS example"
$ optuna create-study --study-name "distributed-example" --storage "mysql://root@localhost/example"
[I 2020-07-21 13:43:39,642] A new study created with name: distributed-example

Then, write an optimization script. Let’s assume that foo.py contains the following code.

import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


if __name__ == "__main__":
    study = optuna.load_study(
        study_name="distributed-example", storage="mysql://root@localhost/example"
    )
    study.optimize(objective, n_trials=100)

Note

Click here to download the full example code

5. Quick Visualization for Hyperparameter Optimization Analysis

Optuna provides various visualization features in optuna.visualization to analyze optimization results visually.

This tutorial walks you through this module by visualizing the history of lightgbm model for breast cancer dataset.

For visualizing multi-objective optimization (i.e., the usage of optuna.visualization.plot_pareto_front()), please refer to the tutorial of Multi-objective Optimization with Optuna.

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna

# You can use Matplotlib instead of Plotly for visualization by simply replacing `optuna.visualization` with
# `optuna.visualization.matplotlib` in the following examples.
from optuna.visualization import plot_contour
from optuna.visualization import plot_edf
from optuna.visualization import plot_intermediate_values
from optuna.visualization import plot_optimization_history
from optuna.visualization import plot_parallel_coordinate
from optuna.visualization import plot_param_importances
from optuna.visualization import plot_slice

SEED = 42

np.random.seed(SEED)

Define the objective function.

def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "bagging_fraction": trial.suggest_float("bagging_fraction", 0.4, 1.0),
        "bagging_freq": trial.suggest_int("bagging_freq", 1, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    # Add a callback for pruning.
    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
    gbm = lgb.train(
        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
    )

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

study = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=SEED),
    pruner=optuna.pruners.MedianPruner(n_warmup_steps=10),
)
study.optimize(objective, n_trials=100, timeout=600)

'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.

Plot functions

Visualize the optimization history. See plot_optimization_history() for the details.

plot_optimization_history(study)

Visualize the learning curves of the trials. See plot_intermediate_values() for the details.

plot_intermediate_values(study)

Visualize high-dimensional parameter relationships. See plot_parallel_coordinate() for the details.

plot_parallel_coordinate(study)

Select parameters to visualize.

plot_parallel_coordinate(study, params=["bagging_freq", "bagging_fraction"])

Visualize hyperparameter relationships. See plot_contour() for the details.

plot_contour(study)

Select parameters to visualize.

plot_contour(study, params=["bagging_freq", "bagging_fraction"])

Visualize individual hyperparameters as slice plot. See plot_slice() for the details.

plot_slice(study)

Select parameters to visualize.

plot_slice(study, params=["bagging_freq", "bagging_fraction"])

Visualize parameter importances. See plot_param_importances() for the details.

plot_param_importances(study)

Learn which hyperparameters are affecting the trial duration with hyperparameter importance.

optuna.visualization.plot_param_importances(
    study, target=lambda t: t.duration.total_seconds(), target_name="duration"
)

Visualize empirical distribution function. See plot_edf() for the details.

plot_edf(study)

Total running time of the script: ( 0 minutes 4.392 seconds)

Gallery generated by Sphinx-Gallery

Recipes

Showcases the recipes that might help you using Optuna with comfort.

Saving/Resuming Study with RDB Backend

Multi-objective Optimization with Optuna

User Attributes

Command-Line Interface

User-Defined Sampler

User-Defined Pruner

Callback for Study.optimize

Specify Hyperparameters Manually

Ask-and-Tell Interface

Re-use the best trial

Note

Click here to download the full example code

Saving/Resuming Study with RDB Backend

An RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies. In addition, we can run multi-node optimization tasks with this feature, which is described in 4. Easy Parallelization.

In this section, let’s try simple examples running on a local environment with SQLite DB.

Note

You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB’s URL. Please refer to SQLAlchemy’s document for how to set up the URL.

New Study

We can create a persistent study by calling create_study() function as follows. An SQLite file example.db is automatically initialized with a new study record.

import logging
import sys

import optuna

# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study_name = "example-study"  # Unique identifier of the study.
storage_name = "sqlite:///{}.db".format(study_name)
study = optuna.create_study(study_name=study_name, storage=storage_name)

A new study created in RDB with name: example-study

To run a study, call optimize() method passing an objective function.

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


study.optimize(objective, n_trials=3)

Trial 0 finished with value: 0.01468308085175024 and parameters: {'x': 2.1211737630502174}. Best is trial 0 with value: 0.01468308085175024.
Trial 1 finished with value: 18.947933515750766 and parameters: {'x': 6.352922410950001}. Best is trial 0 with value: 0.01468308085175024.
Trial 2 finished with value: 24.447033765382926 and parameters: {'x': 6.944394175769457}. Best is trial 0 with value: 0.01468308085175024.

Resume Study

To resume a study, instantiate a Study object passing the study name example-study and the DB URL sqlite:///example-study.db.

study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)
study.optimize(objective, n_trials=3)

Using an existing study with name 'example-study' instead of creating a new one.
Trial 3 finished with value: 44.52698148667395 and parameters: {'x': -4.672854073533599}. Best is trial 0 with value: 0.01468308085175024.
Trial 4 finished with value: 48.13798453280508 and parameters: {'x': -4.938154259801744}. Best is trial 0 with value: 0.01468308085175024.
Trial 5 finished with value: 1.2138110455403865 and parameters: {'x': 0.8982690684471155}. Best is trial 0 with value: 0.01468308085175024.

Experimental History

We can access histories of studies and trials via the Study class. For example, we can get all trials of example-study as:

study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)
df = study.trials_dataframe(attrs=("number", "value", "params", "state"))

Using an existing study with name 'example-study' instead of creating a new one.

The method trials_dataframe() returns a pandas dataframe like:

print(df)

   number      value  params_x     state
     0   0.014683  2.121174  COMPLETE
     1  18.947934  6.352922  COMPLETE
     2  24.447034  6.944394  COMPLETE
     3  44.526981 -4.672854  COMPLETE
     4  48.137985 -4.938154  COMPLETE
     5   1.213811  0.898269  COMPLETE

A Study object also provides properties such as trials, best_value, best_params (see also 1. Lightweight, versatile, and platform agnostic architecture).

print("Best params: ", study.best_params)
print("Best value: ", study.best_value)
print("Best Trial: ", study.best_trial)
print("Trials: ", study.trials)

Best params:  {'x': 2.1211737630502174}
Best value:  0.01468308085175024
Best Trial:  FrozenTrial(number=0, values=[0.01468308085175024], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 54, 921374), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 54, 942707), params={'x': 2.1211737630502174}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE, value=None)
Trials:  [FrozenTrial(number=0, values=[0.01468308085175024], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 54, 921374), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 54, 942707), params={'x': 2.1211737630502174}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE, value=None), FrozenTrial(number=1, values=[18.947933515750766], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 54, 966952), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 54, 982803), params={'x': 6.352922410950001}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=2, state=TrialState.COMPLETE, value=None), FrozenTrial(number=2, values=[24.447033765382926], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 54, 998894), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 55, 14604), params={'x': 6.944394175769457}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE, value=None), FrozenTrial(number=3, values=[44.52698148667395], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 55, 77947), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 55, 97462), params={'x': -4.672854073533599}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=4, state=TrialState.COMPLETE, value=None), FrozenTrial(number=4, values=[48.13798453280508], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 55, 117775), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 55, 134217), params={'x': -4.938154259801744}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=5, state=TrialState.COMPLETE, value=None), FrozenTrial(number=5, values=[1.2138110455403865], datetime_start=datetime.datetime(2022, 12, 19, 8, 18, 55, 151664), datetime_complete=datetime.datetime(2022, 12, 19, 8, 18, 55, 169587), params={'x': 0.8982690684471155}, distributions={'x': FloatDistribution(high=10.0, log=False, low=-10.0, step=None)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=6, state=TrialState.COMPLETE, value=None)]

Total running time of the script: ( 0 minutes 0.681 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Multi-objective Optimization with Optuna

This tutorial showcases Optuna’s multi-objective optimization feature by optimizing the validation accuracy of Fashion MNIST dataset and the FLOPS of the model implemented in PyTorch.

We use thop to measure FLOPS.

import thop
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision

import optuna


DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
DIR = ".."
BATCHSIZE = 128
N_TRAIN_EXAMPLES = BATCHSIZE * 30
N_VALID_EXAMPLES = BATCHSIZE * 10


def define_model(trial):
    n_layers = trial.suggest_int("n_layers", 1, 3)
    layers = []

    in_features = 28 * 28
    for i in range(n_layers):
        out_features = trial.suggest_int("n_units_l{}".format(i), 4, 128)
        layers.append(nn.Linear(in_features, out_features))
        layers.append(nn.ReLU())
        p = trial.suggest_float("dropout_{}".format(i), 0.2, 0.5)
        layers.append(nn.Dropout(p))

        in_features = out_features

    layers.append(nn.Linear(in_features, 10))
    layers.append(nn.LogSoftmax(dim=1))

    return nn.Sequential(*layers)


# Defines training and evaluation.
def train_model(model, optimizer, train_loader):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
        optimizer.zero_grad()
        F.nll_loss(model(data), target).backward()
        optimizer.step()


def eval_model(model, valid_loader):
    model.eval()
    correct = 0
    with torch.no_grad():
        for batch_idx, (data, target) in enumerate(valid_loader):
            data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
            pred = model(data).argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    accuracy = correct / N_VALID_EXAMPLES

    flops, _ = thop.profile(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),), verbose=False)
    return flops, accuracy

Define multi-objective objective function. Objectives are FLOPS and accuracy.

def objective(trial):
    train_dataset = torchvision.datasets.FashionMNIST(
        DIR, train=True, download=True, transform=torchvision.transforms.ToTensor()
    )
    train_loader = torch.utils.data.DataLoader(
        torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))),
        batch_size=BATCHSIZE,
        shuffle=True,
    )

    val_dataset = torchvision.datasets.FashionMNIST(
        DIR, train=False, transform=torchvision.transforms.ToTensor()
    )
    val_loader = torch.utils.data.DataLoader(
        torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))),
        batch_size=BATCHSIZE,
        shuffle=True,
    )
    model = define_model(trial).to(DEVICE)

    optimizer = torch.optim.Adam(
        model.parameters(), trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    )

    for epoch in range(10):
        train_model(model, optimizer, train_loader)
    flops, accuracy = eval_model(model, val_loader)
    return flops, accuracy

Run multi-objective optimization

If your optimization problem is multi-objective, Optuna assumes that you will specify the optimization direction for each objective. Specifically, in this example, we want to minimize the FLOPS (we want a faster model) and maximize the accuracy. So we set directions to ["minimize", "maximize"].

study = optuna.create_study(directions=["minimize", "maximize"])
study.optimize(objective, n_trials=30, timeout=300)

print("Number of finished trials: ", len(study.trials))

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to ../FashionMNIST/raw/train-images-idx3-ubyte.gz

  0%|          | 0/26421880 [00:00<?, ?it/s]
  0%|          | 40960/26421880 [00:00<01:13, 360947.40it/s]
  0%|          | 104448/26421880 [00:00<01:15, 348118.53it/s]
  1%|          | 224256/26421880 [00:00<00:40, 645704.15it/s]
  2%|1         | 483328/26421880 [00:00<00:20, 1294406.64it/s]
  4%|3         | 968704/26421880 [00:00<00:10, 2442989.86it/s]
  7%|7         | 1931264/26421880 [00:00<00:05, 4709185.14it/s]
 13%|#3        | 3549184/26421880 [00:00<00:02, 8272847.67it/s]
 20%|#9        | 5177344/26421880 [00:00<00:01, 10733881.19it/s]
 25%|##5       | 6704128/26421880 [00:00<00:01, 12100658.88it/s]
 32%|###1      | 8334336/26421880 [00:01<00:01, 13374828.79it/s]
 38%|###7      | 9996288/26421880 [00:01<00:01, 14353515.73it/s]
 44%|####4     | 11666432/26421880 [00:01<00:00, 15046896.03it/s]
 50%|#####     | 13341696/26421880 [00:01<00:00, 15553213.25it/s]
 57%|#####6    | 15016960/26421880 [00:01<00:00, 15912756.77it/s]
 63%|######3   | 16700416/26421880 [00:01<00:00, 16189620.75it/s]
 70%|######9   | 18391040/26421880 [00:01<00:00, 16404109.57it/s]
 76%|#######5  | 20065280/26421880 [00:01<00:00, 16505575.09it/s]
 82%|########2 | 21740544/26421880 [00:01<00:00, 16579585.50it/s]
 89%|########8 | 23421952/26421880 [00:02<00:00, 16649683.21it/s]
 95%|#########5| 25102336/26421880 [00:02<00:00, 16695907.21it/s]
26422272it [00:02, 12480211.34it/s]
Extracting ../FashionMNIST/raw/train-images-idx3-ubyte.gz to ../FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to ../FashionMNIST/raw/train-labels-idx1-ubyte.gz

  0%|          | 0/29515 [00:00<?, ?it/s]
29696it [00:00, 297897.09it/s]
Extracting ../FashionMNIST/raw/train-labels-idx1-ubyte.gz to ../FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to ../FashionMNIST/raw/t10k-images-idx3-ubyte.gz

  0%|          | 0/4422102 [00:00<?, ?it/s]
  1%|          | 40960/4422102 [00:00<00:12, 346845.94it/s]
  2%|2         | 98304/4422102 [00:00<00:09, 470713.96it/s]
  4%|4         | 195584/4422102 [00:00<00:06, 690706.44it/s]
  9%|9         | 405504/4422102 [00:00<00:03, 1232276.32it/s]
 19%|#8        | 824320/4422102 [00:00<00:01, 2280059.22it/s]
 38%|###7      | 1666048/4422102 [00:00<00:00, 4341044.67it/s]
 73%|#######3  | 3229696/4422102 [00:00<00:00, 8001420.26it/s]
4422656it [00:00, 5480649.06it/s]
Extracting ../FashionMNIST/raw/t10k-images-idx3-ubyte.gz to ../FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to ../FashionMNIST/raw/t10k-labels-idx1-ubyte.gz

  0%|          | 0/5148 [00:00<?, ?it/s]
6144it [00:00, 32414847.52it/s]
Extracting ../FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to ../FashionMNIST/raw

Number of finished trials:  30

Check trials on Pareto front visually.

optuna.visualization.plot_pareto_front(study, target_names=["FLOPS", "accuracy"])

Fetch the list of trials on the Pareto front with best_trials.

For example, the following code shows the number of trials on the Pareto front and picks the trial with the highest accuracy.

print(f"Number of trials on the Pareto front: {len(study.best_trials)}")

trial_with_highest_accuracy = max(study.best_trials, key=lambda t: t.values[1])
print(f"Trial with highest accuracy: ")
print(f"\tnumber: {trial_with_highest_accuracy.number}")
print(f"\tparams: {trial_with_highest_accuracy.params}")
print(f"\tvalues: {trial_with_highest_accuracy.values}")

Number of trials on the Pareto front: 6
Trial with highest accuracy:
        number: 24
        params: {'n_layers': 1, 'n_units_l0': 112, 'dropout_0': 0.20966436978178923, 'lr': 0.0033186365417282727}
        values: [88928.0, 0.8515625]

Learn which hyperparameters are affecting the flops most with hyperparameter importance.

optuna.visualization.plot_param_importances(
    study, target=lambda t: t.values[0], target_name="flops"
)

Total running time of the script: ( 2 minutes 0.481 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

User Attributes

This feature is to annotate experiments with user-defined attributes.

Adding User Attributes to Studies

A Study object provides set_user_attr() method to register a pair of key and value as an user-defined attribute. A key is supposed to be a str, and a value be any object serializable with json.dumps.

import sklearn.datasets
import sklearn.model_selection
import sklearn.svm

import optuna


study = optuna.create_study(storage="sqlite:///example.db")
study.set_user_attr("contributors", ["Akiba", "Sano"])
study.set_user_attr("dataset", "MNIST")

We can access annotated attributes with user_attr property.

study.user_attrs  # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

{'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

StudySummary object, which can be retrieved by get_all_study_summaries(), also contains user-defined attributes.

study_summaries = optuna.get_all_study_summaries("sqlite:///example.db")
study_summaries[0].user_attrs  # {"contributors": ["Akiba", "Sano"], "dataset": "MNIST"}

{'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

See also

optuna study set-user-attr command, which sets an attribute via command line interface.

Adding User Attributes to Trials

As with Study, a Trial object provides set_user_attr() method. Attributes are set inside an objective function.

def objective(trial):
    iris = sklearn.datasets.load_iris()
    x, y = iris.data, iris.target

    svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
    clf = sklearn.svm.SVC(C=svc_c)
    accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean()

    trial.set_user_attr("accuracy", accuracy)

    return 1.0 - accuracy  # return error for minimization


study.optimize(objective, n_trials=1)

We can access annotated attributes as:

study.trials[0].user_attrs

{'accuracy': 0.96}

Note that, in this example, the attribute is not annotated to a Study but a single Trial.

Total running time of the script: ( 0 minutes 0.282 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Command-Line Interface

Command	Description
ask	Create a new trial and suggest parameters.
best-trial	Show the best trial.
best-trials	Show a list of trials located at the Pareto front.
create-study	Create a new study.
delete-study	Delete a specified study.
storage upgrade	Upgrade the schema of a storage.
studies	Show a list of studies.
study optimize	Start optimization of a study.
study set-user-attr	Set a user attribute to a study.
tell	Finish a trial, which was created by the ask command.
trials	Show a list of trials.

Optuna provides command-line interface as shown in the above table.

Let us assume you are not in IPython shell and writing Python script files instead. It is totally fine to write scripts like the following:

import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


if __name__ == "__main__":
    study = optuna.create_study()
    study.optimize(objective, n_trials=100)
    print("Best value: {} (params: {})\n".format(study.best_value, study.best_params))

Best value: 1.8367394724891643e-05 (params: {'x': 2.0042857198607575})

However, we can reduce boilerplate codes by using our optuna command. Let us assume that foo.py contains only the following code.

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2

Even so, we can invoke the optimization as follows. (Don’t care about --storage sqlite:///example.db for now, which is described in Saving/Resuming Study with RDB Backend.)

$ cat foo.py
def objective(trial):
    x = trial.suggest_float('x', -10, 10)
    return (x - 2) ** 2

$ STUDY_NAME=`optuna create-study --storage sqlite:///example.db`
$ optuna study optimize foo.py objective --n-trials=100 --storage sqlite:///example.db --study-name $STUDY_NAME
[I 2018-05-09 10:40:25,196] Finished a trial resulted in value: 54.353767789264026. Current best value is 54.353767789264026 with parameters: {'x': -5.372500782588228}.
[I 2018-05-09 10:40:25,197] Finished a trial resulted in value: 15.784266965526376. Current best value is 15.784266965526376 with parameters: {'x': 5.972941852774387}.
...
[I 2018-05-09 10:40:26,204] Finished a trial resulted in value: 14.704254135013741. Current best value is 2.280758099793617e-06 with parameters: {'x': 1.9984897821018828}.

Please note that foo.py only contains the definition of the objective function. By giving the script file name and the method name of objective function to optuna study optimize command, we can invoke the optimization.

Total running time of the script: ( 0 minutes 0.348 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

User-Defined Sampler

Thanks to user-defined samplers, you can:

experiment your own sampling algorithms,
implement task-specific algorithms to refine the optimization performance, or
wrap other optimization libraries to integrate them into Optuna pipelines (e.g., SkoptSampler).

This section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler.

Overview of Sampler

A sampler has the responsibility to determine the parameter values to be evaluated in a trial. When a suggest API (e.g., suggest_float()) is called inside an objective function, the corresponding distribution object (e.g., FloatDistribution) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the suggest API and evaluated in the objective function.

To create a new sampler, you need to define a class that inherits BaseSampler. The base class has three abstract methods; infer_relative_search_space(), sample_relative(), and sample_independent().

As the method names imply, Optuna supports two types of sampling: one is relative sampling that can consider the correlation of the parameters in a trial, and the other is independent sampling that samples each parameter independently.

At the beginning of a trial, infer_relative_search_space() is called to provide the relative search space for the trial. Then, sample_relative() is invoked to sample relative parameters from the search space. During the execution of the objective function, sample_independent() is used to sample parameters that don’t belong to the relative search space.

Note

Please refer to the document of BaseSampler for further details.

An Example: Implementing SimulatedAnnealingSampler

For example, the following code defines a sampler based on Simulated Annealing (SA):

import numpy as np
import optuna


class SimulatedAnnealingSampler(optuna.samplers.BaseSampler):
    def __init__(self, temperature=100):
        self._rng = np.random.RandomState()
        self._temperature = temperature  # Current temperature.
        self._current_trial = None  # Current state.

    def sample_relative(self, study, trial, search_space):
        if search_space == {}:
            return {}

        # Simulated Annealing algorithm.
        # 1. Calculate transition probability.
        prev_trial = study.trials[-2]
        if self._current_trial is None or prev_trial.value <= self._current_trial.value:
            probability = 1.0
        else:
            probability = np.exp(
                (self._current_trial.value - prev_trial.value) / self._temperature
            )
        self._temperature *= 0.9  # Decrease temperature.

        # 2. Transit the current state if the previous result is accepted.
        if self._rng.uniform(0, 1) < probability:
            self._current_trial = prev_trial

        # 3. Sample parameters from the neighborhood of the current point.
        # The sampled parameters will be used during the next execution of
        # the objective function passed to the study.
        params = {}
        for param_name, param_distribution in search_space.items():
            if (
                not isinstance(param_distribution, optuna.distributions.FloatDistribution)
                or (param_distribution.step is not None and param_distribution.step != 1)
                or param_distribution.log
            ):
                msg = (
                    "Only suggest_float() with `step` `None` or 1.0 and"
                    " `log` `False` is supported"
                )
                raise NotImplementedError(msg)

            current_value = self._current_trial.params[param_name]
            width = (param_distribution.high - param_distribution.low) * 0.1
            neighbor_low = max(current_value - width, param_distribution.low)
            neighbor_high = min(current_value + width, param_distribution.high)
            params[param_name] = self._rng.uniform(neighbor_low, neighbor_high)

        return params

    # The rest are unrelated to SA algorithm: boilerplate
    def infer_relative_search_space(self, study, trial):
        return optuna.samplers.intersection_search_space(study)

    def sample_independent(self, study, trial, param_name, param_distribution):
        independent_sampler = optuna.samplers.RandomSampler()
        return independent_sampler.sample_independent(study, trial, param_name, param_distribution)

Note

In favor of code simplicity, the above implementation doesn’t support some features (e.g., maximization). If you’re interested in how to support those features, please see examples/samplers/simulated_annealing.py.

You can use SimulatedAnnealingSampler in the same way as built-in samplers as follows:

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -5, 5)
    return x**2 + y


sampler = SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)

best_trial = study.best_trial
print("Best value: ", best_trial.value)
print("Parameters that achieve the best value: ", best_trial.params)

Best value:  -4.828927703061493
Parameters that achieve the best value:  {'x': -0.017465347411791843, 'y': -4.829232741421707}

In this optimization, the values of x and y parameters are sampled by using SimulatedAnnealingSampler.sample_relative method.

Note

Strictly speaking, in the first trial, SimulatedAnnealingSampler.sample_independent method is used to sample parameter values. Because intersection_search_space() used in SimulatedAnnealingSampler.infer_relative_search_space cannot infer the search space if there are no complete trials.

Total running time of the script: ( 0 minutes 0.374 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

User-Defined Pruner

In optuna.pruners, we described how an objective function can optionally include calls to a pruning feature which allows Optuna to terminate an optimization trial when intermediate results do not appear promising. In this document, we describe how to implement your own pruner, i.e., a custom strategy for determining when to stop a trial.

Overview of Pruning Interface

The create_study() constructor takes, as an optional argument, a pruner inheriting from BasePruner. The pruner should implement the abstract method prune(), which takes arguments for the associated Study and Trial and returns a boolean value: True if the trial should be pruned and False otherwise. Using the Study and Trial objects, you can access all other trials through the get_trial() method and, and from a trial, its reported intermediate values through the intermediate_values() (a dictionary which maps an integer step to a float value).

You can refer to the source code of the built-in Optuna pruners as templates for building your own. In this document, for illustration, we describe the construction and usage of a simple (but aggressive) pruner which prunes trials that are in last place compared to completed trials at the same step.

Note

Please refer to the documentation of BasePruner or, for example, ThresholdPruner or PercentilePruner for more robust examples of pruner implementation, including error checking and complex pruner-internal logic.

An Example: Implementing LastPlacePruner

We aim to optimize the loss and alpha hyperparameters for a stochastic gradient descent classifier (SGDClassifier) run on the sklearn iris dataset. We implement a pruner which terminates a trial at a certain step if it is in last place compared to completed trials at the same step. We begin considering pruning after a “warmup” of 1 training step and 5 completed trials. For demonstration purposes, we print() a diagnostic message from prune when it is about to return True (indicating pruning).

It may be important to note that the SGDClassifier score, as it is evaluated on a holdout set, decreases with enough training steps due to overfitting. This means that a trial could be pruned even if it had a favorable (high) value on a previous training set. After pruning, Optuna will take the intermediate value last reported as the value of the trial.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDClassifier

import optuna
from optuna.pruners import BasePruner
from optuna.trial._state import TrialState


class LastPlacePruner(BasePruner):
    def __init__(self, warmup_steps, warmup_trials):
        self._warmup_steps = warmup_steps
        self._warmup_trials = warmup_trials

    def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool:
        # Get the latest score reported from this trial
        step = trial.last_step

        if step:  # trial.last_step == None when no scores have been reported yet
            this_score = trial.intermediate_values[step]

            # Get scores from other trials in the study reported at the same step
            completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,))
            other_scores = [
                t.intermediate_values[step]
                for t in completed_trials
                if step in t.intermediate_values
            ]
            other_scores = sorted(other_scores)

            # Prune if this trial at this step has a lower value than all completed trials
            # at the same step. Note that steps will begin numbering at 0 in the objective
            # function definition below.
            if step >= self._warmup_steps and len(other_scores) > self._warmup_trials:
                if this_score < other_scores[0]:
                    print(f"prune() True: Trial {trial.number}, Step {step}, Score {this_score}")
                    return True

        return False

Lastly, let’s confirm the implementation is correct with the simple hyperparameter optimization.

def objective(trial):
    iris = load_iris()
    classes = np.unique(iris.target)
    X_train, X_valid, y_train, y_valid = train_test_split(
        iris.data, iris.target, train_size=100, test_size=50, random_state=0
    )

    loss = trial.suggest_categorical("loss", ["hinge", "log", "perceptron"])
    alpha = trial.suggest_float("alpha", 0.00001, 0.001, log=True)
    clf = SGDClassifier(loss=loss, alpha=alpha, random_state=0)
    score = 0

    for step in range(0, 5):
        clf.partial_fit(X_train, y_train, classes=classes)
        score = clf.score(X_valid, y_valid)

        trial.report(score, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return score


pruner = LastPlacePruner(warmup_steps=1, warmup_trials=5)
study = optuna.create_study(direction="maximize", pruner=pruner)
study.optimize(objective, n_trials=50)

The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 9, Step 1, Score 0.66
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 10, Step 1, Score 0.42
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 11, Step 1, Score 0.64
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 14, Step 1, Score 0.7
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 17, Step 1, Score 0.62
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 20, Step 1, Score 0.66
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 21, Step 2, Score 0.34
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 24, Step 1, Score 0.64
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 27, Step 2, Score 0.36
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 32, Step 1, Score 0.62
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 33, Step 1, Score 0.62
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 34, Step 1, Score 0.62
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 38, Step 1, Score 0.64
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 40, Step 1, Score 0.66
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
prune() True: Trial 46, Step 4, Score 0.66
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.
The loss 'log' was deprecated in v1.1 and will be removed in version 1.3. Use `loss='log_loss'` which is equivalent.

Total running time of the script: ( 0 minutes 0.746 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Callback for Study.optimize

This tutorial showcases how to use & implement Optuna Callback for optimize().

Callback is called after every evaluation of objective, and it takes Study and FrozenTrial as arguments, and does some work.

MLflowCallback is a great example.

Stop optimization after some trials are pruned in a row

This example implements a stateful callback which stops the optimization if a certain number of trials are pruned in a row. The number of trials pruned in a row is specified by threshold.

import optuna


class StopWhenTrialKeepBeingPrunedCallback:
    def __init__(self, threshold: int):
        self.threshold = threshold
        self._consequtive_pruned_count = 0

    def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None:
        if trial.state == optuna.trial.TrialState.PRUNED:
            self._consequtive_pruned_count += 1
        else:
            self._consequtive_pruned_count = 0

        if self._consequtive_pruned_count >= self.threshold:
            study.stop()

This objective prunes all the trials except for the first 5 trials (trial.number starts with 0).

def objective(trial):
    if trial.number > 4:
        raise optuna.TrialPruned

    return trial.suggest_float("x", 0, 1)

Here, we set the threshold to 2: optimization finishes once two trials are pruned in a row. So, we expect this study to stop after 7 trials.

import logging
import sys

# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))

study_stop_cb = StopWhenTrialKeepBeingPrunedCallback(2)
study = optuna.create_study()
study.optimize(objective, n_trials=10, callbacks=[study_stop_cb])

A new study created in memory with name: no-name-8ff871ff-9659-4dc5-90b4-6feb03ad9162
Trial 0 finished with value: 0.4526899773080182 and parameters: {'x': 0.4526899773080182}. Best is trial 0 with value: 0.4526899773080182.
Trial 1 finished with value: 0.17853805031952186 and parameters: {'x': 0.17853805031952186}. Best is trial 1 with value: 0.17853805031952186.
Trial 2 finished with value: 0.3746099795164709 and parameters: {'x': 0.3746099795164709}. Best is trial 1 with value: 0.17853805031952186.
Trial 3 finished with value: 0.3025717263844696 and parameters: {'x': 0.3025717263844696}. Best is trial 1 with value: 0.17853805031952186.
Trial 4 finished with value: 0.2083890335904287 and parameters: {'x': 0.2083890335904287}. Best is trial 1 with value: 0.17853805031952186.
Trial 5 pruned.
Trial 6 pruned.

As you can see in the log above, the study stopped after 7 trials as expected.

Total running time of the script: ( 0 minutes 0.008 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Specify Hyperparameters Manually

It’s natural that you have some specific sets of hyperparameters to try first such as initial learning rate values and the number of leaves. Also, it’s possible that you’ve already tried those sets before having Optuna find better sets of hyperparameters.

Optuna provides two APIs to support such cases:

Passing those sets of hyperparameters and let Optuna evaluate them - enqueue_trial()
Adding the results of those sets as completed Trials - add_trial()

First Scenario: Have Optuna evaluate your hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters but have not evaluated them yet and decided to use Optuna to find better sets of hyperparameters.

Optuna has optuna.study.Study.enqueue_trial() which lets you pass those sets of hyperparameters to Optuna and Optuna will evaluate them.

This section walks you through how to use this lit API with LightGBM.

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna

Define the objective function.

def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1),
        "bagging_freq": trial.suggest_int("bagging_freq", 0, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    # Add a callback for pruning.
    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
    gbm = lgb.train(
        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
    )

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

Then, construct Study for hyperparameter optimization.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())

Here, we get Optuna evaluate some sets with larger "bagging_fraq" value and the default values.

study.enqueue_trial(
    {
        "bagging_fraction": 1.0,
        "bagging_freq": 0,
        "min_child_samples": 20,
    }
)

study.enqueue_trial(
    {
        "bagging_fraction": 0.75,
        "bagging_freq": 5,
        "min_child_samples": 20,
    }
)

import logging
import sys

# Add stream handler of stdout to show the messages to see Optuna works expectedly.
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study.optimize(objective, n_trials=100, timeout=600)

'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 0 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 1.0, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 0 with value: 0.972027972027972.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 1 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.75, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 2 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.9060975249671693, 'bagging_freq': 2, 'min_child_samples': 71}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 3 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4310982251079994, 'bagging_freq': 0, 'min_child_samples': 57}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 4 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6859656577616365, 'bagging_freq': 5, 'min_child_samples': 68}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 5 pruned. Trial was pruned at iteration 0.
Trial 6 pruned. Trial was pruned at iteration 1.
Trial 7 pruned. Trial was pruned at iteration 0.
Trial 8 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5217150677825523, 'bagging_freq': 4, 'min_child_samples': 36}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 0.
Trial 11 pruned. Trial was pruned at iteration 0.
Trial 12 pruned. Trial was pruned at iteration 0.
Trial 13 pruned. Trial was pruned at iteration 1.
Trial 14 pruned. Trial was pruned at iteration 3.
Trial 15 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7347261947960549, 'bagging_freq': 1, 'min_child_samples': 46}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 16 pruned. Trial was pruned at iteration 0.
Trial 17 pruned. Trial was pruned at iteration 1.
Trial 18 pruned. Trial was pruned at iteration 1.
Trial 19 pruned. Trial was pruned at iteration 54.
Trial 20 pruned. Trial was pruned at iteration 0.
Trial 21 pruned. Trial was pruned at iteration 1.
Trial 22 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9439115556607582, 'bagging_freq': 1, 'min_child_samples': 20}. Best is trial 1 with value: 0.986013986013986.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 23 pruned. Trial was pruned at iteration 1.
Trial 24 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.7780376220231672, 'bagging_freq': 3, 'min_child_samples': 44}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 25 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.773234485324975, 'bagging_freq': 3, 'min_child_samples': 48}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 26 pruned. Trial was pruned at iteration 0.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 0.
Trial 30 pruned. Trial was pruned at iteration 0.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 pruned. Trial was pruned at iteration 0.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 0.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 pruned. Trial was pruned at iteration 0.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 0.
Trial 42 pruned. Trial was pruned at iteration 13.
Trial 43 pruned. Trial was pruned at iteration 8.
Trial 44 pruned. Trial was pruned at iteration 0.
Trial 45 pruned. Trial was pruned at iteration 4.
Trial 46 pruned. Trial was pruned at iteration 38.
Trial 47 pruned. Trial was pruned at iteration 0.
Trial 48 pruned. Trial was pruned at iteration 0.
Trial 49 pruned. Trial was pruned at iteration 0.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 0.
Trial 53 pruned. Trial was pruned at iteration 0.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7483469634393837, 'bagging_freq': 3, 'min_child_samples': 40}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 56 pruned. Trial was pruned at iteration 4.
Trial 57 pruned. Trial was pruned at iteration 0.
Trial 58 pruned. Trial was pruned at iteration 1.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 0.
Trial 63 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.747646495537022, 'bagging_freq': 1, 'min_child_samples': 29}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 64 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8232739883521802, 'bagging_freq': 2, 'min_child_samples': 16}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 0.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7801074358187047, 'bagging_freq': 1, 'min_child_samples': 26}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 72 pruned. Trial was pruned at iteration 0.
Trial 73 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7524186612588695, 'bagging_freq': 1, 'min_child_samples': 33}. Best is trial 24 with value: 0.993006993006993.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 74 pruned. Trial was pruned at iteration 0.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 0.
Trial 77 pruned. Trial was pruned at iteration 0.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 0.
Trial 81 pruned. Trial was pruned at iteration 0.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 pruned. Trial was pruned at iteration 0.
Trial 84 pruned. Trial was pruned at iteration 0.
Trial 85 pruned. Trial was pruned at iteration 0.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 0.
Trial 93 pruned. Trial was pruned at iteration 0.
Trial 94 pruned. Trial was pruned at iteration 0.
Trial 95 pruned. Trial was pruned at iteration 0.
Trial 96 pruned. Trial was pruned at iteration 0.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 0.
Trial 99 pruned. Trial was pruned at iteration 0.

Second scenario: Have Optuna utilize already evaluated hyperparameters

In this scenario, let’s assume you have some out-of-box sets of hyperparameters and you have already evaluated them but the results are not desirable so that you are thinking of using Optuna.

Optuna has optuna.study.Study.add_trial() which lets you register those results to Optuna and then Optuna will sample hyperparameters taking them into account.

In this section, the objective is the same as the first scenario.

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 1.0,
            "bagging_freq": 0,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.94,
    )
)
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 0.75,
            "bagging_freq": 5,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.95,
    )
)
study.optimize(objective, n_trials=100, timeout=600)

A new study created in memory with name: no-name-c1eeed57-63bc-42f3-ad80-21a940f9c0eb
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 2 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.9084942488405118, 'bagging_freq': 6, 'min_child_samples': 58}. Best is trial 2 with value: 0.951048951048951.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 3 finished with value: 1.0 and parameters: {'bagging_fraction': 0.6420519695923782, 'bagging_freq': 4, 'min_child_samples': 41}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 4 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5708515203907085, 'bagging_freq': 1, 'min_child_samples': 66}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 5 pruned. Trial was pruned at iteration 0.
Trial 6 pruned. Trial was pruned at iteration 0.
Trial 7 pruned. Trial was pruned at iteration 0.
Trial 8 pruned. Trial was pruned at iteration 0.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 0.
Trial 11 pruned. Trial was pruned at iteration 0.
Trial 12 pruned. Trial was pruned at iteration 0.
Trial 13 pruned. Trial was pruned at iteration 1.
Trial 14 pruned. Trial was pruned at iteration 0.
Trial 15 pruned. Trial was pruned at iteration 0.
Trial 16 pruned. Trial was pruned at iteration 0.
Trial 17 pruned. Trial was pruned at iteration 0.
Trial 18 pruned. Trial was pruned at iteration 0.
Trial 19 pruned. Trial was pruned at iteration 0.
Trial 20 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8147128537761313, 'bagging_freq': 7, 'min_child_samples': 48}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 21 pruned. Trial was pruned at iteration 0.
Trial 22 pruned. Trial was pruned at iteration 0.
Trial 23 pruned. Trial was pruned at iteration 0.
Trial 24 pruned. Trial was pruned at iteration 0.
Trial 25 pruned. Trial was pruned at iteration 1.
Trial 26 pruned. Trial was pruned at iteration 0.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 0.
Trial 30 pruned. Trial was pruned at iteration 0.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 pruned. Trial was pruned at iteration 1.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 0.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 pruned. Trial was pruned at iteration 0.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 0.
Trial 42 pruned. Trial was pruned at iteration 1.
Trial 43 pruned. Trial was pruned at iteration 0.
Trial 44 pruned. Trial was pruned at iteration 0.
Trial 45 pruned. Trial was pruned at iteration 0.
Trial 46 pruned. Trial was pruned at iteration 0.
Trial 47 pruned. Trial was pruned at iteration 0.
Trial 48 pruned. Trial was pruned at iteration 1.
Trial 49 pruned. Trial was pruned at iteration 0.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 1.
Trial 53 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.9220153556338132, 'bagging_freq': 0, 'min_child_samples': 27}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 pruned. Trial was pruned at iteration 1.
Trial 56 pruned. Trial was pruned at iteration 0.
Trial 57 pruned. Trial was pruned at iteration 0.
Trial 58 pruned. Trial was pruned at iteration 0.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 1.
Trial 63 pruned. Trial was pruned at iteration 0.
Trial 64 pruned. Trial was pruned at iteration 0.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 0.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8543410725477059, 'bagging_freq': 5, 'min_child_samples': 16}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 72 pruned. Trial was pruned at iteration 0.
Trial 73 pruned. Trial was pruned at iteration 0.
Trial 74 finished with value: 1.0 and parameters: {'bagging_fraction': 0.9021528361635935, 'bagging_freq': 6, 'min_child_samples': 9}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 0.
Trial 77 pruned. Trial was pruned at iteration 1.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 0.
Trial 81 pruned. Trial was pruned at iteration 1.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 pruned. Trial was pruned at iteration 1.
Trial 84 pruned. Trial was pruned at iteration 0.
Trial 85 pruned. Trial was pruned at iteration 0.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 1.
Trial 93 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.8491207237527549, 'bagging_freq': 5, 'min_child_samples': 58}. Best is trial 3 with value: 1.0.
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 94 pruned. Trial was pruned at iteration 0.
Trial 95 pruned. Trial was pruned at iteration 0.
Trial 96 pruned. Trial was pruned at iteration 1.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 0.
Trial 99 pruned. Trial was pruned at iteration 0.
Trial 100 pruned. Trial was pruned at iteration 1.
Trial 101 pruned. Trial was pruned at iteration 1.

Total running time of the script: ( 0 minutes 6.438 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Ask-and-Tell Interface

Optuna has an Ask-and-Tell interface, which provides a more flexible interface for hyperparameter optimization. This tutorial explains three use-cases when the ask-and-tell interface is beneficial:

Apply Optuna to an existing optimization problem with minimum modifications
Define-and-Run
Batch Optimization

Apply Optuna to an existing optimization problem with minimum modifications

Let’s consider the traditional supervised classification problem; you aim to maximize the validation accuracy. To do so, you train LogisticRegression as a simple model.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

import optuna


X, y = make_classification(n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y)

C = 0.01
clf = LogisticRegression(C=C)
clf.fit(X_train, y_train)
val_accuracy = clf.score(X_test, y_test)  # the objective

Then you try to optimize hyperparameters C and solver of the classifier by using optuna. When you introduce optuna naively, you define an objective function such that it takes trial and calls suggest_* methods of trial to sample the hyperparameters:

def objective(trial):
    X, y = make_classification(n_features=10)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    C = trial.suggest_float("C", 1e-7, 10.0, log=True)
    solver = trial.suggest_categorical("solver", ("lbfgs", "saga"))

    clf = LogisticRegression(C=C, solver=solver)
    clf.fit(X_train, y_train)
    val_accuracy = clf.score(X_test, y_test)

    return val_accuracy


study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=10)

This interface is not flexible enough. For example, if objective requires additional arguments other than trial, you need to define a class as in How to define objective functions that have own arguments?. The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters. The following example is equivalent to the previous code block.

study = optuna.create_study(direction="maximize")

n_trials = 10
for _ in range(n_trials):
    trial = study.ask()  # `trial` is a `Trial` and not a `FrozenTrial`.

    C = trial.suggest_float("C", 1e-7, 10.0, log=True)
    solver = trial.suggest_categorical("solver", ("lbfgs", "saga"))

    clf = LogisticRegression(C=C, solver=solver)
    clf.fit(X_train, y_train)
    val_accuracy = clf.score(X_test, y_test)

    study.tell(trial, val_accuracy)  # tell the pair of trial and objective value

The main difference is to use two methods: optuna.study.Study.ask() and optuna.study.Study.tell(). optuna.study.Study.ask() creates a trial that can sample hyperparameters, and optuna.study.Study.tell() finishes the trial by passing trial and an objective value. You can apply Optuna’s hyperparameter optimization to your original code without an objective function.

If you want to make your optimization faster with a pruner, you need to explicitly pass the state of trial to the argument of optuna.study.Study.tell() method as follows:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna


X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)
n_train_iter = 100

# define study with hyperband pruner.
study = optuna.create_study(
    direction="maximize",
    pruner=optuna.pruners.HyperbandPruner(
        min_resource=1, max_resource=n_train_iter, reduction_factor=3
    ),
)

for _ in range(20):
    trial = study.ask()

    alpha = trial.suggest_float("alpha", 0.0, 1.0)

    clf = SGDClassifier(alpha=alpha)
    pruned_trial = False

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            pruned_trial = True
            break

    if pruned_trial:
        study.tell(trial, state=optuna.trial.TrialState.PRUNED)  # tell the pruned state
    else:
        score = clf.score(X_valid, y_valid)
        study.tell(trial, score)  # tell objective value

Note

optuna.study.Study.tell() method can take a trial number rather than the trial object. study.tell(trial.number, y) is equivalent to study.tell(trial, y).

Define-and-Run

The ask-and-tell interface supports both define-by-run and define-and-run APIs. This section shows the example of the define-and-run API in addition to the define-by-run example above.

Define distributions for the hyperparameters before calling the optuna.study.Study.ask() method for define-and-run API. For example,

distributions = {
    "C": optuna.distributions.FloatDistribution(1e-7, 10.0, log=True),
    "solver": optuna.distributions.CategoricalDistribution(("lbfgs", "saga")),
}

Pass distributions to optuna.study.Study.ask() method at each call. The retuned trial contains the suggested hyperparameters.

study = optuna.create_study(direction="maximize")
n_trials = 10
for _ in range(n_trials):
    trial = study.ask(distributions)  # pass the pre-defined distributions.

    # two hyperparameters are already sampled from the pre-defined distributions
    C = trial.params["C"]
    solver = trial.params["solver"]

    clf = LogisticRegression(C=C, solver=solver)
    clf.fit(X_train, y_train)
    val_accuracy = clf.score(X_test, y_test)

    study.tell(trial, val_accuracy)

Batch Optimization

The ask-and-tell interface enables us to optimize a batched objective for faster optimization. For example, parallelizable evaluation, operation over vectors, etc.

The following objective takes batched hyperparameters xs and ys instead of a single pair of hyperparameters x and y and calculates the objective over the full vectors.

def batched_objective(xs: np.ndarray, ys: np.ndarray):
    return xs**2 + ys

In the following example, the number of pairs of hyperparameters in a batch is \(10\), and batched_objective is evaluated three times. Thus, the number of trials is \(30\). Note that you need to store either trial_numbers or trial to call optuna.study.Study.tell() method after the batched evaluations.

batch_size = 10
study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())

for _ in range(3):

    # create batch
    trial_numbers = []
    x_batch = []
    y_batch = []
    for _ in range(batch_size):
        trial = study.ask()
        trial_numbers.append(trial.number)
        x_batch.append(trial.suggest_float("x", -10, 10))
        y_batch.append(trial.suggest_float("y", -10, 10))

    # evaluate batched objective
    x_batch = np.array(x_batch)
    y_batch = np.array(y_batch)
    objectives = batched_objective(x_batch, y_batch)

    # finish all trials in the batch
    for trial_number, objective in zip(trial_numbers, objectives):
        study.tell(trial_number, objective)

Total running time of the script: ( 0 minutes 0.132 seconds)

Gallery generated by Sphinx-Gallery

Note

Click here to download the full example code

Re-use the best trial

In some cases, you may want to re-evaluate the objective function with the best hyperparameters again after the hyperparameter optimization.

For example,

You have found good hyperparameters with Optuna and want to run a similar objective function using the best hyperparameters found so far to further analyze the results, or
You have optimized with Optuna using a partial dataset to reduce training time. After the hyperparameter tuning, you want to train the model using the whole dataset with the best hyperparameter values found.

best_trial provides an interface to re-evaluate the objective function with the current best hyperparameter values.

This tutorial shows an example of how to re-run a different objective function with the current best values, like the first example above.

Investigating the best model further

Let’s consider a classical supervised classification problem with Optuna as follows:

from sklearn import metrics
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split


import optuna


def objective(trial):
    X, y = make_classification(n_features=10, random_state=1)
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

    C = trial.suggest_float("C", 1e-7, 10.0, log=True)

    clf = LogisticRegression(C=C)
    clf.fit(X_train, y_train)

    return clf.score(X_test, y_test)


study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=10)

print(study.best_trial.value)  # Show the best value.

0.92

Suppose after the hyperparameter optimization, you want to calculate other evaluation metrics such as recall, precision, and f1-score on the same dataset. You can define another objective function that shares most of the objective function to reproduce the model with the best hyperparameters.

def detailed_objective(trial):
    # Use same code objective to reproduce the best model
    X, y = make_classification(n_features=10, random_state=1)
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

    C = trial.suggest_float("C", 1e-7, 10.0, log=True)

    clf = LogisticRegression(C=C)
    clf.fit(X_train, y_train)

    # calculate more evaluation metrics
    pred = clf.predict(X_test)

    acc = metrics.accuracy_score(pred, y_test)
    recall = metrics.recall_score(pred, y_test)
    precision = metrics.precision_score(pred, y_test)
    f1 = metrics.f1_score(pred, y_test)

    return acc, f1, recall, precision

Pass study.best_trial as the argument of detailed_objective.

detailed_objective(study.best_trial)  # calculate acc, f1, recall, and precision

(0.92, 0.9285714285714286, 0.9285714285714286, 0.9285714285714286)

The difference between best_trial and ordinal trials

This uses best_trial, which returns the best_trial as a FrozenTrial. The FrozenTrial is different from an active trial and behaves differently from Trial in some situations. For example, pruning does not work because should_prune always returns False.

Note

For multi-objective optimization as demonstrated by Multi-objective Optimization with Optuna, best_trials returns a list of FrozenTrial on Pareto front. So we can re-use each trial in the list by the similar way above.

Total running time of the script: ( 0 minutes 0.058 seconds)

Gallery generated by Sphinx-Gallery

API Reference

optuna

The optuna module is primarily used as an alias for basic Optuna functionality coded in other modules. Currently, two modules are aliased: (1) from optuna.study, functions regarding the Study lifecycle, and (2) from optuna.exceptions, the TrialPruned Exception raised when a trial is pruned.

`optuna.create_study`	Create a new `Study`.
`optuna.load_study`	Load the existing `Study` that has the specified name.
`optuna.delete_study`	Delete a `Study` object.
`optuna.copy_study`	Copy study from one storage to another.
`optuna.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.TrialPruned`	Exception for pruned trials.

optuna.cli

The cli module implements Optuna’s command-line functionality using the cliff framework.

See also

The Command-Line Interface tutorial provides use-cases with examples.

optuna
    [--version]
    [-v | -q]
    [--log-file LOG_FILE]
    [--debug]
    [--storage STORAGE]

--version: show program’s version number and exit

-v, --verbose: Increase verbosity of output. Can be repeated.

-q, --quiet: Suppress output except warnings and errors.

--log-file <LOG_FILE>: Specify a file to log output. Disabled by default.

--debug: Show tracebacks on errors.

--storage <STORAGE>: DB URL. (e.g. sqlite:///example.db)

ask

Create a new trial and suggest parameters.

optuna ask
    [--study-name STUDY_NAME]
    [--direction {minimize,maximize}]
    [--directions {minimize,maximize} [{minimize,maximize} ...]]
    [--sampler SAMPLER]
    [--sampler-kwargs SAMPLER_KWARGS]
    [--search-space SEARCH_SPACE]
    [-f {json,table,yaml}]
    [--flatten]

--study-name <STUDY_NAME>: Name of study.

--direction <DIRECTION>: Direction of optimization. This argument is deprecated. Please create a study in advance.

--directions <DIRECTIONS>: Directions of optimization, if there are multiple objectives. This argument is deprecated. Please create a study in advance.

--sampler <SAMPLER>: Class name of sampler object to create.

--sampler-kwargs <SAMPLER_KWARGS>: Sampler object initialization keyword arguments as JSON.

--search-space <SEARCH_SPACE>: Search space as JSON. Keys are names and values are outputs from distribution_to_json().

-f <FORMAT>, --format <FORMAT>: Output format.

--flatten: Flatten nested columns such as params.

This command is provided by the optuna plugin.

best-trial

Show the best trial.

optuna best-trial
    --study-name STUDY_NAME
    [-f {json,table,yaml}]
    [--flatten]

--study-name <STUDY_NAME>: The name of the study to get the best trial.

-f <FORMAT>, --format <FORMAT>: Output format.

--flatten: Flatten nested columns such as params and user_attrs.

This command is provided by the optuna plugin.

best-trials

Show a list of trials located at the Pareto front.

optuna best-trials
    --study-name STUDY_NAME
    [-f {json,table,yaml}]
    [--flatten]

--study-name <STUDY_NAME>: The name of the study to get the best trials (trials at the Pareto front).

-f <FORMAT>, --format <FORMAT>: Output format.

--flatten: Flatten nested columns such as params and user_attrs.

This command is provided by the optuna plugin.

create-study

Create a new study.

optuna create-study
    [--study-name STUDY_NAME]
    [--direction {minimize,maximize}]
    [--skip-if-exists]
    [--directions {minimize,maximize} [{minimize,maximize} ...]]

--study-name <STUDY_NAME>: A human-readable name of a study to distinguish it from others.

--direction <DIRECTION>: Set direction of optimization to a new study. Set ‘minimize’ for minimization and ‘maximize’ for maximization.

--skip-if-exists: If specified, the creation of the study is skipped without any error when the study name is duplicated.

--directions <DIRECTIONS>: Set directions of optimization to a new study. Put whitespace between directions. Each direction should be either “minimize” or “maximize”.

This command is provided by the optuna plugin.

delete-study

Delete a specified study.

optuna delete-study [--study-name STUDY_NAME]

--study-name <STUDY_NAME>: The name of the study to delete.

This command is provided by the optuna plugin.

storage upgrade

Upgrade the schema of a storage.

optuna storage upgrade

This command is provided by the optuna plugin.

studies

Show a list of studies.

optuna studies [-f {json,table,yaml}] [--flatten]

-f <FORMAT>, --format <FORMAT>: Output format.

--flatten: Flatten nested columns such as directions.

This command is provided by the optuna plugin.

study optimize

Start optimization of a study. Deprecated since version 2.0.0.

optuna study optimize
    [--n-trials N_TRIALS]
    [--timeout TIMEOUT]
    [--n-jobs N_JOBS]
    [--study STUDY]
    [--study-name STUDY_NAME]
    file
    method

--n-trials <N_TRIALS>: The number of trials. If this argument is not given, as many trials run as possible.

--timeout <TIMEOUT>: Stop study after the given number of second(s). If this argument is not given, as many trials run as possible.

--n-jobs <N_JOBS>: The number of parallel jobs. If this argument is set to -1, the number is set to CPU counts.

--study <STUDY>: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>: The name of the study to start optimization on.

file: Python script file where the objective function resides.

method: The method name of the objective function.

This command is provided by the optuna plugin.

study set-user-attr

Set a user attribute to a study.

optuna study set-user-attr
    [--study STUDY]
    [--study-name STUDY_NAME]
    --key KEY
    --value VALUE

--study <STUDY>: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>: The name of the study to set the user attribute to.

--key <KEY>, -k <KEY>: Key of the user attribute.

--value <VALUE>: Value to be set.

This command is provided by the optuna plugin.

tell

Finish a trial, which was created by the ask command.

optuna tell
    [--study-name STUDY_NAME]
    [--trial-number TRIAL_NUMBER]
    [--values VALUES [VALUES ...]]
    [--state {complete,pruned,fail}]
    [--skip-if-finished]

--study-name <STUDY_NAME>: Name of study.

--trial-number <TRIAL_NUMBER>: Trial number.

--values <VALUES>: Objective values.

--state <STATE>: Trial state.

--skip-if-finished: If specified, tell is skipped without any error when the trial is alreadyfinished.

This command is provided by the optuna plugin.

trials

Show a list of trials.

optuna trials
    --study-name STUDY_NAME
    [-f {json,table,yaml}]
    [--flatten]

--study-name <STUDY_NAME>: The name of the study which includes trials.

-f <FORMAT>, --format <FORMAT>: Output format.

--flatten: Flatten nested columns such as params and user_attrs.

This command is provided by the optuna plugin.

optuna.distributions

The distributions module defines various classes representing probability distributions, mainly used to suggest initial hyperparameter values for an optimization trial. Distribution classes inherit from a library-internal BaseDistribution, and is initialized with specific parameters, such as the low and high endpoints for a IntDistribution.

Optuna users should not use distribution classes directly, but instead use utility functions provided by Trial such as suggest_int().

`optuna.distributions.FloatDistribution`	A distribution on floats.
`optuna.distributions.IntDistribution`	A distribution on integers.
`optuna.distributions.UniformDistribution`	A uniform distribution in the linear domain.
`optuna.distributions.LogUniformDistribution`	A uniform distribution in the log domain.
`optuna.distributions.DiscreteUniformDistribution`	A discretized uniform distribution in the linear domain.
`optuna.distributions.IntUniformDistribution`	A uniform distribution on integers.
`optuna.distributions.IntLogUniformDistribution`	A uniform distribution on integers in the log domain.
`optuna.distributions.CategoricalDistribution`	A categorical distribution.
`optuna.distributions.distribution_to_json`	Serialize a distribution to JSON format.
`optuna.distributions.json_to_distribution`	Deserialize a distribution in JSON format.
`optuna.distributions.check_distribution_compatibility`	A function to check compatibility of two distributions.

optuna.exceptions

The exceptions module defines Optuna-specific exceptions deriving from a base OptunaError class. Of special importance for library users is the TrialPruned exception to be raised if optuna.trial.Trial.should_prune() returns True for a trial that should be pruned.

`optuna.exceptions.OptunaError`	Base class for Optuna specific errors.
`optuna.exceptions.TrialPruned`	Exception for pruned trials.
`optuna.exceptions.CLIUsageError`	Exception for CLI.
`optuna.exceptions.StorageInternalError`	Exception for storage operation.
`optuna.exceptions.DuplicatedStudyError`	Exception for a duplicated study name.

optuna.importance

The importance module provides functionality for evaluating hyperparameter importances based on completed trials in a given study. The utility function get_param_importances() takes a Study and optional evaluator as two of its inputs. The evaluator must derive from BaseImportanceEvaluator, and is initialized as a FanovaImportanceEvaluator by default when not passed in. Users implementing custom evaluators should refer to either FanovaImportanceEvaluator or MeanDecreaseImpurityImportanceEvaluator as a guide, paying close attention to the format of the return value from the Evaluator’s evaluate() function.

`optuna.importance.get_param_importances`	Evaluate parameter importances based on completed trials in the given study.
`optuna.importance.FanovaImportanceEvaluator`	fANOVA importance evaluator.
`optuna.importance.MeanDecreaseImpurityImportanceEvaluator`	Mean Decrease Impurity (MDI) parameter importance evaluator.

optuna.integration

The integration module contains classes used to integrate Optuna with external machine learning frameworks.

For most of the ML frameworks supported by Optuna, the corresponding Optuna integration class serves only to implement a callback object and functions, compliant with the framework’s specific callback API, to be called with each intermediate step in the model training. The functionality implemented in these callbacks across the different ML frameworks includes:

Reporting intermediate model scores back to the Optuna trial using optuna.trial.report(),
According to the results of optuna.trial.Trial.should_prune(), pruning the current model by raising optuna.TrialPruned(), and
Reporting intermediate Optuna data such as the current trial number back to the framework, as done in MLflowCallback.

For scikit-learn, an integrated OptunaSearchCV estimator is available that combines scikit-learn BaseEstimator functionality with access to a class-level Study object.

AllenNLP

`optuna.integration.AllenNLPExecutor`	AllenNLP extension to use optuna with Jsonnet config file.
`optuna.integration.allennlp.dump_best_config`	Save JSON config file with environment variables and best performing hyperparameters.
`optuna.integration.AllenNLPPruningCallback`	AllenNLP callback to prune unpromising trials.

BoTorch

`optuna.integration.BoTorchSampler`	A sampler that uses BoTorch, a Bayesian optimization library built on top of PyTorch.
`optuna.integration.botorch.qei_candidates_func`	Quasi MC-based batch Expected Improvement (qEI).
`optuna.integration.botorch.qehvi_candidates_func`	Quasi MC-based batch Expected Hypervolume Improvement (qEHVI).
`optuna.integration.botorch.qparego_candidates_func`	Quasi MC-based extended ParEGO (qParEGO) for constrained multi-objective optimization.

Catalyst

optuna.integration.CatalystPruningCallback

Catalyst callback to prune unpromising trials.

CatBoost

optuna.integration.CatBoostPruningCallback

Callback for catboost to prune unpromising trials.

Chainer

`optuna.integration.ChainerPruningExtension`	Chainer extension to prune unpromising trials.
`optuna.integration.ChainerMNStudy`	A wrapper of `Study` to incorporate Optuna with ChainerMN.

fast.ai

`optuna.integration.FastAIV1PruningCallback`	FastAI callback to prune unpromising trials for fastai.
`optuna.integration.FastAIV2PruningCallback`	FastAI callback to prune unpromising trials for fastai.
`optuna.integration.FastAIPruningCallback`	alias of `FastAIV2PruningCallback`

Keras

optuna.integration.KerasPruningCallback

Keras callback to prune unpromising trials.

LightGBM

`optuna.integration.LightGBMPruningCallback`	Callback for LightGBM to prune unpromising trials.
`optuna.integration.lightgbm.train`	Wrapper of LightGBM Training API to tune hyperparameters.
`optuna.integration.lightgbm.LightGBMTuner`	Hyperparameter tuner for LightGBM.
`optuna.integration.lightgbm.LightGBMTunerCV`	Hyperparameter tuner for LightGBM with cross-validation.

MLflow

optuna.integration.MLflowCallback

Callback to track Optuna trials with MLflow.

Weights & Biases

optuna.integration.WeightsAndBiasesCallback

Callback to track Optuna trials with Weights & Biases.

MXNet

optuna.integration.MXNetPruningCallback

MXNet callback to prune unpromising trials.

pycma

`optuna.integration.PyCmaSampler`	A Sampler using cma library as the backend.
`optuna.integration.CmaEsSampler`	Wrapper class of PyCmaSampler for backward compatibility.

PyTorch

`optuna.integration.PyTorchIgnitePruningHandler`	PyTorch Ignite handler to prune unpromising trials.
`optuna.integration.PyTorchLightningPruningCallback`	PyTorch Lightning callback to prune unpromising trials.
`optuna.integration.TorchDistributedTrial`	A wrapper of `Trial` to incorporate Optuna with PyTorch distributed.

scikit-learn

optuna.integration.OptunaSearchCV

Hyperparameter search with cross-validation.

scikit-optimize

optuna.integration.SkoptSampler

Sampler using Scikit-Optimize as the backend.

SHAP

optuna.integration.ShapleyImportanceEvaluator

Shapley (SHAP) parameter importance evaluator.

skorch

optuna.integration.SkorchPruningCallback

Skorch callback to prune unpromising trials.

TensorFlow

`optuna.integration.TensorBoardCallback`	Callback to track Optuna trials with TensorBoard.
`optuna.integration.TensorFlowPruningHook`	TensorFlow SessionRunHook to prune unpromising trials.
`optuna.integration.TFKerasPruningCallback`	tf.keras callback to prune unpromising trials.

XGBoost

optuna.integration.XGBoostPruningCallback

Callback for XGBoost to prune unpromising trials.

optuna.logging

The logging module implements logging using the Python logging package. Library users may be especially interested in setting verbosity levels using set_verbosity() to one of optuna.logging.CRITICAL (aka optuna.logging.FATAL), optuna.logging.ERROR, optuna.logging.WARNING (aka optuna.logging.WARN), optuna.logging.INFO, or optuna.logging.DEBUG.

`optuna.logging.get_verbosity`	Return the current level for the Optuna's root logger.
`optuna.logging.set_verbosity`	Set the level for the Optuna's root logger.
`optuna.logging.disable_default_handler`	Disable the default handler of the Optuna's root logger.
`optuna.logging.enable_default_handler`	Enable the default handler of the Optuna's root logger.
`optuna.logging.disable_propagation`	Disable propagation of the library log outputs.
`optuna.logging.enable_propagation`	Enable propagation of the library log outputs.

optuna.multi_objective

This module is deprecated, with former functionality moved to optuna.samplers, optuna.study, optuna.trial and optuna.visualization.

optuna.multi_objective.samplers

`optuna.multi_objective.samplers.BaseMultiObjectiveSampler`	Base class for multi-objective samplers.
`optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler`	Multi-objective sampler using the NSGA-II algorithm.
`optuna.multi_objective.samplers.RandomMultiObjectiveSampler`	Multi-objective sampler using random sampling.
`optuna.multi_objective.samplers.MOTPEMultiObjectiveSampler`	Multi-objective sampler using the MOTPE algorithm.

optuna.multi_objective.study

`optuna.multi_objective.study.MultiObjectiveStudy`	A study corresponds to a multi-objective optimization task, i.e., a set of trials.
`optuna.multi_objective.study.create_study`	Create a new `MultiObjectiveStudy`.
`optuna.multi_objective.study.load_study`	Load the existing `MultiObjectiveStudy` that has the specified name.

optuna.multi_objective.trial

`optuna.multi_objective.trial.MultiObjectiveTrial`	A trial is a process of evaluating an objective function.
`optuna.multi_objective.trial.FrozenMultiObjectiveTrial`	Status and results of a `MultiObjectiveTrial`.

optuna.multi_objective.visualization

Note

optuna.multi_objective.visualization module uses plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

optuna.multi_objective.visualization.plot_pareto_front

Plot the pareto front of a study.

optuna.pruners

The pruners module defines a BasePruner class characterized by an abstract prune() method, which, for a given trial and its associated study, returns a boolean value representing whether the trial should be pruned. This determination is made based on stored intermediate values of the objective function, as previously reported for the trial using optuna.trial.Trial.report(). The remaining classes in this module represent child classes, inheriting from BasePruner, which implement different pruning strategies.

See also

3. Efficient Optimization Algorithms tutorial explains the concept of the pruner classes and a minimal example.

See also

User-Defined Pruner tutorial could be helpful if you want to implement your own pruner classes.

`optuna.pruners.BasePruner`	Base class for pruners.
`optuna.pruners.MedianPruner`	Pruner using the median stopping rule.
`optuna.pruners.NopPruner`	Pruner which never prunes trials.
`optuna.pruners.PatientPruner`	Pruner which wraps another pruner with tolerance.
`optuna.pruners.PercentilePruner`	Pruner to keep the specified percentile of the trials.
`optuna.pruners.SuccessiveHalvingPruner`	Pruner using Asynchronous Successive Halving Algorithm.
`optuna.pruners.HyperbandPruner`	Pruner using Hyperband.
`optuna.pruners.ThresholdPruner`	Pruner to detect outlying metrics of the trials.

optuna.samplers

The samplers module defines a base class for parameter sampling as described extensively in BaseSampler. The remaining classes in this module represent child classes, deriving from BaseSampler, which implement different sampling strategies.

See also

3. Efficient Optimization Algorithms tutorial explains the overview of the sampler classes.

See also

User-Defined Sampler tutorial could be helpful if you want to implement your own sampler classes.

	RandomSampler	GridSampler	TPESampler	CmaEsSampler	NSGAIISampler	QMCSampler	BoTorchSampler
Float parameters	✅	✅	✅	✅	▲	✅	✅
Integer parameters	✅	✅	✅	✅	▲	✅	✅
Categorical parameters	✅	✅	✅	▲	✅	▲	✅
Pruning	✅	✅	✅	▲	❌	✅	▲
Multivariate optimization	▲	▲	✅	✅	▲	▲	✅
Conditional search space	✅	▲	✅	▲	▲	▲	▲
Multi-objective optimization	✅	▲	✅	❌	✅ (▲ for single-objective)	▲	✅
Batch optimization	✅	✅	✅	✅	✅	✅	▲
Distributed optimization	✅	✅	✅	✅	✅	✅	▲
Constrained optimization	❌	❌	✅	❌	✅	❌	✅
Time complexity (per trial) (*)	\(O(d)\)	\(O(dn)\)	\(O(dn \log n)\)	\(O(d^3)\)	\(O(mnp)\)	\(O(dn)\)	\(O(n^3)\)
Recommended budgets (#trials) (**)	as many as one likes	number of combinations	100 – 1000	1000 – 10000	100 – 10000	as many as one likes	10 – 100

Note

✅: Supports this feature. ▲ : Works, but inefficiently. ❌: Causes an error, or has no interface.

(*): We assumes that \(d\) is the dimension of the search space, \(n\) is the number of finished trials, \(m\) is the number of objectives, and \(p\) is the population size (algorithm specific parameter). This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including \(O(d)\) to call the sampling methods and \(O(n)\) to collect the completed trials. This means that, for example, the actual time complexity of RandomSampler is \(O(d+n+d) = O(d+n)\). From another perspective, with the exception of NSGAIISampler, all time complexity is written for single-objective optimization.

(**): The budget depends on the number of parameters and the number of objectives.

Note

For float, integer, or categorical parameters, see 2. Pythonic Search Space tutorial.

For pruning, see 3. Efficient Optimization Algorithms tutorial.

For multivariate optimization, see BaseSampler. The multivariate optimization is implemented as sample_relative() in Optuna. Please check the concrete documents of samplers for more details.

For conditional search space, see 2. Pythonic Search Space tutorial and TPESampler. The group option of TPESampler allows TPESampler to handle the conditional search space.

For multi-objective optimization, see Multi-objective Optimization with Optuna tutorial.

For batch optimization, see Batch Optimization tutorial. Note that the constant_liar option of TPESampler allows TPESampler to handle the batch optimization.

For distributed optimization, see 4. Easy Parallelization tutorial. Note that the constant_liar option of TPESampler allows TPESampler to handle the distributed optimization.

For constrained optimization, see an example.

`optuna.samplers.BaseSampler`	Base class for samplers.
`optuna.samplers.GridSampler`	Sampler using grid search.
`optuna.samplers.RandomSampler`	Sampler using random sampling.
`optuna.samplers.TPESampler`	Sampler using TPE (Tree-structured Parzen Estimator) algorithm.
`optuna.samplers.CmaEsSampler`	A sampler using cmaes as the backend.
`optuna.samplers.PartialFixedSampler`	Sampler with partially fixed parameters.
`optuna.samplers.NSGAIISampler`	Multi-objective sampler using the NSGA-II algorithm.
`optuna.samplers.MOTPESampler`	Multi-objective sampler using the MOTPE algorithm.
`optuna.samplers.QMCSampler`	A Quasi Monte Carlo Sampler that generates low-discrepancy sequences.
`optuna.samplers.IntersectionSearchSpace`	A class to calculate the intersection search space of a `Study`.
`optuna.samplers.intersection_search_space`	Return the intersection search space of the `Study`.

Note

The following optuna.samplers.nsgaii module defines crossover operations used by NSGAIISampler.

optuna.samplers.nsgaii

The nsgaii module defines crossover operations used by NSGAIISampler.

`optuna.samplers.nsgaii.BaseCrossover`	Base class for crossovers.
`optuna.samplers.nsgaii.UniformCrossover`	Uniform Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.BLXAlphaCrossover`	Blend Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.SPXCrossover`	Simplex Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.SBXCrossover`	Simulated Binary Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.VSBXCrossover`	Modified Simulated Binary Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.UNDXCrossover`	Unimodal Normal Distribution Crossover used by `NSGAIISampler`.

optuna.storages

The storages module defines a BaseStorage class which abstracts a backend database and provides library-internal interfaces to the read/write histories of the studies and trials. Library users who wish to use storage solutions other than the default in-memory storage should use one of the child classes of BaseStorage documented below.

`optuna.storages.RDBStorage`	Storage class for RDB backend.
`optuna.storages.RedisStorage`	Storage class for Redis backend.
`optuna.storages.RetryFailedTrialCallback`	Retry a failed trial up to a maximum number of times.
`optuna.storages.fail_stale_trials`	Fail stale trials and run their failure callbacks.

optuna.study

The study module implements the Study object and related functions. A public constructor is available for the Study class, but direct use of this constructor is not recommended. Instead, library users should create and load a Study using create_study() and load_study() respectively.

`optuna.study.Study`	A study corresponds to an optimization task, i.e., a set of trials.
`optuna.study.create_study`	Create a new `Study`.
`optuna.study.load_study`	Load the existing `Study` that has the specified name.
`optuna.study.delete_study`	Delete a `Study` object.
`optuna.study.copy_study`	Copy study from one storage to another.
`optuna.study.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.study.MaxTrialsCallback`	Set a maximum number of trials before ending the study.
`optuna.study.StudyDirection`	Direction of a `Study`.
`optuna.study.StudySummary`	Basic attributes and aggregated results of a `Study`.

optuna.trial

The trial module contains Trial related classes and functions.

A Trial instance represents a process of evaluating an objective function. This instance is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial’s state, and set/get user-defined attributes of the trial, so that Optuna users can define a custom objective function through the interfaces. Basically, Optuna users only use it in their custom objective functions.

`optuna.trial.Trial`	A trial is a process of evaluating an objective function.
`optuna.trial.FixedTrial`	A trial class which suggests a fixed value for each parameter.
`optuna.trial.FrozenTrial`	Status and results of a `Trial`.
`optuna.trial.TrialState`	State of a `Trial`.
`optuna.trial.create_trial`	Create a new `FrozenTrial`.

optuna.visualization

The visualization module provides utility functions for plotting the optimization process using plotly and matplotlib. Plotting functions generally take a Study object and optional parameters are passed as a list to the params argument.

Note

In the optuna.visualization module, the following functions use plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

Note

The plot_param_importances() requires the Python package of scikit-learn.

`optuna.visualization.plot_contour`	Plot the parameter relationship as contour plot in a study.
`optuna.visualization.plot_edf`	Plot the objective value EDF (empirical distribution function) of a study.
`optuna.visualization.plot_intermediate_values`	Plot intermediate values of all trials in a study.
`optuna.visualization.plot_optimization_history`	Plot optimization history of all trials in a study.
`optuna.visualization.plot_parallel_coordinate`	Plot the high-dimensional parameter relationships in a study.
`optuna.visualization.plot_param_importances`	Plot hyperparameter importances.
`optuna.visualization.plot_pareto_front`	Plot the Pareto front of a study.
`optuna.visualization.plot_slice`	Plot the parameter relationship as slice plot in a study.
`optuna.visualization.is_available`	Returns whether visualization with plotly is available or not.

Note

The following optuna.visualization.matplotlib module uses Matplotlib as a backend.

optuna.visualization.matplotlib

Note

The following functions use Matplotlib as a backend.

`optuna.visualization.matplotlib.plot_contour`	Plot the parameter relationship as contour plot in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_edf`	Plot the objective value EDF (empirical distribution function) of a study with Matplotlib.
`optuna.visualization.matplotlib.plot_intermediate_values`	Plot intermediate values of all trials in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_optimization_history`	Plot optimization history of all trials in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_parallel_coordinate`	Plot the high-dimensional parameter relationships in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_param_importances`	Plot hyperparameter importances with Matplotlib.
`optuna.visualization.matplotlib.plot_pareto_front`	Plot the Pareto front of a study.
`optuna.visualization.matplotlib.plot_slice`	Plot the parameter relationship as slice plot in a study with Matplotlib.
`optuna.visualization.matplotlib.is_available`	Returns whether visualization with Matplotlib is available or not.

See also

The 5. Quick Visualization for Hyperparameter Optimization Analysis tutorial provides use-cases with examples.

FAQ

Can I use Optuna with X? (where X is your favorite ML library)
How to define objective functions that have own arguments?
Can I use Optuna without remote RDB servers?
How can I save and resume studies?
How to suppress log messages of Optuna?
How to save machine learning models trained in objective functions?
How can I obtain reproducible optimization results?
How are exceptions from trials handled?
How are NaNs returned by trials handled?
What happens when I dynamically alter a search space?
How can I use two GPUs for evaluating two trials simultaneously?
How can I test my objective functions?
How do I avoid running out of memory (OOM) when optimizing studies?
How can I output a log only when the best value is updated?
How do I suggest variables which represent the proportion, that is, are in accordance with Dirichlet distribution?
How can I optimize a model with some constraints?
How can I parallelize optimization?
Can I monitor trials and make them failed automatically when they are killed unexpectedly?

Can I use Optuna with X? (where X is your favorite ML library)

Optuna is compatible with most ML libraries, and it’s easy to use Optuna with those. Please refer to examples.

How to define objective functions that have own arguments?

There are two ways to realize it.

First, callable classes can be used for that purpose as follows:

import optuna


class Objective:
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_float("x", self.min_x, self.max_x)
        return (x - 2) ** 2


# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

Second, you can use lambda or functools.partial for creating functions (closures) that hold extra arguments. Below is an example that uses lambda:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_float("x", min_x, max_x)
    return (x - 2) ** 2


# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

Please also refer to sklearn_addtitional_args.py example, which reuses the dataset instead of loading it in each trial execution.

Can I use Optuna without remote RDB servers?

Yes, it’s possible.

In the simplest form, Optuna works with in-memory storage:

study = optuna.create_study()
study.optimize(objective)

If you want to save and resume studies, it’s handy to use SQLite as the local storage:

study = optuna.create_study(study_name="foo_study", storage="sqlite:///example.db")
study.optimize(objective)  # The state of `study` will be persisted to the local SQLite file.

Please see Saving/Resuming Study with RDB Backend for more details.

How can I save and resume studies?

There are two ways of persisting studies, which depend if you are using in-memory storage (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using pickle or joblib. For example, using joblib:

study = optuna.create_study()
joblib.dump(study, "study.pkl")

And to resume the study:

study = joblib.load("study.pkl")
print("Best trial until now:")
print(" Value: ", study.best_trial.value)
print(" Params: ")
for key, value in study.best_trial.params.items():
    print(f"    {key}: {value}")

Note that Optuna does not support saving/reloading across different Optuna versions with pickle. To save/reload a study across different Optuna versions, please use RDBs and upgrade storage schema if necessary. If you are using RDBs, see Saving/Resuming Study with RDB Backend for more details.

How to suppress log messages of Optuna?

By default, Optuna shows log messages at the optuna.logging.INFO level. You can change logging levels by using optuna.logging.set_verbosity().

For instance, you can stop showing each trial result as follows:

optuna.logging.set_verbosity(optuna.logging.WARNING)

study = optuna.create_study()
study.optimize(objective)
# Logs like '[I 2020-07-21 13:41:45,627] Trial 0 finished with value:...' are disabled.

Please refer to optuna.logging for further details.

How to save machine learning models trained in objective functions?

Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used.

We recommend saving optuna.trial.Trial.number with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows:

def objective(trial):
    svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
    clf = sklearn.svm.SVC(C=svc_c)
    clf.fit(X_train, y_train)

    # Save a trained model to a file.
    with open("{}.pickle".format(trial.number), "wb") as fout:
        pickle.dump(clf, fout)
    return 1.0 - accuracy_score(y_valid, clf.predict(X_valid))


study = optuna.create_study()
study.optimize(objective, n_trials=100)

# Load the best model.
with open("{}.pickle".format(study.best_trial.number), "rb") as fin:
    best_clf = pickle.load(fin)
print(accuracy_score(y_valid, best_clf.predict(X_valid)))

How can I obtain reproducible optimization results?

To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via seed argument of an instance of samplers as follows:

sampler = TPESampler(seed=10)  # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)
study.optimize(objective)

To make the pruning by HyperbandPruner reproducible, you can specify study_name of Study and hash seed.

However, there are two caveats.

First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result.

Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it.

How are exceptions from trials handled?

Trials that raise exceptions without catching them will be treated as failures, i.e. with the FAIL status.

By default, all exceptions except TrialPruned raised in objective functions are propagated to the caller of optimize(). In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in optimize() which exception types to catch using the catch argument. Exceptions of these types are caught inside the study and will not propagate further.

You can find the failed trials in log messages.

[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')

You can also find the failed trials by checking the trial states as follows:

study.trials_dataframe()

number	state	value	…	params	system_attrs
0	TrialState.FAIL		…	0	Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’)
1	TrialState.COMPLETE	1269	…	1

See also

The catch argument in optimize().

How are NaNs returned by trials handled?

Trials that return NaN (float('nan')) are treated as failures, but they will not abort studies.

Trials which return NaN are shown as follows:

[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.

What happens when I dynamically alter a search space?

Since parameters search spaces are specified in each call to the suggestion API, e.g. suggest_float() and suggest_int(), it is possible to, in a single study, alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually.

Note

Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822

How can I use two GPUs for evaluating two trials simultaneously?

If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used in your script, main.py, the easiest way is to set CUDA_VISIBLE_DEVICES environment variable:

# On a terminal.
#
# Specify to use the first GPU, and run an optimization.
$ export CUDA_VISIBLE_DEVICES=0
$ python main.py

# On another terminal.
#
# Specify to use the second GPU, and run another optimization.
$ export CUDA_VISIBLE_DEVICES=1
$ python main.py

Please refer to CUDA C Programming Guide for further details.

How can I test my objective functions?

When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use FixedTrial, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of \(x\) and \(y\) to the objective function \(x + y\) as follows:

def objective(trial):
    x = trial.suggest_float("x", -1.0, 1.0)
    y = trial.suggest_int("y", -5, 5)
    return x + y


objective(FixedTrial({"x": 1.0, "y": -1}))  # 0.0
objective(FixedTrial({"x": -1.0, "y": -4}))  # -5.0

Using FixedTrial, you can write unit tests as follows:

# A test function of pytest
def test_objective():
    assert 1.0 == objective(FixedTrial({"x": 1.0, "y": 0}))
    assert -1.0 == objective(FixedTrial({"x": 0.0, "y": -1}))
    assert 0.0 == objective(FixedTrial({"x": -1.0, "y": 1}))

How do I avoid running out of memory (OOM) when optimizing studies?

If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify gc_after_trial to True when calling optimize() or call gc.collect() inside a callback.

def objective(trial):
    x = trial.suggest_float("x", -1.0, 1.0)
    y = trial.suggest_int("y", -5, 5)
    return x + y


study = optuna.create_study()
study.optimize(objective, n_trials=10, gc_after_trial=True)

# `gc_after_trial=True` is more or less identical to the following.
study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()])

There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, gc_after_trial is False by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that gc.collect() is called even when errors, including TrialPruned are raised.

Note

ChainerMNStudy does currently not provide gc_after_trial nor callbacks for optimize(). When using this class, you will have to call the garbage collector inside the objective function.

How can I output a log only when the best value is updated?

Here’s how to replace the logging feature of optuna with your own logging callback function. The implemented callback can be passed to optimize(). Here’s an example:

import optuna


# Turn off optuna log notes.
optuna.logging.set_verbosity(optuna.logging.WARN)


def objective(trial):
    x = trial.suggest_float("x", 0, 1)
    return x ** 2


def logging_callback(study, frozen_trial):
    previous_best_value = study.user_attrs.get("previous_best_value", None)
    if previous_best_value != study.best_value:
        study.set_user_attr("previous_best_value", study.best_value)
        print(
            "Trial {} finished with best value: {} and parameters: {}. ".format(
            frozen_trial.number,
            frozen_trial.value,
            frozen_trial.params,
            )
        )


study = optuna.create_study()
study.optimize(objective, n_trials=100, callbacks=[logging_callback])

Note that this callback may show incorrect values when you try to optimize an objective function with n_jobs!=1 (or other forms of distributed optimization) due to its reads and writes to storage that are prone to race conditions.

How do I suggest variables which represent the proportion, that is, are in accordance with Dirichlet distribution?

When you want to suggest \(n\) variables which represent the proportion, that is, \(p[0], p[1], ..., p[n-1]\) which satisfy \(0 \le p[k] \le 1\) for any \(k\) and \(p[0] + p[1] + ... + p[n-1] = 1\), try the below. For example, these variables can be used as weights when interpolating the loss functions. These variables are in accordance with the flat Dirichlet distribution.

import numpy as np
import matplotlib.pyplot as plt
import optuna


def objective(trial):
    n = 5
    x = []
    for i in range(n):
        x.append(- np.log(trial.suggest_float(f"x_{i}", 0, 1)))

    p = []
    for i in range(n):
        p.append(x[i] / sum(x))

    for i in range(n):
        trial.set_user_attr(f"p_{i}", p[i])

    return 0

study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
study.optimize(objective, n_trials=1000)

n = 5
p = []
for i in range(n):
    p.append([trial.user_attrs[f"p_{i}"] for trial in study.trials])
axes = plt.subplots(n, n, figsize=(20, 20))[1]

for i in range(n):
    for j in range(n):
        axes[j][i].scatter(p[i], p[j], marker=".")
        axes[j][i].set_xlim(0, 1)
        axes[j][i].set_ylim(0, 1)
        axes[j][i].set_xlabel(f"p_{i}")
        axes[j][i].set_ylabel(f"p_{j}")

plt.savefig("sampled_ps.png")

This method is justified in the following way: First, if we apply the transformation \(x = - \log (u)\) to the variable \(u\) sampled from the uniform distribution \(Uni(0, 1)\) in the interval \([0, 1]\), the variable \(x\) will follow the exponential distribution \(Exp(1)\) with scale parameter \(1\). Furthermore, for \(n\) variables \(x[0], ..., x[n-1]\) that follow the exponential distribution of scale parameter \(1\) independently, normalizing them with \(p[i] = x[i] / \sum_i x[i]\), the vector \(p\) follows the Dirichlet distribution \(Dir(\alpha)\) of scale parameter \(\alpha = (1, ..., 1)\). You can verify the transformation by calculating the elements of the Jacobian.

How can I optimize a model with some constraints?

When you want to optimize a model with constraints, you can use the following classes, NSGAIISampler or BoTorchSampler. The following example is a benchmark of Binh and Korn function, a multi-objective optimization, with constraints using NSGAIISampler. This one has two constraints \(c_0 = (x-5)^2 + y^2 - 25 \le 0\) and \(c_1 = -(x - 8)^2 - (y + 3)^2 + 7.7 \le 0\) and finds the optimal solution satisfying these constraints.

import optuna


def objective(trial):
    # Binh and Korn function with constraints.
    x = trial.suggest_float("x", -15, 30)
    y = trial.suggest_float("y", -15, 30)

    # Constraints which are considered feasible if less than or equal to zero.
    # The feasible region is basically the intersection of a circle centered at (x=5, y=0)
    # and the complement to a circle centered at (x=8, y=-3).
    c0 = (x - 5) ** 2 + y ** 2 - 25
    c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7

    # Store the constraints as user attributes so that they can be restored after optimization.
    trial.set_user_attr("constraint", (c0, c1))

    v0 = 4 * x ** 2 + 4 * y ** 2
    v1 = (x - 5) ** 2 + (y - 5) ** 2

    return v0, v1


def constraints(trial):
    return trial.user_attrs["constraint"]


sampler = optuna.samplers.NSGAIISampler(constraints_func=constraints)
study = optuna.create_study(
    directions=["minimize", "minimize"],
    sampler=sampler,
)
study.optimize(objective, n_trials=32, timeout=600)

print("Number of finished trials: ", len(study.trials))

print("Pareto front:")

trials = sorted(study.best_trials, key=lambda t: t.values)

for trial in trials:
    print("  Trial#{}".format(trial.number))
    print(
        "    Values: Values={}, Constraint={}".format(
            trial.values, trial.user_attrs["constraint"][0]
        )
    )
    print("    Params: {}".format(trial.params))

If you are interested in the exmaple for BoTorchSampler, please refer to this sample code.

There are two kinds of constrained optimizations, one with soft constraints and the other with hard constraints. Soft constraints do not have to be satisfied, but an objective function is penalized if they are unsatisfied. On the other hand, hard constraints must be satisfied.

Optuna is adopting the soft one and DOES NOT support the hard one. In other words, Optuna DOES NOT have built-in samplers for the hard constraints.

How can I parallelize optimization?

The variations of parallelization are in the following three cases.

Multi-threading parallelization with single node
Multi-processing parallelization with single node
Multi-processing parallelization with multiple nodes

1. Multi-threading parallelization with a single node 

Parallelization can be achieved by setting the argument n_jobs in optuna.study.Study.optimize(). However, the python code will not be faster due to GIL because optuna.study.Study.optimize() with n_jobs!=1 uses multi-threading.

While optimizing, it will be faster in limited situations, such as waiting for other server requests or C/C++ processing with numpy, etc., but it will not be faster in other cases.

For more information about 1., see APIReference.

2. Multi-processing parallelization with single node 

This can be achieved by using file-based RDBs (such as SQLite) and client/server RDBs (such as PostgreSQL and MySQL). However, if you are in the environment where you can not install an RDB, you can not run multi-processing parallelization with single node. When you really want to do it, please request it as a GitHub issue. If we receive a lot of requests, we may provide a solution for it.

For more information about 2., see TutorialEasyParallelization.

3. Multi-processing parallelization with multiple nodes 

This can be achieved by using client/server RDBs (such as PostgreSQL and MySQL). However, if you are in the environment where you can not install a client/server RDB, you can not run multi-processing parallelization with multiple nodes.

For more information about 3., see TutorialEasyParallelization.

Can I monitor trials and make them failed automatically when they are killed unexpectedly?

Note

Heartbeat mechanism is experimental. API would change in the future.

A process running a trial could be killed unexpectedly, typically by a job scheduler in a cluster environment. If trials are killed unexpectedly, they will be left on the storage with their states RUNNING until we remove them or update their state manually. For such a case, Optuna supports monitoring trials using heartbeat mechanism. Using heartbeat, if a process running a trial is killed unexpectedly, Optuna will automatically change the state of the trial that was running on that process to FAIL from RUNNING.

import optuna

def objective(trial):
    (Very time-consuming computation)

# Recording heartbeats every 60 seconds.
# Other processes' trials where more than 120 seconds have passed
# since the last heartbeat was recorded will be automatically failed.
storage = optuna.storages.RDBStorage(url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120)
study = optuna.create_study(storage=storage)
study.optimize(objective, n_trials=100)

Note

The heartbeat is supposed to be used with optimize(). If you use ask() and tell(), please change the state of the killed trials by calling tell() explicitly.

You can also execute a callback function to process the failed trial. Optuna provides a callback to retry failed trials as RetryFailedTrialCallback. Note that a callback is invoked at a beginning of each trial, which means RetryFailedTrialCallback will retry failed trials when a new trial starts to evaluate.

import optuna
from optuna.storages import RetryFailedTrialCallback

storage = optuna.storages.RDBStorage(
    url="sqlite:///:memory:",
    heartbeat_interval=60,
    grace_period=120,
    failed_trial_callback=RetryFailedTrialCallback(max_retry=3),
)

study = optuna.create_study(storage=storage)