Trial

class optuna.trial.Trial(study, trial_id)[source]

A trial is a process of evaluating an objective function.

This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial’s state, and set/get user-defined attributes of the trial.

Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the optuna.study.Study.optimize() method; hence library users do not care about instantiation of this object.

Parameters
  • study – A Study object.

  • trial_id – A trial ID that is automatically generated.

property datetime_start

Return start datetime.

Returns

Datetime where the Trial started.

property distributions

Return distributions of parameters to be optimized.

Returns

A dictionary containing all distributions.

property number

Return trial’s number which is consecutive and unique in a study.

Returns

A trial number.

property params

Return parameters to be optimized.

Returns

A dictionary containing all parameters.

report(value, step)[source]

Report an objective function value for a given step.

The reported values are used by the pruners to determine whether this trial should be pruned.

See also

Please refer to BasePruner.

Note

The reported value is converted to float type by applying float() function internally. Thus, it accepts all float-like types (e.g., numpy.float32). If the conversion fails, a TypeError is raised.

Example

Report intermediate scores of SGDClassifier training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    clf = SGDClassifier(random_state=0)
    for step in range(100):
        clf.partial_fit(X_train, y_train, np.unique(y))
        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step=step)
        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • value – A value returned from the objective function.

  • step – Step of the trial (e.g., Epoch of neural network training).

set_user_attr(key, value)[source]

Set user attributes to the trial.

The user attributes in the trial can be access via optuna.trial.Trial.user_attrs().

Example

Save fixed hyperparameters of neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0)

def objective(trial):
    trial.set_user_attr('BATCHSIZE', 128)
    momentum = trial.suggest_uniform('momentum', 0, 1.0)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50),
                        batch_size=trial.user_attrs['BATCHSIZE'],
                        momentum=momentum, solver='sgd', random_state=0)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
assert 'BATCHSIZE' in study.best_trial.user_attrs.keys()
assert study.best_trial.user_attrs['BATCHSIZE'] == 128
Parameters
  • key – A key string of the attribute.

  • value – A value of the attribute. The value should be JSON serializable.

should_prune(step=None)[source]

Suggest whether the trial should be pruned or not.

The suggestion is made by a pruning algorithm associated with the trial and is based on previously reported values. The algorithm can be specified when constructing a Study.

Note

If no values have been reported, the algorithm cannot make meaningful suggestions. Similarly, if this method is called multiple times with the exact same set of reported values, the suggestions will be the same.

See also

Please refer to the example code in optuna.trial.Trial.report().

Parameters

step – Deprecated since 0.12.0: Step of the trial (e.g., epoch of neural network training). Deprecated in favor of always considering the most recent step.

Returns

A boolean value. If True, the trial should be pruned according to the configured pruning algorithm. Otherwise, the trial should continue.

property study_id

Return the study ID.

Deprecated since version 0.20.0: The direct use of this attribute is deprecated and it is recommended that you use study instead.

Returns

The study ID.

suggest_categorical(name, choices)[source]

Suggest a value for the categorical parameter.

The value is sampled from choices.

Example

Suggest a kernel function of SVC.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    kernel = trial.suggest_categorical('kernel', ['linear', 'poly', 'rbf'])
    clf = SVC(kernel=kernel, gamma='scale', random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • choices – Parameter value candidates.

Returns

A suggested value.

suggest_discrete_uniform(name, low, high, q)[source]

Suggest a value for the discrete parameter.

The value is sampled from the range \([\mathsf{low}, \mathsf{high}]\), and the step of discretization is \(q\). More specifically, this method returns one of the values in the sequence \(\mathsf{low}, \mathsf{low} + q, \mathsf{low} + 2 q, \dots, \mathsf{low} + k q \le \mathsf{high}\), where \(k\) denotes an integer. Note that \(high\) may be changed due to round-off errors if \(q\) is not an integer. Please check warning messages to find the changed values.

Example

Suggest a fraction of samples used for fitting the individual learners of GradientBoostingClassifier.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    subsample = trial.suggest_discrete_uniform('subsample', 0.1, 1.0, 0.1)
    clf = GradientBoostingClassifier(subsample=subsample, random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • low – Lower endpoint of the range of suggested values. low is included in the range.

  • high – Upper endpoint of the range of suggested values. high is included in the range.

  • q – A step of discretization.

Returns

A suggested float value.

suggest_float(name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False)float[source]

Suggest a value for the floating point parameter.

Note that this is a wrapper method for suggest_uniform(), suggest_loguniform() and suggest_discrete_uniform().

New in version 1.3.0.

Example

Suggest a momentum, learning rate and scaling factor of learning rate for neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0)

def objective(trial):
    momentum = trial.suggest_float('momentum', 0.0, 1.0)
    learning_rate_init = trial.suggest_float('learning_rate_init',
                                             1e-5, 1e-3, log=True)
    power_t = trial.suggest_float('power_t', 0.2, 0.8, step=0.1)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50), momentum=momentum,
                        learning_rate_init=learning_rate_init,
                        solver='sgd', random_state=0, power_t=power_t)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • low – Lower endpoint of the range of suggested values. low is included in the range.

  • high – Upper endpoint of the range of suggested values. high is excluded from the range.

  • step – A step of discretization.

  • log – A flag to sample the value from the log domain or not. If log is true, the value is sampled from the range in the log domain. Otherwise, the value is sampled from the range in the linear domain. See also suggest_uniform() and suggest_loguniform().

Returns

A suggested float value.

suggest_int(name, low, high, step=1, log=False)[source]

Suggest a value for the integer parameter.

The value is sampled from the integers in \([\mathsf{low}, \mathsf{high}]\), and the step of discretization is \(\mathsf{step}\). More specifically, this method returns one of the values in the sequence \(\mathsf{low}, \mathsf{low} + \mathsf{step}, \mathsf{low} + 2 * \mathsf{step}, \dots, \mathsf{low} + k * \mathsf{step} \le \mathsf{high}\), where \(k\) denotes an integer. Note that \(\mathsf{high}\) is modified if the range is not divisible by \(\mathsf{step}\). Please check the warning messages to find the changed values.

Example

Suggest the number of trees in RandomForestClassifier.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 50, 400)
    clf = RandomForestClassifier(n_estimators=n_estimators, random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • low – Lower endpoint of the range of suggested values. low is included in the range.

  • high – Upper endpoint of the range of suggested values. high is included in the range.

  • step – A step of discretization.

  • log – A flag to sample the value from the log domain or not. If log is true, at first, the range of suggested values is divided into grid points of width step. The range of suggested values is then converted to a log domain, from which a value is uniformly sampled. The uniformly sampled value is re-converted to the original domain and rounded to the nearest grid point that we just split, and the suggested value is determined. For example, if low = 2, high = 8 and step = 2, then the range of suggested values is divided by step as [2, 4, 6, 8] and lower values tend to be more sampled than higher values.

suggest_loguniform(name, low, high)[source]

Suggest a value for the continuous parameter.

The value is sampled from the range \([\mathsf{low}, \mathsf{high})\) in the log domain. When \(\mathsf{low} = \mathsf{high}\), the value of \(\mathsf{low}\) will be returned.

Example

Suggest penalty parameter C of SVC.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    c = trial.suggest_loguniform('c', 1e-5, 1e2)
    clf = SVC(C=c, gamma='scale', random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • low – Lower endpoint of the range of suggested values. low is included in the range.

  • high – Upper endpoint of the range of suggested values. high is excluded from the range.

Returns

A suggested float value.

suggest_uniform(name, low, high)[source]

Suggest a value for the continuous parameter.

The value is sampled from the range \([\mathsf{low}, \mathsf{high})\) in the linear domain. When \(\mathsf{low} = \mathsf{high}\), the value of \(\mathsf{low}\) will be returned.

Example

Suggest a momentum for neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    momentum = trial.suggest_uniform('momentum', 0.0, 1.0)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50), momentum=momentum,
                        solver='sgd', random_state=0)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
Parameters
  • name – A parameter name.

  • low – Lower endpoint of the range of suggested values. low is included in the range.

  • high – Upper endpoint of the range of suggested values. high is excluded from the range.

Returns

A suggested float value.

property user_attrs

Return user attributes.

Returns

A dictionary containing all user attributes.

class optuna.trial.FixedTrial(params, number=0)[source]

A trial class which suggests a fixed value for each parameter.

This object has the same methods as Trial, and it suggests pre-defined parameter values. The parameter values can be determined at the construction of the FixedTrial object. In contrast to Trial, FixedTrial does not depend on Study, and it is useful for deploying optimization results.

Example

Evaluate an objective function with parameter values given by a user.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

assert objective(optuna.trial.FixedTrial({'x': 1, 'y': 0})) == 1

Note

Please refer to Trial for details of methods and properties.

Parameters
  • params – A dictionary containing all parameters.

  • number – A trial number. Defaults to 0.

class optuna.trial.FrozenTrial(number, state, value, datetime_start, datetime_complete, params, distributions, user_attrs, system_attrs, intermediate_values, trial_id)[source]

Status and results of a Trial.

number

Unique and consecutive number of Trial for each Study. Note that this field uses zero-based numbering.

state

TrialState of the Trial.

value

Objective value of the Trial.

datetime_start

Datetime where the Trial started.

datetime_complete

Datetime where the Trial finished.

params

Dictionary that contains suggested parameters.

user_attrs

Dictionary that contains the attributes of the Trial set with optuna.trial.Trial.set_user_attr().

intermediate_values

Intermediate objective values set with optuna.trial.Trial.report().

property distributions

Dictionary that contains the distributions of params.

property duration

Return the elapsed time taken to complete the trial.

Returns

The duration.

class optuna.trial.TrialState[source]

State of a Trial.

RUNNING

The Trial is running.

COMPLETE

The Trial has been finished without any error.

PRUNED

The Trial has been pruned with TrialPruned.

FAIL

The Trial has failed due to an uncaught error.