Can I use Optuna with X? (where X is your favorite ML library)

Optuna is compatible with most ML libraries, and it’s easy to use Optuna with those. Please refer to examples.

How to define objective functions that have own arguments?

There are two ways to realize it.

First, callable classes can be used for that purpose as follows:

import optuna

class Objective(object):
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_uniform('x', self.min_x, self.max_x)
        return (x - 2) ** 2

# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

Second, you can use lambda or functools.partial for creating functions (closures) that hold extra arguments. Below is an example that uses lambda:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_uniform('x', min_x, max_x)
    return (x - 2) ** 2

# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

Please also refer to sklearn_addtitional_args.py example.

Can I use Optuna without remote RDB servers?

Yes, it’s possible.

In the simplest form, Optuna works with in-memory storage:

study = optuna.create_study()

If you want to save and resume studies, it’s handy to use SQLite as the local storage:

study = optuna.create_study(study_name='foo_study', storage='sqlite:///example.db')
study.optimize(objective)  # The state of `study` will be persisted to the local SQLite file.

Please see Saving/Resuming Study with RDB Backend for more details.

How to suppress log messages of Optuna?

By default, Optuna shows log messages at the optuna.logging.INFO level. You can change logging levels by using optuna.logging.set_verbosity().

For instance, you can stop showing each trial result as follows:


study = optuna.create_study()
# Logs like '[I 2018-12-05 11:41:42,324] Finished a trial resulted in value:...' are disabled.

Please refer to optuna.logging for further details.

How to save machine learning models trained in objective functions?

Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used.

We recommend saving optuna.trial.Trial.number with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows:

def objective(trial):
    svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
    clf = sklearn.svm.SVC(C=svc_c)
    clf.fit(X_train, y_train)

    # Save a trained model to a file.
    with open('{}.pickle'.format(trial.number), 'wb') as fout:
        pickle.dump(clf, fout)
    return 1.0 - accuracy_score(y_test, clf.predict(X_test))

study = optuna.create_study()
study.optimize(objective, n_trials=100)

# Load the best model.
with open('{}.pickle'.format(study.best_trial.number), 'rb') as fin:
    best_clf = pickle.load(fin)
print(accuracy_score(y_test, best_clf.predict(X_test)))

How can I obtain reproducible optimization results?

To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via seed argument of RandomSampler or TPESampler as follows:

sampler = TPESampler(seed=10)  # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)

However, there are two caveats.

First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result.

Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it.

How does Optuna handle NaNs and exceptions reported by the objective function?

Optuna treats such trials as failures (i.e., FAIL) and continues the study. The Optuna’s system process will not be crashed by any objective values or exceptions raised in objective functions.

You can find the failed trials in log messages. Errors raised in objective functions are shown as follows:

[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')

And trials which returned NaN are shown as follows:

[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.

You can also find the failed trials by checking the trial states as follows:

number state value params system_attrs
0 TrialState.FAIL   0 Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’)
1 TrialState.COMPLETE 1269 1  

How can I use two GPUs for evaluating two trials simultaneously?

If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used, the easiest way is to set CUDA_VISIBLE_DEVICES environment variable:

# On a terminal.
# Specify to use the first GPU, and run an optimization.
$ optuna study optimize foo.py objective --study foo --storage sqlite:///example.db

# On another terminal.
# Specify to use the second GPU, and run another optimization.
$ optuna study optimize bar.py objective --study bar --storage sqlite:///example.db

Please refer to CUDA C Programming Guide for further details.

How can I test my objective functions?

When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use FixedTrial, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of \(x\) and \(y\) to the objective function \(x + y\) as follows:

def objective(trial):
    x = trial.suggest_uniform('x', -1.0, 1.0)
    y = trial.suggest_int('y', -5, 5)
    return x + y

objective(FixedTrial({'x': 1.0, 'y': -1}))  # 0.0
objective(FixedTrial({'x': -1.0, 'y': -4}))  # -5.0

Using FixedTrial, you can write unit tests as follows:

# A test function of pytest
def test_objective():
    assert 1.0 == objective(FixedTrial({'x': 1.0, 'y': 0}))
    assert -1.0 == objective(FixedTrial({'x': 0.0, 'y': -1}))
    assert 0.0 == objective(FixedTrial({'x': -1.0, 'y': 1}))