Optuna: A hyperparameter optimization framework¶
Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.
Key Features¶
Optuna has modern functionalities as follows:
Basic Concepts¶
We use the terms study and trial as follows:
Study: optimization based on an objective function
Trial: a single execution of the objective function
Please refer to sample code below. The goal of a study is to find out
the optimal set of hyperparameter values (e.g., classifier
and
svm_c
) through multiple trials (e.g., n_trials=100
). Optuna is
a framework designed for the automation and the acceleration of the
optimization studies.
import ...
# Define an objective function to be minimized.
def objective(trial):
# Invoke suggest methods of a Trial object to generate hyperparameters.
regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
if regressor_name == 'SVR':
svr_c = trial.suggest_loguniform('svr_c', 1e-10, 1e10)
regressor_obj = sklearn.svm.SVR(C=svr_c)
else:
rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)
X, y = sklearn.datasets.load_boston(return_X_y=True)
X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)
regressor_obj.fit(X_train, y_train)
y_pred = regressor_obj.predict(X_val)
error = sklearn.metrics.mean_squared_error(y_val, y_pred)
return error # An objective value linked with the Trial object.
study = optuna.create_study() # Create a new study.
study.optimize(objective, n_trials=100) # Invoke optimization of the objective function.
Communication¶
GitHub Issues for bug reports, feature requests and questions.
Gitter for interactive chat with developers.
Stack Overflow for questions.
Contribution¶
Any contributions to Optuna are welcome! When you send a pull request, please follow the contribution guide.
License¶
MIT License (see LICENSE).
Reference¶
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).
Contents:
- Installation
- Tutorial
- API Reference
- FAQ
- Can I use Optuna with X? (where X is your favorite ML library)
- How to define objective functions that have own arguments?
- Can I use Optuna without remote RDB servers?
- How can I save and resume studies?
- How to suppress log messages of Optuna?
- How to save machine learning models trained in objective functions?
- How can I obtain reproducible optimization results?
- How are exceptions from trials handled?
- How are NaNs returned by trials handled?
- What happens when I dynamically alter a search space?
- How can I use two GPUs for evaluating two trials simultaneously?
- How can I test my objective functions?
- How do I avoid running out of memory (OOM) when optimizing studies?