Optuna: 一个超参数优化框架
Optuna 是一个特别为机器学习设计的自动超参数优化软件框架.它具有命令式的, define-by-run 风格的 API.由于这种 API 的存在, 用 Optuna 编写的代码模块化程度很高, Optuna 的用户因此也可以动态地构造超参数的搜索空间.
主要特点
Optuna 有如下现代化的功能：
-
只需少量依赖, 简单安装完成后便可处理各种任务.
-
利用熟悉的 Python 语法, 如条件语句和循环来定义搜索空间.
-
采用了最先进的超参数采样和最有效的对无望 trial 进行剪枝的算法.
-
仅需少量甚至无需代码修改便可将 study 扩展到数十甚至数百个 worker 上.
-
查询优化记录.
基本概念
我们以如下方式使用 study 和 trial 这两个术语：
Study: 基于目标函数的优化过程
Trial: 目标函数的单次执行过程
请参考下面的示例代码.一个 study 的目的是通过多次 trial (例如 n_trials=100 ) 来找出最佳的超参数值集（比如选择 classifier 还是 svm_c）.而 Optuna 旨在加速和自动化此类 study 优化过程.
import ...
# Define an objective function to be minimized.
def objective(trial):
# Invoke suggest methods of a Trial object to generate hyperparameters.
regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
if regressor_name == 'SVR':
svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
regressor_obj = sklearn.svm.SVR(C=svr_c)
else:
rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)
X, y = sklearn.datasets.load_boston(return_X_y=True)
X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)
regressor_obj.fit(X_train, y_train)
y_pred = regressor_obj.predict(X_val)
error = sklearn.metrics.mean_squared_error(y_val, y_pred)
return error # An objective value linked with the Trial object.
study = optuna.create_study() # Create a new study.
study.optimize(objective, n_trials=100) # Invoke optimization of the objective function.
Communication
可在 GitHub Issues 报告bug、提feature request 和问问题.
可在 Gitter 与开发者互动.
可在 StackOverflow 提问.
贡献
欢迎大家对Optuna的一切贡献！但是在发送 pull request 时请遵从 contribution guide 的规范.
License
MIT License (see LICENSE).
Reference
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).
安装
Optuna 支持 3.6 或者更新版本的 Python.
我们推荐使用 pip 来安装 Optuna：
$ pip install optuna
你也可以在 Git 仓库的 master 分支下用开发模式安装Optuna：
$ pip install git+https://github.com/optuna/optuna.git
你也可以通过 conda 来安装 Optuna：
$ conda install -c conda-forge optuna
教程
如果你不熟悉Optuna或者需要一个一般的介绍，我们强烈推荐下面这个视频。
关键特性
展现 Optuna 的关键特性
备注
Click here to download the full example code
轻量级、多功能和跨平台架构
Optuna 完全由 Python 写成，并且只有少量依赖。这意味着一旦你开始对 Optuna 感兴趣，我们可以快速上手真实的例子。
二次函数的例子
通常，Optuna是用于优化超参数的，但是作为例子，我们这里来优化一个简单的而此函数：\((x - 2)^2\).
首先，导入 optuna.
import optuna
在 Optuna 中，待优化函数一般被命名为 objective.
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
该函数的返回值是 \((x - 2)^2\). 我们的目标是找到一个 x，使 objective 函数的输出最小。这被称为 “optimization” (优化)。 在优化过程中，Optuna 反复调用目标函数，在不同的 x 下对其进行求值。
一个 Trial 对应着目标函数的单次执行。在每次调用该函数的时候，它都被内部实例化一次。
而 suggest API (例如 suggest_uniform()) 在目标函数内部被调用。它被用于获取单个 trial 的参数。在上面的例子中，suggest_uniform() 在给定的范围（-10 到 10）内均匀地选择参数。
为了开始优化过程，我们将创建一个 study 对象，并将目标函数传递给它的一个方法 optimize()：
study = optuna.create_study()
study.optimize(objective, n_trials=100)
You can get the best parameter as follows.
best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))
Out:
Found x: 2.0007453774691357, (x - 2)^2: 5.555875714951739e-07
最佳参数可以通过如下方式获得：
备注
当 Optuna 被用于机器学习时，目标函数通常返回模型的损失或者准确度。
Study 对象
下面是几个常用术语：
Trial: 目标函数的单次调用
Study: 一次优化过程，包含一系列的 trials.
Parameter: 待优化的参数，比如上面例子中的
x.
在 Optuna 中，我们用 study 对象来管理优化过程。 create_study() 方法会返回一个 study 对象。该对象包含若干有用的属性，可以用于分析优化结果。
获得参数名和参数值的字典：
study.best_params
Out:
{'x': 2.0007453774691357}
获得最佳目标函数值：
study.best_value
Out:
5.555875714951739e-07
获得最佳 trial:
study.best_trial
Out:
FrozenTrial(number=46, values=[5.555875714951739e-07], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 266572), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 269818), params={'x': 2.0007453774691357}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=46, state=TrialState.COMPLETE, value=None)
获得所有 trials:
study.trials
Out:
[FrozenTrial(number=0, values=[0.17511330772143616], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 127027), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 127432), params={'x': 2.4184654199828657}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE, value=None), FrozenTrial(number=1, values=[5.55018460646496], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 127925), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 128255), params={'x': 4.355882978092282}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE, value=None), FrozenTrial(number=2, values=[8.18262262981584], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 128683), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 128995), params={'x': -0.8605283829767956}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=2, state=TrialState.COMPLETE, value=None), FrozenTrial(number=3, values=[13.502562948612052], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 129426), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 129744), params={'x': -1.6745833707526696}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE, value=None), FrozenTrial(number=4, values=[4.097234965300769], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 130166), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 130482), params={'x': 4.024162781324854}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=4, state=TrialState.COMPLETE, value=None), FrozenTrial(number=5, values=[53.56822845316632], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 130904), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 131227), params={'x': 9.319031934154019}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=5, state=TrialState.COMPLETE, value=None), FrozenTrial(number=6, values=[18.267873372441038], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 131649), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 131980), params={'x': -2.2740932807369845}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=6, state=TrialState.COMPLETE, value=None), FrozenTrial(number=7, values=[18.646084119202367], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 132429), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 132750), params={'x': -2.318111174947024}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=7, state=TrialState.COMPLETE, value=None), FrozenTrial(number=8, values=[20.994735530038245], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 133174), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 133499), params={'x': -2.5820012581881997}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=8, state=TrialState.COMPLETE, value=None), FrozenTrial(number=9, values=[5.4930707389445725], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 133922), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 134258), params={'x': -0.3437300908902827}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=9, state=TrialState.COMPLETE, value=None), FrozenTrial(number=10, values=[115.85355576747152], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 134692), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 138089), params={'x': -8.763528964399711}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=10, state=TrialState.COMPLETE, value=None), FrozenTrial(number=11, values=[6.969189419890846], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 138587), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 141822), params={'x': 4.639922237470423}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=11, state=TrialState.COMPLETE, value=None), FrozenTrial(number=12, values=[4.1748186083217185], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 142312), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 145362), params={'x': 4.04323728634775}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=12, state=TrialState.COMPLETE, value=None), FrozenTrial(number=13, values=[30.18355405734446], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 145902), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 148934), params={'x': 7.49395613900807}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=13, state=TrialState.COMPLETE, value=None), FrozenTrial(number=14, values=[0.025113897016394486], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 149425), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 152476), params={'x': 1.8415263522966847}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=14, state=TrialState.COMPLETE, value=None), FrozenTrial(number=15, values=[74.38366290953937], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 152964), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 155952), params={'x': -6.624596391109519}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=15, state=TrialState.COMPLETE, value=None), FrozenTrial(number=16, values=[0.09703268629031321], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 156469), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 159535), params={'x': 1.6884992996953085}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=16, state=TrialState.COMPLETE, value=None), FrozenTrial(number=17, values=[0.4125357190870272], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 160043), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 163045), params={'x': 1.3577105643971503}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=17, state=TrialState.COMPLETE, value=None), FrozenTrial(number=18, values=[21.22010016253533], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 163540), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 166568), params={'x': 6.606527994328845}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=18, state=TrialState.COMPLETE, value=None), FrozenTrial(number=19, values=[46.90390780414091], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 167057), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 170131), params={'x': -4.848642770954031}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=19, state=TrialState.COMPLETE, value=None), FrozenTrial(number=20, values=[0.7775069811123316], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 170619), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 173810), params={'x': 1.1182364369558404}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=20, state=TrialState.COMPLETE, value=None), FrozenTrial(number=21, values=[0.012413584718127094], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 174300), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 177367), params={'x': 1.8885837322554417}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=21, state=TrialState.COMPLETE, value=None), FrozenTrial(number=22, values=[0.07511133650160198], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 177865), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 180915), params={'x': 2.2740644750813246}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=22, state=TrialState.COMPLETE, value=None), FrozenTrial(number=23, values=[0.5466776677921873], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 181420), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 184640), params={'x': 2.7393765399254884}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=23, state=TrialState.COMPLETE, value=None), FrozenTrial(number=24, values=[19.25099140158749], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 185130), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 188191), params={'x': 6.387595172937846}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=24, state=TrialState.COMPLETE, value=None), FrozenTrial(number=25, values=[4.3749649721785415], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 188685), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 191725), params={'x': -0.09164169306756298}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=25, state=TrialState.COMPLETE, value=None), FrozenTrial(number=26, values=[11.312722846060762], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 192310), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 195313), params={'x': 5.363439139639777}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=26, state=TrialState.COMPLETE, value=None), FrozenTrial(number=27, values=[43.06511215794055], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 195808), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 198887), params={'x': 8.562401401769062}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=27, state=TrialState.COMPLETE, value=None), FrozenTrial(number=28, values=[38.688448825253964], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 199385), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 202518), params={'x': -4.220003924858405}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=28, state=TrialState.COMPLETE, value=None), FrozenTrial(number=29, values=[0.5913731357096472], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 203020), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 206167), params={'x': 2.7690078905379627}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=29, state=TrialState.COMPLETE, value=None), FrozenTrial(number=30, values=[0.37483742192540864], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 206673), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 209816), params={'x': 2.61223967686308}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=30, state=TrialState.COMPLETE, value=None), FrozenTrial(number=31, values=[0.5161944340333385], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 210314), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 213499), params={'x': 1.2815332756255593}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=31, state=TrialState.COMPLETE, value=None), FrozenTrial(number=32, values=[0.7617263879426132], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 213996), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 217153), params={'x': 1.1272306215599603}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=32, state=TrialState.COMPLETE, value=None), FrozenTrial(number=33, values=[1.0794389251182408], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 217650), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 220818), params={'x': 3.038960502193534}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=33, state=TrialState.COMPLETE, value=None), FrozenTrial(number=34, values=[0.13289479251062056], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 221316), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 224805), params={'x': 1.6354526196629298}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=34, state=TrialState.COMPLETE, value=None), FrozenTrial(number=35, values=[11.446686965195555], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 225304), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 228950), params={'x': -1.3832952819988316}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=35, state=TrialState.COMPLETE, value=None), FrozenTrial(number=36, values=[2.9878511097283753], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 229449), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 232615), params={'x': 0.2714598327697515}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=36, state=TrialState.COMPLETE, value=None), FrozenTrial(number=37, values=[10.593822650257883], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 233162), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 236361), params={'x': 5.25481530201913}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=37, state=TrialState.COMPLETE, value=None), FrozenTrial(number=38, values=[2.213699158511425], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 236872), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 240046), params={'x': 3.4878505161848166}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=38, state=TrialState.COMPLETE, value=None), FrozenTrial(number=39, values=[30.215990740533847], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 240557), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 243670), params={'x': -3.4969073796575696}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=39, state=TrialState.COMPLETE, value=None), FrozenTrial(number=40, values=[7.588837313086679], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 244202), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 247391), params={'x': -0.7547844404030379}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=40, state=TrialState.COMPLETE, value=None), FrozenTrial(number=41, values=[0.00290412150271473], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 247903), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 251074), params={'x': 1.9461100983233897}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=41, state=TrialState.COMPLETE, value=None), FrozenTrial(number=42, values=[0.00027037325027566095], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 251580), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 254812), params={'x': 1.9835569695531614}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=42, state=TrialState.COMPLETE, value=None), FrozenTrial(number=43, values=[2.031819158341656], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 255317), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 258515), params={'x': 0.5745810586562083}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=43, state=TrialState.COMPLETE, value=None), FrozenTrial(number=44, values=[6.38628859133542], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 259029), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 262271), params={'x': 4.527110720038879}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=44, state=TrialState.COMPLETE, value=None), FrozenTrial(number=45, values=[3.097905002315559], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 262782), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 266067), params={'x': 3.760086646252269}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=45, state=TrialState.COMPLETE, value=None), FrozenTrial(number=46, values=[5.555875714951739e-07], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 266572), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 269818), params={'x': 2.0007453774691357}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=46, state=TrialState.COMPLETE, value=None), FrozenTrial(number=47, values=[2.620533320149347], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 270328), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 273579), params={'x': 0.3811938596146396}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=47, state=TrialState.COMPLETE, value=None), FrozenTrial(number=48, values=[0.01866236563727553], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 274089), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 277309), params={'x': 2.1366102691501467}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=48, state=TrialState.COMPLETE, value=None), FrozenTrial(number=49, values=[12.125057471377989], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 277828), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 281250), params={'x': -1.4821053216951938}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=49, state=TrialState.COMPLETE, value=None), FrozenTrial(number=50, values=[6.953694426075021], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 281764), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 285008), params={'x': -0.6369858600445739}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=50, state=TrialState.COMPLETE, value=None), FrozenTrial(number=51, values=[0.005642228727751361], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 285520), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 288791), params={'x': 2.07511477037009}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=51, state=TrialState.COMPLETE, value=None), FrozenTrial(number=52, values=[2.481284166330171], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 289307), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 292558), params={'x': 3.5752092452528874}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=52, state=TrialState.COMPLETE, value=None), FrozenTrial(number=53, values=[0.07527163820774595], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 293077), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 296308), params={'x': 2.2743567717548556}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=53, state=TrialState.COMPLETE, value=None), FrozenTrial(number=54, values=[8.353249507608016], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 296825), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 300075), params={'x': 4.890198869906363}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=54, state=TrialState.COMPLETE, value=None), FrozenTrial(number=55, values=[15.473715966276579], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 300621), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 303887), params={'x': 5.933664445053312}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=55, state=TrialState.COMPLETE, value=None), FrozenTrial(number=56, values=[1.9737793578202127], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 304471), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 307654), params={'x': 0.5950874198654876}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=56, state=TrialState.COMPLETE, value=None), FrozenTrial(number=57, values=[0.019405400182998345], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 308234), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 311425), params={'x': 1.8606967330498012}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=57, state=TrialState.COMPLETE, value=None), FrozenTrial(number=58, values=[4.401077263262311], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 312020), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 315182), params={'x': 4.09787446317989}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=58, state=TrialState.COMPLETE, value=None), FrozenTrial(number=59, values=[1.889278069298502], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 315736), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 318988), params={'x': 3.374510119751216}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=59, state=TrialState.COMPLETE, value=None), FrozenTrial(number=60, values=[0.011652917487524122], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 319507), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 323082), params={'x': 2.1079486798785614}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=60, state=TrialState.COMPLETE, value=None), FrozenTrial(number=61, values=[0.002115239083029317], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 323638), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 326887), params={'x': 2.045991728419677}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=61, state=TrialState.COMPLETE, value=None), FrozenTrial(number=62, values=[4.206232131689771], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 327442), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 330702), params={'x': -0.05091007401342962}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=62, state=TrialState.COMPLETE, value=None), FrozenTrial(number=63, values=[1.0872842056866994], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 331258), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 334529), params={'x': 3.0427292101436016}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=63, state=TrialState.COMPLETE, value=None), FrozenTrial(number=64, values=[0.9340137931005128], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 335071), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 338365), params={'x': 1.0335561096988026}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=64, state=TrialState.COMPLETE, value=None), FrozenTrial(number=65, values=[0.015887482920919283], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 338921), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 342205), params={'x': 2.1260455589099405}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=65, state=TrialState.COMPLETE, value=None), FrozenTrial(number=66, values=[0.9656106661563976], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 342763), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 346023), params={'x': 1.017345093048227}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=66, state=TrialState.COMPLETE, value=None), FrozenTrial(number=67, values=[0.3009435278843816], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 346581), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 349868), params={'x': 2.5485832005123576}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=67, state=TrialState.COMPLETE, value=None), FrozenTrial(number=68, values=[17.230203294458455], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 350415), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 353681), params={'x': -2.1509280040080747}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=68, state=TrialState.COMPLETE, value=None), FrozenTrial(number=69, values=[4.9434761059690215], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 354240), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 357570), params={'x': 4.223392926580685}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=69, state=TrialState.COMPLETE, value=None), FrozenTrial(number=70, values=[0.0451969426865006], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 358129), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 361433), params={'x': 1.787404274063422}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=70, state=TrialState.COMPLETE, value=None), FrozenTrial(number=71, values=[1.0626697305956325], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 361997), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 365290), params={'x': 3.0308587345488385}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=71, state=TrialState.COMPLETE, value=None), FrozenTrial(number=72, values=[0.014831854679878848], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 365851), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 369147), params={'x': 2.12178610216227}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=72, state=TrialState.COMPLETE, value=None), FrozenTrial(number=73, values=[0.25958207934605604], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 369705), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 373011), params={'x': 1.490508018369223}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=73, state=TrialState.COMPLETE, value=None), FrozenTrial(number=74, values=[5.565757067348065], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 373569), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 376872), params={'x': -0.3591856788621077}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=74, state=TrialState.COMPLETE, value=None), FrozenTrial(number=75, values=[1.371920143202178], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 377433), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 380744), params={'x': 0.828710051608835}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=75, state=TrialState.COMPLETE, value=None), FrozenTrial(number=76, values=[0.297429692830697], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 381304), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 384613), params={'x': 2.5453711514470645}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=76, state=TrialState.COMPLETE, value=None), FrozenTrial(number=77, values=[1.4285372905339677], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 385175), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 388481), params={'x': 3.195214328283412}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=77, state=TrialState.COMPLETE, value=None), FrozenTrial(number=78, values=[0.21050282840328488], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 389040), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 392336), params={'x': 1.5411941277584986}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=78, state=TrialState.COMPLETE, value=None), FrozenTrial(number=79, values=[9.694970600227476], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 392896), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 396523), params={'x': -1.113674774318518}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=79, state=TrialState.COMPLETE, value=None), FrozenTrial(number=80, values=[3.5500247307859993], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 397085), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 400407), params={'x': 0.11584906900057512}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=80, state=TrialState.COMPLETE, value=None), FrozenTrial(number=81, values=[0.028527127006948625], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 400967), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 404287), params={'x': 2.1688997543128723}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=81, state=TrialState.COMPLETE, value=None), FrozenTrial(number=82, values=[3.451733794781739], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 404847), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 408168), params={'x': 3.8578842253439096}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=82, state=TrialState.COMPLETE, value=None), FrozenTrial(number=83, values=[0.010691879022229478], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 408720), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 412075), params={'x': 2.1034015426491766}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=83, state=TrialState.COMPLETE, value=None), FrozenTrial(number=84, values=[0.5005413489193208], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 412627), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 415945), params={'x': 2.7074894691225593}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=84, state=TrialState.COMPLETE, value=None), FrozenTrial(number=85, values=[0.28584821590567494], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 416523), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 419843), params={'x': 1.465352250630684}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=85, state=TrialState.COMPLETE, value=None), FrozenTrial(number=86, values=[131.74886759988212], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 420432), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 423745), params={'x': -9.478190955019093}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=86, state=TrialState.COMPLETE, value=None), FrozenTrial(number=87, values=[7.923881860381277], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 424344), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 427660), params={'x': 4.814939050917671}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=87, state=TrialState.COMPLETE, value=None), FrozenTrial(number=88, values=[1.6991662178012814], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 428251), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 431584), params={'x': 0.6964792990514952}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=88, state=TrialState.COMPLETE, value=None), FrozenTrial(number=89, values=[0.007305279722261716], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 432173), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 435494), params={'x': 1.9145290708938898}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=89, state=TrialState.COMPLETE, value=None), FrozenTrial(number=90, values=[2.2525044712393463], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 436091), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 439416), params={'x': 3.5008345915654218}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=90, state=TrialState.COMPLETE, value=None), FrozenTrial(number=91, values=[0.013611941728996074], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 440012), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 443347), params={'x': 1.8833297735967052}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=91, state=TrialState.COMPLETE, value=None), FrozenTrial(number=92, values=[0.034969095499672066], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 443932), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 447402), params={'x': 1.8129997446534576}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=92, state=TrialState.COMPLETE, value=None), FrozenTrial(number=93, values=[0.5920434719980951], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 447982), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 451344), params={'x': 1.2305563880321735}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=93, state=TrialState.COMPLETE, value=None), FrozenTrial(number=94, values=[4.972439816515154], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 451924), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 455286), params={'x': -0.2298968174593088}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=94, state=TrialState.COMPLETE, value=None), FrozenTrial(number=95, values=[0.954822669288232], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 455858), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 459242), params={'x': 2.9771502797872147}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=95, state=TrialState.COMPLETE, value=None), FrozenTrial(number=96, values=[2.892535897293333], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 459805), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 463367), params={'x': 0.2992543113994576}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=96, state=TrialState.COMPLETE, value=None), FrozenTrial(number=97, values=[0.29336229215312165], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 463947), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 467380), params={'x': 2.541629294031556}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=97, state=TrialState.COMPLETE, value=None), FrozenTrial(number=98, values=[1.3957190045137042], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 467957), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 471337), params={'x': 0.8185944792266695}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=98, state=TrialState.COMPLETE, value=None), FrozenTrial(number=99, values=[0.0020225435919977103], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 35, 471909), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 35, 475297), params={'x': 1.9550273017042816}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=99, state=TrialState.COMPLETE, value=None)]
获得 trial 的数目：
len(study.trials)
Out:
100
再次执行 optimize()，我们可以继续优化过程。
study.optimize(objective, n_trials=100)
获得更新后的的 trial 数量：
len(study.trials)
Out:
200
由于此目标函数非常简单，所以后面的 100 个trial 并没有提升结果。然而，我们可以再一次检查该结果：
best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))
Out:
Found x: 2.0007453774691357, (x - 2)^2: 5.555875714951739e-07
Total running time of the script: ( 0 minutes 0.799 seconds)
备注
Click here to download the full example code
Python 式的搜索空间
对于超参数采样，Optuna 提供了以下特性：
通过可选的 step 与 log 参数，我们可以对整形或者浮点型参数进行离散化或者取对数操作。
import optuna
def objective(trial):
# Categorical parameter
optimizer = trial.suggest_categorical("optimizer", ["MomentumSGD", "Adam"])
# Integer parameter
num_layers = trial.suggest_int("num_layers", 1, 3)
# Integer parameter (log)
num_channels = trial.suggest_int("num_channels", 32, 512, log=True)
# Integer parameter (discretized)
num_units = trial.suggest_int("num_units", 10, 100, step=5)
# Floating point parameter
dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0)
# Floating point parameter (log)
learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True)
# Floating point parameter (discretized)
drop_path_rate = trial.suggest_float("drop_path_rate", 0.0, 1.0, step=0.1)
定义参数空间
在 Optuna 中，我们使用和 Python 语法类似的方式来定义搜索空间，其中包含条件和循环语句。
类似地，你也可以根据参数值采用分支或者循环。
更多用法见 examples.
分支：
import sklearn.ensemble
import sklearn.svm
def objective(trial):
classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"])
if classifier_name == "SVC":
svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
classifier_obj = sklearn.svm.SVC(C=svc_c)
else:
rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True)
classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)
循环：
import torch
import torch.nn as nn
def create_model(trial, in_size):
n_layers = trial.suggest_int("n_layers", 1, 3)
layers = []
for i in range(n_layers):
n_units = trial.suggest_int("n_units_l{}".format(i), 4, 128, log=True)
layers.append(nn.Linear(in_size, n_units))
layers.append(nn.ReLU())
in_size = n_units
layers.append(nn.Linear(in_size, 10))
return nn.Sequential(*layers)
随着参数个数的增长，优化的难度约呈指数增长。也就是说，当你增加参数的个数的时候，优化所需要的 trial 个数会呈指数增长。因此我们不推荐增加不必要的参数。
Total running time of the script: ( 0 minutes 0.001 seconds)
备注
Click here to download the full example code
高效的优化算法
通过采用最先进的超参数采样算法和对无望 trial 的剪枝， Optuna使得高效的超参数优化成为可能。
采样算法
利用 suggested 参数值和评估的目标值的记录，采样器基本上不断缩小搜索空间，直到找到一个最佳的搜索空间，其产生的参数会带来 更好的目标函数值。关于采样器如何 suggest 参数的更详细的解释见 optuna.samplers.BaseSampler.
Optuna 提供了下列采样算法：
optuna.samplers.TPESampler实现的 Tree-structured Parzen Estimator 算法optuna.samplers.CmaEsSampler实现的 CMA-ES 算法optuna.samplers.GridSampler实现的网格搜索optuna.samplers.RandomSampler实现的随机搜索
默认的采样器是 optuna.samplers.TPESampler.
切换采样器
import optuna
默认情况下， Optuna 这样使用 TPESampler.
study = optuna.create_study()
print(f"Sampler is {study.sampler.__class__.__name__}")
Out:
Sampler is TPESampler
如果你希望使用其他采样器，比如 RandomSampler 和 CmaEsSampler,
study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")
study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")
Out:
Sampler is RandomSampler
Sampler is CmaEsSampler
剪枝算法
Pruners 自动在训练的早期（也就是自动化的 early-stopping）终止无望的 trial.
Optuna 提供以下剪枝算法：
optuna.pruners.SuccessiveHalvingPruner实现的 Asynchronous Successive Halving 算法。optuna.pruners.HyperbandPruner实现的 Hyperband 算法。optuna.pruners.MedianPruner实现的中位数剪枝算法optuna.pruners.ThresholdPruner实现的阈值剪枝算法
在大多数例子中我们采用的 optuna.pruners.MedianPruner, 尽管其性能基本上会被 optuna.pruners.SuccessiveHalvingPruner 和 optuna.pruners.HyperbandPruner 超过，就像在 这个基准测试结果 中那样.
激活 Pruner
要打开剪枝特性的话，你需要在迭代式训练的每一步后调用 report() 和 should_prune(). report() 定期监控目标函数的中间值. should_prune() 确定终结那些没有达到预先设定条件的 trial.
我们推荐在主流机器学习框架中使用集成模块，全部的模块列表在 optuna.integration 里，用例见 optuna/examples.
import logging
import sys
import sklearn.datasets
import sklearn.linear_model
import sklearn.model_selection
def objective(trial):
iris = sklearn.datasets.load_iris()
classes = list(set(iris.target))
train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(
iris.data, iris.target, test_size=0.25, random_state=0
)
alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True)
clf = sklearn.linear_model.SGDClassifier(alpha=alpha)
for step in range(100):
clf.partial_fit(train_x, train_y, classes=classes)
# Report intermediate objective value.
intermediate_value = 1.0 - clf.score(valid_x, valid_y)
trial.report(intermediate_value, step)
# Handle pruning based on the intermediate value.
if trial.should_prune():
raise optuna.TrialPruned()
return 1.0 - clf.score(valid_x, valid_y)
将中位数终止规则设置为剪枝条件。
# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
study.optimize(objective, n_trials=20)
Out:
A new study created in memory with name: no-name-2550f085-0161-4804-aabe-dafd3d2a69fc
Trial 0 finished with value: 0.23684210526315785 and parameters: {'alpha': 0.0056457619503721985}. Best is trial 0 with value: 0.23684210526315785.
Trial 1 finished with value: 0.10526315789473684 and parameters: {'alpha': 0.0004415029135399882}. Best is trial 1 with value: 0.10526315789473684.
Trial 2 finished with value: 0.2894736842105263 and parameters: {'alpha': 0.003557613371651606}. Best is trial 1 with value: 0.10526315789473684.
Trial 3 finished with value: 0.10526315789473684 and parameters: {'alpha': 1.9394951442030858e-05}. Best is trial 1 with value: 0.10526315789473684.
Trial 4 finished with value: 0.39473684210526316 and parameters: {'alpha': 2.6138481035133756e-05}. Best is trial 1 with value: 0.10526315789473684.
Trial 5 pruned.
Trial 6 finished with value: 0.26315789473684215 and parameters: {'alpha': 0.006413605309772462}. Best is trial 1 with value: 0.10526315789473684.
Trial 7 pruned.
Trial 8 pruned.
Trial 9 pruned.
Trial 10 finished with value: 0.3421052631578947 and parameters: {'alpha': 0.0003506682152386342}. Best is trial 1 with value: 0.10526315789473684.
Trial 11 pruned.
Trial 12 pruned.
Trial 13 pruned.
Trial 14 pruned.
Trial 15 pruned.
Trial 16 pruned.
Trial 17 pruned.
Trial 18 pruned.
Trial 19 pruned.
如你所见，有几个 trial 在其迭代完成之前被剪枝（终止）了。消息格式是 "Trial <Trial Number> pruned.".
应该使用哪个 pruner 呢？
对于非深度学习来说，根据 optuna/optuna - wiki “Benchmarks with Kurobako” 里的基准测试结果，我们推荐
对
optuna.samplers.RandomSampler而言optuna.pruners.MedianPruner是最好的。对于
optuna.samplers.TPESampler而言optuna.pruners.Hyperband是最好的。
不过，注意这个基准测试不是深度学习。对于深度学习而言，请参考 Ozaki et al, Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 里的这张表格。
Parallel Compute Resource |
Categorical/Conditional Hyperparameters |
Recommended Algorithms |
|---|---|---|
Limited |
No |
TPE. GP-EI if search space is low-dimensional and continuous. |
Yes |
TPE. GP-EI if search space is low-dimensional and continuous |
|
Sufficient |
No |
CMA-ES, Random Search |
Yes |
Random Search or Genetic Algorithm |
用于剪枝的集成模块
为了用最简单的形式实现剪枝算法，Optuna 为以下库提供了集成模块。
关于 Optuna 集成模块的完整列表，参见 optuna.integration.
For example, XGBoostPruningCallback introduces pruning without directly changing the logic of training iteration.
(See also example for the entire script.)
pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])
Total running time of the script: ( 0 minutes 1.443 seconds)
备注
Click here to download the full example code
简单的并行化
并行化非常直接 optuna.study.Study.optimize().
如果你想要手动执行 Optuna 的优化：
启动一个 RDB 服务器（在本例中我们用 MySQL）
通过 –storage 参数创建 study
在多个节点和进程之间共享 study
当然，你可以像 the kubernetes examples 里一样使用 Kubernetes.
要看并行优化怎么进行的化，请查看下面这个视频。
创建 study
你可以通过 optuna create-study 来创建 study.或者在 Python 脚本中通过 optuna.create_study() 也行。
$ mysql -u root -e "CREATE DATABASE IF NOT EXISTS example"
$ optuna create-study --study-name "distributed-example" --storage "mysql://root@localhost/example"
[I 2020-07-21 13:43:39,642] A new study created with name: distributed-example
然后写一个优化脚本。假设 foo.py 包含了以下代码。
import optuna
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
if __name__ == "__main__":
study = optuna.load_study(
study_name="distributed-example", storage="mysql://root@localhost/example"
)
study.optimize(objective, n_trials=100)
备注
Click here to download the full example code
用于超参数优化分析的快速可视化
Optuna 在 optuna.visualization 里提供了各种可视化特性用于分析优化结果。
通过可视化乳腺癌数据集 上的 lightgbm 历史记录，本教程将带你体验此模块。
import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split
import optuna
from optuna.visualization import plot_contour
from optuna.visualization import plot_edf
from optuna.visualization import plot_intermediate_values
from optuna.visualization import plot_optimization_history
from optuna.visualization import plot_parallel_coordinate
from optuna.visualization import plot_param_importances
from optuna.visualization import plot_slice
SEED = 42
np.random.seed(SEED)
定义目标函数
def objective(trial):
data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
dtrain = lgb.Dataset(train_x, label=train_y)
dvalid = lgb.Dataset(valid_x, label=valid_y)
param = {
"objective": "binary",
"metric": "auc",
"verbosity": -1,
"boosting_type": "gbdt",
"bagging_fraction": trial.suggest_float("bagging_fraction", 0.4, 1.0),
"bagging_freq": trial.suggest_int("bagging_freq", 1, 7),
"min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
}
# Add a callback for pruning.
pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
gbm = lgb.train(
param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
)
preds = gbm.predict(valid_x)
pred_labels = np.rint(preds)
accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
return accuracy
study = optuna.create_study(
direction="maximize",
sampler=optuna.samplers.TPESampler(seed=SEED),
pruner=optuna.pruners.MedianPruner(n_warmup_steps=10),
)
study.optimize(objective, n_trials=100, timeout=600)
Out:
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
绘图函数
对优化历史进行可视化。细节见 plot_optimization_history().
plot_optimization_history(study)
绘制各 trial 的学习曲线。细节见 plot_intermediate_values().
plot_intermediate_values(study)
绘制高维参数关系。细节见 plot_parallel_coordinate().
plot_parallel_coordinate(study)
选择要绘制的参数
plot_parallel_coordinate(study, params=["bagging_freq", "bagging_fraction"])
绘制超参数关系。细节见 plot_contour().
plot_contour(study)
选择要绘制的参数
plot_contour(study, params=["bagging_freq", "bagging_fraction"])
绘制各超参数的切片图。细节见 plot_slice().
plot_slice(study)
选择要绘制的参数
plot_slice(study, params=["bagging_freq", "bagging_fraction"])
绘制超参数重要性。细节见 plot_param_importances().
plot_param_importances(study)
绘制经验分布函数。 plot_edf().
plot_edf(study)
Total running time of the script: ( 0 minutes 4.634 seconds)
用法
展现可能对你轻松使用 Optuna 更有帮助的一些用法
备注
Click here to download the full example code
用 RDB 后端保存/恢复 Study
RDB后端可以实现持久化实验（即保存和恢复 study）以及访问 study 的历史记录。此外，我们还可以利用这个特点来进行分布式优化。具体描述见 简单的并行化.
在本部分中，我们将尝试一个在本地环境下运行SQLite DB的简单例子。
备注
通过设置 DB 的 storage URL 参数，你也可以使用其他的 RDB 后端，比如 PostgreSQL 或者 MySQL. 设置 URL 的方式参见 SQLAlchemy 的文档.
新建 Study
通过调用函数 create_study()，我们可以创建一个持久化的 study. 创建新 study 会自动初始化一个 SQLite 文件 example.db.
import logging
import sys
import optuna
# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study_name = "example-study" # Unique identifier of the study.
storage_name = "sqlite:///{}.db".format(study_name)
study = optuna.create_study(study_name=study_name, storage=storage_name)
Out:
A new study created in RDB with name: example-study
为了运行一个 study, 我们需要将目标函数传入 optimize() 方法并调用它。
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
study.optimize(objective, n_trials=3)
Out:
Trial 0 finished with value: 1.5067618568079724 and parameters: {'x': 3.2275022838300433}. Best is trial 0 with value: 1.5067618568079724.
Trial 1 finished with value: 24.96165877167141 and parameters: {'x': -2.996164405988999}. Best is trial 0 with value: 1.5067618568079724.
Trial 2 finished with value: 0.45814470516718514 and parameters: {'x': 2.676863874916652}. Best is trial 2 with value: 0.45814470516718514.
恢复 Study
为了恢复 study, 首先需要初始化一个 Study 对象， 并将该study 的名字 example-study 和 DB URL参数 sqlite:///example.db 传入其中。
study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)
study.optimize(objective, n_trials=3)
Out:
Using an existing study with name 'example-study' instead of creating a new one.
Trial 3 finished with value: 54.55663920492726 and parameters: {'x': 9.38624662497315}. Best is trial 2 with value: 0.45814470516718514.
Trial 4 finished with value: 14.274094669669244 and parameters: {'x': -1.7781072866806262}. Best is trial 2 with value: 0.45814470516718514.
Trial 5 finished with value: 99.93271079015234 and parameters: {'x': -7.996634973337395}. Best is trial 2 with value: 0.45814470516718514.
实验历史记录
我们可以通过 Study 类来获得 study 和对应 trials的历史记录。比如，下面的语句可以获取 example-study 的所有 trials.
study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)
df = study.trials_dataframe(attrs=("number", "value", "params", "state"))
Out:
Using an existing study with name 'example-study' instead of creating a new one.
trials_dataframe() 方法会返回一个如下的 pandas dataframe：
print(df)
Out:
number value params_x state
0 0 1.506762 3.227502 COMPLETE
1 1 24.961659 -2.996164 COMPLETE
2 2 0.458145 2.676864 COMPLETE
3 3 54.556639 9.386247 COMPLETE
4 4 14.274095 -1.778107 COMPLETE
5 5 99.932711 -7.996635 COMPLETE
Study 对象也有一些其他属性，比如 trials, best_value 和 best_params (见 轻量级、多功能和跨平台架构).
print("Best params: ", study.best_params)
print("Best value: ", study.best_value)
print("Best Trial: ", study.best_trial)
print("Trials: ", study.trials)
Out:
Best params: {'x': 2.676863874916652}
Best value: 0.45814470516718514
Best Trial: FrozenTrial(number=2, values=[0.45814470516718514], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 42, 892134), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 42, 908612), params={'x': 2.676863874916652}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE, value=None)
Trials: [FrozenTrial(number=0, values=[1.5067618568079724], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 42, 796078), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 42, 816506), params={'x': 3.2275022838300433}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE, value=None), FrozenTrial(number=1, values=[24.96165877167141], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 42, 852903), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 42, 867951), params={'x': -2.996164405988999}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=2, state=TrialState.COMPLETE, value=None), FrozenTrial(number=2, values=[0.45814470516718514], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 42, 892134), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 42, 908612), params={'x': 2.676863874916652}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE, value=None), FrozenTrial(number=3, values=[54.55663920492726], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 42, 976636), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 42, 995327), params={'x': 9.38624662497315}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=4, state=TrialState.COMPLETE, value=None), FrozenTrial(number=4, values=[14.274094669669244], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 43, 26577), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 43, 42780), params={'x': -1.7781072866806262}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=5, state=TrialState.COMPLETE, value=None), FrozenTrial(number=5, values=[99.93271079015234], datetime_start=datetime.datetime(2022, 5, 26, 12, 5, 43, 66882), datetime_complete=datetime.datetime(2022, 5, 26, 12, 5, 43, 81061), params={'x': -7.996634973337395}, distributions={'x': UniformDistribution(high=10.0, low=-10.0)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=6, state=TrialState.COMPLETE, value=None)]
Total running time of the script: ( 0 minutes 0.622 seconds)
备注
Click here to download the full example code
用 Optuna 进行多目标优化
This tutorial showcases Optuna’s multi-objective optimization feature by optimizing the validation accuracy of Fashion MNIST dataset and the FLOPS of the model implemented in PyTorch.
我们采用 thop 来测量 FLOPS.
import thop
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import optuna
DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
DIR = ".."
BATCHSIZE = 128
N_TRAIN_EXAMPLES = BATCHSIZE * 30
N_VALID_EXAMPLES = BATCHSIZE * 10
def define_model(trial):
n_layers = trial.suggest_int("n_layers", 1, 3)
layers = []
in_features = 28 * 28
for i in range(n_layers):
out_features = trial.suggest_int("n_units_l{}".format(i), 4, 128)
layers.append(nn.Linear(in_features, out_features))
layers.append(nn.ReLU())
p = trial.suggest_float("dropout_{}".format(i), 0.2, 0.5)
layers.append(nn.Dropout(p))
in_features = out_features
layers.append(nn.Linear(in_features, 10))
layers.append(nn.LogSoftmax(dim=1))
return nn.Sequential(*layers)
# Defines training and evaluation.
def train_model(model, optimizer, train_loader):
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
optimizer.zero_grad()
F.nll_loss(model(data), target).backward()
optimizer.step()
def eval_model(model, valid_loader):
model.eval()
correct = 0
with torch.no_grad():
for batch_idx, (data, target) in enumerate(valid_loader):
data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
pred = model(data).argmax(dim=1, keepdim=True)
correct += pred.eq(target.view_as(pred)).sum().item()
accuracy = correct / N_VALID_EXAMPLES
flops, _ = thop.profile(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),), verbose=False)
return flops, accuracy
Out:
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning:
CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
定义多目标函数. 其目标为 FLOPS 和 准确度.
def objective(trial):
train_dataset = torchvision.datasets.FashionMNIST(
DIR, train=True, download=True, transform=torchvision.transforms.ToTensor()
)
train_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))),
batch_size=BATCHSIZE,
shuffle=True,
)
val_dataset = torchvision.datasets.FashionMNIST(
DIR, train=False, transform=torchvision.transforms.ToTensor()
)
val_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))),
batch_size=BATCHSIZE,
shuffle=True,
)
model = define_model(trial).to(DEVICE)
optimizer = torch.optim.Adam(
model.parameters(), trial.suggest_float("lr", 1e-5, 1e-1, log=True)
)
for epoch in range(10):
train_model(model, optimizer, train_loader)
flops, accuracy = eval_model(model, val_loader)
return flops, accuracy
运行多目标优化.
如果你的优化问题是多目标的, Optuna 预设你会为每一个目标指定优化方向. 具体在本例中, 我们希望最小化 FLOPS (我们想要更快的模型) 和最大化准确度. 所以我们将 directions 设置为 ["minimize", "maximize"].
study = optuna.create_study(directions=["minimize", "maximize"])
study.optimize(objective, n_trials=30, timeout=300)
print("Number of finished trials: ", len(study.trials))
Out:
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to ../FashionMNIST/raw/train-images-idx3-ubyte.gz
0it [00:00, ?it/s]
0%| | 0/26421880 [00:00<?, ?it/s]
0%| | 49152/26421880 [00:00<01:47, 245800.96it/s]
0%| | 114688/26421880 [00:00<01:03, 415451.53it/s]
1%| | 212992/26421880 [00:00<00:41, 626641.77it/s]
2%|1 | 442368/26421880 [00:00<00:21, 1205029.17it/s]
3%|3 | 892928/26421880 [00:00<00:11, 2298475.10it/s]
7%|6 | 1794048/26421880 [00:00<00:05, 4454796.77it/s]
13%|#2 | 3342336/26421880 [00:01<00:02, 7919072.06it/s]
18%|#8 | 4882432/26421880 [00:01<00:02, 10227247.85it/s]
24%|##4 | 6471680/26421880 [00:01<00:01, 11960396.41it/s]
30%|### | 8052736/26421880 [00:01<00:01, 13096020.86it/s]
37%|###6 | 9650176/26421880 [00:01<00:01, 13949535.11it/s]
43%|####2 | 11239424/26421880 [00:01<00:01, 14530982.26it/s]
49%|####8 | 12836864/26421880 [00:01<00:00, 14918256.39it/s]
55%|#####4 | 14442496/26421880 [00:01<00:00, 15253596.43it/s]
61%|###### | 16048128/26421880 [00:01<00:00, 15465740.17it/s]
67%|######6 | 17678336/26421880 [00:01<00:00, 15707752.46it/s]
73%|#######3 | 19316736/26421880 [00:02<00:00, 15890648.33it/s]
79%|#######9 | 20963328/26421880 [00:02<00:00, 16058160.88it/s]
86%|########5 | 22609920/26421880 [00:02<00:00, 16155043.42it/s]
92%|#########1| 24264704/26421880 [00:02<00:00, 16263152.14it/s]
98%|#########8| 25919488/26421880 [00:02<00:00, 16325161.67it/s]
26427392it [00:02, 10708587.67it/s]
Extracting ../FashionMNIST/raw/train-images-idx3-ubyte.gz to ../FashionMNIST/raw
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to ../FashionMNIST/raw/train-labels-idx1-ubyte.gz
0it [00:00, ?it/s]
0%| | 0/29515 [00:00<?, ?it/s]
32768it [00:00, 99247.73it/s]
Extracting ../FashionMNIST/raw/train-labels-idx1-ubyte.gz to ../FashionMNIST/raw
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to ../FashionMNIST/raw/t10k-images-idx3-ubyte.gz
0it [00:00, ?it/s]
0%| | 0/4422102 [00:00<?, ?it/s]
1%| | 40960/4422102 [00:00<00:12, 361996.97it/s]
2%|2 | 90112/4422102 [00:00<00:09, 434259.18it/s]
4%|3 | 163840/4422102 [00:00<00:07, 568345.76it/s]
8%|8 | 368640/4422102 [00:00<00:03, 1141556.84it/s]
18%|#8 | 802816/4422102 [00:00<00:01, 2281822.61it/s]
35%|###4 | 1531904/4422102 [00:00<00:00, 3971141.49it/s]
67%|######6 | 2957312/4422102 [00:00<00:00, 7314825.62it/s]
4423680it [00:01, 4209306.93it/s]
Extracting ../FashionMNIST/raw/t10k-images-idx3-ubyte.gz to ../FashionMNIST/raw
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to ../FashionMNIST/raw/t10k-labels-idx1-ubyte.gz
0it [00:00, ?it/s]
0%| | 0/5148 [00:00<?, ?it/s]
8192it [00:00, 34965.80it/s]
Extracting ../FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to ../FashionMNIST/raw
Processing...
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/torchvision/datasets/mnist.py:480: UserWarning:
The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
Done!
Number of finished trials: 30
可视化地检查位于帕累托前沿上的 trials
optuna.visualization.plot_pareto_front(study, target_names=["FLOPS", "accuracy"])
Out:
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/002_multi_objective.py:123: ExperimentalWarning:
plot_pareto_front is experimental (supported from v2.4.0). The interface can change in the future.
Total running time of the script: ( 1 minutes 51.154 seconds)
备注
Click here to download the full example code
用户定义属性
利用用户自定义属性，这个功能可以给实验做注解。
将用户定义属性添加到 Study
Study 对象提供了一个将键-值对设置为用户自定义属性的方法： set_user_attr(). 这里的键应该属于 str 类型， 而值可以是任何能用 json.dumps 来序列化的对象。
import sklearn.datasets
import sklearn.model_selection
import sklearn.svm
import optuna
study = optuna.create_study(storage="sqlite:///example.db")
study.set_user_attr("contributors", ["Akiba", "Sano"])
study.set_user_attr("dataset", "MNIST")
我们可以利用 user_attr 属性来获取所有定义过的属性。
study.user_attrs # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}
Out:
{'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}
StudySummary 对象中也包含了用户的自定义属性。我们可以从 get_all_study_summaries() 中获取它。
study_summaries = optuna.get_all_study_summaries("sqlite:///example.db")
study_summaries[0].user_attrs # {"contributors": ["Akiba", "Sano"], "dataset": "MNIST"}
Out:
{'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}
参见
在命令行界面里，optuna study set-user-attr 可用于设置用户定义属性。
将用户属性添加到 Trial 中
和 Study 类似，Trial 对象也提供了一个设置属性的方法 set_user_attr() 方法。这些属性是在目标函数内部设置的。
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
clf = sklearn.svm.SVC(C=svc_c)
accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean()
trial.set_user_attr("accuracy", accuracy)
return 1.0 - accuracy # return error for minimization
study.optimize(objective, n_trials=1)
可以用如下方式获取这些注解的属性：
study.trials[0].user_attrs
Out:
{'accuracy': 0.9266666666666667}
注意，在本例中，属性不是被注解到 Study 的，它属于一个单独的 Trial.
Total running time of the script: ( 0 minutes 0.282 seconds)
备注
Click here to download the full example code
命令行界面
Command |
Description |
|---|---|
create-study |
创建一个新的 study. |
delete-study |
删除指定的 study. |
dashboard |
启动 web 面板 (beta). |
storage upgrade |
升级数据库 schema. |
studies |
输出 study 列表 |
study optimize |
针对一个 study 开始优化过程。 |
study set-user-attr |
设置 study 的用户属性。 |
Optuna 提供的命令行界面包括上表中的所有命令。
如果你不使用 IPython shell, 而是写 Python 脚本文件的话，编写像下面这样的脚本是完全可以的：
import optuna
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
if __name__ == "__main__":
study = optuna.create_study()
study.optimize(objective, n_trials=100)
print("Best value: {} (params: {})\n".format(study.best_value, study.best_params))
Out:
Best value: 0.0010273417403227686 (params: {'x': 1.9679478278376836})
然而，通过 optuna 命令，我们可以少写一些上面这样的模版代码。假设有一个 foo.py 文件，它只包含如下代码：
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
即使在这种情况下，我们也可以利用如下命令启动优化过程（请暂时忽略 --storage sqlite:///example.db 的作用, 关于它的具体描述见 用 RDB 后端保存/恢复 Study）。
$ cat foo.py
def objective(trial):
x = trial.suggest_float('x', -10, 10)
return (x - 2) ** 2
$ STUDY_NAME=`optuna create-study --storage sqlite:///example.db`
$ optuna study optimize foo.py objective --n-trials=100 --storage sqlite:///example.db --study-name $STUDY_NAME
[I 2018-05-09 10:40:25,196] Finished a trial resulted in value: 54.353767789264026. Current best value is 54.353767789264026 with parameters: {'x': -5.372500782588228}.
[I 2018-05-09 10:40:25,197] Finished a trial resulted in value: 15.784266965526376. Current best value is 15.784266965526376 with parameters: {'x': 5.972941852774387}.
...
[I 2018-05-09 10:40:26,204] Finished a trial resulted in value: 14.704254135013741. Current best value is 2.280758099793617e-06 with parameters: {'x': 1.9984897821018828}.
请注意，foo.py 中只包含目标函数的定义。通过向 optuna study optimize 命令传递该文件名和对应目标函数的方法名，我们就可以启动优化过程。
Total running time of the script: ( 0 minutes 0.360 seconds)
备注
Click here to download the full example code
用户定义的采样器 (Sampler)
你可以用用户定义的 sampler 来实现：
试验你自己的采样算法，
实现具体任务对应的算法来改进优化性能，或者
将其他的优化框架包装起来，整合进 Optuna 的流水线中 (比如
SkoptSampler).
本节将介绍 sampler 类的内部行为，并展示一个实现用户自定义 sampler 的例子。
Sampler 概述
A sampler has the responsibility to determine the parameter values to be evaluated in a trial.
When a suggest API (e.g., suggest_float()) is called inside an objective function, the corresponding distribution object (e.g., UniformDistribution) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the suggest API and evaluated in the objective function.
为了创建一个新的 sampler, 你所定义的类需继承 BaseSampler.该基类提供三个抽象方法：infer_relative_search_space(), sample_relative() 和 sample_independent().
从这些方法名可以看出，Optuna 支持两种类型的采样过程：一种是 relative sampling, 它考虑了单个 trial 内参数之间的相关性， 另一种是 independent sampling, 它对各个参数的采样是彼此独立的。
在一个 trial 刚开始时，infer_relative_search_space() 会被调用，它向该 trial 提供一个相对搜索空间。之后， sample_relative() 会被触发，它从该搜索空间中对相对参数进行采样。在目标函数的执行过程中，sample_independent() 用于对不属于该相对搜索空间的参数进行采样。
备注
更多细节参见 BaseSampler 的文档。
案例: 实现模拟退火 Sampler (SimulatedAnnealingSampler)
下面的代码根据 Simulated Annealing (SA) 定义类一个 sampler：
import numpy as np
import optuna
class SimulatedAnnealingSampler(optuna.samplers.BaseSampler):
def __init__(self, temperature=100):
self._rng = np.random.RandomState()
self._temperature = temperature # Current temperature.
self._current_trial = None # Current state.
def sample_relative(self, study, trial, search_space):
if search_space == {}:
return {}
# Simulated Annealing algorithm.
# 1. Calculate transition probability.
prev_trial = study.trials[-2]
if self._current_trial is None or prev_trial.value <= self._current_trial.value:
probability = 1.0
else:
probability = np.exp(
(self._current_trial.value - prev_trial.value) / self._temperature
)
self._temperature *= 0.9 # Decrease temperature.
# 2. Transit the current state if the previous result is accepted.
if self._rng.uniform(0, 1) < probability:
self._current_trial = prev_trial
# 3. Sample parameters from the neighborhood of the current point.
# The sampled parameters will be used during the next execution of
# the objective function passed to the study.
params = {}
for param_name, param_distribution in search_space.items():
if not isinstance(param_distribution, optuna.distributions.UniformDistribution):
raise NotImplementedError("Only suggest_float() is supported")
current_value = self._current_trial.params[param_name]
width = (param_distribution.high - param_distribution.low) * 0.1
neighbor_low = max(current_value - width, param_distribution.low)
neighbor_high = min(current_value + width, param_distribution.high)
params[param_name] = self._rng.uniform(neighbor_low, neighbor_high)
return params
# The rest are unrelated to SA algorithm: boilerplate
def infer_relative_search_space(self, study, trial):
return optuna.samplers.intersection_search_space(study)
def sample_independent(self, study, trial, param_name, param_distribution):
independent_sampler = optuna.samplers.RandomSampler()
return independent_sampler.sample_independent(study, trial, param_name, param_distribution)
备注
为了代码的简洁性，上面的实现没有支持一些特性 (比如 maximization). 如果你对如何实现这些特性感兴趣，请看 examples/samplers/simulated_annealing.py.
你可以像使用内置的 sampler 一样使用 SimulatedAnnealingSampler:
def objective(trial):
x = trial.suggest_float("x", -10, 10)
y = trial.suggest_float("y", -5, 5)
return x ** 2 + y
sampler = SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)
best_trial = study.best_trial
print("Best value: ", best_trial.value)
print("Parameters that achieve the best value: ", best_trial.params)
Out:
Best value: -4.945941226368364
Parameters that achieve the best value: {'x': 0.05100620365716724, 'y': -4.948542859179881}
在上面这个优化过程中，参数 x 和 y 的值都是由 SimulatedAnnealingSampler.sample_relative 方法采样得出的。
备注
严格意义上说，在第一个 trial 中，SimulatedAnnealingSampler.sample_independent 用于采样参数值。因为，如果没有已经完成的 trial 的话， SimulatedAnnealingSampler.infer_relative_search_space 中的 intersection_search_space() 是无法对搜索空间进行推断的。
Total running time of the script: ( 0 minutes 0.376 seconds)
备注
Click here to download the full example code
用户定义的 Pruner
在 optuna.pruners 中, 我们描述了一个目标函数如何可选地包含一些 pruning 函数调用, 这些函数允许 Optuna 去终结中间结果无望的 tiral. 在本文档中, 我们描述了如何实现一个你自己的 pruner, 也即一个自定义 trial 终止条件的策略.
Pruning 界面概述
The create_study() constructor takes, as an optional
argument, a pruner inheriting from BasePruner. The
pruner should implement the abstract method
prune(), which takes arguments for the
associated Study and Trial and
returns a boolean value: True if the trial should be pruned and False
otherwise. Using the Study and Trial objects, you can access all other trials
through the get_trial() method and, and from a trial,
its reported intermediate values through the
intermediate_values() (a
dictionary which maps an integer step to a float value).
你可以参考 Optuna 内置的 pruner 的源代码作为构建你自己的 pruner 的模板. 在本文档中, 我们描述了一个简单 (但是却激进) 的 pruner 实现过程, 它会对那些处于同步骤 trial 中最后一名的 trial 进行剪枝.
备注
Please refer to the documentation of BasePruner or,
for example, ThresholdPruner or
PercentilePruner for more robust examples of pruner
implementation, including error checking and complex pruner-internal logic.
例子: 实现 LastPlacePruner
We aim to optimize the loss and alpha hyperparameters for a stochastic
gradient descent classifier (SGDClassifier) run on the sklearn iris dataset. We
implement a pruner which terminates a trial at a certain step if it is in last
place compared to completed trials at the same step. We begin considering
pruning after a “warmup” of 1 training step and 5 completed trials. For
demonstration purposes, we print() a diagnostic message from prune when
it is about to return True (indicating pruning).
It may be important to note that the SGDClassifier score, as it is evaluated on
a holdout set, decreases with enough training steps due to overfitting. This
means that a trial could be pruned even if it had a favorable (high) value on a
previous training set. After pruning, Optuna will take the intermediate value
last reported as the value of the trial.
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDClassifier
import optuna
from optuna.pruners import BasePruner
from optuna.trial._state import TrialState
class LastPlacePruner(BasePruner):
def __init__(self, warmup_steps, warmup_trials):
self._warmup_steps = warmup_steps
self._warmup_trials = warmup_trials
def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool:
# Get the latest score reported from this trial
step = trial.last_step
if step: # trial.last_step == None when no scores have been reported yet
this_score = trial.intermediate_values[step]
# Get scores from other trials in the study reported at the same step
completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,))
other_scores = [
t.intermediate_values[step]
for t in completed_trials
if step in t.intermediate_values
]
other_scores = sorted(other_scores)
# Prune if this trial at this step has a lower value than all completed trials
# at the same step. Note that steps will begin numbering at 0 in the objective
# function definition below.
if step >= self._warmup_steps and len(other_scores) > self._warmup_trials:
if this_score < other_scores[0]:
print(f"prune() True: Trial {trial.number}, Step {step}, Score {this_score}")
return True
return False
最后, 让我们通过一个简单的超参数优化来确认该实现是正确的.
def objective(trial):
iris = load_iris()
classes = np.unique(iris.target)
X_train, X_valid, y_train, y_valid = train_test_split(
iris.data, iris.target, train_size=100, test_size=50, random_state=0
)
loss = trial.suggest_categorical("loss", ["hinge", "log", "perceptron"])
alpha = trial.suggest_float("alpha", 0.00001, 0.001, log=True)
clf = SGDClassifier(loss=loss, alpha=alpha, random_state=0)
score = 0
for step in range(0, 5):
clf.partial_fit(X_train, y_train, classes=classes)
score = clf.score(X_valid, y_valid)
trial.report(score, step)
if trial.should_prune():
raise optuna.TrialPruned()
return score
pruner = LastPlacePruner(warmup_steps=1, warmup_trials=5)
study = optuna.create_study(direction="maximize", pruner=pruner)
study.optimize(objective, n_trials=50)
Out:
prune() True: Trial 6, Step 1, Score 0.84
prune() True: Trial 7, Step 1, Score 0.62
prune() True: Trial 10, Step 1, Score 0.62
prune() True: Trial 17, Step 1, Score 0.32
prune() True: Trial 26, Step 1, Score 0.62
prune() True: Trial 38, Step 4, Score 0.68
prune() True: Trial 42, Step 4, Score 0.68
prune() True: Trial 45, Step 4, Score 0.68
prune() True: Trial 49, Step 1, Score 0.62
Total running time of the script: ( 0 minutes 0.705 seconds)
备注
Click here to download the full example code
Study.optimize 的回调
本教程展示了如何使用和实现一个 用于 optimize() 的Optuna Callback .
Callback 会在 objective 每次求值以后被调用一次, 它接受 Study 和 FrozenTrial 作为参数并进行处理.
MLflowCallback 是个好例子.
在特定数量的 trial 被剪枝后终止优化
本例实现了一个有状态的回调函数. 如果特定数目的 trial 被剪枝了, 它将终止优化. 被剪枝的 trial 数是通过 threshold 来指定的.
import optuna
class StopWhenTrialKeepBeingPrunedCallback:
def __init__(self, threshold: int):
self.threshold = threshold
self._consequtive_pruned_count = 0
def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None:
if trial.state == optuna.trial.TrialState.PRUNED:
self._consequtive_pruned_count += 1
else:
self._consequtive_pruned_count = 0
if self._consequtive_pruned_count >= self.threshold:
study.stop()
该目标函数会对除了前五个 trial 之外的所有trial 进行剪枝 (trial.number 从 0 开始计数).
def objective(trial):
if trial.number > 4:
raise optuna.TrialPruned
return trial.suggest_float("x", 0, 1)
在这里, 我们将阈值设置为 2: 优化过程会在一旦两个 trial 被剪枝后发生. 因此, 我们预期该 study 会在 7 个 trial 后停止.
import logging
import sys
# Add stream handler of stdout to show the messages
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study_stop_cb = StopWhenTrialKeepBeingPrunedCallback(2)
study = optuna.create_study()
study.optimize(objective, n_trials=10, callbacks=[study_stop_cb])
Out:
A new study created in memory with name: no-name-df3417c5-834d-42a2-9978-20adf440a28d
Trial 0 finished with value: 0.7727512815488775 and parameters: {'x': 0.7727512815488775}. Best is trial 0 with value: 0.7727512815488775.
Trial 1 finished with value: 0.5717267978885823 and parameters: {'x': 0.5717267978885823}. Best is trial 1 with value: 0.5717267978885823.
Trial 2 finished with value: 0.8552077108941241 and parameters: {'x': 0.8552077108941241}. Best is trial 1 with value: 0.5717267978885823.
Trial 3 finished with value: 0.9001926292198051 and parameters: {'x': 0.9001926292198051}. Best is trial 1 with value: 0.5717267978885823.
Trial 4 finished with value: 0.7865904524969077 and parameters: {'x': 0.7865904524969077}. Best is trial 1 with value: 0.5717267978885823.
Trial 5 pruned.
Trial 6 pruned.
从上面的日志中可以看出, study 如期在 7 个trial 之后停止了.
Total running time of the script: ( 0 minutes 0.008 seconds)
备注
Click here to download the full example code
手动指定超参数
你自然有一些特定的超参数集要先尝试, 比如初始学习率和叶子数量. 另外, 也有可能在让 Optuna 找到更好的超参数集之前，你已经尝试过一些集合.
Optuna 提供 两个API 以应对这种场景:
将这些超参数集合传递过去并让 Optuna 对其求值 -
enqueue_trial()将这些集合的结果标记为已完成的
Trials -add_trial()
第一个场景: 让 Optuna 对你的超参数求值
在这个场景中, 我们假设你已经有了一些开箱即用的超参数但是还没有对他们进行过求值, 你决定用 Optuna 来找到更好的超参数.
Optuna 有一个 API optuna.study.Study.enqueue_trial() , 它允许你将这些超参数传入 Optuna, Optuna 会对他们求值.
这部分将结合 LightGBM 向你介绍如何使用这些API.
import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split
import optuna
定义目标函数.
def objective(trial):
data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
dtrain = lgb.Dataset(train_x, label=train_y)
dvalid = lgb.Dataset(valid_x, label=valid_y)
param = {
"objective": "binary",
"metric": "auc",
"verbosity": -1,
"boosting_type": "gbdt",
"bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1),
"bagging_freq": trial.suggest_int("bagging_freq", 0, 7),
"min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
}
# Add a callback for pruning.
pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
gbm = lgb.train(
param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
)
preds = gbm.predict(valid_x)
pred_labels = np.rint(preds)
accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
return accuracy
然后构建 Study 用于超参数优化.
study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
在这里, 我们让 Optuna 对一些带有更大的 "bagging_fraq" 和默认值的集合求值.
study.enqueue_trial(
{
"bagging_fraction": 1.0,
"bagging_freq": 0,
"min_child_samples": 20,
}
)
study.enqueue_trial(
{
"bagging_fraction": 0.75,
"bagging_freq": 5,
"min_child_samples": 20,
}
)
import logging
import sys
# Add stream handler of stdout to show the messages to see Optuna works expectedly.
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study.optimize(objective, n_trials=100, timeout=600)
Out:
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:81: ExperimentalWarning:
enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/optuna/study.py:857: ExperimentalWarning:
create_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/optuna/study.py:857: ExperimentalWarning:
add_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:89: ExperimentalWarning:
enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 0 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 1.0, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 0 with value: 0.972027972027972.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 1 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.75, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 0 with value: 0.972027972027972.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 2 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.4634984818580007, 'bagging_freq': 4, 'min_child_samples': 43}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 3 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7498262874277628, 'bagging_freq': 7, 'min_child_samples': 48}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 4 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.7947319219659837, 'bagging_freq': 4, 'min_child_samples': 93}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 5 pruned. Trial was pruned at iteration 0.
Trial 6 pruned. Trial was pruned at iteration 0.
Trial 7 pruned. Trial was pruned at iteration 0.
Trial 8 pruned. Trial was pruned at iteration 0.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 0.
Trial 11 pruned. Trial was pruned at iteration 0.
Trial 12 pruned. Trial was pruned at iteration 0.
Trial 13 pruned. Trial was pruned at iteration 0.
Trial 14 pruned. Trial was pruned at iteration 0.
Trial 15 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.661493861887769, 'bagging_freq': 1, 'min_child_samples': 52}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 16 pruned. Trial was pruned at iteration 0.
Trial 17 pruned. Trial was pruned at iteration 0.
Trial 18 pruned. Trial was pruned at iteration 0.
Trial 19 pruned. Trial was pruned at iteration 0.
Trial 20 pruned. Trial was pruned at iteration 0.
Trial 21 pruned. Trial was pruned at iteration 0.
Trial 22 pruned. Trial was pruned at iteration 0.
Trial 23 pruned. Trial was pruned at iteration 0.
Trial 24 pruned. Trial was pruned at iteration 0.
Trial 25 pruned. Trial was pruned at iteration 0.
Trial 26 pruned. Trial was pruned at iteration 0.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 0.
Trial 30 pruned. Trial was pruned at iteration 0.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 pruned. Trial was pruned at iteration 0.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 0.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7395765217168897, 'bagging_freq': 3, 'min_child_samples': 23}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 0.
Trial 42 pruned. Trial was pruned at iteration 0.
Trial 43 pruned. Trial was pruned at iteration 0.
Trial 44 pruned. Trial was pruned at iteration 0.
Trial 45 pruned. Trial was pruned at iteration 0.
Trial 46 pruned. Trial was pruned at iteration 0.
Trial 47 pruned. Trial was pruned at iteration 0.
Trial 48 pruned. Trial was pruned at iteration 0.
Trial 49 pruned. Trial was pruned at iteration 0.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 0.
Trial 53 pruned. Trial was pruned at iteration 0.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8520330576045076, 'bagging_freq': 7, 'min_child_samples': 15}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 56 pruned. Trial was pruned at iteration 0.
Trial 57 pruned. Trial was pruned at iteration 0.
Trial 58 pruned. Trial was pruned at iteration 0.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 0.
Trial 63 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8745866104487076, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 64 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8344316889970204, 'bagging_freq': 5, 'min_child_samples': 9}. Best is trial 2 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 0.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 pruned. Trial was pruned at iteration 0.
Trial 72 pruned. Trial was pruned at iteration 0.
Trial 73 pruned. Trial was pruned at iteration 0.
Trial 74 pruned. Trial was pruned at iteration 0.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 0.
Trial 77 pruned. Trial was pruned at iteration 0.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 0.
Trial 81 pruned. Trial was pruned at iteration 0.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 pruned. Trial was pruned at iteration 0.
Trial 84 pruned. Trial was pruned at iteration 0.
Trial 85 pruned. Trial was pruned at iteration 0.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 0.
Trial 93 pruned. Trial was pruned at iteration 0.
Trial 94 pruned. Trial was pruned at iteration 0.
Trial 95 pruned. Trial was pruned at iteration 0.
Trial 96 pruned. Trial was pruned at iteration 0.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 11.
Trial 99 pruned. Trial was pruned at iteration 0.
第二个场景: 让 Optuna 利用已经求值过的超参数.
在这个场景中, 我们假设你已经有了一些开箱即用的超参数, 而且你已经对他们进行求值过, 但是其结果并不让人满意. 因此你考虑使用 Optuna 来找到更好的超参数.
Optuna 有一个 API optuna.study.Study.add_trial() , 它让你向Optuna 注册这些结果, 之后 Optuna 会在进行超参数采样的时候将它们考虑进去.
在本部分中, objective 和第一个场景中一样.
study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.add_trial(
optuna.trial.create_trial(
params={
"bagging_fraction": 1.0,
"bagging_freq": 0,
"min_child_samples": 20,
},
distributions={
"bagging_fraction": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),
"bagging_freq": optuna.distributions.IntUniformDistribution(0, 7),
"min_child_samples": optuna.distributions.IntUniformDistribution(5, 100),
},
value=0.94,
)
)
study.add_trial(
optuna.trial.create_trial(
params={
"bagging_fraction": 0.75,
"bagging_freq": 5,
"min_child_samples": 20,
},
distributions={
"bagging_fraction": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),
"bagging_freq": optuna.distributions.IntUniformDistribution(0, 7),
"min_child_samples": optuna.distributions.IntUniformDistribution(5, 100),
},
value=0.95,
)
)
study.optimize(objective, n_trials=100, timeout=600)
Out:
A new study created in memory with name: no-name-ebcef9a6-14c7-49dc-8b5d-d516934a1c81
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:126: ExperimentalWarning:
create_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:126: ExperimentalWarning:
add_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:141: ExperimentalWarning:
create_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/checkouts/stable/tutorial/20_recipes/008_specify_params.py:141: ExperimentalWarning:
add_trial is experimental (supported from v2.0.0). The interface can change in the future.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 2 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.5921681795227542, 'bagging_freq': 7, 'min_child_samples': 13}. Best is trial 1 with value: 0.95.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 3 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.9694304841802756, 'bagging_freq': 6, 'min_child_samples': 99}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 4 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.9482649706012053, 'bagging_freq': 6, 'min_child_samples': 5}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 5 pruned. Trial was pruned at iteration 0.
Trial 6 pruned. Trial was pruned at iteration 19.
Trial 7 pruned. Trial was pruned at iteration 0.
Trial 8 pruned. Trial was pruned at iteration 0.
Trial 9 pruned. Trial was pruned at iteration 0.
Trial 10 pruned. Trial was pruned at iteration 0.
Trial 11 pruned. Trial was pruned at iteration 0.
Trial 12 pruned. Trial was pruned at iteration 0.
Trial 13 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.9903914325997821, 'bagging_freq': 6, 'min_child_samples': 36}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 14 pruned. Trial was pruned at iteration 0.
Trial 15 pruned. Trial was pruned at iteration 3.
Trial 16 pruned. Trial was pruned at iteration 0.
Trial 17 pruned. Trial was pruned at iteration 0.
Trial 18 pruned. Trial was pruned at iteration 0.
Trial 19 pruned. Trial was pruned at iteration 0.
Trial 20 pruned. Trial was pruned at iteration 0.
Trial 21 pruned. Trial was pruned at iteration 0.
Trial 22 pruned. Trial was pruned at iteration 0.
Trial 23 pruned. Trial was pruned at iteration 0.
Trial 24 pruned. Trial was pruned at iteration 0.
Trial 25 pruned. Trial was pruned at iteration 0.
Trial 26 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8050779265956571, 'bagging_freq': 7, 'min_child_samples': 13}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 27 pruned. Trial was pruned at iteration 0.
Trial 28 pruned. Trial was pruned at iteration 0.
Trial 29 pruned. Trial was pruned at iteration 0.
Trial 30 pruned. Trial was pruned at iteration 0.
Trial 31 pruned. Trial was pruned at iteration 0.
Trial 32 pruned. Trial was pruned at iteration 70.
Trial 33 pruned. Trial was pruned at iteration 0.
Trial 34 pruned. Trial was pruned at iteration 0.
Trial 35 pruned. Trial was pruned at iteration 0.
Trial 36 pruned. Trial was pruned at iteration 0.
Trial 37 pruned. Trial was pruned at iteration 1.
Trial 38 pruned. Trial was pruned at iteration 0.
Trial 39 pruned. Trial was pruned at iteration 0.
Trial 40 pruned. Trial was pruned at iteration 0.
Trial 41 pruned. Trial was pruned at iteration 41.
Trial 42 pruned. Trial was pruned at iteration 0.
Trial 43 pruned. Trial was pruned at iteration 0.
Trial 44 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7759592345407211, 'bagging_freq': 7, 'min_child_samples': 25}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 45 pruned. Trial was pruned at iteration 1.
Trial 46 pruned. Trial was pruned at iteration 0.
Trial 47 pruned. Trial was pruned at iteration 0.
Trial 48 pruned. Trial was pruned at iteration 1.
Trial 49 pruned. Trial was pruned at iteration 0.
Trial 50 pruned. Trial was pruned at iteration 0.
Trial 51 pruned. Trial was pruned at iteration 0.
Trial 52 pruned. Trial was pruned at iteration 0.
Trial 53 pruned. Trial was pruned at iteration 0.
Trial 54 pruned. Trial was pruned at iteration 0.
Trial 55 pruned. Trial was pruned at iteration 0.
Trial 56 pruned. Trial was pruned at iteration 0.
Trial 57 pruned. Trial was pruned at iteration 1.
Trial 58 pruned. Trial was pruned at iteration 0.
Trial 59 pruned. Trial was pruned at iteration 0.
Trial 60 pruned. Trial was pruned at iteration 0.
Trial 61 pruned. Trial was pruned at iteration 0.
Trial 62 pruned. Trial was pruned at iteration 0.
Trial 63 pruned. Trial was pruned at iteration 0.
Trial 64 pruned. Trial was pruned at iteration 0.
Trial 65 pruned. Trial was pruned at iteration 0.
Trial 66 pruned. Trial was pruned at iteration 3.
Trial 67 pruned. Trial was pruned at iteration 0.
Trial 68 pruned. Trial was pruned at iteration 0.
Trial 69 pruned. Trial was pruned at iteration 0.
Trial 70 pruned. Trial was pruned at iteration 0.
Trial 71 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9970935811879088, 'bagging_freq': 2, 'min_child_samples': 12}. Best is trial 3 with value: 0.986013986013986.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 72 pruned. Trial was pruned at iteration 1.
Trial 73 pruned. Trial was pruned at iteration 0.
Trial 74 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.9979877428860009, 'bagging_freq': 1, 'min_child_samples': 65}. Best is trial 74 with value: 0.993006993006993.
/home/docs/checkouts/readthedocs.org/user_builds/optuna-zh-cn/envs/stable/lib/python3.7/site-packages/lightgbm/engine.py:239: UserWarning:
'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.
Trial 75 pruned. Trial was pruned at iteration 0.
Trial 76 pruned. Trial was pruned at iteration 0.
Trial 77 pruned. Trial was pruned at iteration 0.
Trial 78 pruned. Trial was pruned at iteration 0.
Trial 79 pruned. Trial was pruned at iteration 0.
Trial 80 pruned. Trial was pruned at iteration 3.
Trial 81 pruned. Trial was pruned at iteration 0.
Trial 82 pruned. Trial was pruned at iteration 0.
Trial 83 pruned. Trial was pruned at iteration 1.
Trial 84 pruned. Trial was pruned at iteration 0.
Trial 85 pruned. Trial was pruned at iteration 1.
Trial 86 pruned. Trial was pruned at iteration 0.
Trial 87 pruned. Trial was pruned at iteration 0.
Trial 88 pruned. Trial was pruned at iteration 0.
Trial 89 pruned. Trial was pruned at iteration 0.
Trial 90 pruned. Trial was pruned at iteration 0.
Trial 91 pruned. Trial was pruned at iteration 0.
Trial 92 pruned. Trial was pruned at iteration 0.
Trial 93 pruned. Trial was pruned at iteration 2.
Trial 94 pruned. Trial was pruned at iteration 2.
Trial 95 pruned. Trial was pruned at iteration 0.
Trial 96 pruned. Trial was pruned at iteration 1.
Trial 97 pruned. Trial was pruned at iteration 0.
Trial 98 pruned. Trial was pruned at iteration 1.
Trial 99 pruned. Trial was pruned at iteration 0.
Trial 100 pruned. Trial was pruned at iteration 0.
Trial 101 pruned. Trial was pruned at iteration 1.
Total running time of the script: ( 0 minutes 6.512 seconds)
备注
Click here to download the full example code
Ask-and-Tell 接口
Optuna 带有 Ask-and-Tell 接口, 它为超参数优化提供了一个更灵活的接口.本教程将展示三种可以用到ask-and-tell接口的情况.
只进行最小改动, 就可将 Optuna 应用到一个现存的优化问题上
考虑一个传统的有监督分类问题; 你想最大化验证准确率. 所以, 你训练了一个 LogisticRegression 作为简单模型.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import optuna
X, y = make_classification(n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y)
C = 0.01
clf = LogisticRegression(C=C)
clf.fit(X_train, y_train)
val_accuracy = clf.score(X_test, y_test) # the objective
然后你试图通过 Optuna 来优化超参数 C 和 solver.当你简单地引入 Optuna 后, 你定义了一个 objective 函数, 它接受 trial 并且调用 trial 的 suggest_* 方法 来采样超参数:
def objective(trial):
X, y = make_classification(n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y)
C = trial.suggest_loguniform("C", 1e-7, 10.0)
solver = trial.suggest_categorical("solver", ("lbfgs", "saga"))
clf = LogisticRegression(C=C, solver=solver)
clf.fit(X_train, y_train)
val_accuracy = clf.score(X_test, y_test)
return val_accuracy
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=10)
这个接口并不灵活. 比如, 如果 objective 需要不同于 trial 的额外参数, 你就需要顶一个类, 就像 How to define objective functions that have own arguments? 里做的那样. 而 ask-and-tell 接口提供了更加灵活的语法来优化超参数. 下面的例子等同于上面的代码块.
study = optuna.create_study(direction="maximize")
n_trials = 10
for _ in range(n_trials):
trial = study.ask() # `trial` is a `Trial` and not a `FrozenTrial`.
C = trial.suggest_loguniform("C", 1e-7, 10.0)
solver = trial.suggest_categorical("solver", ("lbfgs", "saga"))
clf = LogisticRegression(C=C, solver=solver)
clf.fit(X_train, y_train)
val_accuracy = clf.score(X_test, y_test)
study.tell(trial, val_accuracy) # tell the pair of trial and objective value
主要区别是这里用了两个方法 optuna.study.Study.ask() 和 optuna.study.Study.tell(). optuna.study.Study.ask() 创建了一个 trial, 它可以采样超参数, 而 optuna.study.Study.tell() 通过传递 trial 和一个目标函数值完成这个 trial. 你可以在没有 objective 函数的情况下对你的原始代码应用 Optuna 的超参数优化.
如果你想用 pruner 让你的优化变得更快, 你需要显式地将 trial 的状态传入到 optuna.study.Study.tell() 的参数中, 就像下面这样:
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
import optuna
X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)
n_train_iter = 100
# define study with hyperband pruner.
study = optuna.create_study(
direction="maximize",
pruner=optuna.pruners.HyperbandPruner(
min_resource=1, max_resource=n_train_iter, reduction_factor=3
),
)
for _ in range(20):
trial = study.ask()
alpha = trial.suggest_uniform("alpha", 0.0, 1.0)
clf = SGDClassifier(alpha=alpha)
pruned_trial = False
for step in range(n_train_iter):
clf.partial_fit(X_train, y_train, classes=classes)
intermediate_value = clf.score(X_valid, y_valid)
trial.report(intermediate_value, step)
if trial.should_prune():
pruned_trial = True
break
if pruned_trial:
study.tell(trial, state=optuna.trial.TrialState.PRUNED) # tell the pruned state
else:
score = clf.score(X_valid, y_valid)
study.tell(trial, score) # tell objective value
备注
optuna.study.Study.tell() 可以接受一个 trial number 而不是 trial 对象本身. study.tell(trial.number, y) 等价于 study.tell(trial, y).
定义-运行 (Define-and-Run)
ask-and-tell 接口同时支持 define-by-run 和 define-and-run API. 在上面 define-by-run的例子之外, 本部分展示 define-and-run 的 API.
在调用 optuna.study.Study.ask() 方法之前为 define-and-run API 定义超参数分布. 例如,
distributions = {
"C": optuna.distributions.LogUniformDistribution(1e-7, 10.0),
"solver": optuna.distributions.CategoricalDistribution(("lbfgs", "saga")),
}
在每次调用时, 将 distributions 传递给 optuna.study.Study.ask() 方法.返回的 trial 里将包含建议的超参数.
study = optuna.create_study(direction="maximize")
n_trials = 10
for _ in range(n_trials):
trial = study.ask(distributions) # pass the pre-defined distributions.
# two hyperparameters are already sampled from the pre-defined distributions
C = trial.params["C"]
solver = trial.params["solver"]
clf = LogisticRegression(C=C, solver=solver)
clf.fit(X_train, y_train)
val_accuracy = clf.score(X_test, y_test)
study.tell(trial, val_accuracy)
批优化
为了更快地完成优化, ask-and-tell 接口使我们可以优化一个批目标函数. 比如, 并行求值, 向量操作等.
下面这个目标函数接受批超参数 xs 而不是一个单独的超参数, 并对整个向量计算目标函数.
def batched_objective(xs: np.ndarray):
return xs ** 2 + 1
在下面的例子中, 一批中包含的超参数个数是 \(10\), 而 batched_objective 进行了三次求值.因此, trial的个数是 \(30\). 注意, 在 批求值以后, 你需要存储 trial_ids 或者 trial 才能调用 optuna.study.Study.tell() 方法
batch_size = 10
study = optuna.create_study()
for _ in range(3):
# create batch
trial_ids = []
samples = []
for _ in range(batch_size):
trial = study.ask()
trial_ids.append(trial.number)
x = trial.suggest_int("x", -10, 10)
samples.append(x)
# evaluate batched objective
samples = np.array(samples)
objectives = batched_objective(samples)
# finish all trials in the batch
for trial_id, objective in zip(trial_ids, objectives):
study.tell(trial_id, objective)
Total running time of the script: ( 0 minutes 0.169 seconds)
备注
Click here to download the full example code
重新使用最佳值
有时候, 在完成超参数优化以后, 你可能要用最佳的超参数对目标函数重新求值.
比如,
你已经用 Optuna 发现了好的超参数, 并且想用已经发现的最佳超参数来运行一个类似的 objective 函数以进一步分析结果, 或者
为节省时间, 你已经用 Optuna 来优化了一个部分数据集. 在超参数调整以后, 你想在整个数据集上用你找到的最佳超参数来训练模型.
best_trial 提供了一个接口, 用当前的最佳超参数值对目标函数进行重新求值.
像上面的第一个例子一样, 本教程展示了利用当前最佳值来重新运行一个不同的 objective 函数的例子.
进一步研究最佳模型
考虑一个如下的采用了 Optuna 的经典有监督分类问题
from sklearn import metrics
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import optuna
def objective(trial):
X, y = make_classification(n_features=10, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
C = trial.suggest_loguniform("C", 1e-7, 10.0)
clf = LogisticRegression(C=C)
clf.fit(X_train, y_train)
return clf.score(X_test, y_test)
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=10)
print(study.best_trial.value) # Show the best value.
Out:
0.92
假设在超参数优化之后, 你想计算出同一个数据集上的其他的测度, 比如召回率, 精确度和 f1-score. 你可以定义另一个目标函数, 令其和 objective 高度相似, 以用最佳超参数来重现模型.
def detailed_objective(trial):
# Use same code objective to reproduce the best model
X, y = make_classification(n_features=10, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
C = trial.suggest_loguniform("C", 1e-7, 10.0)
clf = LogisticRegression(C=C)
clf.fit(X_train, y_train)
# calculate more evaluation metrics
pred = clf.predict(X_test)
acc = metrics.accuracy_score(pred, y_test)
recall = metrics.recall_score(pred, y_test)
precision = metrics.precision_score(pred, y_test)
f1 = metrics.f1_score(pred, y_test)
return acc, f1, recall, precision
将 study.best_trial 传递给 detailed_objective 做参数.
detailed_objective(study.best_trial) # calculate acc, f1, recall, and precision
Out:
(0.92, 0.9285714285714286, 0.9285714285714286, 0.9285714285714286)
best_trial 和普通的 trials 的不同之处
这里使用了 best_trial, 它返回一个 best_trial, 属于 FrozenTrial 类型. FrozenTrial 与活跃的 trial 不同, 而且在某些情况下它的行为和 Trial 也不一样. 比如, 剪枝没法使用, 因为 should_prune 总是返回 False.
Total running time of the script: ( 0 minutes 0.060 seconds)
API Reference
optuna
在 optuna 被用在其他模块中时，optuna 模块主要作为 Optuna 的基础功能的别名来使用。目前有两个模块有别名：(1) optuna.study，包含和 Study 生命周期有关的函数，(2) optuna.exceptions，是当trial 被剪枝时抛出的TrialPruned异常。
创建新 |
|
加载指定名字的 |
|
删除 |
|
获取指定存储内所有 study的历史记录。 |
|
被剪枝的 trial 异常。 |
optuna.cli
cli 模块使用 cliff framework 框架实现了Optuna的命令行功能。
optuna
[--version]
[-v | -q]
[--log-file LOG_FILE]
[--debug]
[--storage STORAGE]
- --version
显示应用的版本号并退出
- -v, --verbose
打印冗余内容。可重复。
- -q, --quiet
禁止打印除了警告和错误之外的输出。
- --log-file <LOG_FILE>
确定用于记录日志的文件名。该选项默认情况下是禁用的。
- --debug
显示异常处理和堆栈跟踪信息。
- --storage <STORAGE>
数据库 URL. (比如
sqlite:///example.db)
create-study
创建新的 study
optuna create-study
[--study-name STUDY_NAME]
[--direction {minimize,maximize}]
[--skip-if-exists]
- --study-name <STUDY_NAME>
设定具有可读性的 study 名，以方便和其他的 study 区分。
- --direction <DIRECTION>
设定新 study 的优化方向。如果方向是最小化的话就设成 ‘minimize’, 如果是最大化就设置成 ‘maximize’.
- --skip-if-exists
如果开启该选项， 当存在一个同名的 study 时，optuna 在优化开始时将跳过创建 study 的过程，并且不报错。
该命令由 optuna 插件提供。
dashboard
启动 web dashboard (beta).
This feature is deprecated since version 2.7.0. Please use optuna-dashboard instead.
optuna dashboard
[--study STUDY]
[--study-name STUDY_NAME]
[--out OUT]
[--allow-websocket-origin BOKEH_ALLOW_WEBSOCKET_ORIGINS]
- --study <STUDY>
该参数已弃用。作为替代，请使用 –study-name.
- --study-name <STUDY_NAME>
The name of the study to show on the dashboard.
- --out <OUT>, -o <OUT>
HTML 文件的输出路径。如果该参数没有设定的话，Optuna 将会启动一个 HTTP server 并host 一个dashboard
- --allow-websocket-origin <BOKEH_ALLOW_WEBSOCKET_ORIGINS>
允许从制定的 host 发起的的 websocket 请求。该选项用法和 bokeh 的
–allow-websocket-origin一样。更多细节见 https://docs.bokeh.org/en/latest/docs/reference/command/subcommands/serve.html.设定具有可读性的 study 名，以方便和其他的 study 区分。
该命令由 optuna 插件提供。
delete-study
删除特定的 study。
optuna delete-study [--study-name STUDY_NAME]
- --study-name <STUDY_NAME>
待删除的 study 名。
该命令由 optuna 插件提供。
storage upgrade
升级数据库架构。
optuna storage upgrade
该命令由 optuna 插件提供。
studies
显示 study 列表。
optuna studies
[-f {csv,json,table,value,yaml}]
[-c COLUMN]
[--quote {all,minimal,none,nonnumeric}]
[--noindent]
[--max-width <integer>]
[--fit-width]
[--print-empty]
[--sort-column SORT_COLUMN]
[--sort-ascending | --sort-descending]
- -f <FORMATTER>, --format <FORMATTER>
输出格式，默认情况下是表格。
- -c COLUMN, --column COLUMN
用于设定要展示的（表格）列。可设置多个 column 参数以同时显示多个column.
- --quote <QUOTE_MODE>
when to include quotes, defaults to nonnumeric
- --noindent
设定是否在 json输出格式中禁用缩进。
- --max-width <integer>
最大展示宽度。如果设置成小于 1 的数值，就代表禁用最大宽度限制。你也可以通过设置
CLIFF_MAX_TERM_WIDTH `` 环境变量来限制最大宽度，但是 ``--max-width参数的优先级比环境变量高。
- --fit-width
将输出表格设置成与显示屏同宽。如果
-max-width> 1 的话，该选项会自动启用。设定是否在 json 中禁用缩进。也可以通过设置CLIFF_FIT_WIDTH环境变量的值为 1 来启用该选项。
- --print-empty
在无数据的情况下也输出空表格
- --sort-column SORT_COLUMN
按照指定列来对表格的行进行排序。越靠前指定的列在排序中的优先级越高。没有被指定的列不参与排序。
- --sort-ascending
sort the column(s) in ascending order
- --sort-descending
sort the column(s) in descending order
该命令由 optuna 插件提供。
study optimize
针对特定 study 开始优化过程。自 2.0.0 版本后弃用。
optuna study optimize
[--n-trials N_TRIALS]
[--timeout TIMEOUT]
[--n-jobs N_JOBS]
[--study STUDY]
[--study-name STUDY_NAME]
file
method
- --n-trials <N_TRIALS>
优化过程将运行的总 trial 数目。如果不设定该数目的话，优化过程会一直持续下去。
- --timeout <TIMEOUT>
到达指定时间（按秒计）以后终止 study. 如果不设定该数目的话，优化过程也会一直持续下去。
- --n-jobs <N_JOBS>
并行运行的任务个数。如果该参数设置为 -1, 则任务的实际数目将等于 CPU 的核心数。
- --study <STUDY>
该参数已弃用。作为替代，请使用 –study-name.
- --study-name <STUDY_NAME>
被优化的 study 名。
- file
用于指定定义了目标函数的 Python 脚本文件名。
- method
目标函数的方法名。
该命令由 optuna 插件提供。
study set-user-attr
针对特定 study 设定用户属性 (user attribute).
optuna study set-user-attr
[--study STUDY]
[--study-name STUDY_NAME]
--key KEY
--value VALUE
- --study <STUDY>
该参数已弃用。作为替代，请使用 –study-name.
- --study-name <STUDY_NAME>
用于设置用户属性的 study 名。
- --key <KEY>, -k <KEY>
设定用户属性中的键 (key).
- --value <VALUE>, -v <VALUE>
设定用户属性中的值 (value).
该命令由 optuna 插件提供。
optuna.distributions
distributions 模块定义了代表各种概率分布的类, 主要用于优化 trial 时进行初始超参数建议.这些分布类继承自库内部的 BaseDistribution, 并且以特定的参数被初始化, 比如对 UniformDistribution 来说是 low 和 high 端点.
Optuna用户不应该直接使用分布类, 而应该使用 Trial 提供的实用函数, 如 suggest_int().
线性域的均匀分布. |
|
Log 均匀分布. |
|
线性离散均匀分布. |
|
整数上的均匀分布. |
|
Log 操作后的整数均匀分布. |
|
分类分布. |
|
将分布序列化成json格式. |
|
将 JSON 格式的分布反序列化. |
|
检查两分布的兼容性. |
optuna.exceptions
exceptions 模块定义了Optuna特有的异常，它派生自一个基本的 OptunaError 类。对于库用户来说，特别重要的是 TrialPruned 异常。此异常在 optuna.trial.Trial.should_prune() 对一个应被剪枝的trial返回 True 时被抛出。
Optuna 异常基类。 |
|
被剪枝 trial 异常。 |
|
命令行界面异常。 |
|
存储异常。 |
|
重复 study 名导致的异常。 |
optuna.importance
importance 模块提供了根据给定 study 中已完成的trials评估超参数重要性的功能。实用函数 get_param_importances() 将一个 Study 和可选的评价器作为它的两个输入。 评价器必须从 BaseImportanceEvaluator 派生，当没有传入时，默认初始化为 FanovaImportanceEvaluator。实现自定义评估器的用户应该参考 FanovaImportanceEvaluator 或 MeanDecreaseImpurityImportanceEvaluator 作为指导，请格外注意 Evaluator的 evaluate() 函数的返回值的格式。
在给定的 study 中根据已完成的 trial 来评估参数重要性。 |
|
fANOVA 重要性求解器。 |
|
Mean Decrease Impurity (MDI) 参数重要性求解器。 |
optuna.integration
integration 模块包含用于将Optuna与外部机器学习框架集成的类。
对于Optuna支持的大多数ML框架来说，相应的Optuna集成类的作用只是实现一个回调对象和函数，符合框架特定的回调API，在模型训练的每一个中间步骤中都会被调用。这些回调在不同ML框架中实现的功能包括：
使用
optuna.trial.Trial.should_prune()向Optuna trial报告模型中间分数，根据
optuna.trial.Trial.should_prune()的结果，通过抛出optuna.TrialPruned()，对当前模型进行剪枝，并且将当前的 trial number 等中间Optuna数据汇报给框架，功能类似
MLflowCallback。
针对scikit-learn，我们提供了一个集成的 OptunaSearchCV 估计器，它结合了scikit-learn BaseEstimator功能和对类级 Study 对象的访问。
AllenNLP
为了让 Jsonnet 配置文件在 Optuna 中可用的 AllenNLP 扩展。 |
|
保存 study 中最佳 trial 的参数更新到 JSON 配置文件。 |
|
用于清除无望 trial 的 AllenNLP 回调函数。 |
BoTorch
采用 BoTorch 的采样器。BoTorch 是一个建立在 PyTorch 上的贝叶斯优化库。 |
|
Quasi MC-based batch Expected Improvement (qEI). |
|
Quasi MC-based batch Expected Hypervolume Improvement (qEHVI). |
|
Quasi MC-based extended ParEGO (qParEGO) for constrained multi-objective optimization. |
Catalyst
用于清除无望 trial 的 Catalyst 回调。 |
Chainer
用于清除无望 trial 的 Chainer 扩展。 |
|
一个将 Optuna 并入 ChainerMN 的 |
fast.ai
用于清除 FastAI 中无望 trial 的回调函数。 |
|
用于清除 FastAI 中无望 trial 的回调函数。 |
|
:py:class:`optuna.integration.fastaiv2.FastAIV2PruningCallback`的别名 |
Keras
用于清除无望 trial 的 Keras 回调函数。 |
LightGBM
用于清除无望 trial 的 LightGBM 回调函数。 |
|
用于超参数调参的 LightGBM training API 的 wrapper. |
|
用于 LightGBM 的超参数调参器。 |
|
带有交叉验证的、用于 LightGBM 的超参数调参器。 |
MLflow
用 MLflow 来追踪 Optuna trial 的回调函数。 |
MXNet
用于清除无望 trial 的 MXNet 回调函数。 |
pycma
使用 cma 库作为后端的采样器。 |
|
用于向后兼容的 PyCmaSampler wrapper 类。 |
PyTorch
用于清除无望 trial 的 PyTorch Ignite handler。 |
|
用于清除无望 trial 的 PyTorch Lighting 回调函数。 |
|
A wrapper of |
scikit-learn
带有交叉验证的超参数搜索。 |
scikit-optimize
以 Scikit-Optimize 为后端的 sampler。 |
skorch
用于清除无望 trial 的 Skorch 回调函数。 |
TensorFlow
使用 TensorBoard 追踪 Optuna trial 的回调函数。 |
|
用于清除无望 trial 的 TensorFlow SessionRunHook 钩子函数。 |
|
用于清除无望 trial 的 tf.keras 回调函数。 |
XGBoost
用于清除无望 trial 的 XGBoost 回调函数。 |
optuna.logging
logging 模块使用Python logging 包来实现日志记录。库用户可使用 set_verbosity() 设置verbosity级别为 optuna.logging.CRITICAL (又名 optuna.logging.FATAL)、 optuna.logging.ERROR、 optuna.logging.WARNING (又名 optuna.logging.WARN)、 optuna.logging.INFO 或 optuna.logging.DEBUG 。
返回目前 Optuna 的 root logger 的 logging 层级。 |
|
设置 Optuna 的 root logger 的 logging 层级。 |
|
禁用 Optuna 的 root logger 的默认句柄。 |
|
启用 Optuna 的 root logger 的默认句柄。 |
|
禁用库 log 输出的传递。 |
|
启用库 log 输出的传递。 |
optuna.multi_objective
该模块已被弃用，其功能已经被迁移到 optuna.samplers, optuna.study, optuna.trial 和 optuna.visualization 中。
multi-objective sampler 类
multi-objective sampler 基类。 |
|
使用 NSGA-II 算法的 Multi-objective sampler。 |
|
Multi-objective sampler using random sampling. |
|
采用 MOTPE 算法的多目标sampler。 |
optuna.multi_objective.study
A study corresponds to a multi-objective optimization task, i.e., a set of trials. |
|
Create a new |
|
Load the existing |
optuna.multi_objective.trial
A trial is a process of evaluating an objective function. |
|
Status and results of a |
optuna.multi_objective.visualization
备注
optuna.multi_objective.visualization module uses plotly to create figures,
but JupyterLab cannot render them by default. Please follow this installation guide to
show figures in JupyterLab.
Plot the pareto front of a study. |
optuna.pruners
pruners 模块定义了一个 BasePruner 类，其特点是有一个抽象的 prune() 方法。该方法对一个给定的trial及其相关的 study，返回一个代表该trial是否应该被剪枝的布尔值。该值由存储的目标函数的中间值确定，这些中间值由前述 optuna.trial.Trial.report() 传入。本模块中其余的类代表子类，继承自 BasePruner，它们实现了不同的剪枝策略。
Pruner 基类 |
|
使用中值停止规则 的 pruner. |
|
不剪枝 trial 的 pruner. |
|
保留到指定百分位 trial 的 pruner. |
|
使用异步连续减半算法的 pruner. |
|
使用 hyperband 的 pruner. |
|
用于检测 trial 的无关度量的 pruner. |
optuna.samplers
samplers 模块定义了一个参数采样的基类，如 BaseSampler 中广泛描述的那样。本模块中其余的类代表从 BaseSampler 派生出来的子类，它们实现了不同的采样策略。
Samplers 基类 |
|
使用网格搜索的 sampler. |
|
使用随机 sampling 的sampler. |
|
使用 TPE (Tree-structured Parzen Estimator) 算法的 sampler. |
|
A sampler using cmaes as the backend. |
|
部分参数固定的 sampler. |
|
使用 NSGA-II 算法的多目标 sampler. |
|
使用 MOTPE 算法的多目标 sampler. |
|
用于计算 |
|
返回 |
optuna.storages
The storages module defines a BaseStorage class which abstracts a backend database and provides library-internal interfaces to the read/write histories of the studies and trials. Library users who wish to use storage solutions other than the default in-memory storage should use one of the child classes of BaseStorage documented below.
RDB 后端的存储类。 |
|
Redis 后端的存储类。 |
optuna.structs
该模块已被弃用，其以前的功能移至 optuna.trial 和 optuna.study 。
- class optuna.structs.TrialState(value)[源代码]
Trial的状态- PRUNED
已经被
TrialPruned剪枝的Trial.
1.4.0 版后已移除: 该类已废弃。请改用
TrialState.
- class optuna.structs.StudyDirection(value)[源代码]
Study的方向。- NOT_SET
方向未设置。
1.4.0 版后已移除: 该类已废弃，请改用
StudyDirection.
- class optuna.structs.FrozenTrial(number, state, value, datetime_start, datetime_complete, params, distributions, user_attrs, system_attrs, intermediate_values, trial_id, *, values=None)[源代码]
警告
在 v1.4.0 中被弃用。该特性将在未来被移除。目前我们计划在 v3.0.0 中移除它，但也可能会改变。参见 https://github.com/optuna/optuna/releases/tag/v1.4.0.
该类已经迁移至
trial, 请改用FrozenTrial.- 参数
number (int) –
state (optuna.trial._state.TrialState) –
datetime_start (Optional[datetime.datetime]) –
datetime_complete (Optional[datetime.datetime]) –
distributions (Dict[str, optuna.distributions.BaseDistribution]) –
trial_id (int) –
- 返回类型
None
- property duration: Optional[datetime.timedelta]
返回完成一个 trial 所消耗的时间。
- 返回
消耗的时间。
- property last_step: Optional[int]
返回 trial 中 intermediate_values 的最大步数。
- 返回
intermediate_values 的最大步数。
- report(value, step)[源代码]
report 函数接口。
由于
FrozenTrial未被剪枝，该函数将不做任何处理。参见
Please refer to
should_prune().- 参数
value (float) – 目标函数返回的值。
step (int) – Trial 的步骤（比如，神经网络训练中的 epoch 数）。注意，pruners 假定
step从零开始计算。比如MedianPruner将仅检查step是否小于n_warmup_steps作为热身机制。
- 返回类型
None
- class optuna.structs.StudySummary(study_name, direction, best_trial, user_attrs, system_attrs, n_trials, datetime_start, study_id, *, directions=None)[源代码]
警告
在 v1.4.0 中被弃用。该特性将在未来被移除。目前我们计划在 v3.0.0 中移除它，但也可能会改变。参见 https://github.com/optuna/optuna/releases/tag/v1.4.0.
该类已经迁移到
study, 请改用StudySummary.- 参数
study_name (str) –
direction (Optional[optuna._study_direction.StudyDirection]) –
best_trial (Optional[optuna.trial._frozen.FrozenTrial]) –
n_trials (int) –
datetime_start (Optional[datetime.datetime]) –
study_id (int) –
directions (Optional[Sequence[optuna._study_direction.StudyDirection]]) –
optuna.study
study 模块实现了 Study 对象和相关函数。Study 类有一个公共构造函数，但不建议直接使用这个构造函数。相反，要创建或加载一个 Study 时, 库用户应该分别使用 create_study 或 load_study() 。
一个 study 对应于一个优化任务，即一组 trials. |
|
创建新的 |
|
加载一个所给名字的已经存在的 |
|
删除一个 |
|
返回指定存储中 所有 study 的历史记录。 |
|
|
|
一个 |
optuna.trial
一个 Trial 实例代表了一个评估目标函数的过程。该实例被传递给目标函数来提供获取参数建议、管理trial状态、设置/获取用户定义的trial属性等接口，Optuna用户可以通过接口定义自定义目标函数。基本上，Optuna用户只在自己的自定义目标函数中使用它。
一个 trial 是一个评估目标函数的过程。 |
|
一个始终为每个参数提供固定 suggestion 值的 trial 类。 |
|
一个 |
|
|
|
创建一个新的 |
optuna.visualization
The visualization module provides utility functions for plotting the optimization process using plotly and matplotlib. Plotting functions generally take a Study object and optional parameters are passed as a list to the params argument.
备注
在 optuna.visualization 模块中，以下函数使用 plotly 来绘制图像，但 JupyterLab 默认情况下不能渲染它们。请按照这个 installation guide 在 JupyterLab 中显示图像。
将 study 中的参数关系绘制成等高线图。 |
|
绘制study的目标值EDF（经验分布函数）。 |
|
绘制一个 study 中所有 trial 的中间值。 |
|
绘制一个 study 中所有 trial 的优化历史记录。 |
|
Plot the high-dimensional parameter relationships in a study. |
|
绘制超参数重要性。 |
|
绘制 study 的帕累托前沿面。 |
|
将study中的参数关系绘制成切片图。 |
|
返回是否可以使用 plotly 进行可视化。 |
备注
optuna.visualization.matplotlib 模块的后端是 Matplotlib。
optuna.visualization.matplotlib
备注
The following functions use Matplotlib as a backend.
使用 Matplotlib 将 study 中参数关系画成等高线图。 |
|
使用 Matplotlib 绘制 study 中目标函数值的 EDF (经验分布函数)。 |
|
使用 Matplotlib 绘制 study 中所有 trial 的中间值。 |
|
使用 Matplotlib 绘制 study 中所有 trial 的优化历史。 |
|
Plot the high-dimensional parameter relationships in a study with Matplotlib. |
|
使用 Matplotlib 绘制超参数重要性。 |
|
使用 Matplotlib 将 study 中的参数关系绘制成切片图。 |
|
返回 Matplotlib 可视化是否可用。 |
常见问题
某某库可以和 Optuna 配合使用吗？（某某是你常用的机器学习库）
Optuna 和绝大多数机器学习库兼容，并且很容易同他们配合使用。参见 examples.
如何定义带有额外参数的目标函数？
有两种方法可以实现这类函数。
首先，如下例所示，可调用的 objective 类具有这个功能：
import optuna
class Objective(object):
def __init__(self, min_x, max_x):
# Hold this implementation specific arguments as the fields of the class.
self.min_x = min_x
self.max_x = max_x
def __call__(self, trial):
# Calculate an objective value by using the extra arguments.
x = trial.suggest_float("x", self.min_x, self.max_x)
return (x - 2) ** 2
# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)
其次，你可以用 lambda 或者 functools.partial 来创建带有额外参数的函数（闭包）。 下面是一个使用了 lambda 的例子：
import optuna
# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
x = trial.suggest_float("x", min_x, max_x)
return (x - 2) ** 2
# Extra arguments.
min_x = -100
max_x = 100
# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)
Please also refer to sklearn_addtitional_args.py example, which reuses the dataset instead of loading it in each trial execution.
没有远程 RDB 的情况下可以使用 Optuna 吗？
可以。
在最简单的情况下，Optuna 使用内存 (in-memory) 存储：
study = optuna.create_study()
study.optimize(objective)
如果想保存和恢复 study 的话，你可以轻松地将 SQLite 用作本地存储。
study = optuna.create_study(study_name="foo_study", storage="sqlite:///example.db")
study.optimize(objective) # The state of `study` will be persisted to the local SQLite file.
更多细节请参考 用 RDB 后端保存/恢复 Study .
如何保存和恢复 study？
有两种方法可以将 study 持久化。具体采用哪种取决于你是使用内存存储 (in-memory) 还是远程数据库存储 (RDB). 通过 pickle 或者 joblib, 采用了内存存储的 study 可以和普通的 Python 对象一样被存储和加载。比如用 joblib 的话：
study = optuna.create_study()
joblib.dump(study, "study.pkl")
恢复 study:
study = joblib.load("study.pkl")
print("Best trial until now:")
print(" Value: ", study.best_trial.value)
print(" Params: ")
for key, value in study.best_trial.params.items():
print(f" {key}: {value}")
如果你用的是 RDB, 具体细节请参考 用 RDB 后端保存/恢复 Study.
如何禁用 Optuna 的日志信息？
默认情况下，Optuna 打印处于 optuna.logging.INFO 层级的日志信息。通过设置 optuna.logging.set_verbosity(), 你可以改变这个层级。
比如，下面的代码可以终止打印每一个trial的结果：
optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study()
study.optimize(objective)
# Logs like '[I 2020-07-21 13:41:45,627] Trial 0 finished with value:...' are disabled.
更多的细节请参考 optuna.logging.
如何在目标函数中保存训练好的机器学习模型？
Optuna 会保存超参数和对应的目标函数值，但是它不会存储诸如机器学习模型或者网络权重这样的中间数据。要保存模型或者权重的话，请利用你正在使用的机器学习库提供的对应功能。
在保存模型的时候，我们推荐将 optuna.trial.Trial.number 一同存储。这样易于之后确认对应的 trial.比如，你可以用以下方式在目标函数中保存训练好的 SVM 模型：
def objective(trial):
svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
clf = sklearn.svm.SVC(C=svc_c)
clf.fit(X_train, y_train)
# Save a trained model to a file.
with open("{}.pickle".format(trial.number), "wb") as fout:
pickle.dump(clf, fout)
return 1.0 - accuracy_score(y_valid, clf.predict(X_valid))
study = optuna.create_study()
study.optimize(objective, n_trials=100)
# Load the best model.
with open("{}.pickle".format(study.best_trial.number), "rb") as fin:
best_clf = pickle.load(fin)
print(accuracy_score(y_valid, best_clf.predict(X_valid)))
如何获得可复现的优化结果？
要让 Optuna 生成的参数可复现的话，你可以通过设置 RandomSampler 或者 TPESampler 中的参数 seed 来指定一个固定的随机数种子：
sampler = TPESampler(seed=10) # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)
study.optimize(objective)
但是这么做的需要注意以下两点。
首先，如果一个 study 的优化过程本身是分布式的或者并行的，那么这个过程中存在着固有的不确定性。因此，在这种情况下我们很难复现出同样的结果。如果你想复现结果的话，我们建议用顺序执行的方式来优化你的 study.
其次，如果你的目标函数的行为本身就是不确定的（也就是说，即使送入同样的参数，其返回值也不是唯一的），那么你就无法复现这个优化过程。要解决这个问题的话，请设置一个选项（比如随机数种子）来让你的优化目标的行为变成确定性的，前提是你用的机器学习库支持这一功能。
Trial 是如何处理抛出异常的？
那些抛出异常却没有对应的捕获机制的 trial 会被视作失败的 trial, 也就是处于 FAIL 状态的 trial.
在默认情况下，除了目标函数中抛出的 TrialPruned, 其他所有异常都会被传回给调用函数 optimize().换句话说，当此类异常被抛出时，对应的 study 就会被终止。但有时候我们希望能用剩余的 trial 将该 study 继续下去。要这么做的话，你得通过 optimize() 函数中的 catch 参数来指定要捕获的异常类型。这样，此类异常就会在 study 内部被捕获，而不会继续向外层传递。
你可以在日志信息里找到失败的 trial.
[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')
你也可以通过查看 trial 的状态来找到它们：
study.trials_dataframe()
number |
state |
value |
… |
params |
system_attrs |
0 |
TrialState.FAIL |
… |
0 |
Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’) |
|
1 |
TrialState.COMPLETE |
1269 |
… |
1 |
参见
The catch argument in optimize().
Trial 返回的 NaN 是如何处理的？
返回 NaN 的 trial 被视为失败的 trial, 但是它们并不会导致 study 被终止。
这些返回 NaN 的 trial 在日志里长这样：
[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.
动态地改变搜索空间会导致怎样的结果？
Since parameters search spaces are specified in each call to the suggestion API, e.g.
suggest_float() and suggest_int(),
it is possible to, in a single study, alter the range by sampling parameters from different search
spaces in different trials.
The behavior when altered is defined by each sampler individually.
备注
关于 TPE sampler 的 讨论：https://github.com/optuna/optuna/issues/822
如何在两块 GPU 上同时跑两个 trial?
如果你的优化目标支持 GPU (CUDA) 加速，你又想指定优化所用的 GPU 的话，设置 CUDA_VISIBLE_DEVICES 环境变量可能是实现这一目标最轻松的方式了：
# On a terminal.
#
# Specify to use the first GPU, and run an optimization.
$ export CUDA_VISIBLE_DEVICES=0
$ optuna study optimize foo.py objective --study-name foo --storage sqlite:///example.db
# On another terminal.
#
# Specify to use the second GPU, and run another optimization.
$ export CUDA_VISIBLE_DEVICES=1
$ optuna study optimize bar.py objective --study-name bar --storage sqlite:///example.db
更多细节见 CUDA C Programming Guide.
如何对目标函数进行测试？
在对目标函数的测试中，我们总倾向于使用固定的，而不是随机采样的参数。这时，你可以选择用 FixedTrial 作为目标函数的输入参数。它会从一个给定的参数字典中送入固定的参数值。比如，针对函数 \(x + y\), 你可以用如下方式送入两个任意的 \(x\) 和 \(y\):
def objective(trial):
x = trial.suggest_float("x", -1.0, 1.0)
y = trial.suggest_int("y", -5, 5)
return x + y
objective(FixedTrial({"x": 1.0, "y": -1})) # 0.0
objective(FixedTrial({"x": -1.0, "y": -4})) # -5.0
如果使用 FixedTrial 的话，你也可以用如下方式写单元测试：
# A test function of pytest
def test_objective():
assert 1.0 == objective(FixedTrial({"x": 1.0, "y": 0}))
assert -1.0 == objective(FixedTrial({"x": 0.0, "y": -1}))
assert 0.0 == objective(FixedTrial({"x": -1.0, "y": 1}))
在优化时, 我该如何避免耗尽内存 (running out of memory, OOM)?
如果内存使用量随着你运行更多的 trial 而增长，请尝试定期运行垃圾回收器。可在调用 optimize() 时指定 gc_after_trial 为 True 或在回调函数中调用 gc.collect().
def objective(trial):
x = trial.suggest_float("x", -1.0, 1.0)
y = trial.suggest_int("y", -5, 5)
return x + y
study = optuna.create_study()
study.optimize(objective, n_trials=10, gc_after_trial=True)
# `gc_after_trial=True` is more or less identical to the following.
study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()])
运行垃圾回收器是有性能损失的，取决于你的目标函数运行的速度，这种损失可能是不能忽略的。因此，gc_after_trial 在默认情况下是 False. 注意，上面这个例子类似于在目标函数内运行垃圾回收器，不同之处在于 gc.collect() 哪怕在出现包括 TrialPruned 这样的错误时也会被调用。
备注
ChainerMNStudy 目前并不提供 gc_after_trial 和用于 optimize() 的回调接口。在使用该类时，你只能在目标函数内部调用垃圾回收器。