# 高效的优化算法

## 采样算法

Optuna 提供了下列采样算法：

## 切换采样器

```import optuna
```

```study = optuna.create_study()
print(f"Sampler is {study.sampler.__class__.__name__}")
```

Out:

```Sampler is TPESampler
```

```study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")

study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())
print(f"Sampler is {study.sampler.__class__.__name__}")
```

Out:

```Sampler is RandomSampler
Sampler is CmaEsSampler
```

## 剪枝算法

`Pruners` 自动在训练的早期（也就是自动化的 early-stopping）终止无望的 trial.

Optuna 提供以下剪枝算法：

## 激活 Pruner

```import logging
import sys

import sklearn.datasets
import sklearn.linear_model
import sklearn.model_selection

def objective(trial):
classes = list(set(iris.target))
train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(
iris.data, iris.target, test_size=0.25, random_state=0
)

alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True)
clf = sklearn.linear_model.SGDClassifier(alpha=alpha)

for step in range(100):
clf.partial_fit(train_x, train_y, classes=classes)

# Report intermediate objective value.
intermediate_value = 1.0 - clf.score(valid_x, valid_y)
trial.report(intermediate_value, step)

# Handle pruning based on the intermediate value.
if trial.should_prune():
raise optuna.TrialPruned()

return 1.0 - clf.score(valid_x, valid_y)
```

```# Add stream handler of stdout to show the messages
study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
study.optimize(objective, n_trials=20)
```

Out:

```A new study created in memory with name: no-name-2550f085-0161-4804-aabe-dafd3d2a69fc
Trial 0 finished with value: 0.23684210526315785 and parameters: {'alpha': 0.0056457619503721985}. Best is trial 0 with value: 0.23684210526315785.
Trial 1 finished with value: 0.10526315789473684 and parameters: {'alpha': 0.0004415029135399882}. Best is trial 1 with value: 0.10526315789473684.
Trial 2 finished with value: 0.2894736842105263 and parameters: {'alpha': 0.003557613371651606}. Best is trial 1 with value: 0.10526315789473684.
Trial 3 finished with value: 0.10526315789473684 and parameters: {'alpha': 1.9394951442030858e-05}. Best is trial 1 with value: 0.10526315789473684.
Trial 4 finished with value: 0.39473684210526316 and parameters: {'alpha': 2.6138481035133756e-05}. Best is trial 1 with value: 0.10526315789473684.
Trial 5 pruned.
Trial 6 finished with value: 0.26315789473684215 and parameters: {'alpha': 0.006413605309772462}. Best is trial 1 with value: 0.10526315789473684.
Trial 7 pruned.
Trial 8 pruned.
Trial 9 pruned.
Trial 10 finished with value: 0.3421052631578947 and parameters: {'alpha': 0.0003506682152386342}. Best is trial 1 with value: 0.10526315789473684.
Trial 11 pruned.
Trial 12 pruned.
Trial 13 pruned.
Trial 14 pruned.
Trial 15 pruned.
Trial 16 pruned.
Trial 17 pruned.
Trial 18 pruned.
Trial 19 pruned.
```

## 应该使用哪个 pruner 呢？

Parallel Compute Resource

Categorical/Conditional Hyperparameters

Recommended Algorithms

Limited

No

TPE. GP-EI if search space is low-dimensional and continuous.

Yes

TPE. GP-EI if search space is low-dimensional and continuous

Sufficient

No

CMA-ES, Random Search

Yes

Random Search or Genetic Algorithm

## 用于剪枝的集成模块

For example, `XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration. (See also example for the entire script.)

```pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])
```

Total running time of the script: ( 0 minutes 1.443 seconds)

Gallery generated by Sphinx-Gallery