optuna.samplers
The samplers
module defines a base class for parameter sampling as described extensively in BaseSampler
. The remaining classes in this module represent child classes, deriving from BaseSampler
, which implement different sampling strategies.
See also
Efficient Optimization Algorithms tutorial explains the overview of the sampler classes.
See also
User-Defined Sampler tutorial could be helpful if you want to implement your own sampler classes.
See also
If you are unsure about which sampler to use, please consider using AutoSampler, which automatically selects a sampler during optimization. For more detail, see the article on AutoSampler.
RandomSampler |
GridSampler |
TPESampler |
CmaEsSampler |
NSGAIISampler |
QMCSampler |
GPSampler |
BoTorchSampler |
BruteForceSampler |
|
---|---|---|---|---|---|---|---|---|---|
Float parameters |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) (\(\color{red}\times\) for infinite domain) |
Integer parameters |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
Categorical parameters |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
Pruning |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{red}\times\) (\(\blacktriangle\) for single-objective) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
Multivariate optimization |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
Conditional search space |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
Multi-objective optimization |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{red}\times\) |
\(\color{green}\checkmark\) (\(\blacktriangle\) for single-objective) |
\(\color{green}\checkmark\) |
\(\color{red}\times\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
Batch optimization |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
Distributed optimization |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
\(\blacktriangle\) |
\(\color{green}\checkmark\) |
\(\color{green}\checkmark\) |
Constrained optimization |
\(\color{red}\times\) |
\(\color{red}\times\) |
\(\color{green}\checkmark\) |
\(\color{red}\times\) |
\(\color{green}\checkmark\) |
\(\color{red}\times\) |
\(\color{red}\times\) |
\(\color{green}\checkmark\) |
\(\color{red}\times\) |
Time complexity (per trial) (*) |
\(O(d)\) |
\(O(dn)\) |
\(O(dn \log n)\) |
\(O(d^3)\) |
\(O(mp^2)\) (***) |
\(O(dn)\) |
\(O(n^3)\) |
\(O(n^3)\) |
\(O(d)\) |
Recommended budgets (#trials) (**) |
as many as one likes |
number of combinations |
100 – 1000 |
1000 – 10000 |
100 – 10000 |
as many as one likes |
– 500 |
10 – 100 |
number of combinations |
Note
\(\color{green}\checkmark\): Supports this feature. \(\blacktriangle\): Works, but inefficiently. \(\color{red}\times\): Causes an error, or has no interface.
(*): We assumes that \(d\) is the dimension of the search space, \(n\) is the number of finished trials, \(m\) is the number of objectives, and \(p\) is the population size (algorithm specific parameter). This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including \(O(d)\) to call the sampling methods and \(O(n)\) to collect the completed trials. This means that, for example, the actual time complexity of
RandomSampler
is \(O(d+n+d) = O(d+n)\). From another perspective, with the exception ofNSGAIISampler
, all time complexity is written for single-objective optimization.(**): (1) The budget depends on the number of parameters and the number of objectives. (2) This budget includes
n_startup_trials
if a sampler hasn_startup_trials
as one of its arguments.(***): This time complexity assumes that the number of population size \(p\) and the number of parallelization are regular. This means that the number of parallelization should not exceed the number of population size \(p\).
Note
Samplers initialize their random number generators by specifying seed
argument at initialization.
However, samplers reseed them when n_jobs!=1
of optuna.study.Study.optimize()
to avoid sampling duplicated parameters by using the same generator.
Thus we can hardly reproduce the optimization results with n_jobs!=1
.
For the same reason, make sure that use either seed=None
or different seed
values among processes with distributed optimization explained in Easy Parallelization tutorial.
Note
For float, integer, or categorical parameters, see Pythonic Search Space tutorial.
For pruning, see Efficient Optimization Algorithms tutorial.
For multivariate optimization, see BaseSampler
. The multivariate optimization is implemented as sample_relative()
in Optuna. Please check the concrete documents of samplers for more details.
For conditional search space, see Pythonic Search Space tutorial and TPESampler
. The group
option of TPESampler
allows TPESampler
to handle the conditional search space.
For multi-objective optimization, see Multi-objective Optimization with Optuna tutorial.
For batch optimization, see Batch Optimization tutorial. Note that the constant_liar
option of TPESampler
allows TPESampler
to handle the batch optimization.
For distributed optimization, see Easy Parallelization tutorial. Note that the constant_liar
option of TPESampler
allows TPESampler
to handle the distributed optimization.
For constrained optimization, see an example.
Base class for samplers. |
|
Sampler using grid search. |
|
Sampler using random sampling. |
|
Sampler using TPE (Tree-structured Parzen Estimator) algorithm. |
|
A sampler using cmaes as the backend. |
|
Sampler using Gaussian process-based Bayesian optimization. |
|
Sampler with partially fixed parameters. |
|
Multi-objective sampler using the NSGA-II algorithm. |
|
Multi-objective sampler using the NSGA-III algorithm. |
|
A Quasi Monte Carlo Sampler that generates low-discrepancy sequences. |
|
Sampler using brute force. |
Note
The following optuna.samplers.nsgaii
module defines crossover operations used by NSGAIISampler
.