optuna.terminator.EMMREvaluator

class optuna.terminator.EMMREvaluator(deterministic_objective=False, delta=0.1, min_n_trials=2, seed=None)[source]

Evaluates a kind of regrets, called the Expected Minimum Model Regret(EMMR).

EMMR is an upper bound of “expected minimum simple regret” in the optimization process.

Expected minimum simple regret is a quantity that converges to zero only if the optimization process has found the global optima.

For further information about expected minimum simple regret and the algorithm, please refer to the following paper:

Also, there is our blog post explaining this evaluator:

Parameters:
  • deterministic_objective (bool) – A boolean value which indicates whether the objective function is deterministic. Default is False.

  • delta (float) – A float number related to the criterion for termination. Default to 0.1. For further information about this parameter, please see the aforementioned paper.

  • min_n_trials (int) – A minimum number of complete trials to compute the criterion. Default to 2.

  • seed (int | None) – A random seed for EMMREvaluator.

Example

import optuna
from optuna.terminator import EMMREvaluator
from optuna.terminator import MedianErrorEvaluator
from optuna.terminator import Terminator

sampler = optuna.samplers.TPESampler(seed=0)
study = optuna.create_study(sampler=sampler, direction="minimize")
emmr_improvement_evaluator = EMMREvaluator()
median_error_evaluator = MedianErrorEvaluator(emmr_improvement_evaluator)
terminator = Terminator(
    improvement_evaluator=emmr_improvement_evaluator,
    error_evaluator=median_error_evaluator,
)


for i in range(1000):
    trial = study.ask()

    ys = [trial.suggest_float(f"x{i}", -10.0, 10.0) for i in range(5)]
    value = sum(ys[i] ** 2 for i in range(5))

    study.tell(trial, value)

    if terminator.should_terminate(study):
        # Terminated by Optuna Terminator!
        break

Note

Added in v4.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v4.0.0.

Methods

evaluate(trials, study_direction)