PK )Uvy 10_key_features/001_first.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# 1. Lightweight, versatile, and platform agnostic architecture\n\nOptuna is entirely written in Python and has few dependencies.\nThis means that we can quickly move to the real example once you get interested in Optuna.\n\n\n## Quadratic Function Example\n\nUsually, Optuna is used to optimize hyperparameters, but as an example,\nlet's optimize a simple quadratic function: $(x - 2)^2$.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First of all, import :mod:`optuna`.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import optuna"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In optuna, conventionally functions to be optimized are named `objective`.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n return (x - 2) ** 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This function returns the value of $(x - 2)^2$. Our goal is to find the value of ``x``\nthat minimizes the output of the ``objective`` function. This is the \"optimization.\"\nDuring the optimization, Optuna repeatedly calls and evaluates the objective function with\ndifferent values of ``x``.\n\nA :class:`~optuna.trial.Trial` object corresponds to a single execution of the objective\nfunction and is internally instantiated upon each invocation of the function.\n\nThe `suggest` APIs (for example, :func:`~optuna.trial.Trial.suggest_float`) are called\ninside the objective function to obtain parameters for a trial.\n:func:`~optuna.trial.Trial.suggest_float` selects parameters uniformly within the range\nprovided. In our example, from $-10$ to $10$.\n\nTo start the optimization, we create a study object and pass the objective function to method\n:func:`~optuna.study.Study.optimize` as follows.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study()\nstudy.optimize(objective, n_trials=100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can get the best parameter as follows.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"best_params = study.best_params\nfound_x = best_params[\"x\"]\nprint(\"Found x: {}, (x - 2)^2: {}\".format(found_x, (found_x - 2) ** 2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the ``x`` value found by Optuna is close to the optimal value of ``2``.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
Note
When used to search for hyperparameters in machine learning,\n usually the objective function would return the loss or accuracy\n of the model.
\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Study Object\n\nLet us clarify the terminology in Optuna as follows:\n\n* **Trial**: A single call of the objective function\n* **Study**: An optimization session, which is a set of trials\n* **Parameter**: A variable whose value is to be optimized, such as ``x`` in the above example\n\nIn Optuna, we use the study object to manage optimization.\nMethod :func:`~optuna.study.create_study` returns a study object.\nA study object has useful properties for analyzing the optimization outcome.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get the dictionary of parameter name and parameter values:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.best_params"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get the best observed value of the objective function:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.best_value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get the best trial:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.best_trial"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get all trials:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.trials"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get the number of trials:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"len(study.trials)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By executing :func:`~optuna.study.Study.optimize` again, we can continue the optimization.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.optimize(objective, n_trials=100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get the updated number of trials:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"len(study.trials)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As the objective function is so easy that the last 100 trials don't improve the result.\nHowever, we can check the result again:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"best_params = study.best_params\nfound_x = best_params[\"x\"]\nprint(\"Found x: {}, (x - 2)^2: {}\".format(found_x, (found_x - 2) ** 2))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK )UȭY! Y! ' 10_key_features/005_visualization.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# 5. Quick Visualization for Hyperparameter Optimization Analysis\n\nOptuna provides various visualization features in :mod:`optuna.visualization` to analyze optimization results visually.\n\nThis tutorial walks you through this module by visualizing the history of lightgbm model for breast cancer dataset.\n\nFor visualizing multi-objective optimization (i.e., the usage of :func:`optuna.visualization.plot_pareto_front`),\nplease refer to the tutorial of `multi_objective`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import lightgbm as lgb\nimport numpy as np\nimport sklearn.datasets\nimport sklearn.metrics\nfrom sklearn.model_selection import train_test_split\n\nimport optuna\n\n# You can use Matplotlib instead of Plotly for visualization by simply replacing `optuna.visualization` with\n# `optuna.visualization.matplotlib` in the following examples.\nfrom optuna.visualization import plot_contour\nfrom optuna.visualization import plot_edf\nfrom optuna.visualization import plot_intermediate_values\nfrom optuna.visualization import plot_optimization_history\nfrom optuna.visualization import plot_parallel_coordinate\nfrom optuna.visualization import plot_param_importances\nfrom optuna.visualization import plot_slice\n\nSEED = 42\n\nnp.random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define the objective function.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)\n dtrain = lgb.Dataset(train_x, label=train_y)\n dvalid = lgb.Dataset(valid_x, label=valid_y)\n\n param = {\n \"objective\": \"binary\",\n \"metric\": \"auc\",\n \"verbosity\": -1,\n \"boosting_type\": \"gbdt\",\n \"bagging_fraction\": trial.suggest_float(\"bagging_fraction\", 0.4, 1.0),\n \"bagging_freq\": trial.suggest_int(\"bagging_freq\", 1, 7),\n \"min_child_samples\": trial.suggest_int(\"min_child_samples\", 5, 100),\n }\n\n # Add a callback for pruning.\n pruning_callback = optuna.integration.LightGBMPruningCallback(trial, \"auc\")\n gbm = lgb.train(\n param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]\n )\n\n preds = gbm.predict(valid_x)\n pred_labels = np.rint(preds)\n accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)\n return accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(\n direction=\"maximize\",\n sampler=optuna.samplers.TPESampler(seed=SEED),\n pruner=optuna.pruners.MedianPruner(n_warmup_steps=10),\n)\nstudy.optimize(objective, n_trials=100, timeout=600)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plot functions\nVisualize the optimization history. See :func:`~optuna.visualization.plot_optimization_history` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_optimization_history(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize the learning curves of the trials. See :func:`~optuna.visualization.plot_intermediate_values` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_intermediate_values(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize high-dimensional parameter relationships. See :func:`~optuna.visualization.plot_parallel_coordinate` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_parallel_coordinate(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Select parameters to visualize.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_parallel_coordinate(study, params=[\"bagging_freq\", \"bagging_fraction\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize hyperparameter relationships. See :func:`~optuna.visualization.plot_contour` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_contour(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Select parameters to visualize.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_contour(study, params=[\"bagging_freq\", \"bagging_fraction\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize individual hyperparameters as slice plot. See :func:`~optuna.visualization.plot_slice` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_slice(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Select parameters to visualize.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_slice(study, params=[\"bagging_freq\", \"bagging_fraction\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize parameter importances. See :func:`~optuna.visualization.plot_param_importances` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_param_importances(study)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Learn which hyperparameters are affecting the trial duration with hyperparameter importance.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"optuna.visualization.plot_param_importances(\n study, target=lambda t: t.duration.total_seconds(), target_name=\"duration\"\n)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize empirical distribution function. See :func:`~optuna.visualization.plot_edf` for the details.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plot_edf(study)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK )UR#k ( 10_key_features/002_configurations.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# 2. Pythonic Search Space\n\nFor hyperparameter sampling, Optuna provides the following features:\n\n- :func:`optuna.trial.Trial.suggest_categorical` for categorical parameters\n- :func:`optuna.trial.Trial.suggest_int` for integer parameters\n- :func:`optuna.trial.Trial.suggest_float` for floating point parameters\n\nWith optional arguments of ``step`` and ``log``, we can discretize or take the logarithm of\ninteger and floating point parameters.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import optuna\n\n\ndef objective(trial):\n # Categorical parameter\n optimizer = trial.suggest_categorical(\"optimizer\", [\"MomentumSGD\", \"Adam\"])\n\n # Integer parameter\n num_layers = trial.suggest_int(\"num_layers\", 1, 3)\n\n # Integer parameter (log)\n num_channels = trial.suggest_int(\"num_channels\", 32, 512, log=True)\n\n # Integer parameter (discretized)\n num_units = trial.suggest_int(\"num_units\", 10, 100, step=5)\n\n # Floating point parameter\n dropout_rate = trial.suggest_float(\"dropout_rate\", 0.0, 1.0)\n\n # Floating point parameter (log)\n learning_rate = trial.suggest_float(\"learning_rate\", 1e-5, 1e-2, log=True)\n\n # Floating point parameter (discretized)\n drop_path_rate = trial.suggest_float(\"drop_path_rate\", 0.0, 1.0, step=0.1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Defining Parameter Spaces\n\nIn Optuna, we define search spaces using familiar Python syntax including conditionals and loops.\n\nAlso, you can use branches or loops depending on the parameter values.\n\nFor more various use, see [examples](https://github.com/optuna/optuna-examples/).\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Branches:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import sklearn.ensemble\nimport sklearn.svm\n\n\ndef objective(trial):\n classifier_name = trial.suggest_categorical(\"classifier\", [\"SVC\", \"RandomForest\"])\n if classifier_name == \"SVC\":\n svc_c = trial.suggest_float(\"svc_c\", 1e-10, 1e10, log=True)\n classifier_obj = sklearn.svm.SVC(C=svc_c)\n else:\n rf_max_depth = trial.suggest_int(\"rf_max_depth\", 2, 32, log=True)\n classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Loops:\n\n```python\nimport torch\nimport torch.nn as nn\n\n\ndef create_model(trial, in_size):\n n_layers = trial.suggest_int(\"n_layers\", 1, 3)\n\n layers = []\n for i in range(n_layers):\n n_units = trial.suggest_int(\"n_units_l{}\".format(i), 4, 128, log=True)\n layers.append(nn.Linear(in_size, n_units))\n layers.append(nn.ReLU())\n in_size = n_units\n layers.append(nn.Linear(in_size, 10))\n\n return nn.Sequential(*layers)\n```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Note on the Number of Parameters\n\nThe difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.\n\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK )UK1m( ( ; 10_key_features/003_efficient_optimization_algorithms.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# 3. Efficient Optimization Algorithms\n\nOptuna enables efficient hyperparameter optimization by\nadopting state-of-the-art algorithms for sampling hyperparameters and\npruning efficiently unpromising trials.\n\n## Sampling Algorithms\n\nSamplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values,\nleading to an optimal search space which giving off parameters leading to better objective values.\nMore detailed explanation of how samplers suggest parameters is in :class:`~optuna.samplers.BaseSampler`.\n\nOptuna provides the following sampling algorithms:\n\n- Grid Search implemented in :class:`~optuna.samplers.GridSampler`\n\n- Random Search implemented in :class:`~optuna.samplers.RandomSampler`\n\n- Tree-structured Parzen Estimator algorithm implemented in :class:`~optuna.samplers.TPESampler`\n\n- CMA-ES based algorithm implemented in :class:`~optuna.samplers.CmaEsSampler`\n\n- Algorithm to enable partial fixed parameters implemented in :class:`~optuna.samplers.PartialFixedSampler`\n\n- Nondominated Sorting Genetic Algorithm II implemented in :class:`~optuna.samplers.NSGAIISampler`\n\n- A Quasi Monte Carlo sampling algorithm implemented in :class:`~optuna.samplers.QMCSampler`\n\nThe default sampler is :class:`~optuna.samplers.TPESampler`.\n\n## Switching Samplers\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import optuna"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study()\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler`\nand :class:`~optuna.samplers.CmaEsSampler`,\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(sampler=optuna.samplers.RandomSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\nstudy = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pruning Algorithms\n\n``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).\n\nOptuna provides the following pruning algorithms:\n\n- Median pruning algorithm implemented in :class:`~optuna.pruners.MedianPruner`\n\n- Non-pruning algorithm implemented in :class:`~optuna.pruners.NopPruner`\n\n- Algorithm to operate pruner with tolerance implemented in :class:`~optuna.pruners.PatientPruner`\n\n- Algorithm to prune specified percentile of trials implemented in :class:`~optuna.pruners.PercentilePruner`\n\n- Asynchronous Successive Halving algorithm implemented in :class:`~optuna.pruners.SuccessiveHalvingPruner`\n\n- Hyperband algorithm implemented in :class:`~optuna.pruners.HyperbandPruner`\n\n- Threshold pruning algorithm implemented in :class:`~optuna.pruners.ThresholdPruner`\n\nWe use :class:`~optuna.pruners.MedianPruner` in most examples,\nthough basically it is outperformed by :class:`~optuna.pruners.SuccessiveHalvingPruner` and\n:class:`~optuna.pruners.HyperbandPruner` as in [this benchmark result](https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako).\n\n\n## Activating Pruners\nTo turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training.\n:func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values.\n:func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition.\n\nWe would recommend using integration modules for major machine learning frameworks.\nExclusive list is :mod:`~optuna.integration` and usecases are available in [~optuna/examples](https://github.com/optuna/optuna-examples/).\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import logging\nimport sys\n\nimport sklearn.datasets\nimport sklearn.linear_model\nimport sklearn.model_selection\n\n\ndef objective(trial):\n iris = sklearn.datasets.load_iris()\n classes = list(set(iris.target))\n train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n iris.data, iris.target, test_size=0.25, random_state=0\n )\n\n alpha = trial.suggest_float(\"alpha\", 1e-5, 1e-1, log=True)\n clf = sklearn.linear_model.SGDClassifier(alpha=alpha)\n\n for step in range(100):\n clf.partial_fit(train_x, train_y, classes=classes)\n\n # Report intermediate objective value.\n intermediate_value = 1.0 - clf.score(valid_x, valid_y)\n trial.report(intermediate_value, step)\n\n # Handle pruning based on the intermediate value.\n if trial.should_prune():\n raise optuna.TrialPruned()\n\n return 1.0 - clf.score(valid_x, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Set up the median stopping rule as the pruning condition.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Add stream handler of stdout to show the messages\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy = optuna.create_study(pruner=optuna.pruners.MedianPruner())\nstudy.optimize(objective, n_trials=20)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, several trials were pruned (stopped) before they finished all of the iterations.\nThe format of message is ``\"Trial pruned.\"``.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Which Sampler and Pruner Should be Used?\n\nFrom the benchmark results which are available at [optuna/optuna - wiki \"Benchmarks with Kurobako\"](https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako), at least for not deep learning tasks, we would say that\n\n* For :class:`~optuna.samplers.RandomSampler`, :class:`~optuna.pruners.MedianPruner` is the best.\n* For :class:`~optuna.samplers.TPESampler`, :class:`~optuna.pruners.Hyperband` is the best.\n\nHowever, note that the benchmark is not deep learning.\nFor deep learning tasks,\nconsult the below table.\nThis table is from the [Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020](https://doi.org/10.14923/transinfj.2019JDR0003) paper,\nwhich is written in Japanese.\n\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n| Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms |\n+===========================+=========================================+===============================================================+\n| Limited | No | TPE. GP-EI if search space is low-dimensional and continuous. |\n+ +-----------------------------------------+---------------------------------------------------------------+\n| | Yes | TPE. GP-EI if search space is low-dimensional and continuous |\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n| Sufficient | No | CMA-ES, Random Search |\n+ +-----------------------------------------+---------------------------------------------------------------+\n| | Yes | Random Search or Genetic Algorithm |\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Integration Modules for Pruning\nTo implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.\n\nFor the complete list of Optuna's integration modules, see :mod:`~optuna.integration`.\n\nFor example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration.\n(See also [example](https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py) for the entire script.)\n\n```python\npruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')\nbst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])\n```\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK )UjgN N % 10_key_features/004_distributed.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# 4. Easy Parallelization\n\nIt's straightforward to parallelize :func:`optuna.study.Study.optimize`.\n\nIf you want to manually execute Optuna optimization:\n\n 1. start an RDB server (this example uses MySQL)\n 2. create a study with `--storage` argument\n 3. share the study among multiple nodes and processes\n\nOf course, you can use Kubernetes as in [the kubernetes examples](https://github.com/optuna/optuna-examples/tree/main/kubernetes).\n\nTo just see how parallel optimization works in Optuna, check the below video.\n\n.. raw:: html\n\n \n\n\n## Create a Study\n\nYou can create a study using ``optuna create-study`` command.\nAlternatively, in Python script you can use :func:`optuna.create_study`.\n\n\n```bash\n$ mysql -u root -e \"CREATE DATABASE IF NOT EXISTS example\"\n$ optuna create-study --study-name \"distributed-example\" --storage \"mysql://root@localhost/example\"\n[I 2020-07-21 13:43:39,642] A new study created with name: distributed-example\n```\nThen, write an optimization script. Let's assume that ``foo.py`` contains the following code.\n\n```python\nimport optuna\n\n\ndef objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n return (x - 2) ** 2\n\n\nif __name__ == \"__main__\":\n study = optuna.load_study(\n study_name=\"distributed-example\", storage=\"mysql://root@localhost/example\"\n )\n study.optimize(objective, n_trials=100)\n```\n## Share the Study among Multiple Nodes and Processes\n\nFinally, run the shared study from multiple processes.\nFor example, run ``Process 1`` in a terminal, and do ``Process 2`` in another one.\nThey get parameter suggestions based on shared trials' history.\n\nProcess 1:\n\n```bash\n$ python foo.py\n[I 2020-07-21 13:45:02,973] Trial 0 finished with value: 45.35553104173011 and parameters: {'x': 8.73465151598285}. Best is trial 0 with value: 45.35553104173011.\n[I 2020-07-21 13:45:04,013] Trial 2 finished with value: 4.6002397305938905 and parameters: {'x': 4.144816945707463}. Best is trial 1 with value: 0.028194513284051464.\n...\n```\nProcess 2 (the same command as process 1):\n\n```bash\n$ python foo.py\n[I 2020-07-21 13:45:03,748] Trial 1 finished with value: 0.028194513284051464 and parameters: {'x': 1.8320877810162361}. Best is trial 1 with value: 0.028194513284051464.\n[I 2020-07-21 13:45:05,783] Trial 3 finished with value: 24.45966755098074 and parameters: {'x': 6.945671597566982}. Best is trial 1 with value: 0.028194513284051464.\n...\n```\nNote
``n_trials`` is the number of trials each process will run, not the total number of trials across all processes. For example, the script given above runs 100 trials for each process, 100 trials * 2 processes = 200 trials. :class:`optuna.study.MaxTrialsCallback` can ensure how many times trials will be performed across all processes.
\n\nNote
We do not recommend SQLite for distributed optimizations at scale because it may cause deadlocks and serious performance issues. Please consider to use another database engine like PostgreSQL or MySQL.
\n\nNote
Please avoid putting the SQLite database on NFS when running distributed optimizations. See also: https://www.sqlite.org/faq.html#q5
\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *U`( ( ! 20_recipes/009_ask_and_tell.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# Ask-and-Tell Interface\n\nOptuna has an `Ask-and-Tell` interface, which provides a more flexible interface for hyperparameter optimization.\nThis tutorial explains three use-cases when the ask-and-tell interface is beneficial:\n\n- `Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications`\n- `Define-and-Run`\n- `Batch-Optimization`\n\n\n## Apply Optuna to an existing optimization problem with minimum modifications\n\nLet's consider the traditional supervised classification problem; you aim to maximize the validation accuracy.\nTo do so, you train `LogisticRegression` as a simple model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\nfrom sklearn.datasets import make_classification\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\n\nimport optuna\n\n\nX, y = make_classification(n_features=10)\nX_train, X_test, y_train, y_test = train_test_split(X, y)\n\nC = 0.01\nclf = LogisticRegression(C=C)\nclf.fit(X_train, y_train)\nval_accuracy = clf.score(X_test, y_test) # the objective"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then you try to optimize hyperparameters ``C`` and ``solver`` of the classifier by using optuna.\nWhen you introduce optuna naively, you define an ``objective`` function\nsuch that it takes ``trial`` and calls ``suggest_*`` methods of ``trial`` to sample the hyperparameters:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n X, y = make_classification(n_features=10)\n X_train, X_test, y_train, y_test = train_test_split(X, y)\n\n C = trial.suggest_float(\"C\", 1e-7, 10.0, log=True)\n solver = trial.suggest_categorical(\"solver\", (\"lbfgs\", \"saga\"))\n\n clf = LogisticRegression(C=C, solver=solver)\n clf.fit(X_train, y_train)\n val_accuracy = clf.score(X_test, y_test)\n\n return val_accuracy\n\n\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective, n_trials=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This interface is not flexible enough.\nFor example, if ``objective`` requires additional arguments other than ``trial``,\nyou need to define a class as in\n[How to define objective functions that have own arguments?](../../faq.html#how-to-define-objective-functions-that-have-own-arguments).\nThe ask-and-tell interface provides a more flexible syntax to optimize hyperparameters.\nThe following example is equivalent to the previous code block.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(direction=\"maximize\")\n\nn_trials = 10\nfor _ in range(n_trials):\n trial = study.ask() # `trial` is a `Trial` and not a `FrozenTrial`.\n\n C = trial.suggest_float(\"C\", 1e-7, 10.0, log=True)\n solver = trial.suggest_categorical(\"solver\", (\"lbfgs\", \"saga\"))\n\n clf = LogisticRegression(C=C, solver=solver)\n clf.fit(X_train, y_train)\n val_accuracy = clf.score(X_test, y_test)\n\n study.tell(trial, val_accuracy) # tell the pair of trial and objective value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The main difference is to use two methods: :func:`optuna.study.Study.ask`\nand :func:`optuna.study.Study.tell`.\n:func:`optuna.study.Study.ask` creates a trial that can sample hyperparameters, and\n:func:`optuna.study.Study.tell` finishes the trial by passing ``trial`` and an objective value.\nYou can apply Optuna's hyperparameter optimization to your original code\nwithout an ``objective`` function.\n\nIf you want to make your optimization faster with a pruner, you need to explicitly pass the state of trial\nto the argument of :func:`optuna.study.Study.tell` method as follows:\n\n```python\nimport numpy as np\nfrom sklearn.datasets import load_iris\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.model_selection import train_test_split\n\nimport optuna\n\n\nX, y = load_iris(return_X_y=True)\nX_train, X_valid, y_train, y_valid = train_test_split(X, y)\nclasses = np.unique(y)\nn_train_iter = 100\n\n# define study with hyperband pruner.\nstudy = optuna.create_study(\n direction=\"maximize\",\n pruner=optuna.pruners.HyperbandPruner(\n min_resource=1, max_resource=n_train_iter, reduction_factor=3\n ),\n)\n\nfor _ in range(20):\n trial = study.ask()\n\n alpha = trial.suggest_float(\"alpha\", 0.0, 1.0)\n\n clf = SGDClassifier(alpha=alpha)\n pruned_trial = False\n\n for step in range(n_train_iter):\n clf.partial_fit(X_train, y_train, classes=classes)\n\n intermediate_value = clf.score(X_valid, y_valid)\n trial.report(intermediate_value, step)\n\n if trial.should_prune():\n pruned_trial = True\n break\n\n if pruned_trial:\n study.tell(trial, state=optuna.trial.TrialState.PRUNED) # tell the pruned state\n else:\n score = clf.score(X_valid, y_valid)\n study.tell(trial, score) # tell objective value\n```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note
:func:`optuna.study.Study.tell` method can take a trial number rather than the trial object.\n ``study.tell(trial.number, y)`` is equivalent to ``study.tell(trial, y)``.
\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n## Define-and-Run\nThe ask-and-tell interface supports both `define-by-run` and `define-and-run` APIs.\nThis section shows the example of the `define-and-run` API\nin addition to the define-by-run example above.\n\nDefine distributions for the hyperparameters before calling the\n:func:`optuna.study.Study.ask` method for define-and-run API.\nFor example,\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"distributions = {\n \"C\": optuna.distributions.FloatDistribution(1e-7, 10.0, log=True),\n \"solver\": optuna.distributions.CategoricalDistribution((\"lbfgs\", \"saga\")),\n}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pass ``distributions`` to :func:`optuna.study.Study.ask` method at each call.\nThe retuned ``trial`` contains the suggested hyperparameters.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(direction=\"maximize\")\nn_trials = 10\nfor _ in range(n_trials):\n trial = study.ask(distributions) # pass the pre-defined distributions.\n\n # two hyperparameters are already sampled from the pre-defined distributions\n C = trial.params[\"C\"]\n solver = trial.params[\"solver\"]\n\n clf = LogisticRegression(C=C, solver=solver)\n clf.fit(X_train, y_train)\n val_accuracy = clf.score(X_test, y_test)\n\n study.tell(trial, val_accuracy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n## Batch Optimization\nThe ask-and-tell interface enables us to optimize a batched objective for faster optimization.\nFor example, parallelizable evaluation, operation over vectors, etc.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following objective takes batched hyperparameters ``xs`` and ``ys`` instead of a single\npair of hyperparameters ``x`` and ``y`` and calculates the objective over the full vectors.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def batched_objective(xs: np.ndarray, ys: np.ndarray):\n return xs**2 + ys"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following example, the number of pairs of hyperparameters in a batch is $10$,\nand ``batched_objective`` is evaluated three times.\nThus, the number of trials is $30$.\nNote that you need to store either ``trial_numbers`` or ``trial`` to call\n:func:`optuna.study.Study.tell` method after the batched evaluations.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"batch_size = 10\nstudy = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())\n\nfor _ in range(3):\n\n # create batch\n trial_numbers = []\n x_batch = []\n y_batch = []\n for _ in range(batch_size):\n trial = study.ask()\n trial_numbers.append(trial.number)\n x_batch.append(trial.suggest_float(\"x\", -10, 10))\n y_batch.append(trial.suggest_float(\"y\", -10, 10))\n\n # evaluate batched objective\n x_batch = np.array(x_batch)\n y_batch = np.array(y_batch)\n objectives = batched_objective(x_batch, y_batch)\n\n # finish all trials in the batch\n for trial_number, objective in zip(trial_numbers, objectives):\n study.tell(trial_number, objective)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *UҔ ) 20_recipes/005_user_defined_sampler.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# User-Defined Sampler\n\nThanks to user-defined samplers, you can:\n\n- experiment your own sampling algorithms,\n- implement task-specific algorithms to refine the optimization performance, or\n- wrap other optimization libraries to integrate them into Optuna pipelines (e.g., :class:`~optuna.integration.SkoptSampler`).\n\nThis section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler.\n\n\n## Overview of Sampler\n\nA sampler has the responsibility to determine the parameter values to be evaluated in a trial.\nWhen a `suggest` API (e.g., :func:`~optuna.trial.Trial.suggest_float`) is called inside an objective function, the corresponding distribution object (e.g., :class:`~optuna.distributions.FloatDistribution`) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the `suggest` API and evaluated in the objective function.\n\nTo create a new sampler, you need to define a class that inherits :class:`~optuna.samplers.BaseSampler`.\nThe base class has three abstract methods;\n:meth:`~optuna.samplers.BaseSampler.infer_relative_search_space`,\n:meth:`~optuna.samplers.BaseSampler.sample_relative`, and\n:meth:`~optuna.samplers.BaseSampler.sample_independent`.\n\nAs the method names imply, Optuna supports two types of sampling: one is **relative sampling** that can consider the correlation of the parameters in a trial, and the other is **independent sampling** that samples each parameter independently.\n\nAt the beginning of a trial, :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space` is called to provide the relative search space for the trial. Then, :meth:`~optuna.samplers.BaseSampler.sample_relative` is invoked to sample relative parameters from the search space. During the execution of the objective function, :meth:`~optuna.samplers.BaseSampler.sample_independent` is used to sample parameters that don't belong to the relative search space.\n\nNote
Please refer to the document of :class:`~optuna.samplers.BaseSampler` for further details.
\n\n\n## An Example: Implementing SimulatedAnnealingSampler\n\nFor example, the following code defines a sampler based on\n[Simulated Annealing (SA)](https://en.wikipedia.org/wiki/Simulated_annealing):\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\nimport optuna\n\n\nclass SimulatedAnnealingSampler(optuna.samplers.BaseSampler):\n def __init__(self, temperature=100):\n self._rng = np.random.RandomState()\n self._temperature = temperature # Current temperature.\n self._current_trial = None # Current state.\n\n def sample_relative(self, study, trial, search_space):\n if search_space == {}:\n return {}\n\n # Simulated Annealing algorithm.\n # 1. Calculate transition probability.\n prev_trial = study.trials[-2]\n if self._current_trial is None or prev_trial.value <= self._current_trial.value:\n probability = 1.0\n else:\n probability = np.exp(\n (self._current_trial.value - prev_trial.value) / self._temperature\n )\n self._temperature *= 0.9 # Decrease temperature.\n\n # 2. Transit the current state if the previous result is accepted.\n if self._rng.uniform(0, 1) < probability:\n self._current_trial = prev_trial\n\n # 3. Sample parameters from the neighborhood of the current point.\n # The sampled parameters will be used during the next execution of\n # the objective function passed to the study.\n params = {}\n for param_name, param_distribution in search_space.items():\n if (\n not isinstance(param_distribution, optuna.distributions.FloatDistribution)\n or (param_distribution.step is not None and param_distribution.step != 1)\n or param_distribution.log\n ):\n msg = (\n \"Only suggest_float() with `step` `None` or 1.0 and\"\n \" `log` `False` is supported\"\n )\n raise NotImplementedError(msg)\n\n current_value = self._current_trial.params[param_name]\n width = (param_distribution.high - param_distribution.low) * 0.1\n neighbor_low = max(current_value - width, param_distribution.low)\n neighbor_high = min(current_value + width, param_distribution.high)\n params[param_name] = self._rng.uniform(neighbor_low, neighbor_high)\n\n return params\n\n # The rest are unrelated to SA algorithm: boilerplate\n def infer_relative_search_space(self, study, trial):\n return optuna.samplers.intersection_search_space(study)\n\n def sample_independent(self, study, trial, param_name, param_distribution):\n independent_sampler = optuna.samplers.RandomSampler()\n return independent_sampler.sample_independent(study, trial, param_name, param_distribution)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note
In favor of code simplicity, the above implementation doesn't support some features (e.g., maximization).\n If you're interested in how to support those features, please see\n [examples/samplers/simulated_annealing.py](https://github.com/optuna/optuna-examples/blob/main/samplers/simulated_annealing_sampler.py).
\n\n\nYou can use ``SimulatedAnnealingSampler`` in the same way as built-in samplers as follows:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n y = trial.suggest_float(\"y\", -5, 5)\n return x**2 + y\n\n\nsampler = SimulatedAnnealingSampler()\nstudy = optuna.create_study(sampler=sampler)\nstudy.optimize(objective, n_trials=100)\n\nbest_trial = study.best_trial\nprint(\"Best value: \", best_trial.value)\nprint(\"Parameters that achieve the best value: \", best_trial.params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this optimization, the values of ``x`` and ``y`` parameters are sampled by using\n``SimulatedAnnealingSampler.sample_relative`` method.\n\nNote
Strictly speaking, in the first trial,\n ``SimulatedAnnealingSampler.sample_independent`` method is used to sample parameter values.\n Because :func:`~optuna.samplers.intersection_search_space` used in\n ``SimulatedAnnealingSampler.infer_relative_search_space`` cannot infer the search space\n if there are no complete trials.
\n\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *U 20_recipes/004_cli.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# Command-Line Interface\n\n.. csv-table::\n :header: Command, Description\n :widths: 20, 40\n :escape: \\\n\n ask, Create a new trial and suggest parameters.\n best-trial, Show the best trial.\n best-trials, Show a list of trials located at the Pareto front.\n create-study, Create a new study.\n delete-study, Delete a specified study.\n storage upgrade, Upgrade the schema of a storage.\n studies, Show a list of studies.\n study optimize, Start optimization of a study.\n study set-user-attr, Set a user attribute to a study.\n tell, Finish a trial\\, which was created by the ask command.\n trials, Show a list of trials.\n\nOptuna provides command-line interface as shown in the above table.\n\nLet us assume you are not in IPython shell and writing Python script files instead.\nIt is totally fine to write scripts like the following:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import optuna\n\n\ndef objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n return (x - 2) ** 2\n\n\nif __name__ == \"__main__\":\n study = optuna.create_study()\n study.optimize(objective, n_trials=100)\n print(\"Best value: {} (params: {})\\n\".format(study.best_value, study.best_params))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, we can reduce boilerplate codes by using our ``optuna`` command.\nLet us assume that ``foo.py`` contains only the following code.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n return (x - 2) ** 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even so, we can invoke the optimization as follows.\n(Don't care about ``--storage sqlite:///example.db`` for now, which is described in `rdb`.)\n\n```bash\n$ cat foo.py\ndef objective(trial):\n x = trial.suggest_float('x', -10, 10)\n return (x - 2) ** 2\n\n$ STUDY_NAME=`optuna create-study --storage sqlite:///example.db`\n$ optuna study optimize foo.py objective --n-trials=100 --storage sqlite:///example.db --study-name $STUDY_NAME\n[I 2018-05-09 10:40:25,196] Finished a trial resulted in value: 54.353767789264026. Current best value is 54.353767789264026 with parameters: {'x': -5.372500782588228}.\n[I 2018-05-09 10:40:25,197] Finished a trial resulted in value: 15.784266965526376. Current best value is 15.784266965526376 with parameters: {'x': 5.972941852774387}.\n...\n[I 2018-05-09 10:40:26,204] Finished a trial resulted in value: 14.704254135013741. Current best value is 2.280758099793617e-06 with parameters: {'x': 1.9984897821018828}.\n```\nPlease note that ``foo.py`` only contains the definition of the objective function.\nBy giving the script file name and the method name of objective function to\n``optuna study optimize`` command, we can invoke the optimization.\n\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK )U}(7K 20_recipes/001_rdb.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# Saving/Resuming Study with RDB Backend\n\nAn RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies.\nIn addition, we can run multi-node optimization tasks with this feature, which is described in `distributed`.\n\nIn this section, let's try simple examples running on a local environment with SQLite DB.\n\nNote
You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB's URL.\n Please refer to [SQLAlchemy's document](https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls) for how to set up the URL.
\n\n\n## New Study\n\nWe can create a persistent study by calling :func:`~optuna.study.create_study` function as follows.\nAn SQLite file ``example.db`` is automatically initialized with a new study record.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import logging\nimport sys\n\nimport optuna\n\n# Add stream handler of stdout to show the messages\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy_name = \"example-study\" # Unique identifier of the study.\nstorage_name = \"sqlite:///{}.db\".format(study_name)\nstudy = optuna.create_study(study_name=study_name, storage=storage_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To run a study, call :func:`~optuna.study.Study.optimize` method passing an objective function.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n x = trial.suggest_float(\"x\", -10, 10)\n return (x - 2) ** 2\n\n\nstudy.optimize(objective, n_trials=3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Resume Study\n\nTo resume a study, instantiate a :class:`~optuna.study.Study` object\npassing the study name ``example-study`` and the DB URL ``sqlite:///example-study.db``.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)\nstudy.optimize(objective, n_trials=3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experimental History\n\nWe can access histories of studies and trials via the :class:`~optuna.study.Study` class.\nFor example, we can get all trials of ``example-study`` as:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)\ndf = study.trials_dataframe(attrs=(\"number\", \"value\", \"params\", \"state\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The method :func:`~optuna.study.Study.trials_dataframe` returns a pandas dataframe like:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"print(df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A :class:`~optuna.study.Study` object also provides properties\nsuch as :attr:`~optuna.study.Study.trials`, :attr:`~optuna.study.Study.best_value`,\n:attr:`~optuna.study.Study.best_params` (see also `first`).\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"print(\"Best params: \", study.best_params)\nprint(\"Best value: \", study.best_value)\nprint(\"Best Trial: \", study.best_trial)\nprint(\"Trials: \", study.trials)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *Ul $ 20_recipes/002_multi_objective.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# Multi-objective Optimization with Optuna\n\nThis tutorial showcases Optuna's multi-objective optimization feature by\noptimizing the validation accuracy of Fashion MNIST dataset and the FLOPS of the model implemented in PyTorch.\n\nWe use [thop](https://github.com/Lyken17/pytorch-OpCounter) to measure FLOPS.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import thop\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\n\nimport optuna\n\n\nDEVICE = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\nDIR = \"..\"\nBATCHSIZE = 128\nN_TRAIN_EXAMPLES = BATCHSIZE * 30\nN_VALID_EXAMPLES = BATCHSIZE * 10\n\n\ndef define_model(trial):\n n_layers = trial.suggest_int(\"n_layers\", 1, 3)\n layers = []\n\n in_features = 28 * 28\n for i in range(n_layers):\n out_features = trial.suggest_int(\"n_units_l{}\".format(i), 4, 128)\n layers.append(nn.Linear(in_features, out_features))\n layers.append(nn.ReLU())\n p = trial.suggest_float(\"dropout_{}\".format(i), 0.2, 0.5)\n layers.append(nn.Dropout(p))\n\n in_features = out_features\n\n layers.append(nn.Linear(in_features, 10))\n layers.append(nn.LogSoftmax(dim=1))\n\n return nn.Sequential(*layers)\n\n\n# Defines training and evaluation.\ndef train_model(model, optimizer, train_loader):\n model.train()\n for batch_idx, (data, target) in enumerate(train_loader):\n data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)\n optimizer.zero_grad()\n F.nll_loss(model(data), target).backward()\n optimizer.step()\n\n\ndef eval_model(model, valid_loader):\n model.eval()\n correct = 0\n with torch.no_grad():\n for batch_idx, (data, target) in enumerate(valid_loader):\n data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)\n pred = model(data).argmax(dim=1, keepdim=True)\n correct += pred.eq(target.view_as(pred)).sum().item()\n\n accuracy = correct / N_VALID_EXAMPLES\n\n flops, _ = thop.profile(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),), verbose=False)\n return flops, accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define multi-objective objective function.\nObjectives are FLOPS and accuracy.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n train_dataset = torchvision.datasets.FashionMNIST(\n DIR, train=True, download=True, transform=torchvision.transforms.ToTensor()\n )\n train_loader = torch.utils.data.DataLoader(\n torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))),\n batch_size=BATCHSIZE,\n shuffle=True,\n )\n\n val_dataset = torchvision.datasets.FashionMNIST(\n DIR, train=False, transform=torchvision.transforms.ToTensor()\n )\n val_loader = torch.utils.data.DataLoader(\n torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))),\n batch_size=BATCHSIZE,\n shuffle=True,\n )\n model = define_model(trial).to(DEVICE)\n\n optimizer = torch.optim.Adam(\n model.parameters(), trial.suggest_float(\"lr\", 1e-5, 1e-1, log=True)\n )\n\n for epoch in range(10):\n train_model(model, optimizer, train_loader)\n flops, accuracy = eval_model(model, val_loader)\n return flops, accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run multi-objective optimization\n\nIf your optimization problem is multi-objective,\nOptuna assumes that you will specify the optimization direction for each objective.\nSpecifically, in this example, we want to minimize the FLOPS (we want a faster model)\nand maximize the accuracy. So we set ``directions`` to ``[\"minimize\", \"maximize\"]``.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(directions=[\"minimize\", \"maximize\"])\nstudy.optimize(objective, n_trials=30, timeout=300)\n\nprint(\"Number of finished trials: \", len(study.trials))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check trials on Pareto front visually.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"optuna.visualization.plot_pareto_front(study, target_names=[\"FLOPS\", \"accuracy\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Fetch the list of trials on the Pareto front with :attr:`~optuna.study.Study.best_trials`.\n\nFor example, the following code shows the number of trials on the Pareto front and picks the trial with the highest accuracy.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"print(f\"Number of trials on the Pareto front: {len(study.best_trials)}\")\n\ntrial_with_highest_accuracy = max(study.best_trials, key=lambda t: t.values[1])\nprint(f\"Trial with highest accuracy: \")\nprint(f\"\\tnumber: {trial_with_highest_accuracy.number}\")\nprint(f\"\\tparams: {trial_with_highest_accuracy.params}\")\nprint(f\"\\tvalues: {trial_with_highest_accuracy.values}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Learn which hyperparameters are affecting the flops most with hyperparameter importance.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"optuna.visualization.plot_param_importances(\n study, target=lambda t: t.values[0], target_name=\"flops\"\n)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *Ux[sF # 20_recipes/008_specify_params.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# Specify Hyperparameters Manually\n\nIt's natural that you have some specific sets of hyperparameters to try first such as initial learning rate\nvalues and the number of leaves.\nAlso, it's possible that you've already tried those sets before having Optuna find better\nsets of hyperparameters.\n\nOptuna provides two APIs to support such cases:\n\n1. Passing those sets of hyperparameters and let Optuna evaluate them - :func:`~optuna.study.Study.enqueue_trial`\n2. Adding the results of those sets as completed ``Trial``\\s - :func:`~optuna.study.Study.add_trial`\n\n\n## First Scenario: Have Optuna evaluate your hyperparameters\n\nIn this scenario, let's assume you have some out-of-box sets of hyperparameters but have not\nevaluated them yet and decided to use Optuna to find better sets of hyperparameters.\n\nOptuna has :func:`optuna.study.Study.enqueue_trial` which lets you pass those sets of\nhyperparameters to Optuna and Optuna will evaluate them.\n\nThis section walks you through how to use this lit API with [LightGBM](https://lightgbm.readthedocs.io/en/latest/).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import lightgbm as lgb\nimport numpy as np\nimport sklearn.datasets\nimport sklearn.metrics\nfrom sklearn.model_selection import train_test_split\n\nimport optuna"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define the objective function.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)\n dtrain = lgb.Dataset(train_x, label=train_y)\n dvalid = lgb.Dataset(valid_x, label=valid_y)\n\n param = {\n \"objective\": \"binary\",\n \"metric\": \"auc\",\n \"verbosity\": -1,\n \"boosting_type\": \"gbdt\",\n \"bagging_fraction\": min(trial.suggest_float(\"bagging_fraction\", 0.4, 1.0 + 1e-12), 1),\n \"bagging_freq\": trial.suggest_int(\"bagging_freq\", 0, 7),\n \"min_child_samples\": trial.suggest_int(\"min_child_samples\", 5, 100),\n }\n\n # Add a callback for pruning.\n pruning_callback = optuna.integration.LightGBMPruningCallback(trial, \"auc\")\n gbm = lgb.train(\n param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]\n )\n\n preds = gbm.predict(valid_x)\n pred_labels = np.rint(preds)\n accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)\n return accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then, construct ``Study`` for hyperparameter optimization.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we get Optuna evaluate some sets with larger ``\"bagging_fraq\"`` value and\nthe default values.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.enqueue_trial(\n {\n \"bagging_fraction\": 1.0,\n \"bagging_freq\": 0,\n \"min_child_samples\": 20,\n }\n)\n\nstudy.enqueue_trial(\n {\n \"bagging_fraction\": 0.75,\n \"bagging_freq\": 5,\n \"min_child_samples\": 20,\n }\n)\n\nimport logging\nimport sys\n\n# Add stream handler of stdout to show the messages to see Optuna works expectedly.\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy.optimize(objective, n_trials=100, timeout=600)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n## Second scenario: Have Optuna utilize already evaluated hyperparameters\n\nIn this scenario, let's assume you have some out-of-box sets of hyperparameters and\nyou have already evaluated them but the results are not desirable so that you are thinking of\nusing Optuna.\n\nOptuna has :func:`optuna.study.Study.add_trial` which lets you register those results\nto Optuna and then Optuna will sample hyperparameters taking them into account.\n\nIn this section, the ``objective`` is the same as the first scenario.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\nstudy.add_trial(\n optuna.trial.create_trial(\n params={\n \"bagging_fraction\": 1.0,\n \"bagging_freq\": 0,\n \"min_child_samples\": 20,\n },\n distributions={\n \"bagging_fraction\": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),\n \"bagging_freq\": optuna.distributions.IntDistribution(0, 7),\n \"min_child_samples\": optuna.distributions.IntDistribution(5, 100),\n },\n value=0.94,\n )\n)\nstudy.add_trial(\n optuna.trial.create_trial(\n params={\n \"bagging_fraction\": 0.75,\n \"bagging_freq\": 5,\n \"min_child_samples\": 20,\n },\n distributions={\n \"bagging_fraction\": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),\n \"bagging_freq\": optuna.distributions.IntDistribution(0, 7),\n \"min_child_samples\": optuna.distributions.IntDistribution(5, 100),\n },\n value=0.95,\n )\n)\nstudy.optimize(objective, n_trials=100, timeout=600)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *Ut& & 20_recipes/003_attributes.ipynb{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n\n# User Attributes\n\nThis feature is to annotate experiments with user-defined attributes.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding User Attributes to Studies\n\nA :class:`~optuna.study.Study` object provides :func:`~optuna.study.Study.set_user_attr` method\nto register a pair of key and value as an user-defined attribute.\nA key is supposed to be a ``str``, and a value be any object serializable with ``json.dumps``.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import sklearn.datasets\nimport sklearn.model_selection\nimport sklearn.svm\n\nimport optuna\n\n\nstudy = optuna.create_study(storage=\"sqlite:///example.db\")\nstudy.set_user_attr(\"contributors\", [\"Akiba\", \"Sano\"])\nstudy.set_user_attr(\"dataset\", \"MNIST\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can access annotated attributes with :attr:`~optuna.study.Study.user_attr` property.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.user_attrs # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":class:`~optuna.struct.StudySummary` object, which can be retrieved by\n:func:`~optuna.study.get_all_study_summaries`, also contains user-defined attributes.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study_summaries = optuna.get_all_study_summaries(\"sqlite:///example.db\")\nstudy_summaries[0].user_attrs # {\"contributors\": [\"Akiba\", \"Sano\"], \"dataset\": \"MNIST\"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
".. seealso::\n ``optuna study set-user-attr`` command, which sets an attribute via command line interface.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding User Attributes to Trials\n\nAs with :class:`~optuna.study.Study`, a :class:`~optuna.trial.Trial` object provides\n:func:`~optuna.trial.Trial.set_user_attr` method.\nAttributes are set inside an objective function.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def objective(trial):\n iris = sklearn.datasets.load_iris()\n x, y = iris.data, iris.target\n\n svc_c = trial.suggest_float(\"svc_c\", 1e-10, 1e10, log=True)\n clf = sklearn.svm.SVC(C=svc_c)\n accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean()\n\n trial.set_user_attr(\"accuracy\", accuracy)\n\n return 1.0 - accuracy # return error for minimization\n\n\nstudy.optimize(objective, n_trials=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can access annotated attributes as:\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"study.trials[0].user_attrs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that, in this example, the attribute is not annotated to a :class:`~optuna.study.Study`\nbut a single :class:`~optuna.trial.Trial`.\n\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK *U