{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBP-KRCq6G6b"
      },
      "source": [
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/creating_benchmarks.ipynb)\n",
        "\n",
        "# Creating Benchmarks\n",
        "We provide a guide below on creating benchmarks, through the use of either:\n",
        "*   Standard search space primitives.\n",
        "*   Metadata for complex search spaces."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f76xE1Os5DFD"
      },
      "source": [
        "## Installation and reference imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BUfxI1EPwXfH"
      },
      "outputs": [],
      "source": [
        "!pip install google-vizier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "giEspdLe5Ux1"
      },
      "outputs": [],
      "source": [
        "import abc\n",
        "import random\n",
        "from typing import Sequence\n",
        "from vizier import pyvizier as vz\n",
        "from vizier.benchmarks import experimenters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M4H9Q3y85dwC"
      },
      "source": [
        "## Experimenters\n",
        "The core base class of any objective function is the `Experimenter` class, which simply contains a method to evaluate a `Trial` and a `ProblemStatement` to describe its search space and metrics. The exact entry into the class can be found [here](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py).\n",
        "\n",
        "```python\n",
        "class Experimenter(metaclass=abc.ABCMeta):\n",
        "  \"\"\"Abstract base class for Experimenters.\"\"\"\n",
        "\n",
        "  @abc.abstractmethod\n",
        "  def evaluate(self, suggestions: Sequence[vz.Trial]) -\u003e None:\n",
        "    \"\"\"Evaluates and mutates the Trials in-place.\"\"\"\n",
        "\n",
        "  @abc.abstractmethod\n",
        "  def problem_statement(self) -\u003e vz.ProblemStatement:\n",
        "    \"\"\"The search configuration generated by this experimenter.\"\"\"\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXyC_E_v5lDv"
      },
      "source": [
        "Below is an example of a basic 1D objective function $f(x) = x^{2}$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jDQNz57RYymT"
      },
      "outputs": [],
      "source": [
        "class Basic1DExperimenter(experimenters.Experimenter):\n",
        "\n",
        "  def evaluate(self, suggestions: Sequence[vz.Trial]) -\u003e None:\n",
        "    for suggestion in suggestions:\n",
        "      x = suggestion.parameters['x'].value\n",
        "      objective = x**2\n",
        "      measurement = pyvizier.Measurement(metrics={'obj': objective})\n",
        "      suggestion.complete(measurement)\n",
        "\n",
        "  def problem_statement(self) -\u003e vz.ProblemStatement:\n",
        "    problem_statement = vz.ProblemStatement()\n",
        "    root = problem_statement.search_space.root\n",
        "    root.add_float_param(name='x', min_value=-1.0, max_value=1.0)\n",
        "    metric = vz.MetricInformation(name='obj', goal=vz.ObjectiveMetricGoal.MAXIMIZE)\n",
        "    problem_statement.metric_information.append(metric)\n",
        "    return problem_statement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWdSIA1daKQB"
      },
      "source": [
        "We may thus evaluate a suggestion. Note that such suggestions are actually `Trial`s, to allow maximum flexibility."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MytRchwhau0z"
      },
      "outputs": [],
      "source": [
        "basic_experimenter = Basic1DExperimenter()\n",
        "trial = vz.Trial()\n",
        "trial.parameters['x'] = 0.1\n",
        "\n",
        "basic_experimenter.evaluate([trial])\n",
        "assert trial.final_measurement.metrics['obj'].value == 0.1 ** 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OdXa6kDM5pOb"
      },
      "source": [
        "## Metadata-based Experimenters\n",
        "Similar to using the `Metadata` primitive to create custom algorithms and\n",
        "complex search spaces, creating custom `Experimenter`s provides the freedom to\n",
        "define custom objective functions.\n",
        "\n",
        "As an example, suppose our search space consisted of unbounded-length sequences\n",
        "consisting of some vocabulary (e.g. the letters 'A' to 'Z' if considering the\n",
        "space of English words), and we wish to maximize the sequence's average ASCII\n",
        "value."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KRufPFLc7rwT"
      },
      "outputs": [],
      "source": [
        "class VocabularyExperimenter(experimenters.Experimenter):\n",
        "\n",
        "  def evaluate(self, suggestions: Sequence[vz.Trial]):\n",
        "    for suggestion in suggestions:\n",
        "      x = suggestion.metadata['word']\n",
        "      objective = float(sum([ord(c) for c in x])) / len(x)\n",
        "      measurement = vz.Measurement(metrics={'obj': objective})\n",
        "      suggestion.complete(measurement)\n",
        "\n",
        "  def problem_statement(self) -\u003e pyvizier.ProblemStatement:\n",
        "    problem_statement = vz.ProblemStatement()\n",
        "    problem_statement.metadata['vocab'] = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'\n",
        "    metric = vz.MetricInformation(name='obj', goal=vz.ObjectiveMetricGoal.MAXIMIZE)\n",
        "    problem_statement.metric_information.append(metric)\n",
        "    return problem_statement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ZKtke9pdjlU"
      },
      "source": [
        "Below is an example of constructing a valid suggestion and evaluating it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s_nuLfYsdtz6"
      },
      "outputs": [],
      "source": [
        "vocab_experimenter = VocabularyExperimenter()\n",
        "vocabulary = vocab_experimenter.problem_statement().metadata['vocab']\n",
        "trial = vz.Trial()\n",
        "trial.metadata['word'] = str(\n",
        "    [random.randint(0, len(vocabulary)) for _ in range(10)]\n",
        ")\n",
        "\n",
        "vocab_experimenter.evaluate([trial])\n",
        "print('Average ASCII value is:', trial.final_measurement.metrics['obj'].value)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Creating Benchmarks.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}