Creating Benchmarks

We provide a guide below on creating benchmarks, through the use of either:

• Standard search space primitives.

• Metadata for complex search spaces.

Installation and reference imports

!pip install google-vizier

import abc
import random
from typing import Sequence
from vizier import pyvizier as vz
from vizier.benchmarks import experimenters

Experimenters

The core base class of any objective function is the Experimenter class, which simply contains a method to evaluate a Trial and a ProblemStatement to describe its search space and metrics. The exact entry into the class can be found here.

class Experimenter(metaclass=abc.ABCMeta):
  """Abstract base class for Experimenters."""

  @abc.abstractmethod
  def evaluate(self, suggestions: Sequence[vz.Trial]) -> None:
    """Evaluates and mutates the Trials in-place."""

  @abc.abstractmethod
  def problem_statement(self) -> vz.ProblemStatement:
    """The search configuration generated by this experimenter."""

Below is an example of a basic 1D objective function \(f(x) = x^{2}\).

class Basic1DExperimenter(experimenters.Experimenter):

  def evaluate(self, suggestions: Sequence[vz.Trial]) -> None:
    for suggestion in suggestions:
      x = suggestion.parameters['x'].value
      objective = x**2
      measurement = pyvizier.Measurement(metrics={'obj': objective})
      suggestion.complete(measurement)

  def problem_statement(self) -> vz.ProblemStatement:
    problem_statement = vz.ProblemStatement()
    root = problem_statement.search_space.root
    root.add_float_param(name='x', min_value=-1.0, max_value=1.0)
    metric = vz.MetricInformation(name='obj', goal=vz.ObjectiveMetricGoal.MAXIMIZE)
    problem_statement.metric_information.append(metric)
    return problem_statement

We may thus evaluate a suggestion. Note that such suggestions are actually Trials, to allow maximum flexibility.

basic_experimenter = Basic1DExperimenter()
trial = vz.Trial()
trial.parameters['x'] = 0.1

basic_experimenter.evaluate([trial])
assert trial.final_measurement.metrics['obj'].value == 0.1 ** 2

Metadata-based Experimenters

Similar to using the Metadata primitive to create custom algorithms and complex search spaces, creating custom Experimenters provides the freedom to define custom objective functions.

As an example, suppose our search space consisted of unbounded-length sequences consisting of some vocabulary (e.g. the letters ‘A’ to ‘Z’ if considering the space of English words), and we wish to maximize the sequence’s average ASCII value.

class VocabularyExperimenter(experimenters.Experimenter):

  def evaluate(self, suggestions: Sequence[vz.Trial]):
    for suggestion in suggestions:
      x = suggestion.metadata['word']
      objective = float(sum([ord(c) for c in x])) / len(x)
      measurement = vz.Measurement(metrics={'obj': objective})
      suggestion.complete(measurement)

  def problem_statement(self) -> pyvizier.ProblemStatement:
    problem_statement = vz.ProblemStatement()
    problem_statement.metadata['vocab'] = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    metric = vz.MetricInformation(name='obj', goal=vz.ObjectiveMetricGoal.MAXIMIZE)
    problem_statement.metric_information.append(metric)
    return problem_statement

Below is an example of constructing a valid suggestion and evaluating it.

vocab_experimenter = VocabularyExperimenter()
vocabulary = vocab_experimenter.problem_statement().metadata['vocab']
trial = vz.Trial()
trial.metadata['word'] = str(
    [random.randint(0, len(vocabulary)) for _ in range(10)]
)

vocab_experimenter.evaluate([trial])
print('Average ASCII value is:', trial.final_measurement.metrics['obj'].value)