Open in Colab

Search Spaces

Below, we provide examples of how to:

  • Setup a flat search space consisting of all four parameter types and additional auxiliary parameter types.

  • Setup a conditional search space correctly.

  • Reparameterize search spaces, which is useful for combinatorial search spaces.

  • Use infeasibility to define shaped search spaces.

Installation and reference imports

!pip install google-vizier
import math
from typing import List

from vizier import pyvizier as vz

Flat search spaces

Below are the core primitive parameter types and their specifications:

  • DOUBLE: Continuous range of possible values in the closed interval \([a,b]\) for some real numbers \(a \le b\).

  • INTEGER: Integer range of possible values in \([a,b] \subset \mathbb{Z}\) for some integers \(a \le b\).

  • DISCRETE: Finite, ordered set of values from \(\mathbb{R}\).

  • CATEGORICAL: Unordered list of strings.

flat_problem = vz.ProblemStatement()
flat_problem_root = flat_problem.search_space.root
flat_problem_root.add_float_param(name='double', min_value=0.0, max_value=1.0)
flat_problem_root.add_int_param(name='int', min_value=1, max_value=10)
flat_problem_root.add_discrete_param(
    name='discrete', feasible_values=[0.1, 0.3, 0.5])
flat_problem_root.add_categorical_param(
    name='categorical', feasible_values=['a', 'b', 'c'])

PyVizier also has a BOOLEAN parameter which under-the-hood, is a binary CATEGORICAL parameter with values 'True' and 'False'.

flat_problem_root.add_bool_param(name='bool')

A default value for seeding the study may be used when constructing a parameter.

flat_problem_root.add_float_param(
    name='double_with_default', min_value=0.0, max_value=1.0, default_value=0.5)

Scaling

Each of the numerical parameter types (DOUBLE, INTEGER, DISCRETE) may also have a scaling type, which toggles whether optimization occurs over a transformed space.

# Default scaling used.
flat_problem_root.add_float_param(
    name='double_uniform',
    min_value=0.0,
    max_value=1.0,
    scale_type=vz.ScaleType.LINEAR)

# Points near min_value are more important.
flat_problem_root.add_float_param(
    name='double_log',
    min_value=0.0,
    max_value=1.0,
    scale_type=vz.ScaleType.LOG)

# Points near the max_value are more important.
flat_problem_root.add_float_param(
    name='double_reverse_log',
    min_value=0.0,
    max_value=1.0,
    scale_type=vz.ScaleType.REVERSE_LOG)

# Default scaling used for DISCRETE parameters.
flat_problem_root.add_discrete_param(
    name='discrete_uniform',
    feasible_values=[0.1, 0.3, 0.5],
    scale_type=vz.ScaleType.UNIFORM_DISCRETE)

Conditional search spaces

Sometimes, child parameters only exist in specific scenarios or conditions when a parent parameter is equal to one or more specific values.

Example: Momentum hyperparameters are used by the Adam optimizer, but not stochastic gradient descent (SGD).

Caveat: Since the value of a “learning rate” depends strongly on the optimizer being used (e.g. a learning rate of 0.1 to SGD means completely differently to Adam), we must create two separate child parameters, rather than sharing a single one.

conditional_problem = vz.ProblemStatement()
conditional_problem_root = conditional_problem.search_space.root
optimizer = conditional_problem_root.add_categorical_param(
    name='optimizer', feasible_values=['sgd', 'adam'])

# SGD child parameters
optimizer.select_values(['sgd']).add_float_param(
    'sgd_learning_rate',
    min_value=0.0001,
    max_value=1.0,
    scale_type=vz.ScaleType.LOG)

# Adam child parameters
optimizer.select_values(['adam']).add_float_param(
    'adam_learning_rate',
    min_value=0.0001,
    max_value=1.0,
    scale_type=vz.ScaleType.LOG)
optimizer.select_values(['adam']).add_float_param(
    'adam_beta1',
    min_value=0.0,
    max_value=1.0,
    scale_type=vz.ScaleType.REVERSE_LOG)
optimizer.select_values(['adam']).add_float_param(
    'adam_beta2',
    min_value=0.0,
    max_value=1.0,
    scale_type=vz.ScaleType.REVERSE_LOG)

Combinatorial Reparamterization

When dealing with a combinatorial search space \(X\), one way to easily deal with such cases is to construct a reparameterization. Mathematically, this means finding a practical search space \(Z\) and surjective mapping \(\Phi: Z \rightarrow X\).

Below is an example over the space of permutations of size \(N\), where our mapping utilizes the Lehmer code.

N = 10

# Setup search space.
permutation_problem = vz.ProblemStatement()
for n in range(N):
  permutation_problem.search_space.root.add_int_param(
      name=str(n), min_value=0, max_value=n)


def compute_index(trial: vz.Trial) -> int:
  """Computes index from Lehmer code."""
  index = 0
  for n in range(N):
    index += trial.parameters.get_value(str(n)) * math.factorial(n)
  return index


def compute_permutation(index: int) -> List[int]:
  """Outputs a N-permutation as a list of indices."""
  all_indices = list(range(N))
  temp_index = index
  output = []
  for k in range(1, N + 1):
    factorial_value = math.factorial(N - k)
    value = all_indices[temp_index // factorial_value]
    output.append(value)
    all_indices.remove(value)
    temp_index = temp_index % factorial_value
  return output


def phi(trial: vz.Trial) -> List[int]:
  """Maps a suggestion to a permutation."""
  return compute_permutation(compute_index(trial))

Infeasibility

Consider an optimization problem where we only consider float parameters \((x,y)\) from the unit disk \(x^{2} + y^{2} \le 1\). For such a scenario, we may denote any parameters outside of this area to be infeasible.

disk_problem = vz.ProblemStatement()
disk_problem_root = disk_problem.search_space.root
disk_problem_root.add_float_param(name='x', min_value=-1.0, max_value=1.0)
disk_problem_root.add_float_param(name='y', min_value=-1.0, max_value=1.0)


def evaluate(trial: vz.Trial) -> vz.Trial:
  x = trial.parameters['x']
  y = trial.parameters['y']
  if x**2 + y**2 <= 1:
    trial.complete(vz.Measurement(metrics={'sum': x + y}))
  else:
    trial.complete(vz.Measurement(), infeasibility_reason='Outside of range.')
  return trial