Search Spaces
Below, we provide examples of how to:
Setup a flat search space consisting of all four parameter types and additional auxiliary parameter types.
Setup a conditional search space correctly.
Reparameterize search spaces, which is useful for combinatorial search spaces.
Use infeasibility to define shaped search spaces.
Installation and reference imports
!pip install google-vizier
import math
from typing import List
from vizier import pyvizier as vz
Flat search spaces
Below are the core primitive parameter types and their specifications:
DOUBLE
: Continuous range of possible values in the closed interval \([a,b]\) for some real numbers \(a \le b\).INTEGER
: Integer range of possible values in \([a,b] \subset \mathbb{Z}\) for some integers \(a \le b\).DISCRETE
: Finite, ordered set of values from \(\mathbb{R}\).CATEGORICAL
: Unordered list of strings.
flat_problem = vz.ProblemStatement()
flat_problem_root = flat_problem.search_space.root
flat_problem_root.add_float_param(name='double', min_value=0.0, max_value=1.0)
flat_problem_root.add_int_param(name='int', min_value=1, max_value=10)
flat_problem_root.add_discrete_param(
name='discrete', feasible_values=[0.1, 0.3, 0.5])
flat_problem_root.add_categorical_param(
name='categorical', feasible_values=['a', 'b', 'c'])
PyVizier also has a BOOLEAN
parameter which under-the-hood, is a binary CATEGORICAL
parameter with values 'True'
and 'False'
.
flat_problem_root.add_bool_param(name='bool')
A default value for seeding the study may be used when constructing a parameter.
flat_problem_root.add_float_param(
name='double_with_default', min_value=0.0, max_value=1.0, default_value=0.5)
Scaling
Each of the numerical parameter types (DOUBLE
, INTEGER
, DISCRETE
) may also have a scaling type, which toggles whether optimization occurs over a transformed space.
# Default scaling used.
flat_problem_root.add_float_param(
name='double_uniform',
min_value=0.0,
max_value=1.0,
scale_type=vz.ScaleType.LINEAR)
# Points near min_value are more important.
flat_problem_root.add_float_param(
name='double_log',
min_value=0.0,
max_value=1.0,
scale_type=vz.ScaleType.LOG)
# Points near the max_value are more important.
flat_problem_root.add_float_param(
name='double_reverse_log',
min_value=0.0,
max_value=1.0,
scale_type=vz.ScaleType.REVERSE_LOG)
# Default scaling used for DISCRETE parameters.
flat_problem_root.add_discrete_param(
name='discrete_uniform',
feasible_values=[0.1, 0.3, 0.5],
scale_type=vz.ScaleType.UNIFORM_DISCRETE)
Conditional search spaces
Sometimes, child parameters only exist in specific scenarios or conditions when a parent parameter is equal to one or more specific values.
Example: Momentum hyperparameters are used by the Adam optimizer, but not stochastic gradient descent (SGD).
Caveat: Since the value of a “learning rate” depends strongly on the optimizer being used (e.g. a learning rate of 0.1 to SGD means completely differently to Adam), we must create two separate child parameters, rather than sharing a single one.
conditional_problem = vz.ProblemStatement()
conditional_problem_root = conditional_problem.search_space.root
optimizer = conditional_problem_root.add_categorical_param(
name='optimizer', feasible_values=['sgd', 'adam'])
# SGD child parameters
optimizer.select_values(['sgd']).add_float_param(
'sgd_learning_rate',
min_value=0.0001,
max_value=1.0,
scale_type=vz.ScaleType.LOG)
# Adam child parameters
optimizer.select_values(['adam']).add_float_param(
'adam_learning_rate',
min_value=0.0001,
max_value=1.0,
scale_type=vz.ScaleType.LOG)
optimizer.select_values(['adam']).add_float_param(
'adam_beta1',
min_value=0.0,
max_value=1.0,
scale_type=vz.ScaleType.REVERSE_LOG)
optimizer.select_values(['adam']).add_float_param(
'adam_beta2',
min_value=0.0,
max_value=1.0,
scale_type=vz.ScaleType.REVERSE_LOG)
Combinatorial Reparamterization
When dealing with a combinatorial search space \(X\), one way to easily deal with such cases is to construct a reparameterization. Mathematically, this means finding a practical search space \(Z\) and surjective mapping \(\Phi: Z \rightarrow X\).
Below is an example over the space of permutations of size \(N\), where our mapping utilizes the Lehmer code.
N = 10
# Setup search space.
permutation_problem = vz.ProblemStatement()
for n in range(N):
permutation_problem.search_space.root.add_int_param(
name=str(n), min_value=0, max_value=n)
def compute_index(trial: vz.Trial) -> int:
"""Computes index from Lehmer code."""
index = 0
for n in range(N):
index += trial.parameters.get_value(str(n)) * math.factorial(n)
return index
def compute_permutation(index: int) -> List[int]:
"""Outputs a N-permutation as a list of indices."""
all_indices = list(range(N))
temp_index = index
output = []
for k in range(1, N + 1):
factorial_value = math.factorial(N - k)
value = all_indices[temp_index // factorial_value]
output.append(value)
all_indices.remove(value)
temp_index = temp_index % factorial_value
return output
def phi(trial: vz.Trial) -> List[int]:
"""Maps a suggestion to a permutation."""
return compute_permutation(compute_index(trial))
Infeasibility
Consider an optimization problem where we only consider float parameters \((x,y)\) from the unit disk \(x^{2} + y^{2} \le 1\). For such a scenario, we may denote any parameters outside of this area to be infeasible.
disk_problem = vz.ProblemStatement()
disk_problem_root = disk_problem.search_space.root
disk_problem_root.add_float_param(name='x', min_value=-1.0, max_value=1.0)
disk_problem_root.add_float_param(name='y', min_value=-1.0, max_value=1.0)
def evaluate(trial: vz.Trial) -> vz.Trial:
x = trial.parameters['x']
y = trial.parameters['y']
if x**2 + y**2 <= 1:
trial.complete(vz.Measurement(metrics={'sum': x + y}))
else:
trial.complete(vz.Measurement(), infeasibility_reason='Outside of range.')
return trial