Optunity API

Optunity

Provides
  1. Routines to efficiently optimize hyperparameters
  2. Function decorators to implicitly log evaluations, constrain the domain and more.
  3. Facilities for k-fold cross-validation to estimate generalization performance.

Available modules

solvers
contains all officially supported solvers
functions
a variety of useful function decorators for hyperparameter tuning
cross_validation
k-fold cross-validation

Available subpackages

tests
regression test suite
solvers
solver implementations and auxillary functions

Utilities

__version__
Optunity version string
__revision__
Optunity revision string
__author__
Main authors of the package
optunity.manual(solver_name=None)[source]

Prints the manual of requested solver.

Parameters:solver_name – (optional) name of the solver to request a manual from. If none is specified, a general manual is printed.

Raises KeyError if solver_name is not registered.

optunity.maximize(f, num_evals=50, solver_name=None, pmap=<built-in function map>, **kwargs)[source]

Basic function maximization routine. Maximizes f within the given box constraints.

Parameters:
  • f – the function to be maximized
  • num_evals – number of permitted function evaluations
  • solver_name (string) – name of the solver to use (optional)
  • pmap (callable) – the map function to use
  • kwargs – box constraints, a dict of the following form {'parameter_name': [lower_bound, upper_bound], ...}
Returns:

retrieved maximum, extra information and solver info

This function will implicitly choose an appropriate solver and its initialization based on num_evals and the box constraints.

optunity.minimize(f, num_evals=50, solver_name=None, pmap=<built-in function map>, **kwargs)[source]

Basic function minimization routine. Minimizes f within the given box constraints.

Parameters:
  • f – the function to be minimized
  • num_evals – number of permitted function evaluations
  • solver_name (string) – name of the solver to use (optional)
  • pmap (callable) – the map function to use
  • kwargs – box constraints, a dict of the following form {'parameter_name': [lower_bound, upper_bound], ...}
Returns:

retrieved minimum, extra information and solver info

This function will implicitly choose an appropriate solver and its initialization based on num_evals and the box constraints.

optunity.optimize(solver, func, maximize=True, max_evals=0, pmap=<built-in function map>, decoder=None)[source]

Optimizes func with given solver.

Parameters:
  • solver – the solver to be used, for instance a result from optunity.make_solver()
  • func (callable) – the objective function
  • maximize (bool) – maximize or minimize?
  • max_evals (int) – maximum number of permitted function evaluations
  • pmap (function) – the map() function to use, to vectorize use optunity.pmap()

Returns the solution and a namedtuple with further details.

Result details includes the following:

optimum
optimal function value f(solution)
stats
statistics about the solving process
call_log
the call log
report
solver report, can be None

Statistics gathered while solving a problem:

num_evals
number of function evaluations
time
wall clock time needed to solve
optunity.wrap_call_log(f, call_dict)[source]

Wraps an existing call log (as dictionary) around f.

This allows you to communicate known function values to solvers. (currently available solvers do not use this info)

optunity.wrap_constraints(f, default=None, ub_o=None, ub_c=None, lb_o=None, lb_c=None, range_oo=None, range_co=None, range_oc=None, range_cc=None, custom=None)[source]

Decorates f with given input domain constraints.

Parameters:
  • f (callable) – the function that will be constrained
  • default (number) – function value to default to in case of constraint violations
  • ub_o (dict) – open upper bound constraints, e.g. \(x < c\)
  • ub_c (dict) – closed upper bound constraints, e.g. \(x \leq c\)
  • lb_o (dict) – open lower bound constraints, e.g. \(x > c\)
  • lb_c (dict) – closed lower bound constraints, e.g. \(x \geq c\)
  • range_oo (dict with 2-element lists as values ([lb, ub])) – range constraints (open lb and open ub) \(lb < x < ub\)
  • range_co (dict with 2-element lists as values ([lb, ub])) – range constraints (closed lb and open ub) \(lb \leq x < ub\)
  • range_oc (dict with 2-element lists as values ([lb, ub])) – range constraints (open lb and closed ub) \(lb < x \leq ub\)
  • range_cc (dict with 2-element lists as values ([lb, ub])) – range constraints (closed lb and closed ub) \(lb \leq x \leq ub\)
  • custom (list of constraints) – custom, user-defined constraints

*custom constraints are binary functions that yield False in case of violations.

>>> def f(x):
...     return x
>>> fc = wrap_constraints(f, default=-1, range_oc={'x': [0, 1]})
>>> fc(x=0.5)
0.5
>>> fc(x=1)
1
>>> fc(x=5)
-1
>>> fc(x=0)
-1

We can define any custom constraint that we want. For instance, assume we have a binary function with arguments x and y, and we want to make sure that the provided values remain within the unit circle.

>>> def f(x, y):
...     return x + y
>>> circle_constraint = lambda x, y: (x ** 2 + y ** 2) <= 1
>>> fc = wrap_constraints(f, default=1234, custom=[circle_constraint])
>>> fc(0.0, 0.0)
0.0
>>> fc(1.0, 0.0)
1.0
>>> fc(0.5, 0.5)
1.0
>>> fc(1, 0.5)
1234
optunity.make_solver(solver_name, *args, **kwargs)[source]

Creates a Solver from given parameters.

Parameters:
  • solver_name (string) – the solver to instantiate
  • args – positional arguments to solver constructor.
  • kwargs – keyword arguments to solver constructor.

Use optunity.manual() to get a list of registered solvers. For constructor arguments per solver, please refer to Solver overview.

Raises KeyError if

  • solver_name is not registered
  • *args and **kwargs are invalid to instantiate the solver.
optunity.suggest_solver(num_evals=50, solver_name=None, **kwargs)[source]
optunity.cross_validated(x, num_folds=10, y=None, strata=None, folds=None, num_iter=1, regenerate_folds=False, clusters=None, aggregator=<function mean>)[source]

Function decorator to perform cross-validation as configured.

Parameters:
  • x – data to be used for cross-validation
  • num_folds – number of cross-validation folds (default 10)
  • y – (optional) labels to be used for cross-validation. If specified, len(labels) must equal len(x)
  • strata – (optional) strata to account for when generating folds. Strata signify instances that must be spread across folds. Not every instance must be in a stratum. Specify strata as a list of lists of instance indices.
  • folds – (optional) prespecified cross-validation folds to be used (list of lists (iterations) of lists (folds)).
  • num_iter – (optional) number of iterations to use (default 1)
  • regenerate_folds – (optional) whether or not to regenerate folds on every evaluation (default false)
  • clusters – (optional) clusters to account for when generating folds. Clusters signify instances that must be assigned to the same fold. Not every instance must be in a cluster. Specify clusters as a list of lists of instance indices.
  • aggregator – function to aggregate scores of different folds (default: mean)
Returns:

a cross_validated_callable with the proper configuration.

This resulting decorator must be used on a function with the following signature (+ potential other arguments):

Parameters:
  • x_train (iterable) – training data
  • y_train (iterable) – training labels (optional)
  • x_test (iterable) – testing data
  • y_test (iterable) – testing labels (optional)

y_train and y_test must be available of the y argument to this function is not None.

These 4 keyword arguments will be bound upon decoration. Further arguments will remain free (e.g. hyperparameter names).

>>> data = list(range(5))
>>> @cross_validated(x=data, num_folds=5, folds=[[[i] for i in range(5)]], aggregator=identity)
... def f(x_train, x_test, a):
...     return x_test[0] + a
>>> f(a=1)
[1, 2, 3, 4, 5]
>>> f(1)
[1, 2, 3, 4, 5]
>>> f(a=2)
[2, 3, 4, 5, 6]

The number of folds must be less than or equal to the size of the data.

>>> data = list(range(5))
>>> @cross_validated(x=data, num_folds=6) 
... def f(x_train, x_test, a):
...     return x_test[0] + a
AssertionError

The number of labels (if specified) must match the number of data instances.

>>> data = list(range(5))
>>> labels = list(range(3))
>>> @cross_validated(x=data, y=labels, num_folds=2) 
... def f(x_train, x_test, a):
...     return x_test[0] + a
AssertionError
optunity.generate_folds(num_rows, num_folds=10, strata=None, clusters=None)[source]

Generates folds for a given number of rows.

Parameters:
  • num_rows – number of data instances
  • num_folds – number of folds to use (default 10)
  • strata – (optional) list of lists to indicate different sampling strata. Not all rows must be in a stratum. The number of rows per stratum must be larger than or equal to num_folds.
  • clusters – (optional) list of lists indicating clustered instances. Clustered instances must be placed in a single fold to avoid information leaks.
Returns:

a list of folds, each fold is a list of instance indices

>>> folds = generate_folds(num_rows=6, num_folds=2, clusters=[[1, 2]], strata=[[3,4]])
>>> len(folds)
2
>>> i1 = [idx for idx, fold in enumerate(folds) if 1 in fold]
>>> i2 = [idx for idx, fold in enumerate(folds) if 2 in fold]
>>> i1 == i2
True
>>> i3 = [idx for idx, fold in enumerate(folds) if 3 in fold]
>>> i4 = [idx for idx, fold in enumerate(folds) if 4 in fold]
>>> i3 == i4
False

Warning

Instances in strata are not necessarily spread out over all folds. Some folds may already be full due to clusters. This effect should be negligible.

optunity.pmap(f, *args, **kwargs)[source]

Parallel map using multiprocessing.

Parameters:
  • f – the callable
  • args – arguments to f, as iterables
Returns:

a list containing the results

Warning

This function will not work in IPython: https://github.com/claesenm/optunity/issues/8.

Warning

Python’s multiprocessing library is incompatible with Jython.

optunity.available_solvers()[source]

Returns a list of all available solvers.

These can be used in optunity.make_solver().

optunity.call_log2dataframe(log)[source]

Converts a call log into a pandas data frame. This function errors if you don’t have pandas available.

Parameters:log – call log to be converted, as returned by e.g. optunity.minimize
Returns:a pandas data frame capturing the same information as the call log
optunity.maximize_structured(f, search_space, num_evals=50, pmap=<built-in function map>)[source]

Basic function maximization routine. Maximizes f within the given box constraints.

Parameters:
  • f – the function to be maximized
  • search_space – the search space (see Structured search spaces for details)
  • num_evals – number of permitted function evaluations
  • pmap (callable) – the map function to use
Returns:

retrieved maximum, extra information and solver info

This function will implicitly choose an appropriate solver and its initialization based on num_evals and the box constraints.

optunity.minimize_structured(f, search_space, num_evals=50, pmap=<built-in function map>)[source]

Basic function minimization routine. Minimizes f within the given box constraints.

Parameters:
  • f – the function to be maximized
  • search_space – the search space (see Structured search spaces for details)
  • num_evals – number of permitted function evaluations
  • pmap (callable) – the map function to use
Returns:

retrieved maximum, extra information and solver info

This function will implicitly choose an appropriate solver and its initialization based on num_evals and the box constraints.