Tree-structured Parzen Estimator¶
The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.
The TPE approach models \(P(x|y)\) and \(P(y)\) where x represents hyperparameters and y the associated quality score. \(P(x|y)\) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. In this solver, Optunity only supports uniform priors within given box constraints. For more exotic search spaces, please refer to [Hyperopt]. This optimization approach is described in detail in [TPE2011] and [TPE2013].
Optunity provides the TPE solver as is implemented in [Hyperopt]. This solver is only available if Hyperopt is installed, which in turn requires NumPy. Both dependencies must be met to use this solver.
|[TPE2011]||Bergstra, James S., et al. “Algorithms for hyper-parameter optimization.” Advances in Neural Information Processing Systems. 2011.|
|[TPE2013]||Bergstra, James, Daniel Yamins, and David Cox. “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures.” Proceedings of The 30th International Conference on Machine Learning. 2013.|
|[Hyperopt]||(1, 2) http://jaberg.github.io/hyperopt/|