Gradient-based methods

Proximal-Gradient

The proximal-gradient method [BT2009], [N2013] is a method to solve problems of the form

\[\argmin_{\bs{x} \in \mathbb{R}^d} f(\bs{x}) + g(\bs{x})\]

where $f$ is a differentiable function for which we have access to its gradient and $g$ is a potentially non-smooth function for which we have access to its proximal operator.

copt.minimize_proximal_gradient(f_grad, x0)

Proximal gradient descent.

Examples

References

BT2009

Beck, Amir, and Marc Teboulle. “Gradient-based algorithms with applications to signal recovery.” Convex optimization in signal processing and communications (2009)

N2013

Nesterov, Yu. “Gradient methods for minimizing composite functions.” Mathematical Programming 140.1 (2013): 125-161.

Primal-dual hybrid gradient

The primal-dual hybrid gradient method [C2013] [V2013] [CP2016] is a method to solve problems of the form

\[\argmin_{\bs{x} \in \mathbb{R}^d} f(\bs{x}) + g(\bs{x}) + h(\bs{A}\bs{x})\]

where $f$ is a differentiable function for which we have access to its gradient and $g$ and $h$ are potentially non-smooth functions for which we have access to their proximal operator.

copt.minimize_primal_dual(f_grad, x0[, …])

Primal-dual hybrid gradient splitting method.

Examples

References

C2013

Condat, Laurent. “A primal–dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms.” Journal of Optimization Theory and Applications 158.2 (2013): 460-479.

V2013

Vũ, Bằng Công. “A splitting algorithm for dual monotone inclusions involving cocoercive operators.” Advances in Computational Mathematics 38.3 (2013)

CP2016

Chambolle, Antonin, and Thomas Pock. “An introduction to continuous optimization for imaging.” Acta Numerica 25 (2016)

Three-operator splitting

The three operator splitting [DY2017] [PG2018] is a method to solve problems of the form

\[\argmin_{\bs{x} \in \mathbb{R}^d} f(\bs{x}) + g(\bs{x}) + h(\bs{x})\]

where $f$ is a differentiable function for which we have access to its gradient and $g$ and $h$ are potentially non-smooth functions for which we have access to their proximal operator.

copt.minimize_three_split(f_grad, x0[, …])

Davis-Yin three operator splitting method.

Examples

References

DY2017

Davis, Damek, and Wotao Yin. “A three-operator splitting scheme and its optimization applications.” Set-Valued and Variational Analysis, 2017.

PG2018

Pedregosa, Fabian, and Gauthier Gidel. “Adaptive Three Operator Splitting.” Proceedings of the 35th International Conference on Machine Learning, 2018.