Convex function

A convex function $$f:\mathbb R^n \to \mathbb R$$ is a function that satisfies the following inequality for all $$x,y \in \textbf{dom} f$$ and $$0\le \theta \le 1$$

$$f(\theta x + (1-\theta) y) \le \theta f(x) + (1-\theta) f(y)$$.

Convexity is an extremely useful property as it renders optimization problems solvable.

Convexity Criteria
While the above-mentioned inequality is the essential property for convexity, there are other equivalent properties.

For any function
A function $$f$$ is convex iff it is convex when restricted to any line that intersects its domain; i.e., if the function $$g(t) := f(x+tv)$$ is convex for all direction vectors $$v$$.

For differentiable functions
If $$f$$ is differentiable, then it is convex if and only if $$\textbf{dom} f$$ is convex and

$$f(y) \ge f(x) + \nabla f (x)^T(y-x)$$

holds for all $$x,y \in \textbf{dom} f$$. In other words, $$f$$ is convex iff its first-order Taylor approximation is a global underestimator of $$f$$ at every point of its domain. If $$f$$ is twice differentiable, then it is convex iff its Hessian $$\nabla^2 f$$ is positive definite, i.e.,

$$\nabla^2 f(x) \ge 0$$ for all $$x \in \textbf{dom} f$$.

Establishing convexity
There are many practical ways of establishing the convexity of a function (including direct usage of the convexity criteria above). Below is a good list :


 * 1) Verify definition
 * 2) Verify definition by restricting to a line
 * 3) For twice differentiable functions, show that the Hessian is positive
 * 4) Show that the function is obtained from simple convex functions by operations that preserve convexity:
 * 5) * Nonnegative weighted sum: $$f(x)=\alpha f_1(x) + \beta f_2(x)$$ is convex if $$\alpha,\beta\ge 0$$ and $$f_1(x), f_2(x)$$ are convex. Extends to infinite sums and integrals.
 * 6) * Composition with affine function: $$f(Ax+b)$$ is convex if $$f$$ is convex.
 * 7) * Pointwise maximum and supremum: (a) If $$f_1, \dots, f_m$$ are convex, then $$f(x) = \max\{f_1(x),\dots,f_m(x)\}$$ is convex; (b) If $$f(x,y)$$ is convex for each $$y \in \mathcal A$$, then $$g(x) = \sup_{y \in \mathcal A}f(x,y)$$ is convex.
 * 8) * Composition rules: $$f(x) = h(g(x)) = h(g_1(x), \dots, g_k(x))$$ is convex if (a) $$g$$ is convex, $$h$$ is convex, $$\tilde h$$ (extended value function ) is nonincreasing; or (b) $$g$$ is concave, $$h$$ is convex and $$\tilde h$$ is nonincreasing in each argument
 * 9) * Minimization: If $$f(x,y)$$ is convex in (x,y) and $$C$$ is a convex set, then $$g(x) = \inf_{y\in C} f(x,y)$$ is convex.
 * 10) * The perspective of a function $$f$$, $$g(x,t) = tf(x/t)$$ is convex if $$g$$ is convex.

Examples
Some typical convex functions are:


 * Quadratic function: $$f(x) = (1/2)x^T P x + q^T + r$$ where $$P \ge 0$$ and $$P$$ is symmetric.
 * Quadratic-over-linear: $$f(x,y) = x^2/y$$
 * Log-sum-exp: $$f(x) = \log \sum_{k=1}^n e^{x_k}$$
 * Geometric mean: $$f(x) = (\prod_{k=1}^n x_k)^{1/n}$$
 * Any norm: $$f(x) = ||x||$$
 * Distance to farthest point in a set $$C$$: $$f(x) = \sup_{y\in C} ||x-y||$$
 * Distance to a set $$S$$: $$\inf_{y\in S}||x-y||$$.
 * Maximum eigenvalue of symmetric matrix: $$\lambda_{\max} = \sup_{||y||_2 =1}y^T X y$$
 * Relative entropy: $$g(x,t) = t \log t - t\log x$$ is convex on $$\mathbb R^{2}_{++}$$