Optimal Control and Planning

What if we knew the transition dynamics

Often we do know the dynamics

1. Games (e.g. Go)
2. Easily modeled systems (e.g., navigating a car)
3. Simulated environments (e.g, simulated robots, video games) Often we learn the dynamics
4. System identification - fit unknown parameters of a known model
5. Learning - fit a general purpose model to observed transition data

Model-based reinforcement learning Model-based reinforcement learning: learn the transition dynamics, then figure out how to choose actions

In this post, we review the basic policy gradient algorithm for deep reinforcement learning and the actor-critic algorithm. Most of the contents are derived from CS 285 at UC Berkeley.

Policy Gradient Introduction

In this part, we will first focus on the fully observable model with finite horizon, i.e., \(o_t = s_{t}\) and \(t\) is bounded. At a very high level, the policy gradient method is very simple: The policy gradient algorithm parametrized the policy \(\pi_{\theta}(a|s)\) by \(\theta\), and use samples (interactions with the environment) to update the parameter \(\theta\).

Next, we first derive the policy gradient algorithm (REINFORCE) and introduce some techniques to improve the performance of policy graident.

Evaluating the objective

First, we use \(\tau = \{s_1,a_1,s_2,a_2,\dots,s_T,a_T\}\) to denote the trajectory of 1 trial, \(r(\tau)\) to denote the reward of trajectory \(\tau\), and \(p_{\theta}(\tau),\pi_{\theta}(\tau)\) to denote the probability that trajectory \(\tau\) appears under the policy \(\pi_{\theta}(a|s)\).

Recall that our reinforcement learning objective can be written as

We define \(J(\theta) = \mathbb E_{\tau\sim p_{\theta}(\tau)}\left[\sum_t r(s_{t},a_{t})\right]\), and the objective becomes \(\theta^* = \arg\max_{\theta} J(\theta)\). We want to compute the gradient of \(J(\theta)\), and then we can update our policy \(\pi_{\theta}\).

First, we have

From the basic calculus computations, we have

Then, plug into the the derivative of \(J(\theta)\), we can get

Then, recall that from the Markovian assumption,we have

Taking the logarithms on both sides, we can get

Taking the gradient, the first term and the third term are thrown away, and we have the following formula for the policy gradient.

Evaluating the policy graident and REINFORCE algorithm

Now given the formula for the policy gradient, we can estimate the expectation in policy gradient by samples. Specifically, we have the following formula for the estimatation.

where \(i\) denotes the index of the trajectory.

Then, given an estimation of the policy gradient, we can then improve our policy by the gradient method \(\theta\leftarrow\theta + \alpha\nabla_{\theta}J(\theta)\).

Putting them all together, we have a simple REINFORCE algorithm (Monte-Carlo policy gradient).

1. For \(s=1,2,\dots,\)
   1. sample \(\{\tau^i\}\) from \(\pi_{\theta}(a_{t}\|s_{t})\) (run the policy)
   2. \[\nabla_{\theta} J(\theta)\approx \frac{1}{N}\sum_{i=1}^N\left(\nabla_{\theta}\log\pi_{\theta}(a_{i,t}|s_{i,t})\right)\left(\sum_{t=1}^T r(s_{i,t},a_{i,t})\right)\]
   3. \[\theta\leftarrow\theta + \alpha\nabla_{\theta}J(\theta)\]

Note that this algorithm may behave not very well because of the high variance. We need to use variance reduction techniques to improve this algorithm. But before we introduce the variance reduction techniques, we may first see an example of the REINFORCE algorithm and see its generalization into the partially observable case.

Now, we compute the policy gradient for Gaussian policies

\begin{align} \pi_{\theta}(a_{t}|s_{t}) =& \mathcal N(f_{\theta}(s_{t});\Sigma), \newline \log\pi_{\theta}(a_{t}|s_{t}) =& -\frac{1}{2}(f_{\theta}(s_{t})-a_{t})^T\Sigma^{-1}(f_{\theta}(s_{t})-a_{t}) + const, \newline \nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}) =& -\Sigma^{-1}(f_{\theta}(s_{t})-a_{t})\frac{df}{d\theta}. \end{align}

The REINFORCE algorithm simply tends to increase the probability of a good trajectory with high reward, and decrease the probability of a bad trajectory with low reward. The intuition behind REINFORCE can be summarized as: good stuff is made more likely, bad stuff is mode less likely, simply formalized the notion of “trial and error”.

It is also very easy to generalize the REINFORCE algorithm into the partially observable case. For partially observable cases, we just change \(s_{t}\) into \(o_t\) in the policy part. The estimation of the policy gradient is given as follow

Variance Reduction Technqiues

What is the problem with the policy gradient? As mentioned before, the policy gradient suffers from very high variance.

Recall that the policy gradient is

The variance is vary large because \(r(\tau)\) is very large. Moreover, adding the reward function by a constant will change the result. For example if we add a large constant to the reward function and let all rewards to be positive, then in each iteration, the policy gradient method will try to increase the probability of all sampled trajectories.

There are 2 methods that can reduce the variance of the policy gradient method, causality and baseline, and we will introduce them one by one.

Causality

The intuition of causality is very simple: the policy at time \(t'\) cannot affect reward at time \(t < t'\). Then instead of using the whole \(r(\tau)\) to update the \(\pi_{\theta}\) at time \(t\), we can only focus on the reward that happens after time \(t\). Formally, we have the following formula

\begin{align} \nabla_{\theta} J(\theta)\approx & \frac{1}{N}\sum_{i=1}^N\left(\nabla_{\theta}\log\pi_{\theta}(a_{i,t}|s_{i,t})\right)\left(\sum_{t=1}^T r(s_{i,t},a_{i,t})\right) \newline = & \frac{1}{N}\sum_{i=1}^N\nabla_{\theta}\log\pi_{\theta}(a_{i,t}|s_{i,t})\left(\sum_{t’=t}^T r(s_{i,t’},a_{i,t’})\right) \end{align}

The last term can be viewed as “reward to go”, and we use \(\hat Q_{i,t} = \sum_{t'=t}^T r(s_{i,t'},a_{i,t'})\) to denote that quantity.

It is easy to know that the causality helps to reduce the variance, because smaller number leads to smaller variance. In practice this basically always help.

Baselines

The baselines technique simply means to add constants to the reward function.

We want

where \(b\) is some constant. We may simply choose \(b = \frac{1}{N}\sum_{i=1}^N r(\tau)\).

Then, we show that adding constants to the reward function does not change the policy gradient. We have \begin{align} \mathbb E\left[\nabla_{\theta}\log\pi_{\theta}(\tau)b\right] = & \int\pi_{\theta}(\tau)\nabla_{\theta}\log\pi_{\theta}(\tau)bd\tau \newline = & b\nabla_{\theta}\int \pi_{\theta}(\tau)d\tau \newline = & b\nabla_{\theta}1 = 0. \end{align}

The previous argument shows that subtracting a baseline is unbiased in expectation. The average reward is not the best baseline, but it is often good in practice. A more careful analysis will show that \(b = \frac{g(\tau)^2r(\tau)}{\mathbb E[g(\tau)^2]}\) is the best baseline, where \(g(\tau) = \nabla_{\theta}\log\pi_{\theta}(\tau)\).

Off-policy learning and importance sampling

Policy gradient is on-policy \(\nabla_{\theta} J(\theta) = \mathbb E_{\tau\sim\pi_{\theta}(\tau)}\left[\left(\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\right)r(\tau)\right]\)

Changing the policy will change the probability distribution for the expectation. Neural networks change only a litle bit with rach gradient step, and on-policy learning can be extremely inefficient.

Now recall that \(J(\theta) = \mathbb E_{\tau\sim \pi_{\theta}(\tau)}[r(\tau)]\), what if we don’t have samples from \(\pi_{\theta}(\tau)\) but have samples from $$\bar\pi(\tau) instead?

Importance sampling \begin{align} \mathbb E_{s\sim p(x)}[f(x)] =& \int p(x)f(x)dx \newline =& \int\frac{q(x)}{q(x)}p(x)f(x)dx \newline =& \int q(x)\frac{p(x)}{q(x)}f(x)dx \newline =& \mathbb E_{x\sim q(x)}\left[\frac{p(x)}{q(x)}f(x)\right] \end{align}

Now using the importance sampling,

\(J(\theta) = \mathbb E_{\tau\sim\bar\pi(\tau)}\left[\frac{\pi_{\theta}(\tau)}{\hat\pi(\tau)}r(\tau)\right]\).

Deriving the policy gradient with importance sampling

can we estimate the value of some new parameters \(\theta'\)?

Taking the gradient

When \(\theta = \theta'\), it is exactly the policy gradient. If \(\theta \neq\theta'\),

\(\nabla_{\theta'}J(\theta') =\mathbb E_{\tau\sim\bar\pi(\tau)}\left[\frac{\pi_{\theta'}(\tau)}{\frac{\pi_{\theta}(\tau)}\nabla_{\theta'}\log\pi_{\theta'}(\tau)r(\tau)\right]\\ =\mathbb E_{\tau\sim\bar\pi(\tau)}\left[\left(\prod_{t=1}^T\frac{ \pi_{\theta'}(a_{t}|s_{t})}{\pi_{\theta}(a_{t}|s_{t})}\right)\left(\nabla_{\theta'}\log\pi_{\theta'}(a_{i,t}|s_{i,t})\right)\left(\sum_{t=1}^T r(s_{i,t},a_{i,t})\right)\right]\\ =\mathbb E_{\tau\sim\bar\pi(\tau)}\left[\sum_{t=1}^T\nabla_{\theta'}\log\pi_{\theta'}(a_{i,t}|s_{i,t})\left(\prod_{t'=1}^t\frac{ \pi_{\theta'}(a_{t'}|s_{t'})}{\pi_{\theta}(a_{t'}|s_{t'})}\right)\left(\sum_{t''=t}^{t'} r(s_{t'},a_{t'})\right)\left(\prod_{t''=t}^{t'}\frac{ \pi_{\theta'}(a_{t''}|s_{t''})}{\pi_{\theta}(a_{t''}|s_{t''})}\right)\right]\) where the last line uses causality.

Policy gradient with automatic differentiation

Just implement a “pseudo-loss” as a weighted maximum likelihood:

Actor-Critic Introduction

Review of policy gradient and intuition for actor-critic

\(\hat Q_{i,t}\): estimate of expected reward if we take action \(a_{i,t}\) in state \(s_{i,t}\). Can we get a better estimate?

What about the baseline? Previously, we set \(b = \frac{1}{N}\sum_i\hat Q_{i,t}\), and now we want to set the baseline \(b_t = \frac{1}{N}\sum_i Q(s_{i,t},a_{i,t})\).

Note that the Q-functions, the value functions are all related to a policy \(\pi\), and we have

\(A^{\pi}(s_{t},a_{t}) = Q^{\pi}(s_{t},a_{t}) - V^{\pi}(s_{t})\): the advantage, how much \(a_{t}\) is.

Then,

The better the estimate, the lower the variance.

Value function fitting What to fit? \(Q^{\pi},V^{\pi},A^{\pi}\)

where the last step is unbiased. Then, we can write out the advantage \(A^{\pi}\) as

Let’s fit \(V^{\pi}\). We use a neural network to represent \(\hat V^{\pi}(s)\) with parameter \(\phi\). Currently, \(\phi\) and \(\theta\), the parameters for the policy and the advantage are not the same (Later, combine them).

Policy evaluation

how can we perform policy evaluation? Monte Carlo policy evaluation (this is what policy gradient does)

Need to “reset” the simulators.

Monte Carlo evaluation with function approximation

not as good as \(V^{\pi}(s_{t})\approx\frac{1}{N}\sum_{i=1}^N\sum_{t'=t}^T r(s_{t'},a_{t'})\) but is still pretty good.

Training data: \(\{(s_{i,t},\sum_{t'=t}^T r(s_{i,t'},a_{i,t'}))\} = \{(s_{i,t},y_{i,t})\}\)

Can we do better?

The last step means to directly use the previous fitted value function. Smaller variance.

Monte Carlo target: \(y_{i,t} = \sum_{t'=t}^T r(s_{i,t'},a_{i,t'})\)

Training data: \(\{(s_{i,t},r(s_{i,t},a_{i,t}) + \hat V^{\pi}_{\phi})\} = \{(s_{i,t},y_{i,t})\}\)

This is sometimes referred to as a “bootstrapped” estimate.

Actor-critic algorithm

batch actor-critic algorithm:

1. sample \(\{s_i,a_i\}\) from $$\pi_{\theta}(a_{t}

   s_{t})$$ (run the policy)

2. fit \(\hat V^{\pi}_{\phi}(s)\) to sampled reward sums
3. evaluate \(\hat A^{\pi}(s_i,a_i) = r(s_i,a_i) + \hat V^{\pi}_{\phi}(s'_i) - \hat V^{\pi}_{\phi}(s_i)\)
4. \[\nabla_{\theta} J(\theta)\approx \frac{1}{N}\sum_{i=1}^N\left(\nabla_{\theta}\log\pi_{\theta}(a_{i,t}|s_{i,t})\right)\hat A^{\pi}(s_{i},a_{i})\]
5. \[\theta\leftarrow\theta + \alpha\nabla_{\theta}J(\theta)\]

Here, recall that we have training data: \(\{(s_{i,t},r(s_{i,t},a_{i,t}) + \hat V^{\pi}_{\phi})\} = \{(s_{i,t},y_{i,t})\}\)

Aside: discount factors

When the trajectories are likely to go to infinity or become very large, use a discout factor. We set

\(y_{i,t} \approx r(s_{i,t},a_{i,t}) + \gamma\hat V^{\pi}_{\phi})\), discount factor \(\gamma\in [0,1]\) (and 0.99 works well)

with critic

batch actor-critic algorithm with discount:

1. sample \(\{s_i,a_i\}\) from $$\pi_{\theta}(a_{t}

   s_{t})$$ (run the policy)

2. fit \(\hat V^{\pi}_{\phi}(s)\) to sampled reward sums
3. evaluate \(\hat A^{\pi}(s_i,a_i) = r(s_i,a_i) + \gamma\hat V^{\pi}_{\phi}(s'_i) - \hat V^{\pi}_{\phi}(s_i)\)
4. \[\nabla_{\theta} J(\theta)\approx \frac{1}{N}\sum_{i=1}^N\left(\nabla_{\theta}\log\pi_{\theta}(a_{i,t}|s_{i,t})\right)\hat A^{\pi}(s_{i},a_{i})\]
5. \[\theta\leftarrow\theta + \alpha\nabla_{\theta}J(\theta)\]

online actor-critic algorithm with discount:

1. take action $$a\sim \pi_{\theta}(a

   s)\(, get\)(s,a,a’,r)$$

2. update \(\hat V^{\pi}_{\phi}\) using target \(r+\gamma \hat V^{\pi}_{\phi}(s')\)
3. evaluate \(\hat A^{\pi}(s,a) = r(s,a) + \gamma\hat V^{\pi}_{\phi}(s') - \hat V^{\pi}_{\phi}(s)\)
4. \[\nabla_{\theta} J(\theta)\approx \left(\nabla_{\theta}\log\pi_{\theta}(a|s)\right)\hat A^{\pi}(s,a)\]
5. \[\theta\leftarrow\theta + \alpha\nabla_{\theta}J(\theta)\]

Architecture design

two network design: simple and stable, but not feature sharing

one network design: the output contains 2 networks, harder to tune the hyper-parameters

online actor-critic algorithm in practice

online actor-critic will not work for deep neural networks, in step 2, would like to update the critic by a batch of data. In practice, get the data in parallel.

Critics as state-dependent baselines.

For the actor-critic algorithm, the gradient is estimated by

The good thing is this representation has low variance (due to the critic), but it is not unbiased (if the critic is not perfect).

For the original policy gradient

This method has no bias, but has a higher variance (because of the single-sample estimate). Combine them together

There is no bias and has a lower variance.

n-step estimation

choosing \(n > 1\) often works better

Generalized advantage estimation

\(\hat A^{\pi}_{GAE}(s_{t},a_{t}) = \sum_{n=1}^{\infty}w_n\hat A^{\pi}_{n}(s_{t},a_{t})\),

can choose \(w_n\prop \lambda^{n-1}\)

References

[1] CS 285 at UC Berkeley

In this post, we will continue on our discuss of mirror descent. We will present a variant of mirror descent: the lazy mirror descent, also known as Nesterov’s dual averaging.

Lazy Mirror Descent(Nesterov’s Dual Averaging)

Algorithm

In this section, we will provide a more efficient version of mirror descent, named Lazy Mirror Descent. We will show that the convergence rate is the same as that of the original mirror descent(omit constants). The lazy mirror descent changes the update process of \(y^{(k)}\) from

into

Moreover, \(y^{(0)}\) is such that \(\nabla \Phi(y^{(0)}) = 0\).

The Lazy Mirror Descent algorithm works as follow:

1. Initial point \(x^{(0)} \in \mathbb R^n\), step size \(h > 0\).
2. For \(k=0,1,2,\dots\) do
3. Pick any \(g^{(k)}\in\partial f(x^{(k)})\).
4. \[\nabla \Phi(y^{(k+1)}) = \nabla \Phi(y^{(k)}) - \eta\cdot g^{(k)}, \nabla \Phi(y^{(0)}) = 0.\]
5. \[x^{(k+1)} = \Pi_{\mathcal X}^{\Phi}(y^{(k+1)}) := \arg\min_{x\in \mathcal X\cap\mathcal D}D_{\Phi}(x,y^{(k+1)}).\]

Analysis

This analysis is adapted from Convex Optimization: Algorithms and Complexity. We have the following theorem.

Let \(\Phi\) be a mirror map \(\rho\)-strongly convex on \(\mathcal X\cap\mathcal D\) w.r.t \(||\cdot||\) . Let \(R^2 = \sup_{x\in\mathcal X\cap\mathcal D}\Phi(x) - \Phi(x^{(0)})\), and \(f\) be convex and \(L\)-Lipschitz w.r.t \(||\cdot||\) . Then the lazy mirror descent with \(\eta = \frac{R}{L}\sqrt{\frac{\rho}{2t}}\) satisfies

\[f\left(\frac{1}{t}\sum_{s=0}^{t-1}x^{(s)}\right) - f(x^*) \le 2RL\sqrt{\frac{2}{\rho t}}.\]

Proof: We define \(\Psi_t(x) = \eta \sum_{s=0}^{t-1}g_s^Tx + \Phi(x)\), so that \(x^{(t)}\in\arg\min_{\mathcal X\cap\mathcal D}\Psi_{t-1}(x)\). Since \(\Phi\) is \(\rho\)-strongly convex, we have \(\Psi_t\) is \(\rho\)-strongly convex.

Then, we can compute

\begin{align} \Psi_t(x^{(t+1)}) - \Psi_t(x^{(t)}) \le& \nabla \Psi_{t}(x^{(t+1)})^T(x^{(t+1)} - x^{(t)}) - \frac{\rho}{2}||x^{(t+1)} - x^{(t)}||^2 \newline \le& - \frac{\rho}{2}||x^{(t+1)} - x^{(t)}||^2, \end{align}

where the second inequality comes from the first order optimality condition for \(x^{(t+1)}\). Next, we observe that

\begin{align} \Psi_t(x^{(t+1)}) - \Psi_t(x^{(t)}) =& \Psi_{t-1}(x^{(t+1)}) - \Psi_{t-1}(x^{(t)}) + \eta (g^{(t)})^T(x^{(t+1)} - x^{(t)})\newline \ge& \eta (g^{(t)})^T(x^{(t+1)} - x^{(t)}). \end{align}

Putting together the two above displays and using Cauchy-Schwarz (with the assumption \(||g^{(t)}||_{*} \le L\) ) one obtains

This shows that \(||x^{(t+1)} - x^{(t)}|| \le \frac{2\eta L}{\rho}\) and thus with the above display

Then we proof the following result: for every \(x\in\mathcal X\cap\mathcal D\),

which would conclude the proof.

The above equation is equivalent to

We prove this by induction,

\begin{align} &\sum_{s=0}^{t-1} (g^{(s)})^T x^{(s+1)} + \frac{\Phi(x^{(0)})}{\eta} \newline \le & (g^{(s)})^Tx^{(t)} + \sum_{s=0}^{t-2} (g^{(s)})^Tx^{(t)} + \frac{\Phi(x^{(t)})}{\eta}\newline \le & \sum_{s=0}^{t-1} (g^{(s)})^Tx + \frac{\Phi(x)}{\eta}, \end{align}

where the first inequality follows from the induction step and the second follows from the definition of \(x^{(t)}\).

⬜

In this post, we describe a new geometry dependent algorithm that relies on different set of assumptions. The algorithm is called conditional gradient descent, aka Frank-Wolfe.

Frank-Wolfe

Algorithm

Frank-Wolfe algorithm solves the following convex optimization problem

for \(f\) such that \(\nabla f(x)\) is `Lipschitz’ in a certain sense. The algorithm reduces the problem into a sequence of linear optimization problems.

Frank-Wolfe algorithm has the following procedure:

1. Initial point \(x^{(0)}\in\mathbb R^n\), step size \(h>0\).
2. For \(k=0,1,\dots\) do
3. Compute \(y^{(k)} = \arg\min_{y\in\mathcal D}\langle y, \nabla f(x^{(k)})\rangle\).
4. \(x^{(k+1)} \leftarrow (1-h_k)x^{(k)} + h_ky^{(k)}\) with \(h_k = \frac{2}{k+2}\).

Analysis

We have the following theorem for Frank-Wolfe algorithm.

Given a convex function \(f\) on a convex set \(\mathcal D\) and a constant \(C_f\) such that

\[f((1-h)x + hy)\le f(x) + h\langle \nabla f(x), y-x\rangle + \frac{1}{2}C_f h^2\]

for any \(x,y\in\mathcal D\) and \(h\in [0,1]\), we have

\[f(x^{(k)}) - f(x^*) \le \frac{2C_f}{k+2}.\]

Proof: By the definition of \(C_f\), we have

Note that from the convexity of \(f\) and the definition of \(y^{(k)}\), we have

Hence, we have

Let \(\epsilon_k = f(x^{(k)}) - f(x^*)\), we have

Note that \(\epsilon_0 = f^{(0)} - f(x^*) \le \frac{1}{2}C_f\), we can prove the theorem by induction.

⬜

Note that if \(\nabla f(x)\) is L-Lipschitz with respect to \(||\cdot||\), over the domain \(\mathcal D\), then \(C_f \le L\cdot diam_{||\cdot||}(\mathcal D)^2\).

Sparsity Analysis

For the domain \(\mathcal D\) is a simplex, the step \(y^{(k)}\) is always a vertex of the simplex, and hence, each step of Frank-Wolfe increases the sparsity by at most \(1\). We can think that the convergence result of Frank Wolfe proves that there is an approximate sparse solution of the optimization problem.

Given a polytope \(\mathcal P = conv(v_i)\) lies inside an unit ball. For any \(u\in\mathcal P\), there are \(k = O(\frac{1}{\epsilon^2} )\) vertices \(v_1,\dots, v_k\) of \(\mathcal P\) such that

\[||\sum_{i=1}^k\lambda_i v_i -u||_2 \le \epsilon,\]

for some \(\sum_i \lambda_i = 1, \lambda_i \ge 0\).

Proof: Run Frank-Wolfe algorithm with \(f(x) = ||x-u||_2^2\). Note that \(f\) is 1-Lipschitz with respect to \(||x||_2\) and that the diameter of \(\mathcal P\) is bounded by \(1\). Hence, we have \(C_f \le 1\).

Therefore, Frank-Wolfe algorithm shows that

Since \(f(u) = 0\), we have \(||x^{(k)} - u||^2 \le \frac{4}{k+2}\), and this gives the result.

⬜

In this post, we will introduce the Mirror Descent algorithm that solves the convex optimization algorithm.

Mirror Descent

Intuition and Algorithm

In the previous analysis for projected gradient descent, we can find that the PGD works for arbitrary Hilbert space \(\mathcal H\). Suppose that now we want to optimize functions in Banach space \(\mathcal B\), and the gradient descent does not make sence. In fact, \(x^{(t)}\) lies in space \(\mathcal B\) and the gradient \(\nabla f(x^{(t)})\) lies in the dual space \(\mathcal B^*\). There is no problem in the Hilbert space \(\mathcal H\), since by Riesz representation theorem \(\mathcal H^*\) is isometric to \(\mathcal H\).

The great insight of Nemirovski and Yudin(creator of Mirror Descent) is that one can still do a gradient descent by first mapping(by a function \(\Phi\) called mirror map) the point \(x\in\mathcal B\) into the dual space \(\mathcal B^*\), performing the gradient update in dual space, and finally mapping back to \(\mathcal B\) and projecting to the constraint set \(\mathcal X\). We will first introduce the mirror map and then discuss the projection step.

Mirror map: Let \(\mathcal D\subset\mathbb R^n\) be a convex open set such that \(\mathbb X\) is included in its closure, that is \(\mathcal X \subset \mathcal{\bar D}\) and \(\mathcal X\cap\mathcal D\neq\phi\). We say that \(\Phi:\mathcal D\to\mathbb R\) is a mirror map if it satisfies the following properties:

1. \(\Phi\) is strictly convex and differentiable.
2. The gradient of \(\Phi\) takes all possible values, that is \(\nabla\Phi(\mathcal D) = \mathbb R^n\).
3. The gradient of \(\Phi\) diverges on the boundary of \(\mathcal D\).

The projection operator is designed by the following Bergman divergence.

Bregman divergence: For any strictly convex function \(\Phi\), we define the Bregman divergence as

\[D_{\Phi}(y,x) := \Phi(y) - \Phi(x) - \langle\nabla \Phi(x),y-x\rangle.\]

And we define the projection of \(y\) as

The Mirror Descent algorithm works as follow:

1. Initial point \(x^{(0)} \in \mathbb R^n\) such that \(\nabla \Phi(x^{(0)}) = 0\), step size \(h > 0\).
2. For \(k=0,1,2,\dots\) do
3. Pick any \(g^{(k)}\in\partial f(x^{(k)})\).
4. \[\nabla \Phi(y^{(k+1)}) = \nabla \Phi(x^{(k)}) - \eta\cdot g^{(k)}.\]
5. \[x^{(k+1)} = \Pi_{\mathcal X}^{\Phi}(y^{(k+1)}) := \arg\min_{x\in \mathcal X\cap\mathcal D}D_{\Phi}(x,y^{(k+1)}).\]

Below is the illustration for the mirror descent(Picture from Convex Opitmization:Algorithms and Complexity)

Actually, one can also rewrite mirror descent as follows:

\begin{align} x^{(t+1)} =& \arg\min_{x\in\mathcal X\cap\mathcal D}D_{\Phi}(x,y^{(t+1)}) \newline =& \arg\min_{x\in\mathcal X\cap\mathcal D}(\Phi(x) - \Phi(y^{(t+1)}) - \nabla \Phi(y^{(t+1)})^Tx) \newline =& \arg\min_{x\in\mathcal X\cap\mathcal D}(\Phi(x) - (\nabla \Phi (x^{(t)}) - \eta (g^{(t)})^Tx)) \newline =& \arg\min_{x\in\mathcal X\cap\mathcal D}(\eta (g^{(t)})^Tx + D_{\Phi}(x,x^{(t)})) \end{align}

This expression is often taken as the definition of mirror descent. It gives a proximal point of view on mirror descent.

Analysis of Mirror Descent

We first have the following property:

\[(\nabla f(x) - \nabla f(y))^T(x-z) = D_f(x,y) + D_f(z,x) - D_f(z,y).\]

Proof: The proof will be completed by trivial computation.

\begin{align} RHS =& f(x) - f(y) - \nabla f(y)^T(x-y) + f(z) - f(x) \newline &\quad - \nabla f(x)^T(z-x) - f(z) + f(y) + \nabla f(y)^T(z-y) \newline =& \nabla f(x)^T(x-z) - \nabla f(y)^T(x-z) \newline =& LHS. \end{align}

⬜

Then we have the following Pythagorean Theorem.

Pythagorean Theorem: Let \(x\in\mathcal X\cap\mathcal D\) and \(y\in\mathcal D\), then

\[(\nabla\Phi(\Pi_{\mathcal X}^{\Phi}(y)) - \nabla\Phi(y))^T(\Pi_{\mathcal X}^{\Phi}(y)-x)\le 0,\]

which also implies

\[D_{\Phi}(x,\Pi_{\mathcal X}^{\Phi}(y)) + D_{\Phi}(\Pi_{\mathcal X}^{\Phi}(y),y) \le D_{\Phi}(x,y).\]

Proof: First, it is easy to show that

Then it is equivalent to show that

Let \(D_{\Phi}(\cdot,y) := f(\cdot)\), then \(\Pi_{\mathcal X}^{\Phi}(y)\) is the minimizer of \(f\) in set \(\mathcal X\cap\mathcal D\). Then from the optimality of convex functions, we have

which is just the inequality

⬜

Then, we have the following theorem:

Let \(\Phi\) be a mirror map \(\rho\)-strongly convex on \(\mathcal X\cap\mathcal D\) w.r.t \(||\cdot||\) . Let \(R^2 = \sup_{x\in\mathcal X\cap\mathcal D}\Phi(x) - \Phi(x^{(0)})\), and \(f\) be convex and \(L\)-Lipschitz w.r.t \(||\cdot||\) . Then the mirror descent with \(\eta = \frac{R}{L}\sqrt{\frac{2\rho}{t}}\) satisfies

\[f\left(\frac{1}{t}\sum_{s=0}^{t-1}x^{(s)}\right) - f(x^*) \le RL\sqrt{\frac{2}{\rho t}}.\]

Proof: Let \(x\in\mathcal X\cap\mathcal D\). The claimed bound will be obtained by taking a limit \(x\to x^*\). By the convexity of \(f\), the definition of mirror descent, and the previous 2 lemmas, we have

\begin{align} &f(x^{(s)}) - f(x) \newline \le& (g^{(s)})^T(x^{(s)} - x) \newline =& \frac{1}{\eta}(\nabla \Phi(x^{(s)}) - \nabla\Phi(y^{(s+1)}))^T(x^{(s)} - x) \newline =& \frac{1}{\eta}\left(D_{\Phi}(x,x^{(s)})+D_{\Phi}(x^{(s)},y^{(s+1)}) - D_{\Phi}(x,y^{(s+1)})\right) \newline \le & \frac{1}{\eta}\left(D_{\Phi}(x,x^{(s)})+D_{\Phi}(x^{(s)},y^{(s+1)}) - D_{\Phi}(x,x^{(s+1)}) - D_{\Phi}(x^{(s+1)},y^{(s+1)})\right). \end{align}

The term \(D_{\Phi}(x,x^{(s)})-D_{\Phi}(x,x^{(s+1)})\) will lead to a telescopic sum, and it remains to bound the other term. By the \(\rho\)-strongly convexity of the mirror map and \(az-bz^2 \le \frac{a^2}{4b}\), we have

\begin{align} & D_{\Phi}(x^{(s)},y^{(s+1)}) - D_{\Phi}(x^{(s+1)},y^{(s+1)}) \newline =& \Phi(x^{(s)}) - \Phi(x^{(s+1)}) - \nabla \Phi(y^{(s+1)})^T(x^{(s)} - x^{(s+1)}) \newline \le& (\nabla\Phi(x^{(s)})- \nabla \Phi(y^{(s+1)}))^T(x^{(s)} - x^{(s+1)}) - \frac{\rho}{2}||x^{(s)} - x^{(s+1)}||^2 \newline =& \eta (g^{(s)})^T(x^{(s)} - x^{(s+1)}) - \frac{\rho}{2}||x^{(s)} - x^{(s+1)}||^2 \newline \le& \eta L ||x^{(s)} - x^{(s+1)}|| - \frac{\rho}{2}||x^{(s)} - x^{(s+1)}||^2 \newline \le& \frac{(\eta L)^2}{2\rho}. \end{align}

Then we have

which concludes the proof up to trivial computation.

⬜

Standard Setups for Mirror Descent

Ball setup: Taking \(\Phi(x) = \frac{1}{2}||x||^2\) on \(\mathcal D = \mathbb R^n\), the function \(\Phi\) is a mirror map and strongly convex w.r.t \(||\cdot ||_2\), and furthermore, the associated Bregman divergence is given by

\begin{align} D_{\Phi}(x,y) =& \Phi(x) - \Phi(y) - \nabla\Phi(y)^T(x-y) \newline =& \frac{||x||^2}{2} - \frac{||y||^2}{2} - y^T(x-y) \newline =& \frac{1}{2}||x-y||^2. \end{align}

In this case, the mirror descent is exactly equivalent to projected subgradient descent.

Simplex setup: Now we focus on case where \(\Phi(x) = \sum x_i\log x_i\) and the simplex \(\mathcal D = \{x_i\ge 0, \sum x_i = 1\}\).

Step formula: From computation, we can know that

for some normalization constant \(Z\). Note that this update formula is just the multiplicative weight updating.

Strong convexity: From computation, we can know that \(\Phi(x)\) is 1-strongly convex w.r.t 1-norm, i.e.

Hence, we have \(\rho = 1\).

Diameter: Direct computation will show that \(-\log n\le\Phi(x)\le 0\). Hence, \(R^2 \le \log n\).

Final result: We have the following theorem.

Let \(f\) be 1-Lipschitz functions on \(||\cdot||_1\). Then, the mirror descent with the mirror map discussed above outputs \(x\) in \(T\) iterations such that

\[f(x) - \min_x f(x) \le \sqrt{\frac{2\log n}{T}}.\]

In comparison, projected gradient descent gives \(\sqrt{\frac{n}{T}}\), which is much larger.

Related to Online Convex Optimization

Note that the proofs of projected subgradient descent in the previous post and the mirror descent in this post do not require that we are solving the same convex function \(f\). Based on this observation, we can easily extend the previous results to the online setting. For the online mirror descent(online subgradient descent is a special case of online mirror descent), we have

which shows that the regret is \(O(\sqrt{t})\).

In this post, we will continue our analysis for gradient descent. Different from the previous post, we will not assume that the function is smooth. We will only assume that the function is convex and has some Lipschitz constant.

In the previous lecture, we assume that all of the functions has \(L\)-Lipschitz gradient. For general \(L\)-smooth functions, the gradient descent algorithm can get a first order \(\epsilon\)-critical proint in \(O(\frac{1}{\epsilon^2})\) iteration. When the function is convex, we show that we can use \(O(\frac{1}{\epsilon})\) iterations to get a solution differ from the optimal for at most \(\epsilon\). When the function is strongly convex and smooth, we show that the number of iterations can be reduced to \(O(poly(\log \frac{1}{\epsilon}))\).

However, in the previous post, we assume that the function is smooth, which implies that the function has gradient at all points. In this post, we will first assume that the function is convex but not smooth. Besides, in the previous post, we also focus on the unconstraint case, and in this posts, we will also introduce the analysis for constraint minimization.

Projected Subgradient Descent

In this post, we assume that the convex optimization problem has the following form:

In the problem, \(\mathcal X\) is the constraint set, which could be \(\mathbb R^n\).

Subgradient: Let \(\mathcal X\subset \mathbb R^n\), and \(f:\mathcal X\to \mathbb R\). Then \(g\in\mathbb R^n\) is a subgradient of \(f\) at \(x\in\mathcal X\) if for any \(y\in\mathcal X\), one has

\[f(y) \ge f(x) + g^T(y-x).\]

We use \(\partial f(x)\) to denote the set of subgradient at \(x\), i.e.

\[\partial f(x) := \{g\in\mathbb R^n: g \text{ is a subgradient of } f \text{ at } x\}.\]

Note that if \(f\) is differentiable at point \(x\), then \(g\) is just \(\nabla f(x)\). So the notion of subgradient is compatible with that of the original gradient.

We will also introduce the projection operator \(\Pi_{\mathcal X}\) on \(\mathcal X\) by

We have the following lemma for projection operator.

Let \(x\in\mathcal X\) and \(y\in\mathbb R^n\), then

\[(\Pi_{\mathcal X}(y) - x)^T(\Pi_{\mathcal X}(y) - y)\le 0,\]

which also implies \(||\Pi_{\mathcal X}(y) - x||^2 + ||y-\Pi_{\mathcal X}(y)||^2 \le ||y-x||^2.\)

Then, we will introduce the projected gradient descent algorithm. The algorithm works as follow:

1. For \(t = 1,2,\dots\)
2. \(y^{(t+1)} = x^{(t)} - \eta g_t, g_t\in\partial f(x^{(t)})\) and \(x^{(t+1)} = \Pi_{\mathcal X}(y^{(t+1)})\)

Analysis for Lipschitz Functions

Suppose \(f\) is convex. Furthermore, suppose that for any \(x\in\mathcal X, g\in\partial f(x)\), we have \(||g|| \le L\), then the projected subgradient method with \(\eta = \frac{R}{L\sqrt{t}}\) satisfies

\[f\left(\frac{1}{t}\sum_{s=1}^t x_s\right) - f(x^*) \le \frac{RL}{\sqrt{t}}.\]

Proof: By the convexity of \(f\), we have

\begin{align} &f(x^{(s)}) - f(x^) \newline \le& g_s(x^{(s)} - x^) \newline =& \frac{1}{\eta}(x^{(s)} - y^{(s+1)})^T(x^{(s)} - x^) \newline =& \frac{1}{2\eta}(||x^{(s)} - x^||^2 + ||x^{(s)} - y^{(s+1)}||^2 - ||x^{} - y^{(s+1)}||^2) \newline =& \frac{1}{2\eta}(||x^{(s)} - x^||^2 - ||x^{*} - y^{(s+1)}||^2) + \frac{\eta}{2}||g_s||^2. \end{align}

Now, from \(||g|| \le L\) and \(||x^{*} - y^{(s+1)}||^2 \ge ||x^{*} - x^{(s+1)}||^2\), we have

Summing up the equations, we complete the proof.

⬜

Analysis for Smooth Functions(Lipschitz Gradient)

In this section, we assume that \(f\) is convex and \(\beta\)-smooth. In the previous post when the optimization problem is not constraint, we have the following inequality(change parameters)

However, this inequality may not be true in the constraint case, since we need to apply the projection operation. The next lemma shows the `right’ quantity to measure the descent procedure.

Let \(x,y\in\mathcal X,x^{+} = \Pi_{\mathcal X}(x-\frac{1}{\beta}\nabla f(x))\) and \(g_{\mathcal X}(x) = \beta (x-x^{+})\). Then the following holds true:

\[f(x^+) - f(y) \le g_{\mathcal X}(x)^T(x-y) - \frac{1}{2\beta}||g_{\mathcal X}(x)||^2.\]

Proof: We first observe that

since the above inequality is equivalent to

Then, we have

\begin{align} &f(x^+) - f(y) \newline =& f(x^+) -f(x) + f(x) - f(y) \newline \le& \nabla f(x)^T(x^+ - x) + \frac{\beta}{2}||x^+ - x||^2 + \nabla f(x)^T(x-y) \newline =& \nabla f(x)^T(x^+ - y) + \frac{1}{2\beta}||g_{\mathcal X}(x)||^2 \newline \le& g_{\mathcal X}(x)^T(x^+ - y) + \frac{1}{2\beta}||g_{\mathcal X}(x)||^2 \newline =& g_{\mathcal X}(x)^T(x - y) - \frac{1}{2\beta}||g_{\mathcal X}(x)||^2. \end{align}

⬜

Now we can show the following theorem for the convergence of PGD.

Let \(f\) be convex and \(\beta\)-smooth on \(\mathcal X\). Then projected gradient descent with \(\eta = \frac{1}{\beta}\) satisfies

\[f(x^{(t)}) - f(x^*) \le \frac{3\beta ||x^{(1)} - x^*||^2 + f(x^{(1)}) - f(x^*)}{t}.\]

Proof: From the previous lemma, we have

and

Then we show that \(||x^{(s)} - x^*||\) is decreasing with \(s\). From the previous lemma, we can also find

then we have

\begin{align} &||x^{(s+1)} - x^||^2 \newline =& ||x^{(s)} - \frac{1}{\beta}g_{\mathcal X}(x^{(s)}) - x^||^2 \newline =& ||x^{(s)} - x^||^2 - \frac{2}{\beta}g_{\mathcal X}(x^{(s)})^T(x^{(s)} - x^) + \frac{1}{\beta^2}||g_{\mathcal X}(x^{(s)})||^2\newline \le& ||x^{(s)} - x^*||^2. \end{align}

Let \(\epsilon_s = f(x^{(s)}) - f(x^*)\), we have

\begin{align} \epsilon_{s+1} \le& \epsilon_s - \frac{1}{2\beta}||g_{\mathcal X}(x^{(s)})||^2 \newline \le& \epsilon_s - \frac{1}{2\beta||x^{(s)} - x^||^2}\epsilon_{s+1}^2 \newline \le& \epsilon_s - \frac{1}{2\beta||x^{(1)} - x^||^2}\epsilon_{s+1}^2. \end{align}

Then we can finish the proof by simple induction.

⬜

Analysis for Strongly Convex Functions

In this section, we consider the projected gradient descent with time-varying step size \((\eta_t)_{t\ge 1}\), that is

1. For \(t = 1,2,\dots\)
2. \(y^{(t+1)} = x^{(t)} - \eta_t g_t, g_t\in\partial f(x^{(t)})\) and \(x^{(t+1)} = \Pi_{\mathcal X}(y^{(t+1)})\)

Then we have the following theorem

Let \(f\) be \(\alpha\)-strongly convex and \(L\)-Lipschitz on \(\mathcal X\). Then projected subgradient descent with \(\eta_s = \frac{2}{\alpha (s+1)}\) satisfies,

\[f\left(\sum_{s=1}^t\frac{2s}{t(t+1)}x_s\right) - f(x^*) \le \frac{2L^2}{\alpha(t+1)}.\]

Proof: Similar with the projected subgradient descent for lipschitz functions, we have

Multiplying the inequalities by \(s\), we have

Summing up the above inequalities and applying the Jensen’s inequality, we complete the proof.

⬜

Analysis for Strongly Convex and Smooth Functions

The key improvement compared with convex and smooth function is that, one can show

Based on this result, we have the following theorem

Let \(f\) be \(\alpha\)-strongly convex and \(\beta\)-smooth on \(\mathcal X\). Then projected gradient descent with \(\eta = \frac{1}{\beta}\) satisfies,

\[||x^{(t+1)} - x^*||^2 = \left(1-\frac{\alpha}{\beta}\right)^t||x^{(1)} - x^*||^2.\]

Proof: Using the previous inequality with \(y = x^*\), we have

\begin{align} & ||x^{(t+1)} - x^||^2 \newline = & ||x^{(t)} - \frac{1}{\beta}g_{\mathcal X}(x^{(t)}) - x^||^2 \newline = & ||x^{(t)} - x^||^2 - \frac{2}{\beta}g_{\mathcal X}(x^{(t)})^T(x^{(t)} - x^) + \frac{1}{\beta^2}||g_{\mathcal X}(x^{(t)})||^2 \newline \le & ||x^{(t)} - x^||^2 - \frac{\alpha}{\beta}||x^{(t)} - x^||^2 \newline = & \left(1-\frac{\alpha}{\beta}\right)^t||x^{(1)} - x^*||^2. \end{align}

⬜

In this post, we will review the most basic and the most intuitive optimization method – the gradient decent method – in optimization.

Gradient Descent

The gradient descent algorithm works as follow: The algorithm requires an initial point \(x^{(0)}\in\mathbb R^n\) and step size \(h > 0\). Then the algorithm repeats to execute:

until \(||\nabla f(x^{(t)})|| \le \epsilon\). In the following of this section, we will assume that the gradient of \(f\) is L-lipschitz, i.e.

we call \(f\) is L-lipschitz gradient or L-smooth.

Analysis for General Smooth Functions

In order to show the convergence of gradient descent for the general functions, we need the following lemma,

Suppose \(f\) is L-lipschitz gradient, then

\[|f(y) - f(x) - \nabla f(x)^T(y-x)| \le \frac{L}{2}||x - y||^2.\]

Proof: From the basic calculus and the assumption that \(f\) is L-lipschitz gradient, we have

\begub{align} &|f(y) - f(x) - \nabla f(x)^T(y-x)| \newline =& \int_{0}^1 \nabla f(x + t(y-x))^T(y-x)dt - \nabla f(x)^T(y-x)\newline =& |\int_{0}^1 (\nabla f(x + t(y-x))-\nabla f(x))^T(y-x)dt| \newline \le& \int_{0}^1 ||\nabla f(x + t(y-x))-\nabla f(x)||\cdot ||y-x||dt \newline \le& \int_{0}^1 tL||y-x||^2dt \newline =& \frac{L}{2}||y-x||^2. \end{align}

⬜

The previous lemma shows and upper and lower bound of a point if the gradient of the function is lipschitz. From the previous lemma, we can then show the convergence of gradient descent.

Let \(f\) be a function with L-Lipschitz gradient and \(x^*\) be any minimizer of \(f\). The gradient descent with step size \(h = \frac{1}{L}\) outputs a point \(x\) such that \(||\nabla f(x)||\le \epsilon\) in \(\frac{2L}{\epsilon^2}(f(x^{(0)}) - f(x^*))\) iterations.

Proof: From the previous lemma, we have the following equation

\begub{align} f(x^{(t+1)}) \le& f(x^{(t)}) - \nabla f(x^{(t)})^T(x^{(t+1)} - x^{(t)}) + \frac{L}{2}||x^{(t+1)} - x^{(t)}||^2 \newline \le& f(x^{(t)}) - \nabla f(x^{(t)})^T\cdot \frac{1}{L}\nabla f(x^{(t)}) + \frac{L}{2}\frac{1}{L^2}||\nabla f(x^{(t)})||^2 \newline =& f(x^{(t)}) - \frac{1}{2L}||\nabla f(x^{(t)})||^2. \end{align}

Then we have \(||\nabla f(x^{(t)})||^2 \le 2L \cdot f(x^{(t)}) - f(x^{(t+1)}).\)

Sum up the equations for \(t=1,2,\dots\), we can get

From this equation, we can see that the gradient descent outputs a point \(x\) such that \(||\nabla f(x)||\le \epsilon\) in \(\frac{2L}{\epsilon^2}(f(x^{(0)}) - f(x^*))\) iterations.

⬜

Analysis for Convex Functions

In this section, we assume that the function \(f\), which we want to optimize, is convex with L-lipschitz gradient. As a result of the convexity, we can show that the difference between \(f(x^*)\) and \(f(x^{(t)})\) is \(O(\frac{1}{t})\). Before we state the theorem, we prove an auxilary lemma.

For any convex function \(f\in\mathcal C^1(\mathbb R^n)\), we have

\[f(x) - f(y) \le ||\nabla f(x)||_2\cdot ||x-y||_2.\]

Proof: From the first order condition, if \(f\) is convex, we have

Arranging the terms and using the basic property of norms, we have

⬜

With the help of the previous lemma, we can show the convergence bound for convex functions now.

Let \(f\in\mathcal C^2(\mathbb R^n)\) be convex with L-Lipschitz gradient and \(x^*\) be any minimizer of \(f\). With step size \(h = \frac{1}{L}\), the sequence \(x^{(k)}\) in Gradient Descent satises

\[f(x^{(k)}) - f(x^{*}) \le \frac{2LD^2}{k+4},\]

where \(D = \max_{f(x) \le f(x^{(0)})}||x-x^*||_2.\)

Proof: Let \(\epsilon_k = f(x^{(k)}) - f(x^*)\). Since \(f\) is L-lipschitz gradient, we have

Then from the previous lemma, we have

Therefore, we have

Moreover, we can also bound \(\epsilon_0\) by the following inequality,

Then we have

Then

which completes the proof.

⬜

Strongly Convex Functions

In the previous analysis for general smooth functions and convex functions, we show that the gradient descent algorithm needs \(O(\frac{1}{\epsilon^2})\) and \(O(\frac{1}{\epsilon})\) iterations to converge to an \(\epsilon\)-approximation(Note: Here, \(\epsilon\)-approximation does not mean the formal definition of \(\epsilon\)-approximation in approximation algorithms. In the sense of convergence of general functions, it means the gradient is small, and in the sense of convergence of convex functions, it means the error to the optimal is small). Now we introduce the strongly convex functions, and analysis the convergence of gradient descent algorithm for strongly convex functions. In particular, we will show that it will use \(poly(\log\frac{1}{\epsilon})\) iterations to converge to an \(\epsilon\)-approximation.

Strongly convex: We call a function \(f \in \mathcal C^1(\mathbb R^n)\) is \(\mu\)-strongly convex if for any \(x, y \in \mathbb R^n\),

\[f(y) \ge f(x) + \nabla f(x)^T(y-x) + \frac{\mu}{2}||y-x||^2.\]

The following theorem gives an equivalent definition for strongly convex funtions if the function is twice differentiable.

Strongly convex: Let \(f \in \mathcal C^2(\mathbb R^n)\). Then \(f\) is \(\mu\)-strongly convex if and only if

\[\nabla^2 f(x)\succeq \mu\cdot I,\forall x\in\mathbb R^n.\]

We will not prove this theorem. This theorem can be shown by the Taylor’s expansion in multi-variable case.

With the definition of strongly convex functions, we can now prove the following theorem.

Let \(f\in\mathcal C^2(\mathbb R^n)\) be \(\mu\)-strongly convex with L-Lipschitz gradient and \(x^*\) be any minimizer of \(f\). With step size \(h = \frac{1}{L}\), the sequence \(x^{(k)}\) in GradientDescent satises

\[f(x^{(k)}) - f(x^{*}) \le \left(1-\frac{\mu}{L}\right)^k(f(x^{(0)}) - f(x^{*})).\]

Proof: From the previous analysis, since \(f\) is L-lipschitz gradient, we have

where \(\epsilon_k = f(x^{(k)}) - f(x^*)\).

Then since \(f\) is \(\mu\)-strongly convex, we have

Rearranging the terms, we have

\begub{align} f(x^{(k)}) - f(x^) \le& \nabla f(x^{(k)})^T(x^{(k)} - x^) - \frac{\mu}{2}||x^* - x^{(k)}||^2 \newline \le& \max_{\Delta}(\nabla f(x^{(k)})^T\Delta - \frac{\mu}{2}\Delta^2)\newline =& \frac{1}{2\mu}||\nabla f(x^{(k)})||^2. \end{align}

Putting the above inequality into the first inequality in the proof, we have

\begub{align} \epsilon_{k+1} \le& \epsilon_k - \frac{1}{2L}||\nabla f(x^{(k)})||_2^2 \newline \le& \epsilon_k - \frac{\mu}{L}\epsilon_k \newline \le& \left(1-\frac{\mu}{L}\right)^{k+1}\epsilon_0. \end{align}

⬜

Recently, I find an interesting course taught by Prof. Yin Tat Lee at UW. The course is called `Theory of Optimization and Continuous Algorithms’, and the lecture notes are available under the homepage of this courseuw-cse535-winter19. As a great fan of optimization theory and algorithm design, I think I will follow this course and write a bunch of blogs to record my study of this course.

Most of the materials in this series of blogs will follow the lecture notes of the course, and and interesting optimization book Convex Optimization: Algorithms and Complexity by Sebastien Bubeck.

Since this is the first blog about this course, I will present the preliminaries of the optimization theory, and some basic knowledge about convex optimization, including some basic properties of convex functions.

Preliminaries

First, we will introduce some of the preliminaries about theory of optimization. All of the definitions will be used later.

Functions

First, we will introduce the notion of lipschitz and differentiable

Lipschitz: A function \(f:V\to W\) is L-lipschitz if

\[||f(x) - f(y)||_W \le L\cdot ||x - y||_V,\]

where the norms \(||\cdot||_{W}\) and \(||\cdot||_{V}\) are \(l_2\) norms if unspecified.

Differentiable: A function \(f\in\mathcal C^k(\mathbb R^n)\) is k-differentiable if its \(k^{th}\) derivative is continuous.

Linear Algebra

The notion of positive semi-definite and the norm of matrices are also important in the world of optimization. We will introduce these notions.

Positive semi-definite: A symmetric matrix \(A\) is positive semi-definite (PSD) if \(x^T Ax \ge 0\) for all \(x \in \mathbb R^n\). We write \(A \succeq B\) if \(A - B\) is PSD.

Trace, norm of matrix: For any matrix A, we define

\[trA = \sum A_{ii}, ||A||_F^2 = tr(A^TA), ||A||_{op} = \max_{||x||_2\le 1}||Ax||_2.\]

For symmetric matrix \(A\), we have \(trA = \sum \lambda_i, ||A||_F = \sum \lambda_i^2\) and \(||A||_{op} = \max_i | \lambda_i |\), where \(\lambda_i\) are the eigenvalues of matrix \(A\).

Calculus

As far as I know, we can get some intuitions of the design and proofs of some optimization algorithm through the Taylor’s expansion.

Taylor’s Remainder Theorem: For any \(g \in \mathcal C^{k+1}(\mathbb R)\), and any \(x\) and \(y\), there is a \(\xi\in [x, y]\) such that

\[g(y) = \sum_{j=0}^k g^{(j)}(x)\frac{(y-x)^j}{j!} + g^{(k+1)}(\xi)\frac{(y-x)^{k+1}}{(k+1)!}.\]

Probability

KL-divergence: The KL-divergence of a density \(\rho\) with respect to another density \(\nu\) is defined as

\[D_{KL}(\rho || \nu ) = \int \rho(x) \log \frac{\rho(x)}{\nu(x)}dx.\]

Ito’s lemma: For any process \(x_t \in \mathbb R^n\) satisfying \(dx_i = \mu(x_t)dt + \sigma(x_i)dW_t\), where \(\mu(x_t)\in\mathbb R^n\) and \(\sigma(x_t)\in\mathbb R^{n\times m}\), we have that

\[df(x_t) = \nabla f(x_t)^T\mu(x_t)dt + \nabla f(x_t)^T\sigma(x_t)dW_t + \frac{1}{2}tr(\sigma(x_t)^T\nabla^2f(x_t)\sigma(x_t))dt.\]

Introduction–Basic properties of convex functions

In this section, we will introduce some basic properties of convex functions, and we will also give some examples.

Definitions

Here will briefly define the convexity of sets and functions.

Convex set: A set \(K\) in \(\mathbb R^n\) is convex if for every pair of points \(x,y\in K\), we have \([x,y] \subseteq K\).

Convex function: A function \(f:\mathbb R^n\to\mathbb R\) is

1. Convex: if for any \(\lambda\in [0,1]\), we have \(f(\lambda x + (1-\lambda) y) \le \lambda f(x) + (1-\lambda) f(y)\).
2. Concave: if for any \(\lambda\in [0,1]\), we have \(f(\lambda x + (1-\lambda) y) \ge \lambda f(x) + (1-\lambda) f(y)\).
3. Logconcave: if \(f\) is nonnegative and \(\log f\) is concave.
4. Quasiconvex: if the level sets of \(f\), defined by \(\{x:f(x)\le t\}\) are convex for all \(t\).

Seperation theorems

Why is convex functions so useful? Maybe convexity is not only general enough to model different kinds of problem, and also simple enough to be solved efficiently. In this section, we will introduce the seperation theorem for convex sets, which allows us to do binary search to find a point in a convex set. This is the basis for polynomial-time algorithms for optimizing general convex functions.

Seperation of a point and a closed convex set: Let \(K\) be a closed convex set in \(\mathbb R^n\) and \(y\notin K\). There is a non-zero \(\theta\in\mathbb R^n\) such that

\[\langle \theta, y \rangle > \max_{x\in K}\langle \theta, x \rangle\]

Proof: Let \(x^*\) be a point in \(K\) closest to \(y\), i.e.

(Such a minimizer always exists for closed convex sets, this is sometimes calle Hilbert’s projection theorem). Using convexity of \(K\), for any \(x \in K\) and any \(0 \le t \le 1\) , we have that

Expand the LHS, we have

\begin{align} ||(1 - t)x^* + tx - y||^2 =& ||x^* - y + t(x - x^)||^2 \newline =& ||x^ - y||^2 + 2t\langle x^* - y, x - x^*\rangle +O(t^2). \end{align}

Taking \(t\to 0^{+}\), we have \(\langle x^* - y, x - x^*\rangle\) for all \(x\in K\).

Taking \(\theta = y - x^*\), we have

⬜

From the above theorem, we know that a polytope is as general as a convex set. As a corollary, we have

Intersection of halfspaces: Any closed convex set \(K\) can be written as the intersection of halfspaces as follows:

\[K = \bigcap_{\theta\in\mathbb R^n}\left\{x:\langle \theta,x\rangle\le\max_{y\in K}\langle\theta,y\rangle\right\}.\]

In other words, any convex set is a limit of a sequence of polyhedra. Next, we will prove a `seperation theorem’ for convex functions. Based on this theorem, one can design polynomial algorithms for optimizing convex functions.

First order definition: Let \(f\in\mathcal C^1(\mathbb R^n)\) be convex. Then for any \(x,y\in\mathbb R^n\), we have

\[f(y) \ge f(x) + \nabla f(x)^T(y-x).\]

Proof: Fix any \(x,y\in\mathbb R^n\). Let \(g(t) = f((1-t)x + ty)\). Since \(f\) is convex, so is \(g\)(any convex function is also convex when restricted to a line), and we have

which implies that

Taking \(t\to 0^{+}\), we have \(g(1) \ge g(0) + g'(0)\). Applying the chain rule, we have

⬜

Actually, the above theorem can be viewed as an equivalent definition of the convex functions. It is easy to show that, when

holds for all \(x,y\in K\)(\(K = \text{dom}f\), which is the domain of \(f\), is convex), we have \(f\) is convex.

Then from the previous theorem, we have the first order optimality condition, which is shown as follow

First order condition for unconstraint problems: Let \(f\in\mathcal C^1(\mathbb R^n)\) be convex. Then \(x\) is the minimizer of \(\min_{x\in\mathbb R^n}f(x)\) if and only if \(\nabla f(x) = 0\).

Proof: If \(\nabla f(x) = 0\), then

for small enough \(\epsilon\). Hence, such a point cannot be the minimizer.

On the other hand, if \(\nabla f(x) = 0\), the previous theorem shows that

⬜