# 数学代写|机器学习中的优化理论代写OPTIMIZATION FOR MACHINE LEARNING代考|STOR712

## 数学代写|机器学习中的优化理论代写OPTIMIZATION FOR MACHINE LEARNING代考|Iterative Soft Thresholding

We now derive an algorithm using a classical technic of surrogate function minimization. We aim at minimizing
$$f(x) \stackrel{\text { def. }}{=} \frac{1}{2}|y-A x|^2+\lambda \mid x |_1$$
and we introduce for any fixed $x^{\prime}$ the function
$$f_\tau\left(x, x^{\prime}\right) \stackrel{\text { def. }}{=} f(x)-\frac{1}{2}\left|A x-\left.A x^{\prime}\right|^2+\frac{1}{2 \tau}\right| x-x^{\prime} |^2$$
We notice that $f_\tau(x, x)=0$ and one the quadratic part of this function reads
$$K\left(x, x^{\prime}\right) \stackrel{\text { def. }}{=}-\frac{1}{2}\left|A x-A x^{\prime} |^2+\frac{1}{2 \tau}\right| x-x^{\prime} \mid=\frac{1}{2}\left\langle\left(\frac{1}{\tau} \operatorname{Id}N-A^{\top} A\right)\left(x-x^{\prime}\right), x-x^{\prime}\right\rangle .$$ This quantity $K\left(x, x^{\prime}\right)$ is positive if $\lambda{\max }\left(A^{\top} A\right) \leqslant 1 / \tau$ (maximum eigenvalue), i.e. $\tau \leqslant 1 /|A|_{\mathrm{op}}^2$, where we recall that $|\left. A\right|{\text {op }}=\sigma{\max }(A)$ is the operator (algebra) norm. This shows that $f_\tau\left(x, x^{\prime}\right)$ is a valid surrogate functional, in the sense that
$$f(x) \leqslant f_\tau\left(x, x^{\prime}\right), \quad f_\tau\left(x, x^{\prime}\right)=0, \quad \text { and } \quad f(\cdot)-f_\tau\left(\cdot, x^{\prime}\right) \text { is smooth. }$$
We also note that this majorant $f_\tau\left(\cdot, x^{\prime}\right)$ is convex. This leads to define
$$x_{k+1} \stackrel{\text { def. }}{=} \underset{x}{\operatorname{argmin}} f_\tau\left(x, x_k\right)$$
which by construction satisfies
$$f\left(x_{k+1}\right) \leqslant f\left(x_k\right)$$

## 数学代写|机器学习中的优化理论代写OPTIMIZATION FOR MACHINE LEARNING代考|Minimizing Sums and Expectation

A large class of functionals in machine learning can be expressed as minimizing large sums of the form
$$\min {x \in \mathbb{R}^P} f(x) \stackrel{\text { def. }}{=} \frac{1}{n} \sum{i=1}^n f_i(x)$$
or even expectations of the form
$$\min {x \in \mathbb{R}^p} f(x) \stackrel{\text { def. }}{=} \mathbb{E}{\mathbf{z} \sim \pi}(f(x, \mathbf{z}))=\int_{\mathcal{Z}} f(x, z) \mathrm{d} \pi(z) .$$
Problem (42) can be seen as a special case of (43), when using a discrete empirical uniform measure $\pi=$ $\sum_{i=1}^n \delta_i$ and setting $f(x, i)=f_i(x)$. One can also viewed (42) as a discretized “empirical” version of (43) when drawing $\left(z_i\right)_i$ i.i.d. according to $\mathbf{z}$ and defining $f_i(x)=f\left(x, z_i\right)$. In this setup, (42) converges to (43) as $n \rightarrow+\infty$.

A typical example of such a class of problems is empirical risk minimization for linear model, where in these cases
$$f_i(x)=\ell\left(\left\langle a_i, x\right\rangle, y_i\right) \text { and } f(x, z)=\ell(\langle a, x\rangle, y)$$
for $z=(a, y) \in \mathcal{Z}=\left(\mathcal{A}=\mathbb{R}^P\right) \times \mathcal{Y}$ (typically $\mathcal{Y}=\mathbb{R}$ or $\mathcal{Y}={-1,+1}$ for regression and classification), where $\ell$ is some loss function. We illustrate below the methods on binary logistic classification, where
$$L(s, y) \stackrel{\text { def }}{=} \log (1+\exp (-s y))$$
But this extends to arbitrary parametric models, and in particular deep neural networks.

# 机器学习中的优化理论代考

## 数学代写|机器学习中的优化理论代写OPTIMIZATION FOR MACHINE LEARNING代考|Iterative Soft Thresholding

$$f(x) \stackrel{\text { def. }}{=} \frac{1}{2}|y-A x|^2+\lambda|x|1$$ 我们介绍任何固定的 $x^{\prime}$ 功能 $$f\tau\left(x, x^{\prime}\right) \stackrel{\text { def. }}{=} f(x)-\frac{1}{2}\left|A x-A x^{\prime}\right|^2+\frac{1}{2 \tau}\left|x-x^{\prime}\right|^2$$

$$K\left(x, x^{\prime}\right) \stackrel{\text { def. }}{=}-\frac{1}{2}\left|A x-A x^{\prime}\right|^2+\frac{1}{2 \tau}\left|x-x^{\prime}\right|=\frac{1}{2}\left\langle\left(\frac{1}{\tau} \operatorname{Id} N-A^{\top} A\right)\left(x-x^{\prime}\right), x-x^{\prime}\right\rangle$$
$|A| \mathrm{op}=\sigma \max (A)$ 是运算符 (代数) 范数。这表明 $f_\tau\left(x, x^{\prime}\right)$ 是一个有效的替代功能，在这个意义上 $f(x) \leqslant f_\tau\left(x, x^{\prime}\right), \quad f_\tau\left(x, x^{\prime}\right)=0, \quad$ and $\quad f(\cdot)-f_\tau\left(\cdot, x^{\prime}\right)$ is smooth.

$$x_{k+1} \stackrel{\text { def. }}{=} \underset{x}{\operatorname{argmin}} f_\tau\left(x, x_k\right)$$

$$f\left(x_{k+1}\right) \leqslant f\left(x_k\right)$$

## 数学代写|机器学习中的优化理论代写OPTIMIZATION FOR MACHINE LEARNING代考|Minimizing Sums and Expectation

$$\min x \in \mathbb{R}^P f(x) \stackrel{\text { def. }}{=} \frac{1}{n} \sum i=1^n f_i(x)$$

$$\min x \in \mathbb{R}^p f(x) \stackrel{\text { def. }}{=} \mathbb{E} \mathbf{z} \sim \pi(f(x, \mathbf{z}))=\int_{\mathcal{Z}} f(x, z) \mathrm{d} \pi(z) .$$

$$f_i(x)=\ell\left(\left\langle a_i, x\right\rangle, y_i\right) \text { and } f(x, z)=\ell(\langle a, x\rangle, y)$$

$$L(s, y) \stackrel{\text { def }}{=} \log (1+\exp (-s y))$$

myassignments-help数学代考价格说明

1、客户需提供物理代考的网址，相关账户，以及课程名称，Textbook等相关资料~客服会根据作业数量和持续时间给您定价~使收费透明，让您清楚的知道您的钱花在什么地方。

2、数学代写一般每篇报价约为600—1000rmb，费用根据持续时间、周作业量、成绩要求有所浮动(持续时间越长约便宜、周作业量越多约贵、成绩要求越高越贵)，报价后价格觉得合适，可以先付一周的款，我们帮你试做，满意后再继续，遇到Fail全额退款。

3、myassignments-help公司所有MATH作业代写服务支持付半款，全款，周付款，周付款一方面方便大家查阅自己的分数，一方面也方便大家资金周转，注意:每周固定周一时先预付下周的定金，不付定金不予继续做。物理代写一次性付清打9.5折。

Math作业代写、数学代写常见问题

myassignments-help擅长领域包含但不是全部: