对机器学习方法的统计证明 1. 模型建立 为了给回归问题建立一个统计模型，我们做出以下假设：
y i = g ( X i ) + ϵ i y_i=g(X_i) + \epsilon_i y i = g ( X i ) + ϵ i 这个公式描述了我们观察到的数据点 ( X i , y i ) (X_i, y_i) ( X i , y i )
g ( X i ) g(X_i) g ( X i ) 在现实中，输入 X X X y y y g g g 。比如，房子的面积和它的“真实”价值之间可能有一个确定的函数关系。我们的目标就是找到一个函数 h h h g g g 。ϵ i \epsilon_i ϵ i 所有影响测量的随机因素 ，比如测量误差、数据记录错误，以及所有 g g g ϵ i \epsilon_i ϵ i 来自某个概率分布 D ′ D' D ′ E [ ϵ ] = 0 E[\epsilon] = 0 E [ ϵ ] = 0 。这意味着，从长期来看，这些随机误差会相互抵消，不会系统性地偏高或偏低。注意，在假设中，噪声分布 ϵ i \epsilon_i ϵ i X X X
然后我们再证明一个简单的结论：我们只需要知道所有 X = x X=x X = x Y Y Y E [ Y ∣ X = x ] E[Y|X=x] E [ Y ∣ X = x ] ，那它就是 g ( x ) g(x) g ( x ) y y y
E ( Y ∣ X = x ) = E ( g ( x ) + ϵ ∣ X = x ) E(Y|X=x)=E(g(x)+\epsilon | X=x) E ( Y ∣ X = x ) = E ( g ( x ) + ϵ ∣ X = x ) 由于 g ( x ) g(x) g ( x ) x x x E ( ϵ ) = 0 E(\epsilon)=0 E ( ϵ ) = 0 E ( g ( x ) + ϵ ∣ X = x ) = g ( x ) E(g(x)+\epsilon | X=x) = g(x) E ( g ( x ) + ϵ ∣ X = x ) = g ( x ) g g g
2. 对最小二乘的推导 下面我们用这个模型解释最小二乘法的合理性。我们假设噪声 ϵ i ∼ N ( 0 , σ 2 ) \epsilon_i \sim \mathcal{N}(0, \sigma^2) ϵ i ∼ N ( 0 , σ 2 ) y i y_i y i
E ( y i ) = E ( g ( X i ) + ϵ i ) = g ( X i ) + 0 = g ( X i ) E(y_i)=E(g(X_i) + \epsilon_i)=g(X_i)+0 = g(X_i) E ( y i ) = E ( g ( X i ) + ϵ i ) = g ( X i ) + 0 = g ( X i ) Var ( y i ) = Var ( ϵ i ) = σ 2 \text{Var}(y_i)=\text{Var}(\epsilon_i)=\sigma^2 Var ( y i ) = Var ( ϵ i ) = σ 2 也即 y ∼ N ( g ( X i ) , σ 2 ) y \sim \mathcal{N}(g(X_i), \sigma^2) y ∼ N ( g ( X i ) , σ 2 ) y y y
P ( y i ∣ g ( X i ) ; σ ) = 1 σ 2 π exp ( − ( y i − g ( X i ) ) 2 2 σ 2 ) P\big(y_i\mid g(X_i);\sigma\big)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{\big(y_i-g(X_i)\big)^2}{2\sigma^2}\right) P ( y i ∣ g ( X i ) ; σ ) = σ 2 π 1 exp ( − 2 σ 2 ( y i − g ( X i ) ) 2 ) 其联合概率分布就是似然函数：
L ( g ) = P ( y 1 , … , y n ∣ X ; g ) = ∏ i = 1 n P ( y i ∣ X i ; g ) = ∏ i = 1 n 1 σ 2 π exp ( − ( y i − g ( X i ) ) 2 2 σ 2 ) \begin{aligned} L(g) &= P\bigl(y_1,\dots,y_n\mid X;g\bigr) \\ &= \prod_{i=1}^n P\bigl(y_i\mid X_i;g\bigr) \\ &= \prod_{i=1}^n \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{\bigl(y_i-g(X_i)\bigr)^2}{2\sigma^2}\right) \end{aligned} L ( g ) = P ( y 1 , … , y n ∣ X ; g ) = i = 1 ∏ n P ( y i ∣ X i ; g ) = i = 1 ∏ n σ 2 π 1 exp ( − 2 σ 2 ( y i − g ( X i ) ) 2 ) 对其取对数：
ℓ ( g ) = ln L ( g ) = ln ∏ i = 1 n P ( y i ∣ X i ; g ) = ln ∏ i = 1 n 1 σ 2 π exp ( − ( y i − g ( X i ) ) 2 2 σ 2 ) = ∑ i = 1 n ln ( 1 σ 2 π exp ( − ( y i − g ( X i ) ) 2 2 σ 2 ) ) = ∑ i = 1 n [ ln 1 σ 2 π + ln exp ( − ( y i − g ( X i ) ) 2 2 σ 2 ) ] = ∑ i = 1 n [ − ln ( σ 2 π ) − ( y i − g ( X i ) ) 2 2 σ 2 ] = − n ln ( σ 2 π ) − 1 2 σ 2 ∑ i = 1 n ( y i − g ( X i ) ) 2 \begin{aligned} \ell(g) &= \ln L(g) \\ &= \ln\prod_{i=1}^n P\bigl(y_i\mid X_i;g\bigr) \\ &= \ln\prod_{i=1}^n \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(y_i-g(X_i))^2}{2\sigma^2}\right) \\ &= \sum_{i=1}^n \ln\left(\frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(y_i-g(X_i))^2}{2\sigma^2}\right)\right) \\ &= \sum_{i=1}^n \left[\ln\frac{1}{\sigma\sqrt{2\pi}} + \ln\exp\left(-\frac{(y_i-g(X_i))^2}{2\sigma^2}\right)\right] \\ &= \sum_{i=1}^n \left[-\ln\bigl(\sigma\sqrt{2\pi}\bigr) -\frac{(y_i-g(X_i))^2}{2\sigma^2}\right] \\ &= -n\ln\bigl(\sigma\sqrt{2\pi}\bigr) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i-g(X_i))^2 \end{aligned} ℓ ( g ) = ln L ( g ) = ln i = 1 ∏ n P ( y i ∣ X i ; g ) = ln i = 1 ∏ n σ 2 π 1 exp ( − 2 σ 2 ( y i − g ( X i ) ) 2 ) = i = 1 ∑ n ln ( σ 2 π 1 exp ( − 2 σ 2 ( y i − g ( X i ) ) 2 ) ) = i = 1 ∑ n [ ln σ 2 π 1 + ln exp ( − 2 σ 2 ( y i − g ( X i ) ) 2 ) ] = i = 1 ∑ n [ − ln ( σ 2 π ) − 2 σ 2 ( y i − g ( X i ) ) 2 ] = − n ln ( σ 2 π ) − 2 σ 2 1 i = 1 ∑ n ( y i − g ( X i ) ) 2 我们的目的是最大化 ℓ ( g ) \ell(g) ℓ ( g ) ∑ i = 1 n ( y i − g ( X i ) ) 2 \sum_{i=1}^n (y_i-g(X_i))^2 ∑ i = 1 n ( y i − g ( X i ) ) 2 g ( X i ) g(X_i) g ( X i ) y i y_i y i
3. 对逻辑回归的推导 我们现在处理的是分类问题，模型 h ( X i ) h(X_i) h ( X i ) X i X_i X i C C C y i y_i y i h ( X i ) h(X_i) h ( X i )
为了能够使用我们之前处理离散变量的方法（计数法），我们设计下面的思想实验：我们想象对于每一个数据点 X i X_i X i β β β β β β h ( X i ) h(X_i) h ( X i ) y i y_i y i 1 − h ( X i ) 1-h(X_i) 1 − h ( X i ) 1 − y i 1-y_i 1 − y i
P = h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β P=h(X_i)^{y_i\beta}\,(1-h(X_i))^{(1-y_i) \beta} P = h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β 对于所有的 n n n
L ( h ) = ∏ i = 1 n h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β L(h)=\prod_{i=1}^n h(X_i)^{y_i\beta}\bigl(1-h(X_i)\bigr)^{(1-y_i)\beta} L ( h ) = i = 1 ∏ n h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β 同样地，我们对上式取对数：
ℓ ( h ) = ln L ( h ) = ∑ i = 1 n ln ( h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β ) = ∑ i = 1 n [ ln ( h ( X i ) y i β ) + ln ( ( 1 − h ( X i ) ) ( 1 − y i ) β ) ] = ∑ i = 1 n [ y i β ln h ( X i ) + ( 1 − y i ) β ln ( 1 − h ( X i ) ) ] = β ∑ i = 1 n [ y i ln h ( X i ) + ( 1 − y i ) ln ( 1 − h ( X i ) ) ] \begin{aligned} \ell(h) &= \ln L(h) \\ &= \sum_{i=1}^n \ln\Big(h(X_i)^{y_i\beta}\big(1-h(X_i)\big)^{(1-y_i)\beta}\Big) \\ &= \sum_{i=1}^n \Big[\ln\big(h(X_i)^{y_i\beta}\big)+\ln\big((1-h(X_i))^{(1-y_i)\beta}\big)\Big] \\ &= \sum_{i=1}^n \Big[y_i\beta\,\ln h(X_i)+(1-y_i)\beta\,\ln(1-h(X_i))\Big] \\ &= \beta\sum_{i=1}^n \Big[y_i\ln h(X_i)+(1-y_i)\ln(1-h(X_i))\Big] \end{aligned} ℓ ( h ) = ln L ( h ) = i = 1 ∑ n ln ( h ( X i ) y i β ( 1 − h ( X i ) ) ( 1 − y i ) β ) = i = 1 ∑ n [ ln ( h ( X i ) y i β ) + ln ( ( 1 − h ( X i ) ) ( 1 − y i ) β ) ] = i = 1 ∑ n [ y i β ln h ( X i ) + ( 1 − y i ) β ln ( 1 − h ( X i )) ] = β i = 1 ∑ n [ y i ln h ( X i ) + ( 1 − y i ) ln ( 1 − h ( X i )) ] 其中 ∑ i = 1 n [ y i ln h ( X i ) + ( 1 − y i ) ln ( 1 − h ( X i ) ) ] \sum_{i=1}^n \Big[y_i\ln h(X_i)+(1-y_i)\ln(1-h(X_i))\Big] ∑ i = 1 n [ y i ln h ( X i ) + ( 1 − y i ) ln ( 1 − h ( X i )) ]
4. 偏差——方差分解 a . a. a . 一个模型的误差主要来自两个方面：
偏差 (Bias): 指的是模型自身的局限性所导致的系统性错误 。 方差 (Variance): 指的是模型对训练数据的微小变化过于敏感所导致的错误 。 b . b. b . 下面我们严格推导误差的来源。我们定义：
数据模型：y i = g ( X i ) + ϵ i y_i = g(X_i) + \epsilon_i y i = g ( X i ) + ϵ i g g g ϵ \epsilon ϵ 学习模型 h h h ( X , y ) (X, y) ( X , y ) h h h 测试点 z z z γ \gamma γ z z z γ = g ( z ) + ϵ γ = g(z) + ϵ γ = g ( z ) + ϵ z z z ϵ ϵ ϵ γ γ γ 注意，h h h 用不同的训练数据去训练，你就会得到一个不同的模型 h h h 。
我们的目标是分解在 z z z R ( h ) = E [ ( h ( z ) − γ ) 2 ] R(h) = E[(h(z) - γ)^2] R ( h ) = E [( h ( z ) − γ ) 2 ] E E E ϵ \epsilon ϵ
R ( h ) = E [ h ( z ) 2 − 2 h ( z ) γ + γ 2 ] = E [ h ( z ) 2 ] − 2 E [ h ( z ) γ ] + E [ γ 2 ] = E [ h ( z ) 2 ] − 2 E [ h ( z ) ] E [ γ ] + E [ γ 2 ] = ( Var ( h ( z ) ) + E [ h ( z ) ] 2 ) − 2 E [ h ( z ) ] E [ γ ] + ( Var ( γ ) + E [ γ ] 2 ) = ( E [ h ( z ) ] 2 − 2 E [ h ( z ) ] E [ γ ] + E [ γ ] 2 ) + Var ( h ( z ) ) + Var ( γ ) = ( E [ h ( z ) ] − E [ γ ] ) 2 + Var ( h ( z ) ) + Var ( γ ) = ( E [ h ( z ) ] − g ( z ) ) 2 + Var ( h ( z ) ) + Var ( ϵ ) \begin{aligned} R(h) &= \mathbb{E}\big[h(z)^2-2h(z)\gamma+\gamma^2\big] \\ &= \mathbb{E}[h(z)^2]-2\mathbb{E}[h(z)\gamma]+\mathbb{E}[\gamma^2] \\ &= \mathbb{E}[h(z)^2]-2\mathbb{E}[h(z)]\mathbb{E}[\gamma]+\mathbb{E}[\gamma^2] \\ &= \big(\operatorname{Var}(h(z))+\mathbb{E}[h(z)]^2\big)-2\mathbb{E}[h(z)]\mathbb{E}[\gamma]+\big(\operatorname{Var}(\gamma)+\mathbb{E}[\gamma]^2\big) \\ &= \big(\mathbb{E}[h(z)]^2-2\mathbb{E}[h(z)]\mathbb{E}[\gamma]+\mathbb{E}[\gamma]^2\big)+\operatorname{Var}(h(z))+\operatorname{Var}(\gamma) \\ &= \big(\mathbb{E}[h(z)]-\mathbb{E}[\gamma]\big)^2+\operatorname{Var}(h(z))+\operatorname{Var}(\gamma) \\ &= \big(\mathbb{E}[h(z)]-g(z)\big)^2+\operatorname{Var}(h(z))+\operatorname{Var}(\epsilon) \end{aligned} R ( h ) = E [ h ( z ) 2 − 2 h ( z ) γ + γ 2 ] = E [ h ( z ) 2 ] − 2 E [ h ( z ) γ ] + E [ γ 2 ] = E [ h ( z ) 2 ] − 2 E [ h ( z )] E [ γ ] + E [ γ 2 ] = ( Var ( h ( z )) + E [ h ( z ) ] 2 ) − 2 E [ h ( z )] E [ γ ] + ( Var ( γ ) + E [ γ ] 2 ) = ( E [ h ( z ) ] 2 − 2 E [ h ( z )] E [ γ ] + E [ γ ] 2 ) + Var ( h ( z )) + Var ( γ ) = ( E [ h ( z )] − E [ γ ] ) 2 + Var ( h ( z )) + Var ( γ ) = ( E [ h ( z )] − g ( z ) ) 2 + Var ( h ( z )) + Var ( ϵ ) 这个式子表明：期望误差 = 偏差2 ^2 2 。这里的偏差和方差和我们在前置概念说的相对应。
c . c. c . 我们以最小二乘法为例，看看偏差与方差在实际算法中是怎么体现的。
为了简化问题，我们进行如下理想化设定：
我们假设模型为 h ( x ) = w T x h(x)=w^Tx h ( x ) = w T x b b b 真实规律 g ( z ) g(z) g ( z ) g ( z ) = v T z g(z)=v^Tz g ( z ) = v T z v v v 真实权重向量 。 训练标签 y y y X v Xv X v e e e y = X v + e y = Xv + e y = X v + e e e e e i e_i e i σ 2 \sigma ^ 2 σ 2 我们的目标是：通过最小二乘法，从带噪声的训练数据 ( X , y ) (X, y) ( X , y ) w w w w w w h ( z ) = w T z h(z) = w^Tz h ( z ) = w T z
我们知道最小二乘线性回归的解为 w = X + y w = X^{+}y w = X + y X + = ( X T X ) − 1 X T X^{+} = (X^TX)^{-1}X^T X + = ( X T X ) − 1 X T X X X
我们把 y y y
w = X + ( X v + e ) = X + X v + X + e = v + X + e \begin{aligned} w &= X^{+}(Xv+e) \\ &= X^{+}Xv + X^{+}e \\ &= v + X^{+}e \end{aligned} w = X + ( X v + e ) = X + X v + X + e = v + X + e 这个式子告诉我们，我们通过最小二乘法学习到的权重 w w w v v v e e e X + e X^{+}e X + e 误差唯一的来源就是训练数据中的噪声 。
我们接着计算偏差 ∣ E [ h ( z ) ] − g ( z ) ∣ |E[h(z)] - g(z)| ∣ E [ h ( z )] − g ( z ) ∣
∣ E [ h ( z ) ] − g ( z ) ∣ = ∣ E [ w T z ] − v T z ∣ = ∣ ( E [ w ] − v ) T z ∣ E [ w ] = E [ v + X + e ] = E [ v ] + E [ X + e ] = v + X + E [ e ] = v + 0 = v ∴ ∣ ( E [ w ] − v ) T z ∣ = ∣ ( v − v ) T z ∣ = 0 \begin{aligned} \bigl|\mathbb{E}[h(z)]-g(z)\bigr| &= \bigl|\mathbb{E}[w^{T}z]-v^{T}z\bigr| \\ &= \bigl|(\mathbb{E}[w]-v)^{T}z\bigr| \\[6pt] \mathbb{E}[w] &= \mathbb{E}\bigl[v+X^{+}e\bigr] \\ &= \mathbb{E}[v]+\mathbb{E}[X^{+}e] \\ &= v + X^{+}\mathbb{E}[e] \\ &= v + 0 = v \\[6pt] \therefore\quad \bigl|(\mathbb{E}[w]-v)^{T}z\bigr| &= \bigl|(v-v)^{T}z\bigr| = 0 \end{aligned} E [ h ( z )] − g ( z ) E [ w ] ∴ ( E [ w ] − v ) T z = E [ w T z ] − v T z = ( E [ w ] − v ) T z = E [ v + X + e ] = E [ v ] + E [ X + e ] = v + X + E [ e ] = v + 0 = v = ( v − v ) T z = 0 意味着，在我们的理想化设定下（模型类别匹配真实规律），最小二乘线性回归是一个无偏估计 (unbiased estimator) 。这并不是说我们随便用一个训练集学到的 w w w v v v w w w 这些 w w w v v v 。模型犯的错（有时偏高，有时偏低）在平均意义上会相互抵消。
我们接着计算方差 V a r ( h ( z ) ) = V a r ( w T z ) Var(h(z)) = Var(w^Tz) Va r ( h ( z )) = Va r ( w T z )
Var ( w T z ) = Var ( ( v + X + e ) T z ) = Var ( v T z + ( X + e ) T z ) = Var ( ( X + e ) T z ) ( since v T z is constant ) = Var ( z T X + e ) = σ 2 ∥ z T X + ∥ 2 = σ 2 ( z T X + ) ( z T X + ) T = σ 2 z T X + ( X + ) T z = σ 2 z T ( X T X ) − 1 X T ( ( X T X ) − 1 X T ) T z = σ 2 z T ( X T X ) − 1 X T X ( ( X T X ) − 1 ) T z = σ 2 z T ( X T X ) − 1 ( X T X ) ( X T X ) − 1 z ( since ( X T X ) − 1 is symmetric ) = σ 2 z T ( X T X ) − 1 z \begin{aligned} \operatorname{Var}(w^{T}z) &= \operatorname{Var}\big((v+X^{+}e)^{T}z\big) \\ &= \operatorname{Var}\big(v^{T}z+(X^{+}e)^{T}z\big) \\ &= \operatorname{Var}\big((X^{+}e)^{T}z\big) \quad(\text{since }v^{T}z\text{ is constant})\\ &= \operatorname{Var}\big(z^{T}X^{+}e\big) \\ &= \sigma^{2}\,\|z^{T}X^{+}\|^{2} \\ &= \sigma^{2}\,(z^{T}X^{+})(z^{T}X^{+})^{T} \\ &= \sigma^{2}\,z^{T}X^{+}(X^{+})^{T}z \\ &= \sigma^{2}\,z^{T}(X^{T}X)^{-1}X^{T}\big((X^{T}X)^{-1}X^{T}\big)^{T}z \\ &= \sigma^{2}\,z^{T}(X^{T}X)^{-1}X^{T}X\big((X^{T}X)^{-1}\big)^{T}z \\ &= \sigma^{2}\,z^{T}(X^{T}X)^{-1}(X^{T}X)(X^{T}X)^{-1}z \quad(\text{since }(X^{T}X)^{-1}\text{ is symmetric})\\ &= \sigma^{2}\,z^{T}(X^{T}X)^{-1}z \end{aligned} Var ( w T z ) = Var ( ( v + X + e ) T z ) = Var ( v T z + ( X + e ) T z ) = Var ( ( X + e ) T z ) ( since v T z is constant ) = Var ( z T X + e ) = σ 2 ∥ z T X + ∥ 2 = σ 2 ( z T X + ) ( z T X + ) T = σ 2 z T X + ( X + ) T z = σ 2 z T ( X T X ) − 1 X T ( ( X T X ) − 1 X T ) T z = σ 2 z T ( X T X ) − 1 X T X ( ( X T X ) − 1 ) T z = σ 2 z T ( X T X ) − 1 ( X T X ) ( X T X ) − 1 z ( since ( X T X ) − 1 is symmetric ) = σ 2 z T ( X T X ) − 1 z 由于真实权重和测试点都是固定的，v T z v^Tz v T z
这个公式虽然精确，但不够直观。通过一些近似（当样本量 n n n
Var ( h ( z ) ) ≈ σ 2 d n \text{Var}(h(z)) \approx \sigma^2 \frac{d}{n} Var ( h ( z )) ≈ σ 2 n d 这个式子揭露了方差的来源与控制，模型的方差：
与 σ 2 \sigma^2 σ 2 与 d d d 特征越多，模型越复杂，越容易拟合噪声，方差越大 。这就是“维度灾难”的一种体现。 与 n n n
Comments