Improve NN

train/dev/test set

0.7/0/0.3 0.6.0.2.0.2 -> 100-10000

0.98/0.01/0.01 … -> big data

Bias/Variance

偏差度量的是单个模型的学习能力，而方差度量的是同一个模型在不同数据集上的稳定性。

high variance ->high dev set error

high bias ->high train set error

basic recipe

high bias -> bigger network / train longer / more advanced optimization algorithms / NN architectures

high variance -> more data / regularization / NN architecture

Regularization

Logistic Regression

$$\begin{gathered} L2regularization: \\ min\mathcal{J}(w,b)\to J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})+\frac{\lambda }{2m}\Vert w{\Vert }_2^2 \end{gathered}$$

Neural network

$$\begin{gathered} Frobeniusnorm \\ \Vert w^{[l]}{\Vert }_F^2=\sum _{i=1}^{n^{[l]}}\sum _{j=1}^{n^{[l-1]}}(w_{i,j}^{[l]}{)}^2 \\ Dropoutregularization: \\ d3=np.randm.rand(a3.shape.shape[0],a3.shape[1]<keep.prob) \\ a3=np.multiply(a3,d3) \\ a3/=keep.prob \end{gathered}$$

other ways

• early stopping
• data augmentation

optimization problem

speed up the training of your neural network

Normalizing inputs

1. subtract mean

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)} \\ x:=x-\mu \end{gathered}$$

2. normalize variance

$$\begin{gathered} {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}{)}^2 \\ x/=\sigma \end{gathered}$$

vanishing/exploding gradients

$$\begin{gathered} y=w^{[l]}{w}^{[l-1]}...w^{[2]}{w}^{[1]}x \\ {w}^{[l]}>I\to (w^{[l]}{)}^L\to \infty \\ {w}^{[l]}<I\to (w^{[l]}{)}^L\to 0 \end{gathered}$$

weight initialize

$$\begin{gathered} var(w)=\frac{1}{n^{(l-1)}} \\ {w}^{[l]}=np.random.randn(shape)\ast np.sqrt(\frac{1}{n^{(l-1)}}) \end{gathered}$$

gradient check

Numerical approximation

$$\begin{gathered} f(\theta )={\theta }^3 \\ {f}^{\prime }(\theta )=\frac{f(\theta +\epsilon )-f(\theta -\epsilon )}{2\epsilon } \end{gathered}$$

grad check

$$\begin{gathered} d{\theta }_{approx}[i]=\frac{J({\theta }_1,...{\theta }_i+\epsilon ...)-J({\theta }_1,...{\theta }_i-\epsilon ...)}{2\epsilon }=d\theta [i] \\ check:\frac{\Vert d{\theta }_{approx}-d\theta {\Vert }_2}{\Vert d{\theta }_{approx}{\Vert }_2+\Vert d\theta {\Vert }_2}<10^{-7} \end{gathered}$$