dm.cs.tu-dortmund.de/en/mlbits/class-nnet-regularization/
Regularization Techniques for Neural Networks – Lecture Notes
additional matrix multiplication with a 0-1 matrix during training.
For final predictions, weights are reduced by \((1-p)\) (where \(p\) is the probability of a 1).
We can also use Gaussian dropout , where [...] simpler models. Simpler models: smaller weights, more zeros.
\[\mathcal{L}_{\text{reg}} = \frac{1}{n}\sum _{i=1}^n\mathcal{L}(y_i,\hat{y}_i) \textcolor[RGB]{132,184,24}{{}+ \frac{\lambda }{n}\left\lVert W [...] \textcolor[RGB]{132,184,24}{{}+\frac{2\lambda }{n} w_{i,j}^{(t)}} \right) \\[3ex] w_{i,j}^{(t+1)} & = w_{i,j}^{(t)} \left(1-\frac{2\lambda }{n}\right) - \eta _i \frac{\partial \mathcal{L}}{\partial w_{i,j}} \e …
