Wk 1

Regression and classification

$y_i \in \mathbb{R}$ -- regression task

• salary prediction

• movie rating prediction

$y_i$ belongs to a finite set -- classification task

• object recognition

• topic classification

Linear model for regression

$a(x) = b + w_1x_1 + w_2x_2 + \dots + w_dx_d$

• $w_1, \dots , w_d$-- coefficients (weights)

• $b$ -- bias

• $d$ + 1 parameters

• to make it simple: there is always a constant feature

Vector notation:

$$a(x) = w^T x$$

For a sample $X$:

$$a(X) = Xw \\ X = \begin{pmatrix}x_{11} \cdots x_{1d} \\ \vdots \ddots \vdots \\ x_{n1} \cdots x_{nd}\end{pmatrix}$$

Loss function

How to measure model quality?

$$\text{Mean squared error:} \\ L(w) = \frac{1}{n}\sum^n_{i=1}(w^Tx_i - y_i)^2 \\ = \frac{1}{n}\Vert Xw - y\Vert^2$$

Training a model

Fitting a model to training data:

$$L(w) = \frac{1}{n} \Vert Xw - y \Vert^2 \rightarrow \underset{w}min$$

Exact solution:

$$w = (X^TX)^{-1} X^Ty$$

But inverting a matrix is hard for high-dimensional data!

Gradient Descent