> For the complete documentation index, see [llms.txt](https://stephanosterburg.gitbook.io/scrapbook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://stephanosterburg.gitbook.io/scrapbook/coding/advanced-machine-learning/wk-1.md).

# Wk 1

## Regression and classification

$$y\_i \in \mathbb{R}$$ -- regression task

* salary prediction
* movie rating prediction

$$y\_i$$ belongs to a finite set -- classification task

* object recognition
* topic classification

### Linear model for regression

$$a(x) = b + w\_1x\_1 + w\_2x\_2 + \dots + w\_dx\_d$$

* $$w\_1, \dots , w\_d$$-- coefficients (weights)
* $$b$$ -- bias
* $$d$$ + 1 parameters
* to make it simple: there is always a constant feature

Vector notation:

$$
a(x) = w^T x
$$

For a sample $$X$$:

$$
a(X) = Xw \\

X = \begin{pmatrix}x\_{11} \cdots x\_{1d} \ \vdots \ddots \vdots \ x\_{n1} \cdots x\_{nd}\end{pmatrix}
$$

### Loss function

How to measure model quality?

$$
\text{Mean squared error:} \\
L(w) = \frac{1}{n}\sum^n\_{i=1}(w^Tx\_i - y\_i)^2 \\
\= \frac{1}{n}\Vert Xw - y\Vert^2
$$

### Training a model

Fitting a model to training data:

$$
L(w) = \frac{1}{n} \Vert Xw - y \Vert^2 \rightarrow \underset{w}min
$$

Exact solution:

$$
w = (X^TX)^{-1} X^Ty
$$

**But inverting a matrix is hard for high-dimensional data!**

### Gradient Descent

![](/files/-LptsPK4YK72RdznITSs)

![](/files/-LptsXrnSbkaLCymRKov)

![](/files/-LpttDLDseZJbjYpCAJQ)