Predictive Modeling

• the goal is predictive modeling
• features: $x \in X$ where $x$ is the tiger image and $X$ is the space of all images of animals
• labels $y \in Y$ and $y = [0, 1, 0]$
• then we could have a function that tells us our prediction $f(x) = \hat{y}$
  • this class is about how to design the function, which is our deep neural network

Data Generating Process

• our features are generated by some unknown model $\mathbb{P}(x)$: $x \sim \mathbb{P}(x)$, so we want to predict the target label based on the input features $y \sim \mathbb{P}(y|x)$
  • Because $\mathbb{P}$ are distributions we don't know / are unknown but assume exist, we want our predictive model to match this: $f(x) \approx \mathbb{P}(y|x)$.

Training Data

• $\mathbb{D} = {\{(x_n, y_n)\}}^{N}_{n=1}$
• $x_n \sim \mathbb{P}(x)$
• $y_n \sim \mathbb{P}(y|x_n)$

Linear Regression

Univariate Linear Regression

• $x \in \mathbb{R}$, $y \in \mathbb{R}$
• $f(x;w) = w \cdot x$
• This is the simplest model you can think of
• $w$ is the slope. It's the parameter that we want to determine from the training data

Squared Loss

We want to come up with a loss function to determine how far we are off of the actual data.

• $\displaystyle P(w;\mathbb{D}) = \frac{1}{N}\sum_{n=1}^{N}l(w, x_n, y_n) = \frac{1}{N}\sum_{n=1}^{N}(f(x, w) - y_n)^2$
• aka mean squared error
• think hard about what you are encoding. what assumptions is this loss function baking in. in this case, we ignore the negative, which means it would not be optimal for like an auction bidding model because it will think overbidding is okay.

$w^* = \arg\min_w l(w; \mathbb{D}) = \arg\min_w \frac{1}{N}\sum_{N}(w \cdot X_n - y_n)^2$ where $w \cdot X_n$ is $f(x;w)$

To solve for $w$:

step 1: take the derivative

$\frac{d}{dw} l(w; D) = \frac{d}{dw}[\frac{1}{N}\sum_{N}(w \cdot X_n - y_n)^2] = \frac{2}{N}\{(\sum_{n=1}^{N}w\cdot {x_n}^2)-(\sum_{n=1}^{N}y_n \cdot x_n)\}$

step 2: set to 0 and solve for $w$

$0 = \frac{d}{dw} l(w; D)$ $\Rightarrow w^* = \frac{\sum_{n=1}^{N}y_n \cdot x_n}{\sum_{n=1}^{N}{x_n}^2}$

You can take this $w$ and use it to make predictions !

Vectorized Version

$\vec{x} = [x_1 \cdots x_N]^T$

$\vec{y} = [y_1 \cdots y_N]^T$

$w^* = \frac{X^TY}{X^TX}$

Maximum Likelihood

Statistical Divergences

Ways of computing divergences between values

• kullback-leibler divergence (KLD)
• $KLD[p(z) || q(z)] \triangleq \mathbb{E}_{p(z)}[log\frac{p(z)}{q(z)}]$ = $\int_z p(z) log \frac{p(z)}{q(z)}dz$
• isn't a true distance metric but is a good one to use
• the amount (bits) of information loss between p(z) and q(z).
• there are other divergences like Hellinger divergence (a type of f divergences)
• Use this to quantify how close the predictive model is to $y \sim \mathbb{P}$ (the true model)

Predictive Models as Distributions

$p(y;\Theta = f(x))$

normal distribution $N(y, \mu, \sigma^2) = N(y, \mu = f(x), {\theta_0}^2$

• imagine at every point in that $y=wx$ there is a normal distribution. The mean is governed by that line.
• Entropy is how uncertain a distribution in. E.g., the fair coin has a high entropy, a skewed coin has a low entropy because you can predict what value the coin will take. Entropy: $\mathbb{H}[p(z)]\triangleq\mathbb{E}_{p(z)}[-log p(z)]$

Maximum Likelihood Estimation

$KLD[\mathbb{P}(y|x) || p(y;\Theta = f(x))]$ between true distribution and the model we have defined where the mean is governed by our $f(x)$ function. this whole equation represents the bits or information lost

$=\mathbb{E}_{\mathbb{P}(y|x)}[log\frac{\mathbb{P}(y|x)}{p(y;\Theta = f(x))}]=\mathbb{E}_{\mathbb{P}(y|x)}[log\mathbb{P}(y|x)] - \mathbb{E}_{\mathbb{P}(y|x)}[log p(y;\Theta = f(x))] = \mathbb{E}_{\mathbb{P}(y|x)}[-log p(y;\Theta = f(x))] - \mathbb{H}[\mathbb{P}(y|x)]$

• Note that $\mathbb{H}[\mathbb{P}(y|x)]$ doesnt involve f(x) !

Now we set up an optimization problem with respect to $f(x)$.

$w^* = \arg\min_{w} \mathbb{E}_{p(x)}\mathbb{E}_{\mathbb{P}(y|x)}[-log p(y;f(x);w)] + \text{entropy constant}$

• entropy constant will go away when you take the derivative
• you can't do this because it assumes you are doing sums or integrals over distributions we don't know.

Monte Carlo Approximation

$\mathbb{E}_{p(x)}[\psi(x)]\approx \frac{1}{S} \sum_S \psi(\hat{x}_s)$

Maximum Likelihood: Final Form

A very important formula:

$w^* = \arg\min_w KLD[\mathbb{P}(y|x) || p(y;\Theta = f(x,w))]$

$\approx \arg\min_w -\frac{1}{N} \sum^N_{n=1} \log p(y_n; f(x_n, w))$

• if you have only 5 samples to learn from, then it's bad
• dataset size is the engine that drives how good you are learning
• KLD must be zero for it to be perfect predictor
• simplifying this last form gets you the squared error function.