\section{Intro}
\textbf{Data Space $\calX$}: Space of objects under investigation. \emph{Example} RGB images in $128 \times 128$ res with 8-bit color depth per channel, $\calX = \{0,1, \ldots, 255\}^{128 \times 128 \times 3};$\\
\textbf{Label Space $\calY$:} Space of objects, kinds, types, attributes relevant to the pattern recognition problem. Examples: \emph{Classification:} Discrete Sets like $\calY = \{0,1\},\{-1,1\},\{jaszz,classic,\ldots\}$\\
\emph{Regression:} Intervals like predicted temp $\calY = [-30, 50]$\\
\textbf{Hypothesis Space $\calH$:} Space (set) of functions/ algorithms that take inputs from $\calX$ and yield outputs in $\calY$.
E.g. set of binary classifiers, set of $K$-NN classifiers, set of neural network classifiers.\\
\textbf{Loss function $\ell$}: To chose algo / func from $\calH$, need measure of "best". $\ell: \calY \times \calY$: Given true label $y$ and estimate $\wh{y}$, $\ell(y \wh{y})$ measures how far they are from each other. \\
\textbf{(Joint) probability distribution over data and labels $\PP$:}
We model the data--label pairs $(\vx, y)$ (elements in $\calX \times \calY$) as being (independent) samples from some probability distribution $\PP$ over $\calX \times \calY$. This probability distribution is a priori just a convenient way to mathematically formalize dealing with large amounts of data. It is an ephemeron. It does not exist, or at least it does not have to, in any concrete sense, for the theory and algorithmic machinery of pattern recognition to operate. 
    At this point, our goal is to find a function $h^*$ in $\calH$ which incurs the smallest average loss, with average being the expectation w.r.t. $\mathbb{P}$:
    $
        \label{eq:risk-minimization}
        h^* = \argmin_{h \in \calH} \EE_{(\vX, Y) \sim \PP}[  \ell(h(\vX), Y)],
    $
    where in the subscript of the expectation, we emphasize that $\vX$ and $Y$ are jointly distributed random variables with distribution $\PP$. 
    \footnote{When we want to emphasize that something is a random variable, we will typeset it in capitals, as is standard. Thus $\PP[Y = y]$ is the probability that the random variable $Y$ assumes value $y$.}
    We will use the name \textit{risk} for the average loss of $h$ and give it a nice letter :
    $
        R(h) := \mathbb{E}[\ell(h(\vX), Y)].
    $
    This highlights a fundamental challenge in machine learning: the actual distribution of the probability distribution $\PP$ is \textit{unknown}. We thus introduce the last ingredient in supervised machine learning: its royal highness, the data.

    %
    \footnote{The operation $\min_{h \in \calH} F(h)$ returns the smallest number in the set $\{ F(h) \mid h \in \calH \}$; the operation $\argmin_{h \in \calH}$ tells us for which $h$ this smallest value is achieved. For example, if simply $\calH = \mathbb{R}$ and $F(x) = x^2  - 5x - 2$, then $\min_{x \in \calH} F(x) = -\tfrac{33}{4}$ while $\argmin_{x \in \calH} = \tfrac{5}{2}$}.\\

\textbf{The (training) data set $\calD = \set{ (\vx_1, y_1), (\vx_2, y_2), \ldots, (\vx_N, y_N)}$:} The training data consists of pattern--label pairs drawn independently from $\PP$. Assumption: data are i.i.d. The training data set is a proxy to $\PP$---the larger it is the better.

\subsection{Training and testing:}
Since we don't know $\PP$ we also cannot compute \eqref{eq:risk-minimization}. So instead we use \emph{empirical risk minimization}:
$
\label{eq:empirical-risk-minimization}
TODO
$
