Logistic Regression
$Y_i \sim Ber(\pi_i)$, $f(y_i)=\pi_i^{y_i} (1-\pi_i)^{1-y_i}$
$$\begin{align*} & g(\mu)=g(\pi)= \ln{\frac{\pi_i}{1-\pi_i}} = \mathbb{X}_i \boldsymbol{\beta} \\\\\\ \Rightarrow \ & \pi_i = \frac{\texttt{exp}(\mathbb{X} \boldsymbol{\beta}) }{1+\texttt{exp}(\mathbb{X} \boldsymbol{\beta})} \end{align*}$$$$\begin{align*} L & =\prod_{i=1}^n \pi_i^{y_i} (1-\pi_i)^{1-y_i} \\\\\\ & = \prod_{i=1}^n \left(\frac{\texttt{exp}(\mathbb{X}_i \boldsymbol{\beta}) }{1+\texttt{exp}(\mathbb{X}_i \boldsymbol{\beta})} \right)^{y_i} \left(\frac{1 }{1+\texttt{exp}(\mathbb{X}_i \boldsymbol{\beta})} \right)^{1-y_i} \end{align*}$$$$ \sum_{i=1}^n \left[ y_i \mathbb{X}_i \boldsymbol{\beta} -\ln(1+\texttt{exp}(\mathbb{X}_i \boldsymbol{\beta})) \right] $$MLE of $\boldsymbol{\beta}$ doesn’t has closed form.
Odds ratio
| Sex | Disease | No disease | Probability of disease | Odds |
|---|---|---|---|---|
| male | 217 | 162 | $\frac{217}{217+162} = 0.573$ | $\frac{0.573}{1-0.573} = 1.342$ |
| female | 105 | 136 | $\frac{105}{105+136} = 0.436$ | $\frac{0.436}{1-0.436} = 0.773$ |
代表男性患疾病的機率比女性來的高
損失函數
$$\begin{align*} -\frac{1}{n} \sum_{i=1}^n \left[ y_i \ln p_i + (1 - y_i) \ln (1 - p_i) \right], \quad y_i = 0 \text{ or } 1 \end{align*}$$$$\begin{align*} -\frac{1}{n} \sum_{i=1}^n \sum_{c=1}^k y_{i,c} \ln p_{i,c} \end{align*}$$假如現在有3個類別
| Sample | $y_i$ | $p_i$ |
|---|---|---|
| 1 | (1, 0, 0) | (0.7, 0.2, 0.1) |
| 2 | (0, 1, 0) | (0.1, 0.6, 0.3) |
| 3 | (0, 0, 1) | (0.2, 0.3, 0.5) |