Multiple testing

Background

As Multiple testing， we need to adjust p-value.

Control Type I error

Under $H_0$ ：

\begin{align*} t_1=\frac{\hat{\beta}_1}{\operatorname{se}\left(\hat{\beta}_1\right)} \sim t_{n-p} \end{align*}

As sample size $n$ large， $t_{n-p} \approx N(0,1)$ . Under null $H_0$ ， effect size $\beta \sim N(0,1)$ generally. Suppose $P($ Type I error $)=\alpha$ ，for $m$ test

\begin{align*} \text { FWER }=1-(1-\alpha)^m \approx 1-(1-m \alpha) \end{align*}

In Bonferroni Correction， to control FWER $\leq \alpha$

\begin{align*} \begin{aligned} & \Rightarrow \quad m \alpha_{\text {Bon }} \leq \alpha \\ & \Rightarrow \quad \alpha_{\text {Bon }} \leq \frac{\alpha}{m} \end{aligned} \end{align*}

Control False discovery rate

FDR (False Discovery Rate)

\begin{align*} \operatorname{FDR}\left(q^*\right)=E\left[\frac{F\left(q^*\right)}{S\left(q^*\right)}\right] \end{align*}

$q^*$ : threshold
$S$ : number of significance
$F$ : number of false discovery

For $m$ test，p-value $p_1, \ldots, p_m$

order p-value $p_{(1)} \leq \ldots \leq p_{(m)}$
$k \overset{\underset{\mathrm{def}}{}}{=} \underset{i}{\operatorname{argmax}} \left(p_i \leq \frac{i}{m} q^*\right),\ i=1,2, \ldots, m$
reject $H_{(i)},\ i=1, \ldots, k$

q-value

\begin{align*} \hat{F D R}(t)=\frac{\hat{\pi}_0 m t}{S(t)}=\frac{\hat{\pi}_0 m t}{\sum_i I\left(P_i \leq t\right)} \end{align*}

where $\pi_0=P(H_0 \text{ is true} )$ ， $t$ is cut off

\begin{align*} \hat{\pi}_0(\lambda) &= \frac{\sum_i I\left(P_i \leq t\right)}{m(1-\lambda} \\ &= \frac{\text{number of } H_0}{\text{number of total}} \end{align*}