Conjugate Prior

Conjugate Prior.

• Bernoulli distribution and Beta prior
• Categorical distribution and Dirichlet prior
• Poisson distribution and Gamma prior
• Univariate Gaussian distribution and Normal-Gamma Priors
  • Conjugacy for the mean
  • Conjugacy for the variance
  • Conjugacy for the mean and variance

Bernoulli distribution and Beta prior

The likelihood function

$$p(x|\theta )={\theta }^x(1-\theta {)}^{1-x}$$

The beta distribution is parameterized using ${\alpha }_i-1$

$$p(\theta |\alpha )=K(\alpha ){\theta }^{{\alpha }_1-1}(1-\theta {)}^{{\alpha }_2-1}$$

where the normalization factor $K(\alpha )$ can be obtained analytically:

$$\begin{array}{rl}K(\alpha )& ={(\int {\theta }^{{\alpha }_1-1}(1-\theta {)}^{{\alpha }_2-1}d\theta )}^{-1}\\ & =\frac{\mathrm{\Gamma }({\alpha }_1+{\alpha }_2)}{\mathrm{\Gamma }({\alpha }_1)\mathrm{\Gamma }({\alpha }_2)}\end{array}$$

Consider in particular $N$ i.i.d Bernoulli observations, $x=(x_1,\dots ,x_N{)}^{\mathrm{\top }}$:

$$\begin{array}{rl}p(\theta |\mathbf{x},\alpha )& \propto (\prod _{n=1}^N{\theta }^{x_n}(1-\theta {)}^{1-x_n}){\theta }^{{\alpha }_1-1}(1-\theta {)}^{{\alpha }_2-1}\\ & ={\theta }^{\sum _{n=1}^N{x}_n+{\alpha }_1-1}(1-\theta {)}^{N-\sum _{n=1}^N{x}_n+{\alpha }_2-1}.\end{array}$$

In particular, it is a $\text{Beta}(\theta |\sum _{n=1}^N{x}_n+{\alpha }_1,N-\sum _{n=1}^N{x}_n+{\alpha }_2)$.

$$\mathbb{E}[\theta |\alpha ]=\int \theta \cdot \text{Beta}(\theta |{\alpha }_1,{\alpha }_2)\text{d}\theta =\frac{{\alpha }_1}{{\alpha }_1+{\alpha }_2}.$$

A similar calculation ($\mathbb{E}[{\theta }^2|\alpha ]$) yields the variance:

$$\mathrm{Var}[\theta |\alpha ]=\frac{{\alpha }_1{\alpha }_2}{({\alpha }_1+{\alpha }_2+1){({\alpha }_1+{\alpha }_2)}^2}.$$

Applying the results to $\text{Beta}(\theta |\sum _{n=1}^N{x}_n+{\alpha }_1,N-\sum _{n=1}^N{x}_n+{\alpha }_2)$ we obtain

$$\begin{array}{rl}\mathbb{E}[\theta |\mathbf{x},\alpha ]& =\frac{\sum _{n=1}^N{x}_n+{\alpha }_1}{N+{\alpha }_1+{\alpha }_2}\\ \mathrm{Var}[\theta |\mathbf{x},\alpha ]& =\frac{(\sum _{n=1}^N{x}_n+{\alpha }_1)(N-\sum _{n=1}^N{x}_n+{\alpha }_2)}{(N+{\alpha }_1+{\alpha }_2+1){(N+{\alpha }_1+{\alpha }_2)}^2}.\end{array}$$

The predictive probability of a new data point $X$ is calculated as follows:

$$p(X=1|x,\alpha )=\int p(X=1|\theta )p(\theta |x,\alpha )\text{d}\theta =\int \theta p(\theta |x,\alpha )\text{d}\theta =\mathbb{E}[\theta |x,\alpha ].$$

Categorical distribution and Dirichlet prior

In this case the likelihood takes the form

$$p(x|\theta )={\theta }_1^{x_1}{\theta }_2^{x_2}\dots {\theta }_K^{x_K}$$

where $x_k\in \{0,1\},\sum _k{x}_k=1$.

This yields the conjugate prior:

$$p(\theta |\alpha )=K(\alpha ){\theta }_1^{{\alpha }_1-1}{\theta }_2^{{\alpha }_2-1}\dots {\theta }_K^{{\alpha }_K-1}$$

where ${\alpha }_i>0$ and

$$K(\alpha )=\frac{\mathrm{\Gamma }\left(\sum _{k=1}^K{\alpha }_k\right)}{\prod _{k=1}^K\mathrm{\Gamma }\left({\alpha }_k\right)}$$

The posterior distribution of $N$ i.i.d observations, $x=(x_1,\dots ,x_N{)}^{\mathrm{\top }}$

$$p(\theta |x,\alpha )\propto {\theta }_1^{\sum _{n=1}^N{x}_{n1}+{\alpha }_1-1}{\theta }_2^{\sum _{n=1}^N{x}_{n2}+{\alpha }_2-1}\dots {\theta }_K^{\sum _{n=1}^N{x}_{nK}+{\alpha }_K-1}$$

which is $\text{Dir}(\theta |\sum _n{x}_n+\alpha )$ and note that $x_n=(x_{n1},\dots ,x_{nK})$.

The mean and variance are

$$\begin{array}{rl}\mathbb{E}[{\theta }_i|\alpha ]& =\frac{{\alpha }_i}{\alpha .}\\ \mathrm{Var}[{\theta }_i|\alpha ]& =\frac{{\alpha }_i(\alpha .-{\alpha }_i)}{\alpha {.}^2(\alpha .+1)}\end{array}$$

where $\alpha .=\sum _i{\alpha }_i$.

Poisson distribution and Gamma prior

Consider the Poisson distribution:

$$p(x|\theta )=\frac{{\theta }^x{e}^{-\theta }}{x!}$$

The corresponding conjugate prior retains the shape of the likelihood:

$$p(\theta |\alpha )=K(\alpha ){\theta }^{{\alpha }_1-1}{e}^{-{\alpha }_2\theta }$$

where the normalization factor $K(\alpha )=\frac{{\alpha }_2^{{\alpha }_1}}{\mathrm{\Gamma }({\alpha }_1)}$.

The posterior distribution of $N$ i.i.d Poisson observations is:

$$p(\theta |\alpha )\propto {\theta }^{{\alpha }_1-1+\sum _{n=1}^N{x}_n}{e}^{-(N+{\alpha }_2)\theta }$$

which is $\text{Gamma}(\theta |{\alpha }_1+\sum _n{x}_n,{\alpha }_2+N)$.

The mean and variance are readily computed as:

$$\begin{array}{rl}\mathbb{E}[\theta |\alpha ]& =\frac{{\alpha }_1}{{\alpha }_2}\\ \mathrm{Var}[{\theta }_i|\alpha ]& =\frac{{\alpha }_1}{{\alpha }_2^2}\end{array}$$

In the distribution $\text{Gamma}(\theta |{\alpha }_1,{\alpha }_2)$, ${\alpha }_1$ is known as the shape parameter and ${\alpha }_2$ is called the rate parameter ($1/{\alpha }_2$ is called the scale parameter).

Univariate Gaussian distribution and Normal-Gamma Priors

Recall that the Gaussian distribution is a two-parameter exponential family of the following form:

$$p(x\mid \mu ,{\sigma }^2)\propto ({\sigma }^2{)}^{-1/2}\exp \{-\frac{1}{2{\sigma }^2}{(x-\mu )}^2\}$$

Conjugacy for the mean

Inspecting the above density form we see that the exponent is the negative of a quadratic form in $\mu$. Thus we assume

$$p(\mu |{\mu }_0,{\sigma }_0^2)\propto {\left({\sigma }_0^2\right)}^{-1/2}\exp \{-\frac{1}{2{\sigma }_0^2}{(\mu -{\mu }_0)}^2\}$$

where ${\mu }_0$ and ${\sigma }_0^2$ are the mean and variance of $\mu$.

Before deriving the posterior distribution $p(\mu |x)$, let us introduce some useful properties of Gaussian variable.

$$\begin{array}{rl}\mathbb{E}[Z_1|Z_2]& =\mathbb{E}\left[Z_1\right]+\frac{\mathrm{Cov}[Z_1,Z_2]}{\mathrm{Var}\left[Z_2\right]}(Z_2-\mathbb{E}\left[Z_2\right])\\ \mathrm{Var}[Z_1|Z_2]& =\mathrm{Var}\left[Z_1\right]-\frac{{\mathrm{Cov}}^2[Z_1,Z_2]}{\mathrm{Var}\left[Z_2\right]}\end{array}$$

Consider that we only have one observation $x$, we reparameterize $X$ and $\mu$ using the Normal random variable $ϵ,\delta \sim N(0,1)$ as

$$\begin{array}{l}X=\mu +\sigma ϵ\\ \mu ={\mu }_0+{\sigma }_0\delta \end{array}$$

We can now easily calculate:

$$\begin{array}{rl}\mathbb{E}[X]& =\mathbb{E}[\mu ]+\sigma \mathbb{E}[ϵ]={\mu }_0\\ \mathbb{V}[X]& =\mathbb{V}[\mu ]+{\sigma }^2\mathbb{V}[ϵ]={\sigma }_0^2+{\sigma }^2\\ \text{Cov}(X,\mu )& =\mathbb{E}\left[(X-{\mu }_0)(\mu -{\mu }_0)\right]=\mathbb{E}\left[(\mu +\sigma ϵ-{\mu }_0)(\mu -{\mu }_0)\right]={\sigma }_0^2\end{array}$$

Treating $\mu$ as $Z_1$ and $X$ as $Z_2$ we obtain:

$$\begin{array}{rl}\mathbb{E}[\mu |X=x]& ={\mu }_0+\frac{{\sigma }_0^2}{{\sigma }^2+{\sigma }_0^2}(x-{\mu }_0)=\frac{{\sigma }_0^2}{{\sigma }^2+{\sigma }_0^2}x+\frac{{\sigma }^2}{{\sigma }^2+{\sigma }_0^2}{\mu }_0\\ \text{Var}(\mu |X=x)& ={\sigma }_0^2-\frac{{\sigma }_0^4}{{\sigma }^2+{\sigma }_0^2}=\frac{{\sigma }^2}{{\sigma }^2+{\sigma }_0^2}{\sigma }_0^2.\end{array}$$

We can also express the results in terms of the precision. In particular, plugging $\tau =1/{\sigma }^2$ and ${\tau }_0=1/{\sigma }_0^2$ into the above equation yields:

$$\begin{array}{r}\mathbb{E}[\mu |X=x]=\frac{\tau }{\tau +{\tau }_0}x+\frac{{\tau }_0}{\tau +{\tau }_0}{\mu }_0\end{array}$$

The posterior expectation is a convex combination of the observation $x$ and prior mean ${\mu }_0$. More precisely, the precision of the data multiplies the data $x$ and the precision of the prior multiplies the prior mean ${\mu }_0$. If the precision of the data is large relative to the precision of the prior, then the posterior mean is closer to $x$. Also note that the posterior precision ${\tau }_{\text{post}}=\tau +{\tau }_0$, which has a direct interpretation: precision add.

Let us now consider the posterior distribution in the case of involving multiple observed data points, we have:

$$p(\mathbf{x}|\mu ,\tau )\propto {\tau }^{n/2}\exp \{-\frac{\tau }{2}\sum _{i=1}^n{(x_i-\mu )}^2\}$$

We rewrite the exponent using a standard trick:

$$\begin{array}{rl}\sum _{i=1}^n{(x_i-\mu )}^2& =\sum _{i=1}^n{(x_i-\bar{x}+\bar{x}-\mu )}^2\\ & =\sum _{i=1}^n{(x_i-\bar{x})}^2+n(\bar{x}-\mu {)}^2\end{array}$$

The first term yields a constant factor, and we see the problem reduces to an equivalent problem involving only single random variable (treating $\overline{x}$ as random variable), in which $\overline{X}\sim N(\mu ,{\sigma }^2/n)$.

$$\overline{X}=\mu +\frac{{\sigma }^2}{n}ϵ$$

and we have

$$\begin{array}{rl}\mathbb{E}[\mu {\text{vertx}}_1,\dots ,x_n]& =\frac{{\sigma }_0^2}{{\sigma }^2/n+{\sigma }_0^2}\overline{x}+\frac{{\sigma }^2}{{\sigma }^2/n+{\sigma }_0^2}{\mu }_0\\ \text{Var}(\mu |x_1,\dots ,x_n)& =\frac{{\sigma }^2/n}{{\sigma }^2/n+{\sigma }_0^2}{\sigma }_0^2.\end{array}$$

${\tau }_{\text{post}}=n\tau +{\tau }_0$.

Now consider an unseen data point $Y\sim N(\mu ,{\sigma }^2)$, we reparameterize it as

$$\begin{array}{rl}Y& =\mu +\sigma ϵ\\ \mu & ={\mu }_{\text{post}}+{\sigma }_{\text{post}}\delta \end{array}$$

$Y$ again is a Gaussian random variable as it is the sum of two Gaussian random variables $\mu$ and $ϵ$, its predictive mean and variance:

$$\begin{array}{rl}\mathbb{E}[Y]& ={\mu }_{\text{post}}\\ \text{Var}(Y)& ={\sigma }_{\text{post}}^2+{\sigma }^2.\end{array}$$

Conjugacy for the variance

Let us now consider the Gaussian distribution with known mean $\mu$,

$p(\mathbf{x}|\mu ,{\sigma }^2)\propto {\left({\sigma }^2\right)}^a{e}^{-b/{\sigma }^2}$ where $a=-1/2$ and $b=\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2$. This has the flavor of the gamma distribution, but the random variable ${\sigma }^2$ is in the denominator. This is actually an inverse gamma distribution. We thus assume the prior distribution for the variance is an inverse gamma distribution:

$$p({\sigma }^2|\alpha ,\beta )=\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left({\sigma }^2\right)}^{-\alpha -1}{e}^{-\beta /{\sigma }^2}$$

The posterior distribution:

$$\begin{array}{rl}p({\sigma }^2|\mathbf{x},\mu ,\alpha ,\beta )& \propto p(\mathbf{x}|\mu ,{\sigma }^2)p({\sigma }^2|\alpha ,\beta )\\ & \propto {\left({\sigma }^2\right)}^{-n/2}{e}^{-\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2/{\sigma }^2}{\left({\sigma }^2\right)}^{-\alpha -1}{e}^{-\beta /{\sigma }^2}\\ & ={\left({\sigma }^2\right)}^{-(\alpha +\frac{n}{2})-1}{e}^{-(\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2)/{\sigma }^2}\end{array}$$

which is an $\text{Inv-Gamma}(\alpha +\frac{n}{2},\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2)$.

If we derive the posetrior in terms of precision we would obtain $\tau \sim \text{Gamma}(\alpha +\frac{n}{2},\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2)$.

The predictive distribution for an unseen data point $Y$:

$$\begin{array}{rl}p(Y|\mathbf{x},\mu ,\alpha ,\beta )& =\int p(Y|\mathbf{x},\mu ,\tau )p(\tau |\mathbf{x},\mu ,\alpha ,\beta )d\tau \\ & =\int p(Y|\mu ,\tau )p(\tau |\mathbf{x},\mu ,\alpha ,\beta )d\tau \end{array}$$

which turns out to be a t-distribution:

$$Y\sim \text{St}(\mu ,\frac{\alpha +n/2}{\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\mu )}^2},2\alpha +n).$$

Conjugacy for the mean and variance

We make the following specifications:

$$\begin{array}{rl}{X}_i& \sim N(\mu ,\tau )i=1,\dots ,n\\ \mu & \sim N({\mu }_0,n_0\tau )\\ \tau & \sim \text{Gamma}(\alpha ,\beta )\end{array}$$

where the $X_i$ are assumed independent given $\mu ,\tau$. We refer to this prior as normal-gamma distribution.

To compute the posterior distribution $p(\mu ,\tau |x)$, we first compute $p(\mu |\tau ,x)$ and then find $p(\tau |x)$.

$$\begin{array}{rl}\tau & \sim p(\tau |x)\\ \mu & \sim p(\mu |\tau ,x)\end{array}$$

When $\tau$ is fixed, we are back in the setting of conjugacy for the mean and can simply use the results, plugging in $n_0\tau$ in place of ${\tau }_0$:

$$\begin{array}{rl}\mathbb{E}[\mu |\mathbf{x}]& =\frac{n\tau }{n\tau +n_0\tau }\bar{x}+\frac{n_0\tau }{n\tau +n_0\tau }{\mu }_0\\ & =\frac{n}{n+n_0}\bar{x}+\frac{n_0}{n+n_0}{\mu }_0\end{array}$$

and ${\tau }_{\text{post}}=n\tau +n_0\tau$.

We proceed to working out the marginal posterior of $\tau$:

$$\begin{array}{rl}p(\tau ,\mu |\mathbf{x},{\mu }_0,n_0,\alpha ,\beta )& \propto p(\tau |\alpha ,\beta )p(\mu |\tau ,{\mu }_0,n_0)p(\mathbf{x}|\mu ,\tau )\\ & \propto \left({\tau }^{\alpha -1}{e}^{-\beta \tau }\right)\left({\tau }^{1/2}{e}^{-\frac{n_0\tau }{2}(\mu -\mu 0{)}^2}\right)\left({\tau }^{n/2}{e}^{-\frac{\tau }{2}\sum _{i=1}^n{(x_i-\mu )}^2}\right)\\ & \propto {\tau }^{\alpha +n/2-1}{e}^{-(\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\bar{x})}^2)\tau }{\tau }^{1/2}{e}^{-\frac{\tau }{2}[n_0{(\mu -{\mu }_0)}^2+n(\bar{x}-\mu {)}^2]}\end{array}$$

We want to integrate out $\mu$, and the last factor can be viewed as the joint probability distribution of two Gaussian random variables $\mu$ and $\overline{x}$ that own the following reparameterization form:

$$\begin{array}{l}\overline{X}\sim \mu +\frac{1}{\sqrt{n\tau }}ϵ\\ \mu \sim {\mu }_0+\frac{1}{\sqrt{n_0\tau }}\delta \end{array}$$

Thus integrating out $\mu$ leaves us the marginal distribution of $\overline{X}$ which is still a Gaussian random variable with:

$$\mathbb{E}[\overline{X}]={\mu }_0,\text{Var}(\overline{X})=\frac{1}{n\tau }+\frac{1}{n_0\tau }.$$

The probability density function is

$$p(\overline{x})\propto {\tau }^{1/2}\exp \{-\frac{n{n}_0\tau }{2(n+n_0)}{(\bar{x}-{\mu }_0)}^2\}$$

In the end, we have

$$p(\tau |\mathbf{x},{\mu }_0,n_0,\beta )\propto {\tau }^{\alpha +n/2-1}\exp \{-(\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\bar{x})}^2+\frac{n{n}_0}{2(n+n_0)}{(\bar{x}-{\mu }_0)}^2)\tau \}$$

which is a gamma distribution $\text{Gamma}(\alpha +n/2,\beta +\frac{1}{2}\sum _{i=1}^n{(x_i-\bar{x})}^2+\frac{n{n}_0}{2(n+n_0)}{(\bar{x}-{\mu }_0)}^2)$.