## Thursday, August 30, 2012

### Logit Model Maximum Likelihood Estimate

In the previous post, I derived the log likelihood for the logit model.  In this post, I'll pick up where I left off and present the score functions followed by the maximum likelihood estimates (MLEs) for $\alpha$ and $\beta$.

The score vector, $U(\theta) = [U(\theta)_\alpha \: U(\theta)_\beta]^T$, is arrived at by taking the derivative of the log likelihood with respect to $\alpha$ and $\beta$, respectively.

Recall the log likelihood: $\ell(\theta) = (m_1)\alpha + a\:\beta - (n_1)log(1 + e^{\alpha + \beta}) - (n_2)log(1 + e^{\alpha})$

Now take the derivatives (recall the properties of taking derivatives of logs) then substitute the $\alpha$ and $\beta$ expressions with the equivalent $\pi_i$ expressions.

$U(\theta)_\alpha = \frac{\partial \ell}{\partial \alpha} = m_1 - n_1\frac{e^{\alpha + \beta}}{1 + e^{\alpha + \beta}} - n_2\frac{e^{\alpha}}{1 + e^{\alpha}} = m_1 - n_1\pi_1 - n_2\pi_2$

$U(\theta)_\beta = \frac{\partial \ell}{\partial \beta} = a - n_1\frac{e^{\alpha + \beta}}{1 + e^{\alpha + \beta}} = a - n_1\pi_1$

In order to obtain the MLEs, the two score equations above need to be set equal to zero then solved for the desired parameter but because there are two equations with two unknowns, let's first take the difference, $U(\theta)_\alpha - U(\theta)_\beta$.

$U(\theta)_\alpha - U(\theta)_\beta = m_1 - n_1\pi_1 - n_2\pi_2 - a + n_1\pi_1 = m_1 - n_2\pi_2 - a$

Solving for $\pi_2$ we then get $\pi_2 = \frac{m_1 - a}{n_2}$ which can be re-expressed in terms of $\alpha$ and $\beta$ as well as the cell values from a standard 2x2 table configuration.

Substituting, we have $\frac{e^{\alpha}}{1 + e^{\alpha}} = \frac{b}{n_2}$ which is now in a form that can be solved for $\hat a$:

$\frac{e^{\hat \alpha}}{1 + e^{\hat \alpha}} = \frac{b}{n_2} \ \Rightarrow \ e^{\hat \alpha} = \frac{b}{n_2}(1 + e^{\hat \alpha}) \ \Rightarrow \ e^{\hat \alpha} = \frac{b}{n_2} + \frac{b}{n_2}e^{\hat \alpha}$

Following more algebraic manipulation and the recognition that the difference between $n_2 - b$ is $d$ (per the configuration of a standard 2x2 table), you get $e^{\hat \alpha} = \frac{b}{d}$ but since $\hat \alpha$ is sought, the natural log must be taken of both sides, resulting in the MLE for $\alpha$ as $\hat \alpha = log(\frac{b}{d})$.

The MLE for $\beta$ follows in a similar manner (set the $\beta$ element from the score vector to zero then solve by way of substitution and algebraic manipulation).

$U(\theta)_\beta = a - n_1\pi_1 = a - n_1\frac{e^{\hat \alpha + \hat \beta}}{1 + e^{\hat \alpha + \hat \beta}} = 0$

We know that $\hat \alpha = log(\frac{b}{d})$ and if we substitute accordingly, we get:
$a - n_1\frac{e^{log(\frac{b}{d}) + \hat \beta}}{1 + e^{log(\frac{b}{d}) + \hat \beta}} = a - n_1\frac{{(\frac{b}{d})e^{\hat \beta}}}{1 + (\frac{b}{d})e^{\hat \beta}} = 0$

After more algebra, grouping like terms $(e^\beta)$, and making appropriate substitutions, we eventually get $e^{\hat \beta} = \frac{ad}{bc}$ which can be natural logged to obtain $\hat \beta$ such that $\hat \beta = log\Bigl(\frac{ad}{bc}\Bigr)$.  The ratio of $ad$ to $bc$ in a binary logit setting is, of course, the odds ratio, thus the MLE of $\beta$ is the log odds ratio, $\hat \beta = log(OR)$.