Generalized Linear Models

Sometimes $Y$ is neither quantitative or qualitative. Like a non-negative integer value (counts).

Linear Regression

Some of the issues that arise from estimating a non-qualitative observation can be corrected using a transformation:

log (Y) = \sum_{j = 1}^{p} X_{j} β_{j} + ϵ

A transformation leads to challenges in interpretation and can't be applied to situations where the response can take on a value of 0.

Poisson Regression

The Poisson distribution is typically used to model counts

The distribution is modified to accommodate multiple means: $λ = E (Y)$ written as $λ (X_{1}, \dots, X_{p})$ with covariates $X_{1}, \dots, X_{p}$ :

log (λ (X_{1}, \dots, X_{p})) = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}

equivalently λ (X_{1}, \dots, X_{p}) = e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}

where $β_{0}, β_{1}, \dots, β_{p}$ are parameters to be estimated:

l (β_{0}, β_{1}) = \prod_{i = 1}^{n} \frac{e^{- λ (x_{i}) λ (x_{i})^{y_{i}}}}{y_{i}!}

where λ (x_{i}) = e^{β_{0} + β_{1} x_{i 1} + \dots + β_{p} x_{i p}}

Generalized Linear Models in Greater Generality

Comparisons between linear, logistic, and Poisson regression models

Each approach uses predictors $X_{1}, \dots, X_{p}$ to predict a response $Y$ . We assume that, conditional on $X_{1}, \dots, X_{p}$ , $Y$ belongs to a certain family of distributions. For linear regression, we typically assume that $Y$ follows a Gaussian or normal distribution. For logistic regression, we assume that $Y$ follows a Bernoulli distribution. Finally, for Poisson regression, we assume that $Y$ follows a Poisson distribution.
Each approach models the mean of $Y$ as a function of the predictors. In linear regression, the mean of $Y$ takes the form

E (Y | X_{1}, \dots, X_{p}) = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}

For logistic regression, the mean of $Y$ takes the form

E (Y | X_{1}, \dots, X_{p}) = \frac{e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}}{1 + e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}}

Poisson regression takes the form

E (Y | X_{1}, \dots, X_{p}) = λ (X_{1}, \dots, X_{p}) = e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}

These equations can be expressed using a link function $η$ , which applies a transformation to $E (Y | X_{1}, \dots, X_{p})$ so that the transformed mean is a linear function of the predictors. That is,

η (E (Y | X_{1}, \dots, X_{p})) = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}

The link functions for linear, logistic and Poisson regression are $η (μ) = μ$ , $η (μ) = log (μ / (1 - μ))$ , and $η (μ) = log (μ)$ , respectively.

The exponential family of distributions includes Gaussian, Bernoulli, Poisson, exponential distribution, Gamma, and negative binomial.

Any regression approach that follows this very general recipe is known as a generalized linear model (GLM). Thus, linear regression, logistic regression, and Poisson regression are three examples of GLMs. Other examples not covered here include Gamma regression and negative binomial regression.

Sources: 1

Generalized Linear Models

Linear Regression

Poisson Regression

Generalized Linear Models in Greater Generality

Connect With Me!