Correlation and Regression

Correlation and Regression

Correlation measures the strength of association between variables, and regression models one variable as a function of others. Both are foundational tools for exploring relationships and making predictions from data.

Scatter Plots

Before any numerical summary, a scatter plot of Y versus X reveals shape, direction, and outliers. Linear trends call for Pearson correlation and linear regression; nonlinear patterns suggest transformations or alternative models.

Pearson Correlation Coefficient

The Pearson correlation coefficient r measures linear association: r = Σ(x − x̅)(y − y̅) / √(Σ(x − x̅)² Σ(y − y̅)²). It lies in [−1, 1]. Values near +1 or −1 indicate strong linear association; near 0 indicates weak linear association. r says nothing about nonlinear relationships.

Spearman's Rank Correlation

Spearman's ρ applies Pearson correlation to ranks. It detects monotone relationships and is robust to outliers. Useful when data are ordinal or when relationships are nonlinear but monotone.

Simple Linear Regression

Simple linear regression fits y = β₀ + β₁ x + ε. Ordinary least squares minimizes Σ ε_i². The slope β̂₁ = r ⋅ (s_y/s_x); the intercept is β̂₀ = y̅ − β̂₁ x̅. Fitted values and residuals are straightforward to compute.

Coefficient of Determination

R² is the fraction of variance in y explained by the model: R² = 1 − SS_res/SS_tot. For simple linear regression, R² = r². Higher values indicate better fit, but high R² alone does not prove causation or guarantee predictive accuracy.

Inference for Regression

Under standard assumptions (linearity, independent normal errors with constant variance), the slope estimator is normal. t-tests assess whether β₁ differs from zero; confidence intervals measure uncertainty about the slope. Prediction intervals quantify uncertainty in future observations.

Multiple Linear Regression

Multiple linear regression models y = β₀ + β₁ x₁ + ... + β_k x_k + ε. Matrix form β̂ = (X^TX)⁻¹X^Ty gives the OLS estimator. Adjusted R² penalizes adding irrelevant predictors.

Assumptions and Diagnostics

Residual plots check linearity, constant variance, and normality. The Durbin–Watson test detects autocorrelation in residuals. Multicollinearity (high correlation among predictors) inflates variance of estimates and is detected by variance inflation factors.

Nonlinear and Logistic Regression

When y is binary, logistic regression models the log-odds as linear. When y is count data, Poisson regression applies. More general nonlinear models are fitted by iterative methods like Gauss–Newton or Levenberg–Marquardt.

Correlation Is Not Causation

Strong correlation does not imply that one variable causes the other. Lurking variables, reverse causation, and selection bias can all create spurious correlations. Causal inference requires experimental design or specialized methods like instrumental variables.

Summary

Correlation quantifies linear or monotone association; regression models relationships for explanation or prediction. Together they are the workhorses of statistical analysis, machine learning, and applied data science.