Curve Fitting and Regression

Curve Fitting and Regression

Experimental data is always noisy. Curve fitting constructs a mathematical model that represents the underlying trend while tolerating random error. Unlike interpolation, the fitted curve need not pass through every data point.

Method of Least Squares

The dominant technique is the method of least squares, which minimizes the sum of squared residuals S = Σ (y_i − f(x_i))². Least squares is computationally tractable, has a unique solution for linear models, and has sound statistical interpretations when errors are normally distributed.

Linear Regression

The simplest model is a line y = a + b x. Minimizing S leads to the normal equations: b = (nΣ x_iy_i − Σ x_i Σ y_i) / (nΣ x_i² − (Σ x_i)²), and a = (Σ y_i − b Σ x_i) / n. This linear regression is implemented in every statistics package.

Polynomial Regression

When linear models are inadequate, a polynomial y = a₀ + a₁x + ... + a_mx^m can be fit. The parameters still satisfy a linear system, though the system becomes ill-conditioned for high degrees. Orthogonal polynomials mitigate this.

Multiple Regression

With multiple independent variables, multiple linear regression fits y = β₀ + β₁x₁ + ... + β_kx_k. The solution is given in matrix form by β = (X^TX)^-1X^Ty. Modern methods use QR decomposition or singular value decomposition for numerical stability.

Nonlinear Regression

When the model cannot be made linear by a transformation, we minimize S iteratively. Gauss–Newton and Levenberg–Marquardt algorithms are common. They require initial guesses and can converge to local minima, demanding careful use.

Linearization of Nonlinear Models

Some nonlinear models become linear after transformation. Exponential y = ae^bx yields ln y = ln a + bx; power law y = ax^b yields log y = log a + b log x. Transformation alters the error structure, so results differ slightly from direct nonlinear fits.

Goodness of Fit

The coefficient of determination R² measures the fraction of variance explained by the fit, ranging from 0 (no fit) to 1 (perfect). The correlation coefficient r is its square root in linear regression. Residual plots reveal whether the model captures systematic trends or merely fits noise.

Overfitting and Model Selection

Adding more parameters always decreases residual sum of squares but can capture noise rather than signal—overfitting. Information criteria (AIC, BIC), cross-validation, and statistical hypothesis tests help choose a model with an appropriate complexity.

Outliers and Robust Regression

Least squares is sensitive to outliers; a single bad data point can dominate the fit. Robust regression methods, such as least absolute deviations (L1) or M-estimators like Huber's, reduce outlier influence.

Summary

Curve fitting extracts structure from noisy data. Linear and polynomial regression solve most practical problems; nonlinear regression handles the rest. Beyond fitting, thoughtful analysis of residuals and model complexity is essential for reliable conclusions.