Linear Regression

Statistical regression method used for predictive analysis by modeling the linear relationship between independent and dependent variables.

How it works

Takes independent variables (i.e. covariate, features, predictors, input variables) and dependent variables (i.e. response, output, “thing to be estimated”) and produces the coefficient(s) and intercept for a linear equation (e.g. (β1, β0) for y = β1x + β0) which predicts the relationship between the independent and independent variables by minimizing a cost function, Mean Squared Error, either directly in the case of univariate linear regression or by gradient descent* in the case of multivariate linear regression.

Considerations

• There are four principal assumptions required for good results using linear regression (the first letters of the four principal assumptions form the "LINE" mnemonic):
  • Linearity and Additivity
  • Independent Residuals
  • Normal Residual Distributions
  • Equal Variances (i.e. homoscedasticity)
• Linear regression is a low variance/high bias model.
• Optimizers like Adam, Batch, and Mini-Batch and others are available for certain applications and data sets.
• A large learning ratio or training coefficient may lead to divergent behavior of the model and too small of values may lead to long run times and inefficiency.

Verification Approach

• Models are often evaluated by examining one or more of the metrics of R2, Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE).
• While there is no generally accepted single best performance metric as a criterion, users of linear regression should consider the suitability of one or more of these metrics for assessing the performance of their model.
• Use well known data sets to verify model execution.

Validation Approach

• Violating the principal assumptions of linear regression results in poor or misleading results.
• Ensure data is truly representative and if there are any known biases.

References

1. Gawali, Suvarna. “Linear Regression Algorithm to Make Predictions Easily.” Analytics Vidhya, 22 July 2022, https://www.analyticsvidhya.com/blog/2021/06/linear-regression-in-machine-learning/.
2. Nau, Robert. “Statistical Forecasting: Notes On Regression and Time Series Analysis.” Introduction to Linear Regression Analysis, Duke University Fuqua School of Business, 18 Aug. 2020, https://people.duke.edu/~rnau/regintro.htm.
3. Ng, Ritchie. “Evaluating a Linear Regression Model.” Ritchieng.github.io, 8 Jan. 2023, https://www.ritchieng.com/machine-learning-evaluate-linear-regression-model/.
4. Bochkarev, Alexei. "A New Typology Design of Performance Metrics to Measure Errors in Machine Learning Regression Algorithms", 2019, https://www.researchgate.net/publication/330661543_A_New_Typology_Design_of_Performance_Metrics_to_Measure_Errors_in_Machine_Learning_Regression_Algorithms.