- \n
- Understanding the goals of the simple linear regression model and what its limitations are.<\/li>\n
- Assumptions of Linear Regression<\/li>\n
- Preparing the data for a simple linear model by examining covariance and performing correlation analysis.<\/li>\n
- Understanding the basic formulae and mathematical notations and formulas used in linear modeling<\/li>\n
- Estimating parameters, minimizing residuals, calculating the coefficient of correlations, and generating other statistics.<\/li>\n
- Performing Least Squares Regression to find the line of best fit for a set of data<\/li>\n
- Interpreting Results and making predictions.<\/li>\n<\/ul>\n
Goals of Linear Regression<\/h2>\n
Regression analysis is a statistical framework for quantifying the relationship between a dependent variable and one or more independent variables. Regression analysis comes in many forms\u2014linear, logistic, ride, polynomial, and more\u2014many more! Each has an application for datasets with specific characteristics. Generally, these models can be categorized as linear regression<\/strong>, multiple linear regression<\/strong>, and nonlinear regression<\/strong>.<\/p>\n
Understanding Variable Relationships<\/h3>\n
The goal of linear regression is to predict the value of the dependent variable based on the observed value of an independent variable. These relationships are often described in terms of the coefficient of correlation (r2<\/sup>) and can range from positive to negative, inclusive of a completely uncorrelated relationship.<\/p>\n
In the case of simple linear regression<\/strong>, this goal is achieved via modeling the relationship between a dependent variable and a single independent variable.<\/p>\n
In the case of multiple linear regression<\/strong>, the relationship between the dependent variable is considered with respect to two or more independent variables. The focus of this article is primarily on simple linear regression.<\/p>\n
![\"coefficient](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/coefficient-of-determination-comparison-illustration-r2-1024x1024.jpg\" "\\"coefficient-of-determination-comparison-illustration-r2\\"")
<\/a>Here we see 3-charts showing data representing different coefficients of correlation (r2<\/sup>) values ranging from strong positive relationships to strong negative ones. (click to enlarge)<\/figcaption><\/figure>\n Linear regression is the simplest and most-used form of regression analysis. It fits a line of best fit<\/strong> such that the distance of predicted values from the mean of observed values is minimized. The formulae for varying Linear regression models are based on the algebraic slope-intercept form<\/strong>. Notation and form vary slightly but the concept of having a slope and passing through the y-axis when the predictor variable equals zero translates entirely.<\/p>\n
This formula generates a line that represents the minimal distance between a predicted value for an independent variable and its actual value. The goal of linear regression is to minimize the total squared quantity for all these values. This is done through the method of least squares<\/strong> and can be calculated formulaically by methods such as the ordinary least squares (OLS) or iteratively by using machine-learning algorithms such as gradient descent. Before these calculations can be made, however, linear regression requires certain assumptions about data to be validated.<\/p>\n
Assumptions of Linear Regression<\/h2>\n
For linear regression analysis to be effective, data much abide by a certain number of rules. These rules, which we’ll call assumptions<\/em>, are necessary for linear regression to be accurate. Generally, these are the five primary assumptions of linear regression analysis with respect to the data:<\/p>\n
- \n
- Linearity<\/li>\n
- Homoscedasticity<\/li>\n
- Independence<\/li>\n
- Normality<\/li>\n
- No Multicollinearity<\/li>\n<\/ol>\n
Before proceeding to implement a linear regression model, one needs to examine data to ensure these assumptions hold true. If any of these assumptions prove to be false Linear Regression will not provide accurate conclusions. Before we talk about applying linear regression we’ll take a moment to discuss each of these assumptions and how to validate them.<\/p>\n
Linearity<\/h3>\n
![\"gauss](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/gauss-markov-theorum-1-linearity-1024x673.jpg\" "\\"gauss-markov-theorum-1-linearity\\"")
<\/a>An illustration of a non-linear relationship between variables in violation of the first Gauss-Markov assumption.<\/figcaption><\/figure>\n Linear regression models the linear relationships between variables. As one might imagine, if no such linear relationship exists there is little cause for linear regression analysis. Your computer isn’t going to explode from doing a linear regression on data without a linear relationship\u2014it will simply produce a model that is underfit<\/em><\/strong>. Fortunately, checking for linearity can be done simplify enough by any of the following methods (in descending order computational complexity).<\/p>\n
- \n
- Scatterplots<\/strong>: A plot of the data, combined with the infallible eyeball<\/em> test, is often good enough to make a quick assessment of data linearity (Correll, 2017).<\/li>\n
- Partial Residuals<\/strong>: A plot of the partial residuals can help assess the linearity of data as well, but requires a bit more computational effort than just plotting the data (Hastie, 1990). Ultimately, it falls to the eyeball test for validation as well.<\/li>\n
- Covariance Analysis (COV)<\/strong>: Provides a quantitative description of the direction<\/em> of linearity among variables. If COV returns a near-zero value it suggests there is no linear relationship between data (Kim, 2008).<\/li>\n
- Correlation Analysis (COR)<\/strong>: Methods such as Pearson’s Correlation Coefficient can help describe the linearity of data. However, by the time one calculates this a full-on regression analysis has been completed. This calculation is better used to verify final results rather than initial assumptions.<\/li>\n<\/ul>\n
Homoscedasticity (constant variance)<\/h3>\n
![\"gauss](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/gauss-markov-theorum-2-homoscedasticityjpg.jpg\" "\\"gauss-markov-theorum-2-homoscedasticityjpg\\"")
<\/a>An illustration of a heteroscedastic relationship between variables in violation of the second Gauss-Markov assumption.<\/figcaption><\/figure>\n Homoscedasticity describes a situation where the variance<\/em> of all observed values for the response variable has equal residual values\u2014a measure of distance between an observed value and the line of best fit. Data is said to be homoscedastic <\/strong>when these residual values are the nearish same for all observed values of X and heteroscedastic<\/strong> when the residuals values vary significantly. These differences can be assessed via the infallible eyeball test once again as a scatterplot. Also, several mathematical approaches<\/a> are available as well:<\/p>\n
- \n
- Levene’s Test<\/li>\n
- Park Test<\/li>\n
- Glejser Test<\/li>\n
- Breusch-Pagan Test<\/li>\n
- White Test<\/li>\n
- Goldfield-Quandt Test<\/li>\n<\/ul>\n
When heteroscedasticity is present ordinary least squares linear regression is not an appropriate method. There are a number of ways to deal with these cases but they can be quite technical in nature. For a deeper look, check out this great post on heteroscedasticity by Keita Miyaki<\/a>.<\/p>\n
Independence<\/h3>\n
![\"gauss](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/gauss-markov-theorum-3-indepdendece.jpg\" "\\"gauss-markov-theorum-3-indepdendece\\"")
<\/a>An autocorrelation plot showing a dependency among predictor variables in violation of the third Guass-Markov assumption<\/figcaption><\/figure>\n Data can come in many forms but often can be separated into one of two types: longitudinal or cross-sectional. Longitudinal data reflects multiple measures of a single entity over a period of time. For example, the closing price of a stock being recorded every day for a month. Cross-sectional data reflects a single measure in time for multiple entities. For example, the average yearly income reported for a sample population of 100 day-traders.<\/p>\n
Longitudinal data has a property described as autocorrelation<\/a> which describes a self-correlative relationship among the residuals. Autocorrelation can be checked for by the use of the Durbin-Watson statistic<\/a>, the Breusch-Godfrey test, or the Ljung-Box test. The Durbin Watson test is the most common and has implementations in many common computing packages such as the
statsmodels<\/code> in Python (see here<\/a>.) The Durbin-Watson statistics is scored on a range of 0-4 where the autocorrelation is assessed as follows;<\/p>\n- \n
- 0 \u2264 Score < 2<\/strong>: Positive autocorrelation<\/li>\n
- Score == 0<\/strong>: No autocorrelation<\/li>\n
- 2 \u2264 Score < 4<\/strong>: Negative autocorrelation<\/li>\n<\/ul>\n
Avoiding independence in data is a key assumption to linear regression, characterized by the absence of autocorrelation. Do not be fooled though; autocorrelation analyses are incredible approaches to drawing a different range of insights from data\u2014just not via regression analysis.<\/p>\n
Normality<\/h3>\n
![\"gauss](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/gauss-markov-theorum-4-normality.jpg\" "\\"gauss-markov-theorum-4-normality\\"")
<\/a>Q-Q Plot of residuals illustrating an approximately-normal distribution in accordance with the fourth Gauss-Markov assumption.<\/figcaption><\/figure>\n Linear regression models seek to minimize the difference between an expected value predicted by the model and the observed value. The difference in these two values is referred to as the error<\/em> (a.k.a residual.) How these terms are distributed across a series of observed values can predict issues with a linear model’s ability to accurately predict response values. Testing for normality helps ensure that the dependent variable has no effect on observed values for independent variables. The following tests can be used:<\/p>\n
- \n
- Kolmogorov-Smirnov<\/li>\n
- Shapiro-Wilk<\/li>\n
- Jarque-Bera<\/li>\n
- Anderson-Darling<\/li>\n<\/ul>\n
Normality testing is not technically<\/em> a requirement for linear regression validation but is certainly both common<\/em> and best<\/em> practice. Normality testing can be skipped on the assumption that one’s model equation is correct and the only purpose is to estimate coefficients and generate predictions about the mean squared error. In some cases, transformations resulting from correcting for normality issues can cause even larger problems, especially in large datasets (Schmidt, 2018).<\/p>\n
Issues with normality arise when the distributions of observed values for variables are non-normal or when the assumption of linearity is not true. Also possible are cases where only a few very large errors exist in a dataset, thus skewing the summary statistics, or in cases where multiple subsets of data are present (see bimodal distributions.)<\/p>\n
Multicollinearity<\/h3>\n
![\"gauss](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/gauss-markov-theorum-5-collinearity.jpg\" "\\"gauss-markov-theorum-5-collinearity\\"")
<\/a>A correlation matrix showing several strong linear relationships among predictor variables in violation of the Gauss-Markov assumption of non-Multicollinearity<\/figcaption><\/figure>\n Multicollinearity describes the case where correlative relationships exist between two or more independent variables. Linear regression models the linear relationship between a dependent variable and one or more independent variables. When one or more linear relationships\u00a0also<\/em> exist between independent variables there can be problems such as overfitting. The stronger the relationship the bigger the issue. Multicollinearity can be categorized as one of two types:<\/p>\n
- \n
- Structure Multicollinearity<\/strong>: Collinearity results from creating independent variables from other predictive variables such as the square of another variable.<\/li>\n
- Data Multicollinearity<\/strong>: Interrelationships among independent variables that are present in the data before any transformations are applied.<\/li>\n<\/ul>\n
Each of these two forms of collinearity can cause issues but, arguably, the case of data multicollinearity is more problematic. Structural collinearity is often easily remedied by removing transformations to the data whereas data collinearity may require transformations. In the case of the latter, further analysis is usually required. There are several tests and characteristics of data that can hint at issues of multicollinearity. Below are some common indicators:<\/p>\n
- \n
- Large changes in regression coefficients with small additions or subtractions of independent variables<\/li>\n
- When multivariate regression finds little relationship between an independent variable but simple linear regression finds a significant relationship when isolating that variable.<\/li>\n
- Farrar-Gluaber Test<\/li>\n
- Condition Number Test<\/li>\n
- Re-running regression testing with added noise (perturbation)<\/li>\n<\/ul>\n
In addition to these tests, correlation matrices<\/a> can be used to both numerically and visually detect multilinearity. In the illustration above, relationships denoted in either dark burgundy or light indicate strong linear correlations among predictor variables.<\/p>\n
Preparing the Model<\/h2>\n
Simple linear regression seeks to predict the value of the dependent variable given an observed value for an independent variable. As we’ve seen, a number of assumptions must hold true for this to be possible. The first assumption of linearity among our variables can be assessed explicitly by calculating the covariance<\/a> and the correlation coefficient<\/a>. These two measures can provide insight into the direction and magnitude of the relationship between variables and are essential concepts of linear regression.<\/p>\n
Covariance<\/h3>\n
![\"covariance](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/covariance-equation-diagram-1024x576.jpg\" "\\"covariance-equation-diagram\\"")
<\/a>The Covariance of two variables can be calculated by summing the difference of each observed value and the sample mean divided by the total sample population minus one. (click to enlarge)<\/figcaption><\/figure>\n Covariance<\/a> describes the directional relationship of linearity between two variables. This measure can help estimate where a linear relationship exists but offers no insight into the magnitude of that relationship.<\/p>\n
Covariance is calculated by multiplying the difference between observed values of each variable X, Y, and their respective sample means and dividing by the total observed values – 1. This last step is known as Bessel’s Correction<\/a> and is common among sample population statistics.<\/p>\n
Correlation Coefficient<\/h3>\n
![\"correlation](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/correlation-coefficient-equation-diagram-1024x576.jpg\" "\\"correlation-coefficient-equation-diagram\\"")
<\/a>The correlation coefficient provides a standardized measure of the correlation between variables<\/figcaption><\/figure>\n The correlation coefficient<\/a> is an extension of utility over covariance and provides a measure of both the direction and magnitude of the relationship between variables. This measure, standardized to a range of -1 to 1 can indicate a positive correlation<\/strong>, negative correlation<\/strong>, or zero correlation<\/strong> between variables. The correlation can be regarded as the covariance between the standardized variables or the ratio of covariance to the standard deviation of the variables (Chatterjee, 2012).<\/p>\n
Edge Cases<\/h3>\n
![\"anscombes](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/anscombes-quarter-graphed-e1626301356124-1024x717.jpg\" "\\"anscombes-quarter-graphed\\"")
<\/a>Anscombe’s four data sets all possess near-identical summary statistics yet have very different relationships between variables. (click to enlarge)<\/figcaption><\/figure>\n The illustration above shows four sets of data, known as Anscombe’s Quartet, where almost identical summary statistics are calculated for vastly different data sets. Covariance and Correlation tests will fail to spot the issues with these datasets\u2014and likely many more out there. The Anscombe’s Quarter is the canonical warnings of why one should always visually inspect data before diving too deeply into creating a linear model for prediction.<\/p>\n
Understanding the Model<\/h2>\n
One can regard linear regression as an extension to correlation analysis. Whereas correlation analysis only describes the relationship between variables in terms of strength and direction, regression is able to numerically describe that relationship. This allows for powerful predictive ability and has lead to linear regression being among the most utilized statistical models in scientific study (Hyatt, 2017).<\/p>\n
Linear regression, be it simple or multiple regression, uses a linear model that is built atop the classic slope-intercept form
y = mx + b<\/code>. This formula is the cornerstone of linear equations and is introduced in most introductory algebra classes. Here we’ll re-introduce<\/i>\u00a0this formula with slightly different notation allowing for flexibility among different linear models:<\/p>\n![\"slope](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/slope-intercept-vs-linear-regression-equations-1024x576.jpg\" "\\"slope-intercept-vs-linear-regression-equations\\"")
<\/a>The model for linear regression is conceptually based on the slope-intercept model for linear equations. (click to enlarge)<\/figcaption><\/figure>\n Before we proceed it’s important to have a basic recognition of how and why the terms of the slope-intercept form have been adapted for use with linear regression models. If for no other reason, this knowledge is important in helping one understand how the linear model’s notation changes based on new features and analytical methods. Let’s take a quick walk through the terms of this equation.<\/p>\n
- \n
- Y – Representing the\u00a0dependent variable,<\/em> not its value. Comparable to the
y<\/code> term in slope-intercept form.<\/li>\n- \u00df0 – The\u00a0constant<\/em>\u00a0coefficient representing the y-intercept<\/em> where an observed value of X is equal to zero. i.e.
y = 0<\/code>. Comparable to theb<\/code> term in slope-intercept.<\/li>\n- \u00df1 – A coefficient representing the slope<\/em> of the regression line; comparable to the
m<\/code> term in slope-intercept.<\/li>\n- X – Representing the\u00a0independent variable<\/em>, not its value. Comparable to the
x<\/code> term in slope-intercept form.<\/li>\n- \u03b5 – Representing the residual (error) term describing how far away a predicted value is from an observed value. Not represented in slope-intercept form.<\/li>\n<\/ul>\n
These terms represent an estimation of the relationship between variables X and Y such that Y is an approximately linear function of X where \u03b5 represents the error in that estimation.<\/p>\n
Understanding the Notation<\/h2>\n
These values are represented differently across many textbooks. In addition, there are some nuances that one should be aware of. Namely, the equation we have discussed so far is the general equation for linear regression modeling. That is, it describes the formula by which each observed value should be analyzed (Pardoe, 2012).<\/p>\n
As we begin working through our analysis, the equation’s form will change slightly to express which observed values we are currently considering. The capital<\/em> Y represents all observed values<\/strong> of the independent variable whereas\u00a0lowercase<\/em> y represents a single observed value<\/strong> for that variable.<\/p>\n
These may seem like nuances but become quite important to understand as we begin to represent Y differently as we progress through the stages of simple linear regression analysis. For example, this is the formula for a single observed value of X and Y, represented as the ith term.<\/p>\n
![\"single](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/single-term-linear-equation-expression-1024x576.jpg\" "\\"single-term-linear-equation-expression\\"")
<\/a>The linear equation representing a single observed value within a pair of variables X and Y. (click to enlarge)<\/figcaption><\/figure>\n As a regression analysis progresses through a series of variables X and Y, this equation is expressed with
i<\/code> being replaced for whichever term is currently being considered. For example, the following illustration shows the series of all observed values from 1-n where n represents the last observation:<\/p>\n![\"linear](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/linear-equation-n-terms-expression-1024x576.jpg\" "\\"linear-equation-n-terms-expression\\"")
<\/a>The linear equation expressing terms 1 – n where there are n-total observed values for each variable X and Y. (click to enlarge)<\/figcaption><\/figure>\n At this point we can understand how to approach the many observed values for our variables X and Y. Additionally, we can understand how the terms of the linear equation can be compared to the slope-intercept form of linear algebra in a way that a line can be generated through our data points.<\/p>\n
Now we need a way to ensure that this line fits our data in a way that affords the greatest possibility of our predicted values being the same as our observed values. In other words, we need to minimize the error between our guesses and observations. There are a number of ways to approach this but the most common is the Ordinary Least Squares (OLS)<\/strong> method. This method, along with alternatives, represent the general approach of finding the least-squares regression line<\/strong> (Fox, 2015)<\/p>\n
Building the Model<\/h2>\n
To find the line of best fit<\/strong> for our data we can calculate the slope (m, \u00df1<\/sub> coefficient) and intercept (b, \u00df0 <\/sub>coefficient). This will give us the equation of a line that will minimize the distance of our predicted values<\/strong> to the mean of our observed values<\/strong>. This collection of resulting distances are referred to as residuals<\/strong> (a.k.a. errors<\/em>) and can be used to assess the goodness of fit<\/strong> resulting from our regression.<\/p>\n
This slew of new vocabulary represents a lot of moving pieces. Let’s start walking through things step-by-step to get a good idea of how things work. First, let’s consider some sample data:<\/p>\n
- \u00df0 – The\u00a0constant<\/em>\u00a0coefficient representing the y-intercept<\/em> where an observed value of X is equal to zero. i.e.
- Y – Representing the\u00a0dependent variable,<\/em> not its value. Comparable to the
- Data Multicollinearity<\/strong>: Interrelationships among independent variables that are present in the data before any transformations are applied.<\/li>\n<\/ul>\n
- Structure Multicollinearity<\/strong>: Collinearity results from creating independent variables from other predictive variables such as the square of another variable.<\/li>\n
- Score == 0<\/strong>: No autocorrelation<\/li>\n
- 0 \u2264 Score < 2<\/strong>: Positive autocorrelation<\/li>\n
- Partial Residuals<\/strong>: A plot of the partial residuals can help assess the linearity of data as well, but requires a bit more computational effort than just plotting the data (Hastie, 1990). Ultimately, it falls to the eyeball test for validation as well.<\/li>\n
- Scatterplots<\/strong>: A plot of the data, combined with the infallible eyeball<\/em> test, is often good enough to make a quick assessment of data linearity (Correll, 2017).<\/li>\n
![\"coefficient](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1156\/coefficient-of-determination-comparison-illustration-r2.jpg\" "\\"coefficient-of-determination-comparison-illustration-r2\\"")
<\/a>