- \n
- Covariance is used in correlation analysis to describe the direction of correlative relationships between variables.<\/li>\n
- Correlation coefficients lend a hand at describing the magnitude of correlative relationships between variables.<\/li>\n
- Correlation analysis generally results in a standardized value on a scale of -1 to 1 where -1 is a negative correlation, 0 is no correlation, and 1 is a strong positive correlation.<\/li>\n
- Subtle quirks in data can result in functionally inaccurate conclusions drawn from correlation techniques.<\/li>\n
- Correlation analysis is utilized in machine learning to find better features, reduce computational complexity, and select the most appropriate algorithm<\/li>\n
- Correlation coefficients are integral to regression analysis<\/li>\n<\/ul>\n
Introduction<\/h2>\n
Correlation is a term used to describe the relationship between two variables where the change in the value of one affects the value of the other. A common example of correlated values is illustrated by the measure of a person’s height and the height of their parents. Larger measures of height are observed among those with parents that are taller.<\/p>\n
The terms correlation and dependence are synonymous and are used interchangeably. It is important to note that correlation does not imply causation and, in many cases, the causal influence on an observed value may be reflected in another variable\u2014a possibly unknown one.<\/p>\n
Simply put<\/strong>; correlation is present when a change in variable X causes a change in variable Y. Correlation is assumed to describe a linear relationship between variables where one variable reflects a unit increase for change in the value of the other.<\/p>\n
Correlation Analysis<\/h2>\n
In any case, correlation analysis is used to establish if there is a correlative<\/em> relationship between two variables<\/strong> and, if so, what the nature of that relationship is (positive or negative) and what the strength of that relationship is.<\/p>\n
Correlation analysis is used to measure the relationship between two variables, typically an independent<\/a> and dependent variable<\/a>. The term multicollinearity<\/em><\/a> is used to describe cases where two independent variables express a correlation.<\/p>\n
Correlation Coefficients<\/h2>\n
A correlation between two variables describes how\u00a0strong<\/em> the relationship is. This measure is expressed as a standardized value<\/em> on the interval -1 to 1 and is referred to as the correlation coefficient<\/strong> and is commonly represented by the letter “r”. Below are descriptions of the extreme points found on the standardized correlation coefficient interval.<\/p>\n
- \n
- 1 (Positive)<\/strong>: When variable X moves, variable Y moves in the same direction<\/em>.<\/li>\n
- -1 (Negative)<\/strong>: When variable X moves, variable Y moves in the opposite direction<\/em><\/li>\n
- zero No Correlation<\/strong>: When variable X moves, variable Y moves doesn’t move at all <\/em>or moves unpredictably<\/em>.<\/strong><\/li>\n<\/ul>\n
Below are some visual examples of randomly generated data illustrating relationships among X and Y variables described by the -1, 1, and zero types described above.<\/p>\n
Positive Correlation Coefficient<\/h3>\n
![\"positive](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/positive-correlation-scatterplot.jpg\" "\\"positive-correlation-scatterplot\\"")
The relationship between the X and Y variables reflects a strong positive relationship.<\/figcaption><\/figure>\n This scatterplot shows a strong positive linear relationship<\/strong> between variable X and Y such that an increase in observed value for the\u00a0 X (the independent variable) reflects a strong decrease<\/em> in value observed for the Y variable (the dependent variable.)<\/p>\n
Negative Correlation<\/h3>\n
![\"negative](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/negative-correlation-coefficient-scatterplot.jpg\" "\\"negative-correlation-coefficient-scatterplot\\"")
The relationship between the X and Y variables reflects a strong negative relationship.<\/figcaption><\/figure>\n This scatterplot shows a strong negative linear relationship<\/strong> between variable X and Y such that an increase in observed value for the\u00a0 X (the feature) reflects a strong decrease <\/em>\u00a0in value observed for the Y variable (the label.)<\/p>\n
Zero Correlation<\/h3>\n
![\"zero](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/zero-correlation-coefficient-scatterplot.jpg\" "\\"zero-correlation-coefficient-scatterplot\\"")
The relationship between the X and Y variables reflects no correlation in values at all<\/figcaption><\/figure>\n This scatterplot shows a zero\u00a0linear relationship<\/strong> between variable X and Y such that an increase in observed value for the\u00a0 X (the predictor variable) does not reflect a predictable change in the variable Y (the response variable.)<\/p>\n
Note<\/strong>: The X and Y variables are named differently in each of the above illustrations. This variation is only to illustrate how many different names are used for the same variables and does not reflect a necessity of terminology based on correlation coefficient value. i.e. the term “predictor” used to describe the X values in the zero correlation example could also be used to describe the X values in the positive and negative plots as well.<\/p>\n
Calculating Correlation Coefficients<\/h2>\n
Correlation coefficients provide ample information to describe the relationship between the variables of two series of observations. As such, it is a foundational component of many machine learning algorithms such as feature selection (Devaraj, 2015, Chicco, 2020.) The correlation coefficient formula for a sample population is as follows:<\/p>\n
![\"correlation](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/correlation-coefficient-formula-sample-population-e1626301250157-1024x570.jpg\" "\\"correlation-coefficient-formula-sample-population\\"")
The formula for the correlation coefficient of a sample population<\/figcaption><\/figure>\n The topmost part of this formula represents the covariance<\/strong> between two variables. It is a means of mathematical representation for the direction of correlation between variables. Calculating the covariance is the first step in calculating the correlation coefficient. Let’s take a closer look.<\/p>\n
Covariance<\/h2>\n
![\"covariance](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/covariance-quadrants-1-1024x672.jpg\" "\\"covariance-quadrants\\"")
<\/a>Plotting the sample means for variables X and Y illustrates the sign of covariance to help describe the linearity of their relationship. (click to enlarge)<\/figcaption><\/figure>\n Covariance is used to describe the direction of the relationship between two variables. That is, it can be used to predict how an observed value for variable X will move in response to a change in observed value for variable Y. A positive covariance<\/strong> indicates that an increase in observed value for variable X will be reflected by an increase in observed value for variable Y. Likewise, a decrease in X will be reflected by a decrease in Y when a negative covariance<\/strong> exists.<\/p>\n
The covariance has several notable characteristics one should always be mindful of:<\/p>\n
- \n
- It only describes the relationship between two variables.<\/li>\n
- It applies to series data, such that many observed values are available for each variable.<\/li>\n
- The number of observed values for each variable must be identical.<\/li>\n
- Covariance can express a positive or negative relationship between variables.<\/li>\n
- Covariance only describes\u00a0directional<\/em> relationships and not measures of magnitude which limits its broader applications.<\/li>\n<\/ol>\n
Covariance is used alone for the management of financial assets to control risk, can help forecast stock prices, and can be extended for use with multivariate data to tasks such as feature extraction and dimensionality reduction. Read this article<\/a> to learn more about calculating covariance, its applications, and various programmatic implementations in Python with libraries such as NumPy and Pandas. For now, let’s continue calculating the correlation coefficient.<\/p>\n
Correlation Coefficients<\/h2>\n
Covariance doesn’t tell us anything about the strength (magnitude) of the relationship between variables. Knowing one stock price will rise in response to a rise in another is hardly useful information unless one can estimate how much<\/em> the price of the second stock can be expected to move. This is where correlation coefficients come in.<\/p>\n
To avoid this pitfall, we can use standardization<\/em> to relate our variables in a way that allows for calculating direction and<\/em> magnitude. This is done by first calculating the sample standard deviation for both X and Y variables using the following formula:<\/p>\n
![\"standard](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/standard-deviation-sample-population-e1626301291530-1024x575.jpg\" "\\"standard-deviation-sample-population\\"")
The formula for calculating the standard deviation of a sample population.<\/figcaption><\/figure>\n This calculates the sample mean of each observed value for a variable and its distance from that variable’s sample mean. By multiplying the result of each variable and dividing by the sample population (minus one) we fit each variable onto the same scale. Kind of like converting between miles and kilometers.<\/p>\n
Once we have the sample standard deviation for each variable, we can re-write our initial formulate for the correlation coefficient more concisely as such:<\/p>\n
![\"simplified](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/simplified-correlation-coefficient-formula-vs-expanded-e1626301325382-1024x570.jpg\" "\\"simplified-correlation-coefficient-formula-vs-expanded\\"")
<\/a>A comparison of the simplified form of the correlation coefficient formula to the more expansive form. (click to enlarge)<\/figcaption><\/figure>\n This standardization process provides us with a value in the range of
-1 \u2264 Cor(X, Y) \u2264 1<\/code> where both direction and magnitude are represented. Correlation coefficient values of \u2264 0 indicate a negative relationship where values \u2265 0 indicate a positive relationship. It’s important to note that a correlation coefficient of 0 does not mean that two variables are not related. A value ofCor(X, Y) = 0<\/code> only guarantees that variablesX<\/code> andY<\/code> do not have a linear<\/em> relationship.<\/p>\nEdge Cases<\/h2>\n
The correlation coefficient, along with other descriptive statistics, should never be regarded as without error in describing data. The correlation coefficient is used to consider linear relationships between variables in a way that will fail to grasp nonlinear relationships that may be obvious when viewed visually.<\/p>\n
To illustrate this, statistician Francis Anscombe constructed a series of four datasets in the early 1970s that had near-identical summary statistics but were\u00a0very<\/em> different in nature. These are charted below:<\/p>\n
![\"anscombes](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1074\/anscombes-quarter-graphed-e1626301356124-1024x717.jpg\" "\\"anscombes-quarter-graphed\\"")
<\/a>Anscombe’s four data sets all possess near-identical summary statistics yet have very different relationships between variables. (click to enlarge)<\/figcaption><\/figure>\n The first quarter illustrates a common linear relationship, the second illustrates a non-linear relationship where a correlation coefficient would provide a subtly incorrect measure; the third shows a perfectly linear data set with a regression line offset by a single variable; the fourth illustrates how a single outlier variable can indicate a linear relationship where none exists (at least not practically.)<\/p>\n
Note<\/strong>: Sample data for the Anscombe’s Quartet shown above is available via Github<\/a>.<\/em><\/p>\n
Applications<\/h2>\n
The correlation coefficient provides a backbone for broader correlative analysis. This may be applied via traditional statistical means or modern approaches such as machine learning where correlation analysis can help extract meaningful features and improve model efficiency. Below are some common applications of correlation analysis that leverage the correlation coefficient:<\/p>\n
- \n
- Scheduling Analysis<\/li>\n
- Portfolio Analysis<\/li>\n
- Asset Price Forecasting<\/li>\n
- Parallelization Engineering (kind of like scheduling)<\/li>\n
- Cost Estimation<\/li>\n<\/ul>\n
What are the types of Correlation Analysis?<\/h2>\n
Correlation analysis is applied to a diverse range of analytical subjects that require adjustment for context among data such as units, bias, and other assumptions. The differences in data, sometimes subtle sometimes not, have produced a number of different types of approaches for calculating the correlation coefficient over the years. Below is a summary of the most common.<\/p>\n
- \n
- Pearson Correlation<\/strong>:<\/li>\n
- Kendall Rank Correlation<\/strong>:<\/li>\n
- Spearman Correlation<\/strong>:<\/li>\n
- Point-Biserial Correlation<\/strong>:<\/li>\n<\/ul>\n
For a more exhaustive discussion of the many different approaches for calculating the correlation coefficient read this blog post<\/a>. For our discussion, just keep in mind that the Pearson correlation coefficient is the most common method and the one reflected in the calculations presented in this article.<\/p>\n
Final Thoughts<\/h2>\n
The Correlation Coefficient provides insight into the relationship between variables such that meaningful insight can be gained. Estimates of both strength and direction can be applied to forecast stock prices, reduce dimensionality, and feature engineering. The correlation coefficient serves as a foundation to many machine learning models, regression models, and other statistical modeling tools.<\/p>\n
This article has presented only the basics of how this value is calculated and hinted at its applications. Check out this presentation by NASA<\/a> for a deeper, more technical, discussion on correlation coefficients and their role in correlation analysis. For additional insight, check out some of the most popular books on machine learning and artificial intelligence<\/a>.<\/p>\n
References<\/h2>\n
- \n
- Devaraj, Senthilkumar, and S Paulraj. \u201cAn Efficient Feature Subset Selection Algorithm for Classification of Multidimensional Dataset. \u201dTheScientificWorldJournal<\/i>\u00a0vol. 2015 (2015): 821798. doi:10.1155\/2015\/821798<\/li>\n
- Chicco, Davide, and Giuseppe Jurman. “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.”\u00a0BMC genomics<\/i>\u00a021.1 (2020): 1-13.<\/li>\n
- Mukaka, M M. \u201cStatistics corner: A guide to appropriate use of correlation coefficient in medical research.\u201d\u00a0Malawi medical journal : the journal of Medical Association of Malawi<\/i>\u00a0vol. 24,3 (2012): 69-71.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
Correlation analysis is used to quantify the linear relationship between variables. This analysis is integral to many modern statistical methods such as regression analysis. Correlation analysis helps develop more robust machine learning algorithms and more efficient data processing pipelines. Simple correlation calculations like covariance can provide basic insights whereas multivariate correlation matrices identify linear relationships […]<\/p>\n","protected":false},"author":2,"featured_media":1143,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,195,260],"tags":[434,443,433,445,202,385,239,444,238,442,395,193],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1074"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=1074"}],"version-history":[{"count":10,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1074\/revisions"}],"predecessor-version":[{"id":1672,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1074\/revisions\/1672"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/1143"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=1074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=1074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=1074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Chicco, Davide, and Giuseppe Jurman. “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.”\u00a0BMC genomics<\/i>\u00a021.1 (2020): 1-13.<\/li>\n
- Kendall Rank Correlation<\/strong>:<\/li>\n
- Pearson Correlation<\/strong>:<\/li>\n
- -1 (Negative)<\/strong>: When variable X moves, variable Y moves in the opposite direction<\/em><\/li>\n
- 1 (Positive)<\/strong>: When variable X moves, variable Y moves in the same direction<\/em>.<\/li>\n