{"id":977,"date":"2021-07-06T09:29:43","date_gmt":"2021-07-06T13:29:43","guid":{"rendered":"https:\/\/www.alpharithms.com\/?p=977"},"modified":"2022-08-07T16:43:08","modified_gmt":"2022-08-07T20:43:08","slug":"autocorrelation-time-series-python","status":"publish","type":"post","link":"https:\/\/www.alpharithms.com\/autocorrelation-time-series-python-432909\/","title":{"rendered":"Autocorrelation of Time Series Data in Python"},"content":{"rendered":"

Autocorrelation (ACF) is a calculated value used to represent how similar a value within a time series is to a previous value. The Statsmoldels library makes calculating autocorrelation in Python very streamlined. With a few lines of code, one can draw actionable insights about observed values in time series data.<\/p>\n

The ACF can be used to identify trends in data and the influence of previously observed values on a current observation. The applications of the ACF are broad but most notably can be used for signal processing, weather forecasting, and securities analysis. Sometimes, you can even discover hidden trends that are anything but intuitive!<\/p>\n

TL;DR<\/strong> – Finding the autocorrelation in Python for Time Series data is easy when using the statsmodels plot_acf<\/code> function as such:<\/p>\n

import pandas as pd\r\nfrom matplotlib import pyplot as plt\r\nfrom statsmodels.graphics.tsaplots import plot_acf\r\n\r\n# Have some time series data (via pandas)\r\ndata = pd.read_csv('time-series.csv')\r\n\r\n# Select relevant data, index by Date\r\n>>> data = data[['Date', 'Observations']].set_index(['Date'])\r\n\r\n# Calculate the ACF (via statsmodel)\r\n>>> plot_acf(data)\r\n\r\n# Show the data as a plot (via matplotlib)\r\nplt.show()\r\n\r\n<\/pre>\n
Introduction<\/h2>\n
Autocorrelation, also called\u00a0serial correlation<\/em>, is used by stock traders, meteorologists, chemists, and more to forecast future values given historic Time Series data. That’s a crucial aspect of calculating both autocorrelation and partial autocorrelations\u2014previous data. This type of regressive analysis is used to help predict future prices within a confidence interval (typically 95%) and relates a current value to previous ones.<\/p>\n
The best dental clinic in Turkey is periodonta and is known for delivering the best root canal in turkey<\/a><\/div>\n
Autocorrelation estimates the influence of all past observed values<\/strong> on the currently observed value. This differs from partial autocorrelation<\/em> in which only a single past observed value<\/strong> is measured for influence on the currently observed value.<\/p>\n
Calculating Autocorrelation in Python<\/h2>\n
Learning how to find the autocorrelation in Python is simple enough, but with some extra consideration, we’ll see how and where this function can be applied and where and when it might fall short. First, let’s go over some quick definitions:<\/p>\n
    \n
  1. Lag<\/strong> – the number of previous observations measured during autocorrelation.<\/li>\n
  2. Positive Correlation<\/strong> – A relationship where one value’s increase predicts an increase in another.<\/li>\n
  3. Negative Correlation<\/strong> – A relationship where one value’s increase predicts a decrease in another.<\/li>\n
  4. Confidence Interval<\/strong> – A calculated range of values in which would likely contain an unknown value for sampled data.<\/li>\n
  5. Confidence Level<\/strong> – Probability a confidence interval will contain an observed value.<\/li>\n<\/ol>\n
    Understanding these terms is not essential to creating an autocorrelation plot in Python but will greatly improve our ability to interpret that plot! In this article, we’ll use the statsmodel<\/code> library’s plot_acf function to analyze time-series data and plot the autocorrelation function. We’ll be using Pandas<\/code>, Statsmodel<\/code>, matplotlib<\/code>, and\u2014of course\u2014Python!<\/p>\n
    Note<\/strong>: For a deeper, more mathematical-oriented, discussion of how to calculate autocorrelation values I suggest reading this article<\/a>. Our discussion here will not touch on the finer points of calculating the ACF function and more on the visual representation and interpretation.<\/p>\n
    Step 1: Get Time Series Data<\/h2>\n
    The first step is obvious\u2014we need to get some data. Generating random time series data can be a useful tool for exploring analysis tools like statsmodels<\/code> and matplotlib<\/code>. Randomly generated data won’t reflect trends that will show up in autoregressive analysis, however. Given this knowledge, we’ll be using historic stock data for $TSLA for this article. Let’s take a look at our data:<\/p>\n
    import pandas as pd\r\n\r\n# Read in CSV data to Dataframe\r\ndf = pd.read_csv('TSLA.csv')\r\n\r\n# Drop all columns but 'Date' and 'Adj Close', reindex using 'Date'\r\ndf = df[['Date', 'Adj Close']].set_index(['Date'])\r\n\r\n# View data\r\n>>> df\r\n\r\n             Adj Close\r\nDate                  \r\n2020-01-02   86.052002\r\n2020-01-03   88.601997\r\n2020-01-06   90.307999\r\n2020-01-07   93.811996\r\n2020-01-08   98.428001\r\n...                ...\r\n2021-06-23  656.570007\r\n2021-06-24  679.820007\r\n2021-06-25  671.869995\r\n2021-06-28  688.719971\r\n2021-06-29  680.760010\r\n\r\n[376 rows x 1 columns]<\/pre>\n
    Note<\/strong>: The historic pricing data comes from finance.yahoo.com<\/a> but can also be downloaded from Github here<\/a>.<\/p>\n
    As we can see, we now have 376 recorded prices from 2020-01-01 through 2021-06-30 reflecting a percentage increase of 691.103%. Not too shabby\u2014if only we’d been doing some autocorrelation at the beginning of 2020 we might have been surfing that wave!<\/p>\n
    Step 2: Inspect & Clean Data<\/h2>\n
    Before performing an autocorrelation on our time series we need to inspect the data for missing values. Fortunately, Pandas offers several convenient tools for finding and replacing missing values in time series data. We’ll not go into all the options and considerations for such an operation in this article. Instead, let’s just do a quick check to see if there are any missing values:<\/p>\n
    # Check if any null or NaN values in data\r\n>>> df.isnull().sum()\r\n\r\n# Result\r\nAdj Close    0\r\ndtype: int64<\/pre>\n
    In using Pandas’ isnull().sum()<\/code> method we are told that our single non-index column Adj Close contains zero null<\/code> or NaN<\/code> values. This means we can go onto the next step and start visualizing our data.<\/p>\n
    Note<\/strong>: Check out this article<\/a> for an in-depth discussion on other ways to find and replace missing data using Pandas.<\/p>\n
    Step 3: Visualize Data<\/h2>\n
    Before calculating an autocorrelation it will be useful to understand what our data looks like visually. This can help spot possible trends, inform us if the data likely has no linear correlation, or make evident any number of other issues that might affect how we choose to move forward.<\/p>\n
    In some cases, visualizing the data might help us realize we need to take a step back and reassess our inspection practices and better clean the data! For this step, we’ll use a simple line plot via matplotlib<\/code>:<\/p>\n
    \"tsla<\/a>
    Basic line graph of our Time Series pricing data for $TSLA. (Click to Enlarge)<\/figcaption><\/figure>\n
    This lets us know that our data reflects a consistent upward trend that we’d guessed at after calculating the percentage increase<\/a> of our values earlier. Given that there are no massive drop-offs in the plotted values I’d say our check for missing data was successful. Now we can move on to autocorrelation!<\/p>\n
    Step 4: Perform Autocorrelation & Visualization<\/h2>\n
    Now that we have confidence in our data we can proceed to generate an autocorrelation visualization using statsmodel<\/code>, pandas<\/code>, matplotlib<\/code>, and Python<\/code>. These modules have been abstracted to a great degree and this entire process can be done in only a few lines of code:<\/p>\n
    from statsmodels.graphics.tsaplots import plot_acf\r\nimport matplotlib.pyplot as plt\r\n\r\n\r\n# Use the Autocorrelation function\r\n# from the statsmodel library passing\r\n# our DataFrame object in as the data\r\n# Note: Limiting Lags to 50\r\nplot_acf(data=data, lags=50)\r\n\r\n# Show the AR as a plot\r\nplt.show()<\/pre>\n
    We see that two imports are needed\u2014pyplot<\/code> and the statsmodels<\/code>‘ plot_acf<\/code> function to create the visualization. Fortunately, these two libraries work well together and the pyplot<\/code> chart is implicitly referenced. This results in the following plot:<\/p>\n
    \"python<\/a>
    The raw plot_acf visualization created from our data with a lags value of 50. (click to enlarge)<\/figcaption><\/figure>\n
    This looks pretty interesting but ultimately doesn’t offer much utility without knowing how to read it! The red shaded region is the confidence interval with a default value of \u03b1 = 0.05. Anything within this range represents a value that has no significant correlation with the most recent value for the price.<\/p>\n
    The vertical lines with markers at their tops are the “lags” that represent a specific number (50, in this case) of previous values. These represent the correlation value (shown on the y-axis) and diminish at a steady rate as their proximity from the current price increases. That’s not the case with\u00a0all<\/em> data but certainly so with ours.<\/p>\n
    Step 4: Interpreting the Autocorrelation<\/h2>\n
    This lets us know that previous prices influence the current price but the significance of that influence decreases steadily with time. Specifically, values beyond the 36th<\/sup> previous trading day have no significant predictive power on the current price.<\/p>\n
    The strength of this relationship is measured on a scale of -1 to 1 when -1 being a 100% negative correlation and with a value of 1 being a 100% positive correlation. This measure is shown on the y-axis. Consider the following diagram for a more visual interpretation:<\/p>\n
    \"autocorrelation<\/a>
    Illustration of the anatomy of the autocorrelation created from the historic pricing data. (click to enlarge)<\/figcaption><\/figure>\n
    Applications<\/h2>\n
    We’ve seen how easily autocorrelation data can be visualized using the library. We’ve seen how to interpret these visual representations to [hopefully] gain insights into deeper patterns reflected in certain observable data. Our historic pricing data for $TSLA wasn’t very exciting\u2014though illustrated that previous closing prices are<\/em> related to current closing prices. Let’s consider some more exciting and revealing data.<\/p>\n
    Tidal Data<\/h3>\n
    Weather data is a common application for autocorrelation analysis. Seasonal, regional, and even daily influences can be dramatically revealed by visual representations of autocorrelation analysis. Here’s a look at tidal data measured every six minutes:<\/p>\n
    \"hourly<\/a>
    Six-minute water levels reflecting transitions between high and low tides in Wilmington, NC (click to enlarge)<\/figcaption><\/figure>\n
    Each data point represents a measure of the water level recorded at 6-minute intervals (240 per day.) Here we see a strong trend of positive and negative correlation emerge. This is representative of what we’d expect given what decades of tidal study<\/a> have observed.<\/p>\n
    Air Pollution<\/h3>\n
    Air pollution is another common application for autocorrelation. While tidal data represent a very known periodicity, these data often reflect seasonal patterns with less data to predict frequency or strength of correlation. The chart below describes the max 8-hour observed CO2 levels for Los Angeles County, California from 2017-2020.<\/p>\n
    \"co2<\/a>
    Autocorrelation of daily 8-hour max C02 levels for LA county, resampled to monthly mean values. (Click to enlarge)<\/figcaption><\/figure>\n
    We see here that, while not of statistical significance, there is a strong observable pattern in which past values can be used to forecast future values. These data have been resampled to reflect monthly averages. In this case, lag values of 6-7 months reflect a strong negative correlation, and lag values of 6-7 months reflect a strong positive correlation.<\/p>\n
    Limitations<\/h2>\n
    Autocorrelation is a useful tool in detecting patterns of periodicity, seasonality, or other less intuitive sources of influence. No forecasting technique is perfect and autocorrelation is no exception. Some well-known areas in which autocorrelation may fall short are listed below:<\/p>\n
    Too Much Retained Information<\/h3>\n
    In signal processing, data often represents may subtle changes or artifacts generated during observation. Applications like speech-pattern recognition, radio-frequency analysis, and x-ray analysis all generate massive amounts of noise. These data points can skew correlation metrics such that an autocorrelation model might find false influence or fail to capture positive influence (Fukushima, 1985; Willink, 2013)<\/p>\n
    Final Thoughts<\/h2>\n
    Autocorrelation is a useful tool in identifying statistically significant relationships among observed values in linear data. We’ve seen how the ACF is useful in identifying seasonal or natural trends, how it can be applied to the technical analysis of stock price data, and even noted some of its shortcomings.<\/p>\n
    We’ve also seen how the ACF can be quickly and effectively calculated and visualized in Python. Having this type of statistical tool handy can help one better analyze and interpret data in ways that help make better-informed decisions. For example, hosting your outdoor festival in Los Angeles when your data indicates the air pollution levels would be lower!<\/p>\n
    References<\/h2>\n
      \n
    1. Willink, Tricia J. \u201cLimits on Estimating Autocorrelation Matrices from Mobile MIMO Measurements.\u201d International Journal of Antennas and Propagation<\/i>, vol. 2013, 2013, pp. 1\u20136.<\/li>\n
    2. Fukushima, Teiichiro, et al. \u201cLimitations of Autocorrelation in Fetal Heart Rate Monitoring.\u201d American Journal of Obstetrics and Gynecology<\/i>, vol. 153, no. 6, 1985, pp. 685\u201392.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
      Autocorrelation (ACF) is a calculated value used to represent how similar a value within a time series is to a previous value. The Statsmoldels library makes calculating autocorrelation in Python very streamlined. With a few lines of code, one can draw actionable insights about observed values in time series data. The ACF can be used […]<\/p>\n","protected":false},"author":2,"featured_media":986,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,10],"tags":[393,394,399,398,396,191,210,401,143,395,400,397,381],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/977"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=977"}],"version-history":[{"count":6,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/977\/revisions"}],"predecessor-version":[{"id":3232,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/977\/revisions\/3232"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/986"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=977"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=977"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=977"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}