pandas_ta<\/code> package, how to troubleshoot some common errors, and finally let our trained model loose with a basic trading strategy to assess its predictive power. This article focuses primarily on the implementation of the scikit-learn LinearRegression<\/code>\u00a0model and assumes the reader has a basic working knowledge of the Python language.<\/p>\nHighlights<\/h2>\n\n- We’ll get load historic pricing data into a Pandas’
DataFrame<\/code> and add technical indicators to use as features in our Linear Regression model.<\/li>\n- We’ll extract only the data we intend to use from the DataFrame<\/li>\n
- We’ll cover some common mistakes in how data is handled prior to training our model and show how some simple “reshaping” can solve a nagging error message.<\/li>\n
- We’ll train a simple linear regression model using a 10-day exponential moving average as a predictor for the closing price.<\/li>\n
- We’ll analyze the accuracy of our model, plot the results, and consider the magnitude of our errors<\/li>\n
- Finally, we’ll run a simulated trading strategy to see what kind of returns we could make by leveraging the predictive power of our model. Spoiler alert: it turned out pretty decent!<\/li>\n<\/ul>\n
Introduction<\/h2>\n
Linear regression is utilized in business, science, and just about any other field where predictions and forecasting are relevant. It helps identify the relationships between a dependent variable<\/a> and one or more independent variables<\/a>. Simple linear regression is defined by using a feature to predict an outcome. That’s what we’ll be doing here.<\/p>\nStock market forecasting is an attractive application of linear regression. Modern machine learning packages like scikit-learn<\/code> make implementing these analyses possible in a few lines of code. Sounds like an easy way to make money, right? Well, don’t cash in your 401k\u00a0just<\/em> yet<\/strong>.<\/p>\nAs easy as these analyses are to implement, selecting features with ample enough predictive power to turn a profit is more of an art than science. In training our model, we’ll take a look at how to easily add common technical indicators to our data to use as features in training our model. Let’s take this in a step-by-step approach starting with getting our historic pricing data.<\/p>\n
Note<\/strong>: The information in this article is for informational purposes only and does not constitute financial advice. See our financial disclosure<\/a> for more information.<\/p>\nStep 1: Get Historic Pricing Data<\/h2>\n
To get started we need data. This will come in the form of historic pricing data for Tesla Motor’s (TSLA<\/code>). I’m getting this as a direct .csv<\/code> download from the finance.yahoo.com<\/code> website and loading it into memory as a pandas data frame. See this post on getting stock prices with Python<\/a> for a more detailed walkthrough.<\/p>\nimport pandas as pd\r\n\r\n# Load local .csv file as DataFrame\r\ndf = pd.read_csv('TSLA.csv')\r\n\r\n# Inspect the data\r\nprint(df)\r\n\r\n# List of entries\r\n Date Open High ... Close Adj Close Volume\r\n0 2020-01-02 84.900002 86.139999 ... 86.052002 86.052002 47660500\r\n1 2020-01-03 88.099998 90.800003 ... 88.601997 88.601997 88892500\r\n2 2020-01-06 88.094002 90.311996 ... 90.307999 90.307999 50665000\r\n3 2020-01-07 92.279999 94.325996 ... 93.811996 93.811996 89410500\r\n4 2020-01-08 94.739998 99.697998 ... 98.428001 98.428001 155721500\r\n.. ... ... ... ... ... ... ...\r\n248 2020-12-24 642.989990 666.090027 ... 661.770020 661.770020 22865600\r\n249 2020-12-28 674.510010 681.400024 ... 663.690002 663.690002 32278600\r\n250 2020-12-29 661.000000 669.900024 ... 665.989990 665.989990 22910800\r\n251 2020-12-30 672.000000 696.599976 ... 694.780029 694.780029 42846000\r\n252 2020-12-31 699.989990 718.719971 ... 705.669983 705.669983 49649900\r\n\r\n[253 rows x 7 columns]\r\n\r\n# Show some summary statistics\r\nprint(df.describe())\r\n\r\n Open High Low Close Adj Close Volume\r\ncount 253.000000 253.000000 253.000000 253.000000 253.000000 2.530000e+02\r\nmean 289.108428 297.288412 280.697937 289.997067 289.997067 7.530795e+07\r\nstd 167.665389 171.702889 163.350196 168.995613 168.995613 4.013706e+07\r\nmin 74.940002 80.972000 70.101997 72.244003 72.244003 1.735770e+07\r\n25% 148.367996 154.990005 143.222000 149.792007 149.792007 4.713450e+07\r\n50% 244.296005 245.600006 237.119995 241.731995 241.731995 7.025550e+07\r\n75% 421.390015 430.500000 410.579987 421.200012 421.200012 9.454550e+07\r\nmax 699.989990 718.719971 691.119995 705.669983 705.669983 3.046940e+08\r\n<\/pre>\nNote<\/strong>: This data is available for download via Github<\/a>.<\/em><\/p>\nStep 2: Prepare the data<\/h2>\n
Before we start developing our regression model we are going to trim our data some. The ‘Date’ column will be converted to a DatetimeIndex and the ‘Adj Close’ will be the only numerical values we keep. Everything else is getting dropped.<\/p>\n
# Reindex data using a DatetimeIndex\r\ndf.set_index(pd.DatetimeIndex(df['Date']), inplace=True)\r\n\r\n# Keep only the 'Adj Close' Value\r\ndf = df[['Adj Close']]\r\n\r\n# Re-inspect data\r\nprint(df)\r\n\r\n Adj Close\r\nDate \r\n2020-01-02 86.052002\r\n2020-01-03 88.601997\r\n2020-01-06 90.307999\r\n2020-01-07 93.811996\r\n2020-01-08 98.428001\r\n... ...\r\n2020-12-24 661.770020\r\n2020-12-28 663.690002\r\n2020-12-29 665.989990\r\n2020-12-30 694.780029\r\n2020-12-31 705.669983\r\n\r\n[253 rows x 1 columns]\r\n\r\n# Print Info\r\nprint(df.info())\r\n\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 253 entries, 2020-01-02 to 2020-12-31\r\nData columns (total 1 columns):\r\n # Column Non-Null Count Dtype \r\n--- ------ -------------- ----- \r\n 0 Adj Close 253 non-null float64\r\ndtypes: float64(1)\r\nmemory usage: 4.0 KB\r\n\r\n<\/pre>\nWhat we see here is our ‘Date’ column having been converted to a DatetimeIndex with 253 entries and the ‘Adj Close’ column being the only retained value of type float64 (np.float64.) Let’s plot our data to get a visual picture of what we’ll be working with from here on out.<\/p>\n![\"tsla](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1178\/tsla-2020-2021-historical-prices-chart-pandas-plot-1024x576.jpg\" "\\"tsla-2020-2021-historical-prices-chart-pandas-plot\\"")
<\/a>The plot of $TSLA historic pricing from 2020-2021. (Click to enlarge)<\/figcaption><\/figure>\nWe can see a significant upward trend here reflecting a 12-month price increase from 86.052002 to 705.669983. That’s a relative increase of ~720%<\/a>. Let’s see if we can’t develop a linear regression model that might help predict upward trends like this!<\/p>\nAside: Linear Regression Assumptions & Autocorrelation<\/h3>\n
Before we proceed we need to discuss a technical limitation of linear regression. Linear regression requires a series of assumptions to be made to be effective. One can certainly apply a linear model without validating these assumptions but useful insights are not likely to be had.<\/p>\n
One of these assumptions is that variables in the data are independent.\u00a0<\/em>Namely, this dictates that the residuals (difference between the predicted value and observed value) for any single variable aren’t related.<\/p>\nFor Time Series data this is often a problem since our observed values are longitudinal<\/em> in nature\u2014meaning they are observed values<\/em> for the same thing, recorded in sequence. This produces a characteristic called autocorrelation which describes how a variable is somehow related to itself (self-related.) (Chatterjee, 2012)<\/p>\nAutocorrelation analysis<\/a> is useful in identifying trends like seasonality or weather patterns. When it comes to extrapolating values for price prediction, however, it is problematic. The takeaway here is that our date values aren’t suitable as our independent variable<\/a> and we need to come up with something else and use the adjusted close value as the independent variable. Fortunately, there are some great<\/em> options here.<\/p>\nStep 3: Adding Technical Indicators<\/h2>\n
Technical indicators are calculated values describing movements in historic pricing data for securities like stocks, bonds, and ETFs. Investors use these metrics to predict the movements of stocks to best determine when to buy, sell, or hold.<\/p>\n
Commonly used technical indicators include moving averages<\/strong> (SMA, EMA, MACD), the Relative Strength Index<\/strong> (RSI), Bollinger Bands<\/strong> (BBANDS), and several others. There is certainly no shortage of popular technical indicators out there to choose from. To add our technical indicators we’ll be using the pandas_ta library<\/a>. To get started, let’s add an exponential moving average<\/strong> (EMA) to our data:<\/p>\nimport pandas_ta\r\n\r\n# Add EMA to dataframe by appending\r\n# Note: pandas_ta integrates seamlessly into\r\n# our existing dataframe\r\ndf.ta.ema(close='adj_close', length=10, append=True)\r\n\r\n# Inspect Data once again\r\n adj_close EMA_10\r\ndate \r\n2020-01-02 86.052002 NaN\r\n2020-01-03 88.601997 NaN\r\n2020-01-06 90.307999 NaN\r\n2020-01-07 93.811996 NaN\r\n2020-01-08 98.428001 NaN\r\n... ... ...\r\n2020-12-24 661.770020 643.572394\r\n2020-12-28 663.690002 647.230141\r\n2020-12-29 665.989990 650.641022\r\n2020-12-30 694.780029 658.666296\r\n2020-12-31 705.669983 667.212421\r\n\r\n[253 rows x 2 columns]\r\n\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 253 entries, 2020-01-02 to 2020-12-31\r\nData columns (total 2 columns):\r\n # Column Non-Null Count Dtype \r\n--- ------ -------------- ----- \r\n 0 adj_close 253 non-null float64\r\n 1 EMA_10 244 non-null float64\r\ndtypes: float64(2)\r\n<\/pre>\nAs evident from the printouts above, we now have a new column in our data titled “EMA_10.” This is our newly-calculated value representing the exponential moving average calculated over a 10-day period.<\/p>\n
Note<\/strong>: The pandas_ta library will alter the column names. Here we see the “Adj Close” column renamed to “adj_close.” This is expected behavior but can cause issues if one isn’t aware of this functionality.<\/p>\nThis is great news but also comes with a caveat: the first 9 entries in our data will have a NaN<\/code> value since there weren’t proceeding values from which the EMA could be calculated. Let’s take a closer look at that:<\/p>\n# Print the first 10 entries of our data\r\nprint(df.head(10))\r\n\r\n adj_close EMA_10\r\ndate \r\n2020-01-02 86.052002 NaN\r\n2020-01-03 88.601997 NaN\r\n2020-01-06 90.307999 NaN\r\n2020-01-07 93.811996 NaN\r\n2020-01-08 98.428001 NaN\r\n2020-01-09 96.267998 NaN\r\n2020-01-10 95.629997 NaN\r\n2020-01-13 104.972000 NaN\r\n2020-01-14 107.584000 NaN\r\n2020-01-15 103.699997 96.535599<\/pre>\nWe need to deal with this issue before moving on. There are several approaches we could take to replace the NaN values<\/a> in our data. These include replacing with zeros, the mean for the series, backfilling from the next available, etc. All these approaches seek to replace NaN values with some pseudo values.<\/p>\nGiven our goal of predicting real-world pricing that’s not an attractive option. Instead, we’re going to just drop all the rows where we have NaN values and use a slightly smaller dataset by taking the following approach:<\/p>\n
# Drop the first n-rows\r\ndf = df.iloc[10:]\r\n\r\n# View our newly-formed dataset\r\nprint(df.head(10))\r\n\r\n adj_close EMA_10\r\ndate \r\n2020-01-16 102.697998 97.656035\r\n2020-01-17 102.099998 98.464028\r\n2020-01-21 109.440002 100.459660\r\n2020-01-22 113.912003 102.905540\r\n2020-01-23 114.440002 105.002715\r\n2020-01-24 112.963997 106.450221\r\n2020-01-27 111.603996 107.387271\r\n2020-01-28 113.379997 108.476858\r\n2020-01-29 116.197998 109.880701\r\n2020-01-30 128.162003 113.204574<\/pre>\nNow we’re ready to start developing our regression model to see how effective the EMA is at predicting the price of the stock. First, let’s take a quick look at a plot of our data now to get an idea of how the EMA value tracks with the adjusted closing price.<\/p>\n![\"tsla](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1178\/tsla-adjusted-close-vs-ema-2020-16-2020-1-1024x576.jpg\" "\\"$tsla-adjusted-close-vs-ema-2020-16-2020-1\\"")
<\/a>The plot of $TSLA historic pricing from 2020-2021 with the EMA overlaid. (Click to enlarge)<\/figcaption><\/figure>\nWe can see here the EMA tracks nicely and that we’ve only lost a\u00a0littttttle\u00a0<\/em>bit of our data at the leading edge. Nothing to worry about\u2014our linear model will still have ample data to train on!<\/p>\nStep 4: Test-Train Split<\/h2>\n
Machine learning models require at minimum two sets of data to be effective: the training data<\/strong> and the testing data<\/strong>. Given that new data can be hard to come by, a common approach to generate these subsets of data is to split a single dataset into multiple sets (Xu, 2018).<\/p>\nUsing eighty percent of data for training and the remaining twenty percent for testing is common. This 80\/20 split is the most common approach but more formulaic approaches can be used as well (Guyon, 1997).<\/p>\n
The 80\/20 split is where we’ll be starting out. Rather than mucking about trying to split our DataFrame<\/code> object manually we’ll just the scikit-learn test_train_split<\/code> function to handle the heavy lifting:<\/p>\n# Split data into testing and training sets\r\nX_train, X_test, y_train, y_test = train_test_split(df[['adj_close']], df[['EMA_10']], test_size=.2)\r\n\r\n# Test set\r\nprint(X_test.describe())\r\n\r\n adj_close\r\ncount 49.000000\r\nmean 272.418612\r\nstd 140.741107\r\nmin 86.040001\r\n25% 155.759995\r\n50% 205.009995\r\n75% 408.089996\r\nmax 639.830017\r\n\r\n# Training set\r\nprint(X_train.describe())\r\n\r\n adj_close\r\ncount 194.000000\r\nmean 291.897732\r\nstd 166.033359\r\nmin 72.244003\r\n25% 155.819996\r\n50% 232.828995\r\n75% 421.770004\r\nmax 705.669983\r\n<\/pre>\nWe can see that our data has been split into separate DataFrame objects with the nearest whole-number value of rows reflective of our 80\/20 split (49 test samples, 192 training samples.) Note the test size 0.20<\/code> (20%) was specified as an argument to the train_test_split<\/code> function.<\/p>\nNote<\/strong>: The X_train<\/code>, X_test<\/code>, y_train<\/code>, and y_test<\/code> data are Pandas DataFrame<\/code> objects in memory. This results from the use of double-bracketed access notation df[['adj_close']]<\/code> as opposed to single-bracket notation df['adj_close']<\/code>. The single-bracketed notation would return a Series object and would require reshaping before we could proceed to fit our model. See this post for more details<\/a>.<\/p>\nStep 5: Training the Model<\/h2>\n
We have our data and now we want to see how well it can be fit to a linear model. Scikit-learn’s LinearRegression class makes this simple enough\u2014requiring only 2 lines of code (not including imports):<\/p>\n
from sklearn.linear_model import LinearRegression\r\n\r\n# Create Regression Model\r\nmodel = LinearRegression()\r\n\r\n# Train the model\r\nmodel.fit(X_train, y_train)\r\n\r\n# Use model to make predictions\r\ny_pred = model.predict(X_test)<\/pre>\nThat’s it\u2014our linear model has now been trained on 194 training samples, and we’ve generated predicted values (y_pred<\/code>). Now we can assess how well our model fits our data by examining our model coefficients and some statistics like the mean absolute error (MAE)<\/strong> and coefficient of determination (r2)<\/strong>.<\/p>\nStep 6: Validating the Fit<\/h2>\n
The linear model generates coefficients for each feature during training and returns these values as an array. In our case, we have one feature that will be reflected by a single value. We can access this using the model.regr_<\/code> attribute.<\/p>\nIn addition, we can use the predicted values from our trained model to calculate the mean squared error and the coefficient of determination using other functions from the sklearn.metrics<\/code> module. Let’s see a medley of metrics useful in evaluating our model’s utility.<\/p>\nfrom sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error\r\n\r\n# Printout relevant metrics\r\nprint(\"Model Coefficients:\", model.coef_)\r\nprint(\"Mean Absolute Error:\", mean_absolute_error(y_test, y_pred))\r\nprint(\"Coefficient of Determination:\", r2_score(y_test, y_pred))\r\n\r\n# Results\r\nModel Coefficients: [[0.94540376]]\r\nMean Absolute Error: 12.554147460577513\r\nCoefficient of Determination: 0.9875188616393644<\/pre>\nThe MAE is the arithmetic mean of the absolute errors of our model, calculated by summing the absolute difference between observed values of X and Y and dividing by the total number of observations.<\/p>\n
The MAE can be described as the sum of the absolute error for all observed values divided by the total number of observations. Check out this article by Shravankumar Hiregoudar<\/a> for a deeper look into using the MAE, as well as other metrics, for evaluating regression models.<\/p>\nFor now, let’s just recognize that a lower MAE value is better, and the closer our coefficient of the correlation value is to 1.0 the better. The metrics here suggest that our model fits our data well, though the MAE is slightly high.<\/p>\n
Let’s consider a chart of our observed values compared to the predicted values to see how this is represented visually:<\/p>\n![\"tsla](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1178\/tsla-linear-regression-price-prediciton-python-1024x681.jpg\" "\\"tsla-linear-regression-price-prediciton-python\\"")
<\/a>A chart showing our predicted values overlaying a line of our observed values. (click to enlarge)<\/figcaption><\/figure>\nThis looks like a pretty good fit! Given our relatively high r2 value that’s no surprise. Just for kicks, let’s add some lines to represent the residuals for each predicted value.<\/p>\n![\"linear](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/1178\/linear-regression-price-prediction-plot-with-residual-lines-1024x681.jpg\" "\\"linear-regression-price-prediction-plot-with-residual-lines\\"")
<\/a>The red lines represent the residuals (error) of our predicted values (black line) and our observed values (yellow dots). (Click to enlarge)<\/figcaption><\/figure>\nThis doesn’t tell us anything new but helps to conceptualize what the coefficient of correlation is actually representing\u2014an aggregate statistic for how far off our predicted values are from the actual values. So now we have this linear model\u2014but what is it telling us?<\/p>\n
Step 7: Interpretation<\/h2>\n
At this point, we’ve trained a model on historical pricing data using the Adjusted Closing value and the Exponential Moving Average for a 10-day trading period. Our goal was to develop a model that can use the EMA of any given day (dependent on pricing from the previous 9 days) and accurately predict that day’s closing price. Let’s run a simulation of a very simple trading strategy to assess how well we might have done using this.<\/p>\n
Strategy<\/strong>: If our model predicts a higher closing value than the opening value we make a trade for a single share on that day\u2014buying at market open and selling just before market close.<\/p>\nBelow is a summary of each trading day during our test data period:<\/p>\n