{"id":1984,"date":"2021-08-27T16:57:19","date_gmt":"2021-08-27T20:57:19","guid":{"rendered":"https:\/\/www.alpharithms.com\/?p=1984"},"modified":"2022-06-16T15:23:49","modified_gmt":"2022-06-16T19:23:49","slug":"parse-csv-data-directly-from-download-using-python","status":"publish","type":"post","link":"https:\/\/www.alpharithms.com\/parse-csv-data-directly-from-download-using-python-195716\/","title":{"rendered":"Parse CSV Data Directly from Download Using Python"},"content":{"rendered":"

Python makes downloading files super easy. Parsing data directly from memory is sometimes more convenient than fussing around with temporary files, directories, and the ensuing housekeeping. Fortunately, Python makes parsing data in memory just as easy (if not easier) than actually saving the file to local storage.<\/p>\n

Parsing a CSV File from Memory<\/h2>\n

In this article, we’ll cover how to use the requests library to download a remote CSV file directly into memory and parse it using the csv<\/code> module. This doesn’t involve much code but it will be helpful to outline the steps first. Here is what will happen:<\/p>\n

    \n
  1. Define a URL<\/li>\n
  2. Send HTTP GET request via the requests library<\/li>\n
  3. Convert response data into iterator of lines<\/li>\n
  4. Parse response lines as CSV reader object<\/li>\n
  5. Iterator, save or manipulate as desired<\/li>\n<\/ol>\n

    Listing these steps takes almost as many characters as the actual code. Below is the implementation resulting in iterating over a csv.reader<\/code> object to print each line to the standard output:<\/p>\n

    import request\r\nimport csv\r\n\r\n# Define the remote URL\r\nurl = \"https:\/\/query1.finance.yahoo.com\/v7\/finance\/download\/SPY\"\r\n\r\n# Send HTTP GET request via requests\r\ndata = requests.get(url)\r\n\r\n# Convert to iterator by splitting on \\n chars\r\nlines = data.text.splitlines()\r\n\r\n# Parse as CSV object\r\nreader = csv.reader(lines)\r\n\r\n# View Result\r\nfor row in reader:\r\n    print(row)\r\n\r\n['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume']\r\n['2021-08-27', '447.119995', '450.649994', '447.070007', '450.100006', '450.100006', '58642636']\r\n<\/pre>\n
    Here we see two ‘rows’ of data printing out\u2014the first row being the header and the second being the most recent OHLC quote for $SPY. Converting a response<\/code> object into an iterator is as simple as using the splitlines()<\/code> method. This method is the equivalent of using split('\\n')<\/code> and in either case, will return an iterable (the required argument for the csv.reader<\/code> method.<\/p>\n
    Final Thoughts<\/h2>\n
    Python offers several ways to download a file<\/a>. In any case, the resulting textual data can be parsed\u2014directly from memory\u2014to be used rather than first saving to local storage. This can often be a more efficient approach in cases where temporary file use is preferred and remote files are not excessive in size. Readers interested in retrieving financial data using Python should check out the article 3 Easy Ways to Get Financial Data Using Python<\/a><\/em>.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Python makes downloading files super easy. Parsing data directly from memory is sometimes more convenient than fussing around with temporary files, directories, and the ensuing housekeeping. Fortunately, Python makes parsing data in memory just as easy (if not easier) than actually saving the file to local storage. Parsing a CSV File from Memory In this […]<\/p>\n","protected":false},"author":2,"featured_media":1985,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,10],"tags":[522,523,521,78,459,143,217],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1984"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=1984"}],"version-history":[{"count":1,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1984\/revisions"}],"predecessor-version":[{"id":1986,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1984\/revisions\/1986"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/1985"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=1984"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=1984"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=1984"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}