{"id":517,"date":"2020-12-26T13:35:53","date_gmt":"2020-12-26T18:35:53","guid":{"rendered":"https:\/\/www.alpharithms.com\/?p=517"},"modified":"2022-06-16T15:23:57","modified_gmt":"2022-06-16T19:23:57","slug":"downloading-files-with-python","status":"publish","type":"post","link":"https:\/\/www.alpharithms.com\/downloading-files-with-python-533513\/","title":{"rendered":"Downloading Files with Python"},"content":{"rendered":"

Using Python to download files from the Internet is super easy\u2014a feat possible using only standard library functions if desired. To download a URL with Python one needs little else than a URL\u2014seriously. Of course, this comes as little surprise with Python being one of the most popular programming languages<\/a>.<\/p>\n

Other libraries, most notably the Python requests library, can provide a clearer API for those more concerned with higher-level operations. This article outlines 3 ways to download a file using python with a short discussion of each.<\/p>\n

urllib.request.urlretrieve<\/h2>\n

Pythons’ urllib library<\/a> offers a range of functions designed to handle common URL-related tasks. This includes parsing, requesting, and\u2014you guessed it\u2014downloading files. Let’s consider a basic example of downloading the robots.txt file from Google.com:<\/p>\n

from urllib import request\r\n\r\n# Define the remote file to retrieve\r\nremote_url = 'https:\/\/www.google.com\/robots.txt'\r\n\r\n# Define the local filename to save data\r\nlocal_file = 'local_copy.txt'\r\n\r\n# Download remote and save locally\r\nrequest.urlretrieve(remote_url, local_file)<\/pre>\n
Note<\/strong>: urllib<\/code> is considered “legacy” from Python 2 and, in the words of the Python documentation<\/a>: “might become deprecated at some point in the future.” In my opinion, there’s a big divide between “might” become deprecated and “will” become deprecated. In other words, this is probably a safe approach for the foreseeable future.<\/p>\n
requests.get + manual save<\/h2>\n
The Python requests module<\/a> is a\u00a0super<\/em> friendly library billed as “HTTP for humans.” Offering very<\/em> simplified APIs, requests lives up to its motto for even high-throughput HTTP-related demands. However, it doesn’t feature a one-liner for downloading files. Instead, one must manually save streamed file data as follows:<\/p>\n
import requests\r\n\r\n# Define the remote file to retrieve\r\nremote_url = 'https:\/\/www.google.com\/robots.txt'\r\n\r\n# Define the local filename to save data\r\nlocal_file = 'local_copy.txt'\r\n\r\n# Make http request for remote file data\r\ndata = requests.get(remote_url)\r\n\r\n# Save file data to local copy\r\nwith open(local_file, 'wb')as file:\r\n    file.write(data.content)<\/pre>\n
There are some important aspects of this approach to keep in mind\u2014most notably the binary format<\/strong> of data transfer. When a web browser loads a page (or file) it encodes it using the specified encoding from the host.<\/p>\n
Common encodings include UTF-8 and Latin-1. This is a directive aimed at web browsers that are receiving\u00a0and<\/em> displaying data that isn’t immediately applicable to downloading files.<\/p>\n
Note<\/strong>: downloaded files may\u00a0require<\/em> encoding in order to display properly. That’s beyond the scope of this tutorial.<\/p>\n
wget.download<\/h2>\n
The wget Python library<\/a> offers a method similar to the urllib<\/code> and attracts a lot of attention to its name being identical to the Linux wget<\/code> command. This module was last updated in 2015.<\/p>\n
import wget\r\n\r\n# Define the remote file to retrieve\r\nremote_url = 'https:\/\/www.google.com\/robots.txt'\r\n\r\n# Define the local filename to save data\r\nlocal_file = 'local_copy.txt'\r\n\r\n# Make http request for remote file data\r\nwget.download(remote_url, local_file)<\/pre>\n
Note<\/strong>: The wget.download<\/code> function uses a combination of urllib<\/code>, tempfile<\/code>, and shutil<\/code> to retrieve the downloaded data, save to a temporary file, and then move that file (and rename it) to the specified location.<\/p>\n
Final Thoughts<\/h2>\n
Downloading files with Python is super simple and\u00a0can<\/em> be accomplished using the standard urllib<\/code> functions. I’ve found the requests library to offer the easiest and most versatile APIs for common HTTP-related tasks.\u00a0 One notable exception is the URL parsing features of the urllib<\/a>. Those are\u00a0strictly<\/em> HTTP related though\u2014so I don’t take points away from requests<\/code>!<\/p>\n","protected":false},"excerpt":{"rendered":"
Using Python to download files from the Internet is super easy\u2014a feat possible using only standard library functions if desired. To download a URL with Python one needs little else than a URL\u2014seriously. Of course, this comes as little surprise with Python being one of the most popular programming languages. Other libraries, most notably the […]<\/p>\n","protected":false},"author":2,"featured_media":518,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,10],"tags":[143,220,219,217,218],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/517"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=517"}],"version-history":[{"count":4,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/517\/revisions"}],"predecessor-version":[{"id":522,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/517\/revisions\/522"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/518"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}