Brotli is a modern, open-source compression algorithm developed by Google, designed to enable faster and more efficient data transmission over the web. It offers superior compression ratios and speed compared to previous methods like gzip. While useful, certain HTTP-request libraries aren’t able to handle Brotli without additional consideration. In this article, we’ll discuss how to handle Brotli compression with Python and the requests library.<\/p>\n
Brotli<\/a> is supported by most modern web browsers<\/a> and certainly by Chrome, Firefox, and Safari. It was proposed officially in 2016 via RFC 7932 and since has seen creeping adoption, mostly by large websites due to network compatibility issues<\/a>. It offers a higher compression ratio and — the real kicker — a\u00a0much<\/em> faster decompression speed<\/a>.<\/p>\n Chances are you are seeing Brotli out in the wild while browsing and simply haven’t noticed. Only through inspection of the response headers of an HTTP request is one likely to discern servers using Brotli. The For those who might be web scraping, you’re likely to first notice Brotli via visual inspection of the This is the still-decoded version of the text and, in most cases, may or may not be decoded automatically by many clients like Requests for Python (not) or Axios for JavaScript\/Node (automatic).<\/p>\n Brotli is currently in a grey area of adoption with the majority of friction coming from CDN support being slow. It’s out there however and likely to only become more prevalent given the significant improvements in speed offered. In all likelihood, any widely used HTTP library for any modern language will support Brotli encoding eventually. Python’s most popular third-party HTTP library — requests — however, does not support it out-of-the-box as of 11\/2023. Let’s take a look at a typical requests-based HTTP request via Python:<\/p>\n Chances are the Content-Encoding: br<\/code> header is what tells a client (browser) that the server (website) is serving content (loading pages) using the Brotli compression standard.<\/p>\ncontent<\/code> component of an HTTP response, finding something that resembles the following:<\/p>\n \u0002\ufffdN\u001fVP\ufffd^\ufffd_\ufffd\u000bVV\ufffd`\ufffd\ufffd\t\f\ufffd\ufffd\ufffd\ufffd\u03beb\ufffd\ufffd\ufffd\ufffd\"]\ufffd\u0001\u0002\r\n\u0010\ufffdE@\u00022pZ\ufffdFy\u88ec\ufffd\ufffdV\ufffd\ufffd\u000bV\ufffd\ufffdV\ufffd\u0146\u0007\ufffd\u0001\ufffdY\ufffdbu\ufffdox\\u\u001d\ufffdm\ufffdN\u01d6\ufffd,\u0608\ufffd\u0202\ufffd\u0007\u0249\ufffd\ufffdS\ufffd(\ufffd\ufffd\u001f}\r\n^\ufffd%\u0012S\ufffd\ufffd\ufffd4\ufffd'\ufffd\ufffdR4\ufffd9\ufffd\u019bY\r\n\ufffd\u0012\ufffd)\ufffd\ufffdMyL\ufffdN\ufffd;\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd2r\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdF\ufffd\u0019\u001b\u0017 G\/\ufffddy1\ufffd\u0018\t\ufffd\ufffd\ufffd\ufffd\u0003\ufffdca\u03fd\ufffdC\ufffdr5\u0001h\ufffd\u001a \u0007\ufffdL@\u000f \ufffd#\ufffd\ufffd\ufffdHR\ufffd\ufffd\/\/\ufffdzv\ufffdO\ufffd:\ufffd\u0003\ufffd\u001e\ufffd\ufffd\ufffd\ufffdbR\ufffd \ufffd\ufffd\u0019\ufffd\ufffd\f<%Y\ufffdG\ufffd\r\nb&3\u0012&c,\u00030KW\ufffdf\u0224\ufffd@sv6M\ufffd\u0006\u0018\ufffd\u0003\ufffd\ufffdv\ufffdg\ue438#\u0004\ufffd\u000e\u0001r\ufffd\u0018\ufffd\u04bcR\ufffdpzoR'\ufffd!\ufffdk\ufffd_\u0001\ufffd\ufffdOO\ufffd4-\ufffd\ufffdm\u0010\ufffd\ufffdW\ufffd\ufffd\u01d8)\ufffdM3g\ufffd\ufffd\ufffd\u0013\ufffd\u000e(E\ufffd\u001b\ufffd\ufffd\u001d|\u000b\ufffd\u000f\u0007kO\t\ufffd\r\n\u0442\u0006\ufffd\ufffd\u0012=\ufffd\u0d7fD\u05c7\ufffduM\u0003\ufffd=\ufffd\ufffd\ufffd\u01cd\u0451@\ufffdKf\ufffd\ufffd\ufffd\r\n'\ufffd\ufffd\ufffd,\ufffd;yB\ufffd\\\ufffd\u0019<\u0004\ufffd\ufffd\ufffd\u00f5i\ufffd\ufffd9\u0001\u07fe\ufffd\ufffd\u0005\ufffdX\ufffdy\ufffd\ufffd\ufffdt\ufffd\ufffd^\ufffd\ufffd*\ufffdy\ufffd\ufffd\ufffd-@\ufffd\ufffd\ufffd\ufffd\ufffd}\ufffdi\ufffd9\ufffd\ufffd:\ufffd?B\ufffd{\u0004Wd\u067e\ufffd\ufffdIb\ufffd2\ufffd\ufffdV}\ufffd\ufffd\u001b\ufffd\ufffd\ufffd\ufffd\ufffd\/\ufffd\ufffd\ufffde\ufffd\u000f\ufffdy\ufffd\/{\ufffd\ufffd\ufffd\ufffd[u\ufffdK\ufffd\u07d0\ufffd\ufffdk\u07f7$\u000eSI\ufffd]\ufffd\ufffd}\ufffd'\u001ff\u008c&\ufffd\ufffd\ufffdr\ufffd0\ufffd\u0614\ufffd\ufffd\ufffd\ufffd\ufffd24\ufffd\ufffd24\u0005\ufffd\u0198KA-\ufffd\u0001J\tF\u0001\ufffd\u001by\u0167\ufffd*.\ufffd\ufffd\ufffd\u001e\u000e|q`\ufffdc\u077e\r\np)K\u0001yu\ufffd\ufffdD\ufffd\ufffd\ufffd<\ufffd\ufffd\ufffd}Zl\ufffd\ufffd\ufffd~\ufffd\u0014d3\ufffdO\ufffd\u0142\ufffd\ufffd\ufffd\ufffd\u0012\ufffd\ufffd>\u001a\ufffd\ufffd\u0010q\ufffd\uffd1!\u001a\u001d\ufffdw@\ufffd\u0003\u001a\u001c\ufffd\ufffd\ufffdF\u00074>\ufffd\u0241\ufffd\u001c\ufffd\ufffd\ufffdd@^\ufffdh\ufffdD +\ufffd\ufffd%x\ufffdd\u0011(\u0005|\ufffd\ufffdI\u0010\ufffd\ufffdQ\ufffdg\u0013\ufffd\ufffd\ufffd>4\ufffd)\ufffd\ufffds2\ufffda\ufffd\ufffdw\ufffd4\u00188\ufffdU\ufffd\u0003\u000f\ufffd\u001a\ufffd\u0014\ufffd\ufffd6\ufffd_\ufffda@<\u0004r@ Vw\ufffd\ufffd\ufffd\ufffdu\ufffd\u0007\/}\r\n@$\ufffd\f\u0086\fW\ufffd\ufffdC\ufffd\u000f4.\ufffd\u0012]th\ufffd\ufffd{\ufffd\ufffd\u000e\u0010\ufffd\ufffd\u0653\u031f\u001b\ufffd\ufffd\u00ab\ufffd\u0016\ufffd]\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0003\u0010\ufffd]\ufffd\ufffd \u000f\ufffd\f\u000f\u000ftd 34\ufffd\u041d\ufffd\ufffd\ufffd\ufffd\u0005\u0007\ufffd\ufffdG>p\ufffd\ufffd\ufffd\u0019\ub8a0\u0018\ufffd\ufffdx\ufffd?6\ufffdF\ufffd-\u06c7\ufffdQ\u000f\ufffd\ufffd\ufffdr\ufffd\ufffd7\ufffd8\u000e\ufffd[n\u01337\ufffdq{\ufffd\u0007\ufffd\ufffd\ufffd\ufffd\u0013o\ufffd\ufffdW\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\/ \u0001\ufffdL\u001c\ufffd\ufffd\ufffdMC\ufffd\ufffd\ufffdx\ufffd\u0004\ufffdP\ufffd?VLX\ufffd\ufffd|e\u0016\ufffd\ufffd\ufffd\ufffd\u001bV\\\u04b3\ufffd\ubee7\u01c5\ufffd\ufffdl\ufffd7\ufffd_g\ufffd\ufffd\ufffd,\ufffdWt\ufffd.i\u001fYK\/\ufffd77\u0017\/\ufffd\ufffd\ufffd\u000es3\ufffd\ufffd\ufffd\u04a4e\ufffd\u0462\ufffdu\ufffd\u0007\r\n~\ufffd\u0011\u0015\u0003\ufffd\ufffd`T\fE\u0011H\ufffd'\ufffdVA\ufffd\ufffd\ufffd)\u00071@-v\u0010\t\ufffd?c\ufffd\u0016\ufffd\u02b3A1f\ufffdE%\ufffd\u001b\ufffd\ufffdPT\ufffd\ufffdB\u0628\ufffd\ufffd(\ufffd\ufffd\uabc1cZ\ufffd Rm\u001b\\l\\c\ufffd\ufffdTJ\ufffd\ufffdLMw*\ufffd\r\n\u000b\ufffdJ<\u02b5>\ufffdL\ufffd\ufffdVF)=RC\ufffd\u0005\u0006\ufffd&~u\ufffd\u0007ca\u0003\u000fE\ufffd\u000e@\ufffdL-0\ufffd\u0013\ufffd\ufffd\u0002\ufffd?\ufffdE\u001aqF5\ufffd?<\/pre>\nHandling Brotli in Python with Requests<\/h2>\n
import requests\r\n\r\n# make the HTTP request\r\nresp = requests.get(\"https:\/\/www.google.com\")\r\n\r\n# output the decoded response content\r\nprint(resp.text)<\/pre>\n
resp.text<\/code> is going to come out garbled in stdout. This is because A.)<\/strong> Google is likely using Brotli as the Content-Encoding<\/code> (depending on your geographic location, maybe) and B.)<\/strong> the lack of native support for Brotli via Python’s request library.<\/p>\n