{"id":941,"date":"2021-07-02T16:35:12","date_gmt":"2021-07-02T20:35:12","guid":{"rendered":"https:\/\/www.alpharithms.com\/?p=941"},"modified":"2022-06-16T15:23:53","modified_gmt":"2022-06-16T19:23:53","slug":"python-bytearray-built-in-function","status":"publish","type":"post","link":"https:\/\/www.alpharithms.com\/python-bytearray-built-in-function-123516\/","title":{"rendered":"Python bytearray(): Manipulating Low-Level Data Efficiently"},"content":{"rendered":"

The Python bytearray() function converts strings or collections of integers into a mutable sequence of bytes. It provides developers the usual methods Python affords to both mutable and byte data types. Python’s bytearray() built-in allows for high-efficiency manipulation of data in several common situations.<\/p>\n

The Python bytearray()<\/code> function’s powerful features do come with some responsibility. Developers must be mindful of encodings, be aware of source data format, and have a basic working knowledge of common character sets like ASCII<\/a>.<\/p>\n

In this article, you’ll learn the rationale, common use cases, advanced use cases, and potential pitfalls of Python’s bytearray()<\/code> built-in function. There’s even a section with some notable quirks at the bottom.<\/p>\n

TL;DR<\/strong> \u2014 Python’s bytearray()<\/code> function converts strings and sequences of integers into bytes to provide developers the ability to efficiently update (mutate) data without additional memory allocation.<\/p>\n

# Define a string\r\nx = \"Stay gold Ponyboy.\"\r\n\r\n# Try to mutate it\r\nx[5:9] = 'silver'\r\n\r\n# Throws the following error\r\n>>> TypeError: 'str' object does not support item assignment\r\n\r\n# Define a byte array\r\ny = bytearray(x, encoding='ascii')\r\n\r\n# Try to mutate it\r\ny[5:] = bytearray('silver Ponyboy.', encoding='ascii')\r\n\r\n# Print the resulting type, decoded\r\n# to an ascii-string\r\nprint(y.decode(encoding='ascii'))\r\n\r\n# Results in mutated string\r\n>>> Stay silver Ponyboy.<\/pre>\n
Rationale<\/h2>\n
The bytearray<\/code> object was introduced in Python 2 and has served as a ‘mutable counterpart<\/a>‘ to the bytes object since. Unlike the bytes<\/code> object, the bytearray<\/code> object doesn’t come with a literal notational representation and must be instantiated directly.<\/p>\n
For example, b'string'<\/code> provides literal representation for a bytes object whereas developers must use the syntax bytearray('string', encoding='ascii')<\/code> to create a new bytearray<\/code> object. I regard this as only a minor inconvenience given the benefits Python’s bytearray<\/code> data type provides developers.<\/p>\n
Data like strings are regarded as immutable (more on that below) meaning they can’t be changed directly. To change (mutate) a string, developers must copy the string and assigned the new version to an object located in memory. Consider the following example:<\/p>\n
# Create a string\r\na = \"string one\"\r\nprint(a, \"-\", id(a))\r\n\r\n>>> string one - 3236722728688\r\n\r\n# 'mutate' string\r\na = \"string two\"\r\nprint(a, \"-\", id(a))\r\n\r\n>>> string two - 3236720675504<\/pre>\n
Note that our string has been changed but now has a different id (memory location.) That means that a is now a different object in memory and our original object is garbage. That’s all well-and-good when one is changing around a few strings.<\/p>\n
Such easy string manipulation is one of the language features that make Python such a popular programming language<\/a>. Unfortunately, this approach isn’t efficient for changing large numbers of such data. That’s where the power of the bytearry()<\/code> function shines.<\/p>\n
The official Python documentation<\/a> describes the intended usage of bytearry() as follows:<\/p>\n
The bytearray class is a mutable sequence of integers in the range 0 <= x < 256. It has most of the usual methods of mutable sequences, described in Mutable Sequence Types, as well as most methods that the bytes type has, see Bytes and Bytearray Operations.<\/em><\/p>\n
The reference to the Bytes and Bytearray Operations<\/a> documentation should not be passed over either. That reference provides the following elaboration:<\/p>\n
Since 2 hexadecimal digits correspond precisely to a single byte, hexadecimal numbers are a commonly used format for describing binary data.\u00a0<\/em><\/p>\n
This will make more sense in just a bit when we take a look at cases where the bytearray()<\/code> function displays escaped character sequences. For now, let’s consider some common cases and basic syntax!<\/p>\n
Basic Use & Syntax<\/h2>\n
The bytearray()<\/code> function returns a new array of bytes. This can result from an input string or a collection of integers in the range 0-255 (inclusive). If you are familiar with binary notation you may have already guessed that makes perfect accommodation for eight-bit representations<\/a>.<\/p>\n
Python’s built-in bytearray()<\/code> takes three possible arguments:<\/p>\n
    \n
  1. source<\/strong> – the data to be converted to an array of bytes, either a string or a collection of integers.<\/li>\n
  2. encoding<\/strong> – required for strings, optional for collections of integers<\/li>\n
  3. errors<\/strong> – an optional parameter to specify error handling. See here for options<\/a>.<\/li>\n<\/ol>\n
    Let’s take the bytearray()<\/code> function for a quick spin to demonstrate the basic use of these arguments:<\/p>\n
    # create a bytearray from a string\r\n>>> bytearray(\"byte-sized\")\r\n\r\nTypeError: string argument without an encoding<\/pre>\n
    Well, that didn’t go well. Bytes are essential just numerical representations of information. Without knowing how to interpret that information, Python will make a fuss. Arguably, a default encoding\u00a0could<\/em> be used but that would cause another set of problems. Let’s see what happens when we let Python know how to interpret the data:<\/p>\n
    # create a bytearray from a string, specify encoding\r\n>>> bytearray(\"Byte-sized\", encoding='ascii')\r\n\r\nbytearray(b'Byte-sized')<\/pre>\n
    Now we’ve got a bytearray<\/code> object in memory signifying as much by showing a bytes literal within parenthesis (more on that below.) Let’s take a look at how the bytearray()<\/code> function handles a series of integer values. Remember, only values in the range of 0 < val < 255<\/code> are considered valid here.<\/p>\n
    # creat a bytearry of integers from ordinal values of a string\r\n>>> bytearray([ord(x) for x in \"Byte-sized\"], encoding='ascii')\r\n\r\nTypeError: encoding without a string argument<\/pre>\n
    Well, that didn’t go well either. When given a series of numbers, Python doesn’t need to know the encoding because it limits the values to 0-255. The character representation of any value in this range is assumed to be of the ISO8859-1 standard, more commonly known as the Latin-1 character set (a superset of the ASCII encoding.) Let’s try this again:<\/p>\n
    # creat an array of ints from a string\r\nnums = [ord(x) for x in \"Byte-sized\"]\r\n\r\n[66, 121, 116, 101, 45, 115, 105, 122, 101, 100]\r\n\r\n# Instantiate a byte array\r\n>>> bytearray(nums)\r\n\r\nbytearray(b'Byte-sized')\r\n<\/pre>\n
    Now we’ve got a bytearray<\/code> object from a list of integer values! But what can we do with this? This is where the bytearray()<\/code> function starts demonstrating its utility as a mutable object. We’ll cover what mutable vs. immutable means in just a second. Before we get there, however, let’s stop for a second and consider what the difference between a byte string<\/em> and a byte<\/em> array might be.<\/p>\n
    Byte Strings vs. Byte Arrays<\/h2>\n
    At this point, you may be wondering what the difference between a sequence<\/em> of bytes and a collection<\/em> of bytes might be. In a word\u2014mutability<\/em>. Both represent data in binary format, both can easily handle two-digit hexadecimal inputs, and both rely on developers to be mindful of encodings to ensure proper representation of numerical values and the resulting translations. So what gives?<\/p>\n
    Bytes are treated with more finality than bytearray<\/code> objects. That is, they are suited for applications where one doesn’t need to change the data often<\/strong>. For example, a bytes object would be better suited for storing data received from a network transmission. For large transmissions, maybe even as an array of bytes (but not bytearray<\/code> objects.)<\/p>\n
    The bytearray<\/code> object is better suited for situations where frequent manipulation of data is expected<\/strong>. For example, the updating of data received from a network transmission before retransmission. bytearray<\/code> objects also provide methods afforded to mutable objects such as sizing, slicing, member assignment, insertion, reversal, and many more.<\/p>\n
    To fully appreciate the difference between Python’s bytes and bytearay<\/code> objects we’ll need to consider the difference between mutable vs. immutable data types. Don’t worry, it’s a fairly simple concept and with a basic example one that’s easy to grasp!<\/p>\n
    Mutable vs. Immutable<\/h2>\n
    In Python, strings, numbers, and tuples are immutable\u2014meaning their values can’t be changed. They can be copied or reassigned but not manipulated directly. Data structures like lists, dictionaries<\/a>, and sets<\/a>\u00a0are\u00a0<\/em>mutable and allow much more efficient manipulation of their constituent parts.<\/p>\n
    Specifically, additional memory allocation<\/a> is minimized when updating part of data contained in the middle of a list whereas updating the middle of a string would require a near-doubling. This describes how byte arrays can be applied in representing immutable data types to improve efficiency in certain contexts.<\/p>\n
    Let’s review our original TL;DR<\/strong> example from above, this time with a little more explanation:<\/p>\n
    # Define a string\r\nx = \"stay gold Ponyboy.\"\r\n\r\n# Try to mutate it\r\nx[5:9] = 'silver'\r\n\r\n>>> TypeError: 'str' object does not support item assignment\r\n\r\n# Define a byte array\r\ny = bytearray(x, encoding='ascii')\r\n\r\n# Try to mutate it\r\ny[5:] = bytearray('silver Ponyboy.', encoding='ascii')\r\n\r\n>>> bytearray(b'stay silver Ponyboy.')\r\n\r\n# assign new value to z, check type\r\nz = y.decode(encoding='ascii')\r\n\r\nz, type(z)\r\n\r\n>>> (stay silver Ponyboy., <class 'str'><\/pre>\n
    Here we’ve shown that trying to directly change the text of our string throws a TypeError<\/code>. Storing that string as a byte array allows direct manipulation of the individual character values. After manipulation, we can then convert the resulting values to a string via the decode()<\/code> method\u2014ensuring to specify the proper encoding.<\/p>\n
    Is this an example of efficient<\/em> use of the bytearray()<\/code> function? Probably not. Imagine having to update large sequences of textual data, such as in many database applications or large network data consumptions. The ability to manipulate data without allocating additional memory would be essential. That’s where byte arrays really shine.<\/p>\n
    TL;DR<\/strong> – representing a string as an array of bytes allows one to manipulate it more efficiently. There is an initial overhead cost in memory but a decreasing cost as manipulative operations amass. Always mind your encoding.<\/p>\n
    Notes<\/h2>\n
    The bytearray()<\/code> function deals with some pretty nuanced utility. This makes no surprise of the sometimes spooky behavior exhibited by this built-in. Below are a few cases where the bytearray()<\/code> function might behave in ways that might need some explanation. These certainly aren’t\u00a0bugs<\/em> but should be kept in mind nonetheless.<\/p>\n
    Escaped Character Sequences<\/h3>\n
    Given the bytearray()<\/code> restriction of valid integer arguments to a range of 0-255<\/code> one might expect some quicky behavior to arise. A ValueError<\/code> gets thrown anytime a non-compliant value is attempted\u2014but that’s expected. What’s quirky is how Python will represent ordinal values for non-printable characters. Consider the following:<\/p>\n
    # create an array of nums\r\n>>> nums = [1, 2, 3, 4, 5]\r\n\r\n# Create bytearry by reference\r\n>>> bytearray(nums)\r\n\r\nbytearray(b'\\x01\\x02\\x03\\x04\\x05')\r\n\r\n# Declare a bytearray of list literal\r\n>>> bytearray([1, 2, 3, 4, 5])\r\n\r\nbytearray(b'\\x01\\x02\\x03\\x04\\x05')\r\n\r\n# Declare a bytearray with some known printable ASCII values\r\nprint(bytearray([65, 1, 97, 2, 66, 3, 98]))\r\n\r\nbytearray(b'A\\x01a\\x02B\\x03b')<\/pre>\n
    Note that bytearry()<\/code> will display escaped characters regardless of how an array of integers is passed in (\\x<\/code> prefix.) However, note that the characters with integer values matching ordinal values of\u00a0printable\u00a0<\/em>characters are represented as that character. In other words: 1 is displayed as \\x01<\/code> but 65<\/code> is displayed as A<\/code> since it’s the ordinal value of a printable<\/em> character. Check out the article on Python’s ord() function<\/a> to get a feel for that interoperability.<\/p>\n
    Default Values<\/h3>\n
    The bytearray()<\/code> can be initialized as an empty array or one with default values by passing a non-iterable integer argument. Consider the following:<\/p>\n
    # Create an empty bytearray\r\n>>> bytearray()\r\n\r\nbytearray(b'')\r\n\r\n# Create a byte array of size 5\r\n>>> bytearray(5)\r\n\r\nbytearray(b'\\x00\\x00\\x00\\x00\\x00')\r\n\r\n# Create a byte array with single\r\n# element 5\r\n>>> bytearray([5])\r\n\r\nbytearray(b'\\x05')<\/pre>\n
    Note here that passing the integer value of 5<\/code> as an argument initializes the bytearray()<\/code> object with five values of null<\/code>\u2014represented by the escaped character sequence \\x00<\/code>. The documentation covering valid bytearray()<\/code> arguments isn’t entirely clear on this point in my opinion and can lead to some confusion. Note the last example places the 5 in Python’s bracketed list notation. This results in a bytearray()<\/code> object of length one with the single escaped character sequence of \\x05<\/code>. Again, not a bug but certainly quirky. Note<\/strong>: null<\/code> is not the same as 0<\/code>.<\/p>\n
    Final Thoughts<\/h2>\n
    Python’s bytearray()<\/code> function provides an easy-to-use utility for memory-efficient manipulation of data. Developers of network applications, large string-processing use cases, and programs that encode and decode large swaths of data can all stand in admiration of its high-level accessibility of low-lever data.<\/p>\n
    Python certainly isn’t the canonical language for developing memory-efficient applications\u2014at least not directly. The bytearray()<\/code> function offers the utility that makes a case that, while not the canonical choice, Python is certainly up to the task if needed. For more insight into the handling of binary data check out the article What’s a Byte Stream Anyway<\/a>?<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
    The Python bytearray() function converts strings or collections of integers into a mutable sequence of bytes. It provides developers the usual methods Python affords to both mutable and byte data types. Python’s bytearray() built-in allows for high-efficiency manipulation of data in several common situations. The Python bytearray() function’s powerful features do come with some responsibility. […]<\/p>\n","protected":false},"author":2,"featured_media":947,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,10],"tags":[31,163,165,143,386,363,126],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/941"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=941"}],"version-history":[{"count":3,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/941\/revisions"}],"predecessor-version":[{"id":949,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/941\/revisions\/949"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/947"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}