One of Python’s strongest features are the list-based structures it offers developers. These structures can represent multi-dimensional data of varying types with ease. Flattening a list in Python is a process by which developers can simplify structures like binary search trees and matrices.<\/p>\n
Flattening a list in Python requires some careful consideration, however. Nested lists, strings, byte arrays, and non-iterable sub-items can all produce edge cases that can disrupt a program’s logic. In this article, we’ll walk through many such cases while developing an evolving function to handle flattening lists in Python\u2014no matter their contents.<\/p>\n
TL;DR<\/strong> – One can use the following function to flatten any list in Python while retaining strings and bytes objects as atomic elements:<\/p>\n Python lists are incredibly flexible\u2014they are dynamically sized, can reference elements of any type, and are mutable. In the case of flattening lists, this incredible flexibility can become a pain.\u00a0 Specifically, Python lists can contain a mixture of iterable and non-iterable elements. When considering how to flatten a list, we need to consider the following two cases:<\/p>\n To get a better picture of these two cases, let’s create two such lists in Python:<\/p>\n Note that the irregular list contains both iterable elements, non-iterable elements, and even a callable function. This presents several cases we’ll have to find a way to handle. First, let’s start slowly by considering how to flatten a regular list.<\/p>\n Regular lists provide a degree of predictability that allows certain assumptions to be made. Namely, we can assume that all elements are iterable but without varying depths of order.<\/p>\n That is\u2014if one element is a single sublist with non-iterable items all the others will be as well. We can approach flattening a regular list in several ways. First, let’s consider a method using a simple for loop:<\/p>\n In this approach, we will create a new list to hold each element as we discover them. We’ll loop over each element in the list as well as each element in the sublist:<\/p>\n This showcases the base logic we need to flatten a list: extract each element from any iterable element and copy it to a new list.<\/p>\n List comprehension is a powerful tool in Python to avoid the cumbersome syntax of writing lots of loops.\u00a0 We can use list comprehension to flatten a regular list as such:<\/p>\n For cases of 2-dimensional regular lists we can use a very simplified approach that makes use of Python’s sum() function. This one is a little odd, but can be done as such:<\/p>\n In this approach, we are summing all elements found in our list into a new list, passed as a second argument.<\/p>\n Using the sum function seems pretty slick but is only applicable to cases of 2D regular lists. Let’s consider what happens when we start adding extra dimensions to our data. Let’s take a look at the following regular list:<\/p>\n Here we’ve only succeeded in extracting the sublists from our list rather than extracting all single elements from those sublists. To handle this goal, we’ll need to revert to the use of a for loop:<\/p>\n This works well enough, but what happens when we have an unknown number of nested levels? In other words\u2014what happens when we don’t know how many dimensions our data is represented as beforehand?<\/p>\n It’s not very practical to restrict our approach only to cases where we know the structure of our data beforehand. To tackle this, we would need to take a recursive approach by which we define a new function. Consider the following:<\/p>\n Here we see a function that will iterate into sub-lists when the proper argument is given for the This works and will handle lists with as much nesting as we care to throw its way. It’s not the most elegant solution (also a syntactic nightmare here) and only handles lists where our data is regular in nature. Let’s consider now how we might build upon this solution to handle irregular lists as well.<\/p>\n Our function to flatten a regular list ended up pretty unsightly. It will also throw a This approach involves less syntax and seems more elegant. Recursive functions run the risk of obfuscating logic but, in this case, the benefits outweigh the downside.<\/p>\n Note<\/strong>: this would crash our program with lists that had more than 1000 levels of nesting (Python’s default recursion depth limit).<\/p>\n While not likely to be an issue in most cases, we’ll take a look at an approach that will also handle that issue in a moment. First\u2014let’s deal with strings and other iterables that we might want to extract as single elements.<\/p>\n Strings and bytes often represent conceptually<\/em> atomic objects. For example, the string Our approaches thus far treat Here we note that the strings contained in our irregular list are treated as iterable objects, resulting in single characters being added to our final output. To avoid this issue, we’ll add a single conditional statement to check for strings and bytes:<\/p>\n At this point, we’ve achieved our goal of flattening lists. This function can handle regular lists, irregular lists, avoid breaking strings and bytes objects<\/a> into individual characters, and requires no prior knowledge of our lists’ dimensionality. All in all, this gets the job done. Rather than leave things here, let’s consider how we might make this function a bit more performant, avoid recursion depth errors, tidy up our syntax, and keep things flexible.<\/p>\n We can approach optimizing our function by using generators for performance<\/a>, There are a few new things going on here worth noting:<\/p>\n To view each item in our newly-flattened list, can cast the generator object to a list as such:<\/p>\n Note<\/strong>: This function is available from the Github repo here<\/a>.<\/em><\/p>\n The only caveat with this approach is our use of the Iterable class from the Python collections library. A dependency on Python’s standard library is, in my opinion, a fair trade-off. However, for anyone looking to avoid such a dependency, the The concept of flattening lists can be applied when dealing with matrices<\/a>, binary trees, or any number of other data structures<\/a>. Particularly, cases where a hierarchy is present in data often result in irregularly-shaped lists. Flattening isn’t always needed for meaningful data analysis\u2014though when it is, the approaches we’ve discussed here can be useful in considering both the feasibility, practicality, and performance of such an undertaking.<\/p>\n","protected":false},"excerpt":{"rendered":" One of Python’s strongest features are the list-based structures it offers developers. These structures can represent multi-dimensional data of varying types with ease. Flattening a list in Python is a process by which developers can simplify structures like binary search trees and matrices. Flattening a list in Python requires some careful consideration, however. Nested lists, […]<\/p>\n","protected":false},"author":2,"featured_media":2370,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[186,655,10],"tags":[351,368,456,143,53],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/2363"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=2363"}],"version-history":[{"count":5,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/2363\/revisions"}],"predecessor-version":[{"id":2371,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/2363\/revisions\/2371"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/2370"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=2363"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=2363"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=2363"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}from collections.abc import Iterable\r\n\r\ndef flatten(any_list):\r\n for element in any_list:\r\n if hasattr(element, \"__iter__\") and not isinstance(element, (str, bytes)):\r\n yield from flatten(element)\r\n else:\r\n yield element<\/pre>\n
Introduction: Regular vs. Irregular Lists<\/h2>\n
\n
# regular list of sublists\r\nregular = [\r\n [1, 2, 3],\r\n [4, 5, 6],\r\n [7, 8, 9]\r\n]\r\n\r\n# irregular list containing sublists,\r\n# elements, and sublists of sublists\r\nirregular = [\r\n [1, 2, 3],\r\n [[4, 5, 6], 7],\r\n [8, [9, 10, [11, 12, 13]], 14],\r\n 15,\r\n max\r\n]\r\n\r\n<\/pre>\n
Flattening Regular Lists<\/h2>\n
Method 1: Looping Over Elements<\/h3>\n
# Iterate over list\r\nflat_list = []\r\nfor item in regular:\r\n for subitem in item:\r\n flat_list.append(subitem)\r\n\r\n# View result\r\n>>> flat_list\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9]\r\n<\/pre>\n
Method 2: List Comprehension<\/h3>\n
# Create a flattened list using list comprehension\r\nflat_list = [subitem for item in regular for subitem in item]\r\n\r\n# View result\r\n>>> flat_list\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9]\r\n<\/pre>\n
Method 3: The sum Function<\/h3>\n
sum(regular, [])\r\n\r\n>>> [1, 2, 3, 4, 5, 6, 7, 8, 9]<\/pre>\n
Edge Case 1: 2-Dimensional Data<\/h3>\n
# Create a regular list with iterable sub-items\r\nnested_regular = [\r\n [[1, 2, 3], [4, 5, 6]],\r\n [[7, 8, 9], [10, 11, 12]]\r\n]\r\n\r\n# Attempting to flatten with list comprehension\r\nflat = [subitem for item in nested_regular for subitem in item]\r\n\r\n>>> flat\r\n[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]\r\n\r\n# Attempting to flatten with sum function\r\nsummed = sum(nested_regular, [])\r\n\r\n>>> summed\r\n[[1,\u00a02,\u00a03],\u00a0[4,\u00a05,\u00a06],\u00a0[7,\u00a08,\u00a09],\u00a0[10,\u00a011,\u00a012]]<\/pre>\n
flat_list = []\r\nfor item in regular:\r\n for subitem in item:\r\n for single_item in subitem:\r\n flat_list.append(single_item)\r\n\r\n>>> flat_list\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]\r\n<\/pre>\n
Edge Case 2: Multi-Dimensional Data<\/h3>\n
def flatten_levels(items, levels):\r\n\r\n output = items\r\n for i in range(levels):\r\n tmp = []\r\n for item in output:\r\n for subitem in item:\r\n tmp.append(subitem)\r\n output = tmp\r\n return output\r\n\r\n# Call function with req'd levels argument\r\nflat = flatten_levels(regular, 2)\r\n\r\n>>> flat\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]<\/pre>\n
levels<\/code> parameter. While this solves the dimensionality issue for regular lists, it requires that we know how many dimensions our data has beforehand\u2014not a flexible solution. We can re-work this approach by adding a check for whether a subitem contains an iterable object or not.<\/p>\ndef flatten_with_check(items):\r\n\r\n # Assume items are iterable\r\n is_iterable = True\r\n output = items\r\n\r\n # Get each item in items\r\n while is_iterable is True:\r\n tmp = []\r\n for item in output:\r\n tmp.extend(item) # <--- Will break on irregularity\r\n output = tmp\r\n\r\n # Check if another level of nesting\r\n try:\r\n iter(output[0])\r\n is_iterable = True\r\n except:\r\n is_iterable = False\r\n\r\n return output\r\n\r\nflat = flatten_with_check(regular)\r\n>>> flat\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]\r\n<\/pre>\n
Flattening Irregular Lists<\/h2>\n
TypeError<\/code> if we pass an irregular list in as an argument. To address this, we can use a recursive function to both simplify our syntax and handle irregular lists. Let’s start off with a function that uses a try-catch<\/code> block for determining whether an object is iterable or not.<\/p>\ndef flatten(iterable, flattened):\r\n try:\r\n iter(iterable)\r\n for item in iterable:\r\n flatten(item, flattened)\r\n except:\r\n flattened.append(iterable)\r\n return flattened\r\n\r\n# Flatten our regular list\r\nflat = flatten(regular, [])\r\n>>> flat\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]\r\n\r\n\r\n# Flatten our Irregular list\r\nflat = flatten(irregular, [])\r\n>>> flat\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, <built-in function max>]\r\n\r\n<\/pre>\n
Edge Case 1: Strings & Bytes<\/h3>\n
\"python\"<\/code> isn’t likely to be considered equivalent to the list ['p', 'y', 't', 'h', 'o', 'n']<\/code>. In most cases, this object needs to be retained as a single word rather than a series of characters.<\/p>\nstrings<\/code>, bytes<\/code>, and lists<\/code> all as the iterable objects they are. To ensure our strings<\/code> and bytes<\/code> stay atomic, we can alter our previous recursive function to check for these data types. Also, we’ll create a new irregular list to demonstrate the issue:<\/p>\n# Create a new irregular list\r\nirregular = [\r\n [1, 2, 3],\r\n [4],\r\n [\r\n [4, 5, 6], [5, 6, [11, 12, ['alpha']], max(1, 2)],\r\n [3, 'beta', [['delta', 12, 4 + 5]]]\r\n ],\r\n 'gamma'\r\n]\r\n\r\n# Run our previous function\r\nflat = flatten(irregular, [])\r\n>>> flat\r\n[1, 2, 3, 4, 4, 5, 6, 5, 6, 11, 12, 'a', 'l', 'p', 'h', 'a', 2, 3, 'b', 'e', 't', 'a', 'd', 'e', 'l', 't', 'a', 12, 9, 'g', 'a', 'm', 'm', 'a']\r\n<\/pre>\n
def atomic_flatten(iterable, flattened):\r\n\r\n try:\r\n iter(iterable)\r\n if type(iterable) not in [str, bytes]:\r\n for item in iterable:\r\n atomic_flatten(item, flattened)\r\n else:\r\n flattened.append(iterable)\r\n except:\r\n flattened.append(iterable)\r\n return flattened\r\n\r\nflat = atomic_flatten(irregular, [])\r\n>>> flat\r\n[1, 2, 3, 4, 4, 5, 6, 5, 6, 11, 12, 'alpha', 2, 3, 'beta', 'delta', 12, 9, 'gamma']\r\n<\/pre>\n
Optimized Approach<\/h2>\n
isinstance<\/code> checks rather than try:except<\/code> blocks, and the Python collections<\/code>‘ Iterable<\/code> class to rule out single items. This approach is done as follows:<\/p>\n# import the Iterable class from Python's ABC module\r\nfrom collections.abc import Iterable\r\n\r\n# Define our irregularly-shaped list\r\nirregular = [\r\n [1, 2, 3],\r\n [4],\r\n [\r\n [4, 5, 6], [5, 6, [11, 12, ['alpha']], max(1, 2)],\r\n [3, 'beta', [['delta', 12, 5+4]]]\r\n ],\r\n 'gamma'\r\n]\r\n\r\n# Check every item in our \r\ndef flatten(any_list):\r\n\r\n # Iterate over every possible element\r\n for element in any_list:\r\n\r\n # Check if the object is iterable & not a string or bytes object\r\n if isinstance(element, Iterable) and not isinstance(element, (str, bytes)):\r\n\r\n # recurse on iterable elements\r\n yield from flatten(element)\r\n else:\r\n\r\n # keep single elements\r\n yield element\r\n\r\n# Use function to flatten list\r\nflat = flatten(irregular)\r\n\r\n# View results\r\n>>> flat\r\n<generator object flatten at 0x000001DCA4DDAF20>\r\n<\/pre>\n
\n
# Print each item in our flattened list\r\n>>> list(flat)\r\n[1, 2, 3, 4, 4, 5, 6, 5, 6, 11, 12, 'alpha', 2, 3, 'beta', 'delta', 12, 9, 'gamma']\r\n<\/pre>\n
isinstance(element, Iterable)<\/code> condition can be changed to hasattr(element, \"__iter__\")<\/code>.<\/p>\nFinal Thoughts<\/h2>\n