Broadcasting With Lists of Different Size in Python

list broadcasting in Python

Python comes with a lot of built-in data types to manage sequences. This includes lists, dictionaries, sets, Queues, Deques, and more. The standard comparator operators can be applied to such sequences but trouble arises when comparing sequences of different lengths or dimensions.

In this article, we will take a look at how one can apply a custom function to two sequences of different lengths. This can be used to compare all items in one list to those in another, find the smallest value among multiple sets, or anything else you can imagine.

Getting Started

To get started we are going to define two lists as our sequences to demonstrate things. Then we will compare each item in the first list to determine if it is less than every item in the second list (spoiler: each item is less). Consider the following code:

# Define two equally sized lists
l1 = [1, 2, 3, 4, 5]
l2 = [6, 7, 8, 9, 10]

# Determine if each item in L1 is less than all items in L2
results = [l1[i] < l2[i] for i in range(len(l2))]

# Print results
print(results)
[True, True, True, True, True, True]

This is expected behavior and produces the desired results. Now, let us consider how this might have played out with two lists of different sizes.

Throwing Errors

To change things up, we are going to add an additional item to our second list, the integer value 11, to ensure our lists are of different lengths. We will then attempt the same pairwise comparator evaluation as before. Consider the following example code:

# Define two differently sized lists
l1 = [1, 2, 3, 4, 5]
l2 = [6, 7, 8, 9, 10] 

# Determine if each item in L1 is less than all items in L2 
results = [l1[i] < l2[i] for i in range(len(l2))]

>>> IndexError: list index out of range

Here we see an index error being thrown that results from our use of the len(l2) argument in our comparison logic. This attempts to iterate over our first list for every item in our second. There is one extra element in our second, which means we will be trying to access an extra element in the first list that simply isn’t there.

Broadcasting

The term broadcasting is used by the NumPy library deals with arrays of different sizes. Keep in mind; lists are just 1-dimensional arrays. The concept of broadcasting among arrays has been leveraged by many of the most powerful machine learning libraries such as Tensorflow, Theano, Octave, and others. The term is defined by Scipy as such:

The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.

Skipping over the finer details of broadcasting—it can be used here to resize the smaller of our lists to ensure the IndexError Exception is no longer an issue. We will use the NumPy.take() method to co-opt this feature from NumPy in the following code:

# Define lists of different size
l1 = [1, 2, 3, 4, 5]
l2 = [6, 7, 8, 9, 10, 11]

# Apply NumPy's take() method to each
# Use the max length of each for sizing
l1 = np.take(l1, list(range(max([len(l1), len(l2)]))))
l2 = np.take(l2, list(range(max([len(l1), len(l2)]))))

>>> IndexError: index 5 is out of bounds for axis 0 with size 5

What gives—after all this hullabaloo about broadcasting—we are still getting an IndexError. We need to realize that broadcasting does not perform any magic on our arrays. Some action must be taken to resize one of our arrays to avoid such an error.

This is handled by the take() method by providing an additional mode keyword argument to determine what happens with arrays of different sizes. By default (what was used above) this argument is 'raise' as in raise an Exception.

The documentation for the NumPy take() function details the possibility of using 'clip' or 'wrap' as alternative values for mode. Let’s see how each will affect our output:

# Specify 'clip' as broadcast mode
l1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='clip')
l2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='clip')

# View results
print(l1.tolist())
print(l2.tolist())

[1, 2, 3, 4, 5, 5]
[6, 7, 8, 9, 10, 11]

Here we see our first list having had an additional copy of the last element added. This action is taken as many times as necessary such that our l1 list is resized to match the length of l2. For example, if we had added 5 additional elements in the l2 list—the 'clip' mode of broadcasting would ad 5 duplicates of the last element in the l1 list. Let’s see that in action:

# Define new lists
l1 = [1, 2, 3, 4, 5]
l2 = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

# Broadcast again
l1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='clip')
l2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='clip')

# Print results
print(l1.tolist())
print(l2.tolist())

[1, 2, 3, 4, 5, 5, 5, 5, 5, 5]
[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

Here we find 5 additional 5’s added to list one—just as expected. The NumPy take() method offers 'wrap' as an additional mode of broadcasting. Let’s consider how this handles our data, again using the newly-defined lists:

# Define lists again
l1 = [1, 2, 3, 4, 5]
l2 = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

# Broadcast with NumPy take, using clip as mode
l1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='wrap')
l2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='wrap')

# View results
print(l1.tolist())
print(l2.tolist())

[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]
[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

Here we see our first list having been apparently duplicated within itself. This isn’t exactly the case—though the result would appear as such. What happens is the 'wrap' mode begins wrapping back to the beginning of the l1 list when it runs our index values to match with l2. The first index value of l2 outside the range of l1 (index value of 5) is then wrapped around to the first value of l1. Logically, this is equivalent of 6 % 5 -1.

Emulating with Vanilla Python

There’s not much practical cause for using a vanilla Python implementation of broadcasting. Nonetheless, let us consider how one might approach such as task for the purposes of our collective enlightenment. To implement this in code, we would need to handle the following cases:

  1. Determine the length of the iterable such that we could know if an index value were going to cause an Exception to be thrown;
  2. Handle out-of-bounds index values by delegating to logic for either ‘wrap’ or ‘clip’ modes.
  3. Maintain a container for returning output values taken from the above logic

Consider the following code demonstrating this basic functionality of broadcasting suited for the selection of indices from a 1-dimensional array:

# Define some data
iterable = [1, 2, 3, 4, 5]
indices = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
mode = 'wrap'

# Determine the length of our iterable
# to avoid re-checking during each iteration
iter_length = len(iterable)

# Create an output container
output = []

# Consider each index value from indices
for i in indices:

    # Get element for any idex value within range
    if i < iter_length:
        output.append(iterable[i])

    # If index out-of-range, handle with mode
    elif mode == 'wrap':
        output.append(iterable[i % iter_length - 1])
    elif mode == 'clip':
        output.append(iterable[-1])

# View result
print(output)

[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]
        

Here we see output values identical to those returned via the NumPy.take(mode=’wrap’) approach. The only bit of fancy in this code is the i % iter_length - 1 line. That implements a modulus operation to divide the index by the number of available iterable items, and then subtracts the one to accommodate zero-based indexing. Read this article by Real Python for a in-depth explanation of the modulo operator in Python.

Note: The above code is available as a function via GitHub

Final Thoughts

Dealing with lists of differing lengths can be tricky. The first step in any approach is to carefully consider one’s goals and determine the best way to handle out-of-bounds cases. For example, if comparing all values of a shorter list to those of a larger list duplication of elements may be acceptable. In other cases where an even distribution is required (such as selecting random colors) one might need to make use of an approach similar to the mode=’wrap’ option in the take() method.

For information on basic list comparisons check out the end of the official Python documentation on sequence types. For more on complex use-cases of the take() method check out this article on broadcasting that is an excerpt from the book Python Data Science Handbook. Broadcasting is an incredibly useful concept for managing arrays of differing lengths. Its role in modern machine learning algorithms is no joke and knowing the conceptual basics can pay off in a big way. Check out this article on the best machine learning books for some ideas on how it is implemented!

 

Zαck West
Full-Stack Software Engineer with 10+ years of experience. Expertise in developing distributed systems, implementing object-oriented models with a focus on semantic clarity, driving development with TDD, enhancing interfaces through thoughtful visual design, and developing deep learning agents.