Python comes with a lot of built-in data types to manage sequences. This includes lists, dictionaries, sets, Queues, Deques, and more. The standard comparator operators can be applied to such sequences but trouble arises when comparing sequences of different lengths or dimensions.<\/p>\n
In this article, we will take a look at how one can apply a custom function to two sequences of different lengths. This can be used to compare all items in one list to those in another, find the smallest value among multiple sets, or anything else you can imagine.<\/p>\n
To get started we are going to define two lists as our sequences to demonstrate things. Then we will compare each item in the first list to determine if it is less than every item in the second list (spoiler: each item is less). Consider the following code:<\/p>\n
# Define two equally sized lists\r\nl1 = [1, 2, 3, 4, 5]\r\nl2 = [6, 7, 8, 9, 10]\r\n\r\n# Determine if each item in L1 is less than all items in L2\r\nresults = [l1[i] < l2[i] for i in range(len(l2))]\r\n\r\n# Print results\r\nprint(results)\r\n[True, True, True, True, True, True]\r\n\r\n<\/pre>\nThis is expected behavior and produces the desired results. Now, let us consider how this might have played out with two lists of different sizes.<\/p>\n
Throwing Errors<\/h2>\n
To change things up, we are going to add an additional item to our second list, the integer value 11, to ensure our lists are of different lengths. We will then attempt the same pairwise comparator evaluation as before. Consider the following example code:<\/p>\n
# Define two differently sized lists\r\nl1 = [1, 2, 3, 4, 5]\r\nl2 = [6, 7, 8, 9, 10] \r\n\r\n# Determine if each item in L1 is less than all items in L2 \r\nresults = [l1[i] < l2[i] for i in range(len(l2))]\r\n\r\n>>> IndexError: list index out of range<\/pre>\nHere we see an index error being thrown that results from our use of the
len(l2)<\/code> argument in our comparison logic. This attempts to iterate over our first list for every item in our second. There is one extra element in our second, which means we will be trying to access an extra element in the first list that simply isn’t there.<\/p>\nBroadcasting<\/h2>\n
The term\u00a0broadcasting<\/em> is used by the NumPy library<\/a> deals with arrays of different sizes. Keep in mind; lists are just 1-dimensional arrays. The concept of broadcasting among arrays has been leveraged by many of the most powerful machine learning libraries such as Tensorflow<\/a>, Theano<\/a>, Octave<\/a>, and others. The term is defined by Scipy as such<\/a>:<\/p>\n
The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is \u201cbroadcast\u201d across the larger array so that they have compatible shapes.<\/em><\/p>\n
Skipping over the finer details of broadcasting\u2014it can be used here to resize the smaller of our lists to ensure the IndexError Exception is no longer an issue. We will use the
NumPy.take()<\/code> method to co-opt this feature from NumPy in the following code:<\/p>\n# Define lists of different size\r\nl1 = [1, 2, 3, 4, 5]\r\nl2 = [6, 7, 8, 9, 10, 11]\r\n\r\n# Apply NumPy's take() method to each\r\n# Use the max length of each for sizing\r\nl1 = np.take(l1, list(range(max([len(l1), len(l2)]))))\r\nl2 = np.take(l2, list(range(max([len(l1), len(l2)]))))\r\n\r\n>>> IndexError: index 5 is out of bounds for axis 0 with size 5\r\n<\/pre>\nWhat gives\u2014after all this hullabaloo about broadcasting\u2014<\/em>we are still getting an
IndexError<\/code>. We need to realize that broadcasting does not perform any magic on our arrays. Some action must be taken to\u00a0resize<\/em> one of our arrays to avoid such an error.<\/p>\nThis is handled by the
take()<\/code> method by providing an additionalmode<\/code> keyword argument to determine what happens with arrays of different sizes. By default (what was used above) this argument is'raise'<\/code> as in raise an Exception<\/em>.<\/p>\nThe documentation for the NumPy
take()<\/code> function details the possibility of using'clip'<\/code> or'wrap'<\/code> as alternative values for mode. Let’s see how each will affect our output:<\/p>\n# Specify 'clip' as broadcast mode\r\nl1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='clip')\r\nl2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='clip')\r\n\r\n# View results\r\nprint(l1.tolist())\r\nprint(l2.tolist())\r\n\r\n[1, 2, 3, 4, 5, 5]\r\n[6, 7, 8, 9, 10, 11]\r\n\r\n<\/pre>\nHere we see our first list having had an additional copy of the last element added. This action is taken as many times as necessary such that our l1 list is resized to match the length of
l2<\/code>. For example, if we had added 5 additional elements in thel2<\/code> list\u2014the'clip'<\/code> mode of broadcasting would ad 5 duplicates of the last element in thel1<\/code> list. Let’s see that in action:<\/p>\n# Define new lists\r\nl1 = [1, 2, 3, 4, 5]\r\nl2 = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\r\n\r\n# Broadcast again\r\nl1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='clip')\r\nl2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='clip')\r\n\r\n# Print results\r\nprint(l1.tolist())\r\nprint(l2.tolist())\r\n\r\n[1, 2, 3, 4, 5, 5, 5, 5, 5, 5]\r\n[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\r\n<\/pre>\nHere we find 5 additional 5’s added to list one\u2014just as expected. The NumPy
take()<\/code> method offers'wrap'<\/code> as an additional mode of broadcasting. Let’s consider how this handles our data, again using the newly-defined lists:<\/p>\n# Define lists again\r\nl1 = [1, 2, 3, 4, 5]\r\nl2 = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\r\n\r\n# Broadcast with NumPy take, using clip as mode\r\nl1 = np.take(l1, list(range(max([len(l1), len(l2)]))), mode='wrap')\r\nl2 = np.take(l2, list(range(max([len(l1), len(l2)]))), mode='wrap')\r\n\r\n# View results\r\nprint(l1.tolist())\r\nprint(l2.tolist())\r\n\r\n[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]\r\n[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]<\/pre>\nHere we see our first list having been apparently duplicated within itself. This isn’t exactly the case\u2014though the result would appear as such. What happens is the
'wrap'<\/code> mode begins\u00a0wrapping<\/em> back to the beginning of thel1<\/code> list when it runs our index values to match withl2<\/code>. The first index value ofl2<\/code> outside the range ofl1<\/code> (index value of 5) is then wrapped<\/em> around to the first value ofl1<\/code>. Logically, this is equivalent of6 % 5 -1<\/code>.<\/p>\nEmulating with Vanilla Python<\/h2>\n
There’s not much practical cause for using a vanilla Python implementation of broadcasting. Nonetheless, let us consider how one might approach such as task for the purposes of our collective enlightenment. To implement this in code, we would need to handle the following cases:<\/p>\n
\n
- Determine the length of the iterable such that we could know if an index value were going to cause an Exception to be thrown;<\/li>\n
- Handle out-of-bounds index values by delegating to logic for either ‘wrap’ or ‘clip’ modes.<\/li>\n
- Maintain a container for returning output values taken from the above logic<\/li>\n<\/ol>\n
Consider the following code demonstrating this basic functionality of broadcasting suited for the selection of indices from a 1-dimensional array:<\/p>\n
# Define some data\r\niterable = [1, 2, 3, 4, 5]\r\nindices = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\r\nmode = 'wrap'\r\n\r\n# Determine the length of our iterable\r\n# to avoid re-checking during each iteration\r\niter_length = len(iterable)\r\n\r\n# Create an output container\r\noutput = []\r\n\r\n# Consider each index value from indices\r\nfor i in indices:\r\n\r\n # Get element for any idex value within range\r\n if i < iter_length:\r\n output.append(iterable[i])\r\n\r\n # If index out-of-range, handle with mode\r\n elif mode == 'wrap':\r\n output.append(iterable[i % iter_length - 1])\r\n elif mode == 'clip':\r\n output.append(iterable[-1])\r\n\r\n# View result\r\nprint(output)\r\n\r\n[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]\r\n \r\n<\/pre>\nHere we see output values identical to those returned via the NumPy.take(mode=’wrap’) approach. The only bit of fancy in this code is the
i % iter_length - 1<\/code> line. That implements a modulus operation to divide the index by the number of available iterable items, and then subtracts the one to accommodate zero-based indexing. Read this article by Real Python<\/a> for a in-depth explanation of the modulo operator in Python.<\/p>\nNote<\/strong>: The above code is available as a function via GitHub<\/a><\/em><\/p>\n
Final Thoughts<\/h2>\n
Dealing with lists of differing lengths can be tricky. The first step in any approach is to carefully consider one’s goals and determine the best way to handle out-of-bounds cases. For example, if comparing all values of a shorter list to those of a larger list duplication of elements may be acceptable. In other cases where an even distribution is required (such as selecting random colors) one might need to make use of an approach similar to the mode=’wrap’ option in the take() method.<\/p>\n
For information on basic
list<\/code> comparisons check out the end of the official Python documentation on sequence types<\/a>. For more on complex use-cases of the take() method check out this article on broadcasting<\/a> that is an excerpt from the book Python Data Science Handbook<\/em><\/a>. Broadcasting is an incredibly useful concept for managing arrays of differing lengths. Its role in modern machine learning algorithms is no joke and knowing the conceptual basics can pay off in a big way. Check out this article on the best machine learning books<\/a> for some ideas on how it is implemented!<\/p>\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Python comes with a lot of built-in data types to manage sequences. This includes lists, dictionaries, sets, Queues, Deques, and more. The standard comparator operators can be applied to such sequences but trouble arises when comparing sequences of different lengths or dimensions. In this article, we will take a look at how one can apply […]<\/p>\n","protected":false},"author":2,"featured_media":1955,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[655,10],"tags":[68,510,456,238,511,198,143],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1953"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=1953"}],"version-history":[{"count":2,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1953\/revisions"}],"predecessor-version":[{"id":1956,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/1953\/revisions\/1956"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/1955"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=1953"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=1953"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=1953"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}