Python has a novel framework whereby objects can define how to iterate over the data they encapsulate. This framework, known as the Iterator Protocol, empowers developers to abstract the process of how an object returns members individually. Here we take a dive into understanding Python’s Iterator Protocol, the “magic” methods used, and how to tap into the power of this clear, concise, and convenient framework when designing custom classes.<\/p>\n
__iter__<\/code>, __next__<\/code>, iter()<\/code>, and next()<\/code>.<\/li>\n- Basic iteration examples.<\/li>\n
- Custom Iterables – Basic Use-Cases<\/li>\n
- Custom Iterables – Using our Imagination<\/li>\n<\/ul>\n
Introduction: Understanding Iterables<\/h2>\n
The concept of iteration is language-agnostic. It’s found in every major programming language<\/a> but has roots that run even deeper. Iteration is a conceptual underpinning of approaching repetitive actions.<\/p>\nIteration reduces overhead by providing algorithmic definitions of how to handle situations in which there are multiple things for which multiple actions are required — even if that means repeating the same<\/em> action multiple times! In the context of this article the term “Iterable” will refer to something which provides native support for the process of iteration.<\/strong><\/p>\nThe Python language defines<\/a> an\u00a0Iterable<\/em> as “an object capable of returning its members one at a time.” Examples of Python-native iterables include list<\/code>, tuple<\/code>, dict<\/code>, set<\/code> and even str<\/code> type objects. Very quickly, let’s see an example of iteration using the list<\/code> object:<\/p>\n# defines a list with members\r\nmy_list = [1, 2, 3, 4, 5]\r\n\r\n# uses a for loop to \"iterate\" through the list,\r\n# 1 member at a time\r\nfor item in my_list:\r\n print(item)\r\n<\/pre>\nThis code block outputs the following to stdout<\/code>:<\/p>\n1\r\n2\r\n3\r\n4\r\n5<\/pre>\nThis executes the print<\/code> statement “for each” item within the my_list<\/code> object. In plain English: print each item in my_list<\/code><\/em>. Consider the deeper logic of this statement — how does Python know<\/em> when to stop<\/em> printing members? How does it detect<\/em> when there are\u00a0no more\u00a0<\/em>members to print? If we were to implement this manually it might look something like this:<\/p>\n# defines a list with 5 members\r\nmy_list = [1, 2, 3, 4, 5]\r\n\r\n# loop until the number of members\r\n# having been printed matches the\r\n# number of members present.\r\ncount = 0\r\nwhile count < len(my_list):\r\n print(my_list[count])\r\n count += 1<\/pre>\nJust like before, the output is as follows:<\/p>\n
1\r\n2\r\n3\r\n4\r\n5<\/pre>\nAlternatively, we could attempt to access the next item in the my_list<\/code> collection until we encountered an IndexError<\/code> exception as follows:<\/p>\n# defines a list with 5 members\r\nmy_list = [1, 2, 3, 4, 5]\r\n\r\n# access the next member of the\r\n# list until an exception is thrown.\r\ni = 0\r\nwhile True:\r\n try:\r\n print(my_list[i])\r\n i += 1\r\n except:\r\n break<\/pre>\nOnce again, our output is as follows:<\/p>\n
1\r\n2\r\n3\r\n4\r\n5<\/pre>\nClearly, there are several ways to approach the concept of implementing a for<\/code> loop. The last case — the one that uses exception handling for control flow<\/strong> — is actually the most similar to how Python’s Iterator Protocol works under the hood.<\/p>\nUnderstanding Python Iterables<\/h2>\n
Iterators were introduced in Python version 2 via PEP 234<\/a> \u2014 showing iconic<\/em> number selection. It outlines the use of several methods, special methods, and a StopIteration<\/code> exception to help handle the Iterator Protocol. Over the course of Python’s evolution, the current 3.x<\/code> versions use the following major components for implementing an Iterable object (taken directly from the documentation<\/a>):<\/p>\n\n- __next__():<\/strong> Return the next item from the iterator. If there are no further items, raise the StopIteration exception.<\/li>\n
- __iter__():<\/strong> Return an iterator object. The object is required to support the iterator protocol<\/li>\n
- iter():<\/strong> Built-in function that calls an iterable object’s
__iter__<\/code> method.<\/li>\n- next():\u00a0<\/strong>Built-in function that calls an iterable object’s
__next__<\/code> method.<\/li>\n- StopIteration:\u00a0<\/strong>Exception raised by an iterator’s
__next__<\/code> method to signify that there are no more items.<\/li>\n<\/ul>\nNote<\/strong>: next()<\/code> calls the __next__<\/code>method of the object passed in as an argument. Likewise, iter()<\/code> calls the __iter__<\/code> method of the object passed to it as an argument. In either case, an exception is raised if the object passed as an argument doesn’t implement the required method: __next__<\/code> and __iter__<\/code>, respectively.<\/em><\/p>\nBelow is a manual implementation of a custom Iterable object using the Python Iterator Protocol syntax\/framework:<\/p>\n
from collections.abc import Sequence\r\nfrom typing import Any\r\n\r\n\r\nclass MyIterable:\r\n \"\"\"Custom implementation of an Iterable object via Iterable Protocol\"\"\"\r\n def __init__(self, items: Sequence[Any]):\r\n \"\"\"\r\n Initializes the object with the Sequence specified.\r\n Args:\r\n items: An object that defines the __getitem__ method.\r\n \"\"\"\r\n # internal \"protected\" fields\r\n self._sequence = items\r\n self._counter = 0\r\n\r\n def __iter__(self):\r\n \"\"\"Returns a reference to the Iterable object\"\"\"\r\n return self\r\n\r\n def __next__(self):\r\n \"\"\"Returns the next item or raises StopIteration\"\"\"\r\n # if the indexing variable hasn't exceeded the length of\r\n # the sequence, get the next item -- otherwise raise StopIteration\r\n if self._counter < len(self._sequence):\r\n item = self._sequence[self._counter]\r\n self._counter += 1\r\n return item\r\n raise StopIteration<\/pre>\nNote<\/strong>: The use of Sequence<\/code> here asserts a type hint that the argument should implement the __getitem__<\/code> method that provides self[key]<\/code> type access.<\/em><\/p>\nThis custom MyIterable<\/code> class taps into the power of Python’s Iterator Protocol by implementing a custom __next__<\/code> method. This method allows instances of this custom object to tap into Python-native language tools like for<\/code> and while<\/code> and even\u00a0list comprehensions.<\/em> Let’s take our new object for a test drive:<\/p>\nFor Loops<\/h3>\n
Our custom object allows instances to be accessed via standard for<\/code> loop syntax as follows:<\/p>\n# create new instance encapsulating a list object\r\nmy_list = MyIterable([1, 2, 3, 4, 5])\r\n\r\n# use Python-native for looping to iterate\r\nfor item in my_list:\r\n print(item)<\/pre>\nThis code produces the following to stdout<\/code>, just as if we had used a list<\/code> object:<\/p>\n1\r\n2\r\n3\r\n4\r\n5<\/pre>\nList Comprehensions<\/h3>\n
Python’s list comprehensions are a\u00a0powerful<\/em> abstraction for applying logic to sequences of values. Below, we see our custom object instance can make use of this powerful language-level tool as well:<\/p>\n# create new instance encapsulating a list object\r\nmy_list = MyIterable([1, 2, 3, 4, 5])\r\n\r\n# copy 1-4 into a separate list using list comp.\r\nno_five = [x for x in my_list if x < 5]\r\n\r\n# print results\r\nfor j in no_five:\r\n print(j)<\/pre>\nSimilar to before, this code produces the following to stdout<\/code> minus the last element:<\/p>\n1\r\n2\r\n3\r\n4<\/pre>\nThe real benefit of using custom iterable objects defined via the Python Iterator Protocol is that they are accessible to the languages tools that can interact with built-in iterables like list<\/code>, dict<\/code> and even str<\/code>. Below, we see how the .join<\/code> function can be used on a Custom Iterable object:<\/p>\n# chain the join tool and list comprehension together\r\nstringed = \"\".join([str(x) for x in my_list])<\/pre>\nThe output is a string: 12345<\/code>. Sure, this isn’t a\u00a0direct<\/em> use case but it shows how abiding by the Python Iterable Protocol allows easy folding into Python syntax.<\/p>\nIterator vs. Iterable vs. Iteration<\/h2>\n
Goal<\/strong>: Develop an intuition for Python’s Iterator Protocol<\/p>\nThere are some semantic concepts one needs to be aware of when using iterators in general — but especially when designing and implementing custom Python iterables. Let’s define some terms:<\/p>\n
\n- Iterator<\/strong>: an object with a
__next__<\/code> method (or next<\/code> in Python 2.x).<\/li>\n- Iterable<\/strong>: an object with a
__iter__<\/code> method that maintains state of an iterator.<\/li>\n- Iteration<\/strong>: A\u00a0general<\/em> term used to describe the process of iterating over a sequence of items (language agnostic.)<\/li>\n<\/ul>\n
Those are some easy-access definitions but don’t get us far in terms of developing an intuition of their relevancy in Python. The language from Python’s Iterators documentation<\/a> is helpful:<\/p>\n… the for statement calls iter()<\/code> on the container object. The function returns an iterator object that defines the method __next__()<\/code> which accesses elements in the container one at a time. When there are no more elements, __next__()<\/code> raises a StopIteration<\/code> exception which tells the for loop to terminate.<\/em><\/p>\nNow — a curveball. There is a special method named __getitem__<\/code> that was used to define an Iterable in early 2.x<\/code> versions of Python, before PEP234. This was how an object was defined as being Iterable. This method is still compatible with Python 3.x<\/code> versions and AFAIK has no hints of deprecation in how it is used to qualify an object as an iterable. The __getitem__<\/code> method is used primarily to implement the evaluation of self[key]<\/code> syntax. Read here for more<\/a>. The takeaway:<\/p>\n\n__getitem__<\/code> used to be used to define an Iterable in older versions of Python.<\/li>\n__getitem__<\/code> is now used to implement the self[key]<\/code> access for objects.<\/li>\n- An Iterable implementing
__next__<\/code> and __iter__<\/code> does not have self[key]<\/code> access unless it\u00a0also\u00a0<\/em>implements the __getitem__<\/code> method.<\/li>\n<\/ul>\nRemember that last point — you can iterate over an Iterator but not necessarily index into its values. More specifically: Iterables<\/code> are Sequences<\/code> by default.<\/p>\nCustom Iterable Objects<\/h2>\n
In our example above, we saw a custom implementation of a basic Iterable object. Instances of this object are used similarly to how any native Sequence<\/code> type object might be used and support interoperability with many language features like list comprehension. This object isn’t very exciting though. Let’s implement some more exciting custom Iterables in Python:<\/p>\nExample 1: Fibonacci Iterable<\/h3>\n
Most Computer Science programs will, at some point, ask students to implement a function that calculates the n-th member in the Fibonacci sequence. Commonly, this is introduced as the seminal example of recursion<\/a>. Let’s implement it as a Python Iterable:<\/p>\nclass Fibonacci:\r\n def __init__(self, n: int):\r\n \"\"\"\r\n Args:\r\n n: The number in the Fibonacci sequence that is to be calculated.\r\n \"\"\"\r\n self.n = n\r\n self._i = 0\r\n self._current = 0\r\n self._next = 1\r\n\r\n def __iter__(self):\r\n return self\r\n\r\n def __next__(self):\r\n \"\"\"Calculates and returns the next value in the Fibonacci sequence.\"\"\"\r\n\r\n # returns next digit until the n-th digit is reached\r\n if self._i < self.n:\r\n\r\n # increases reference to members\r\n self._i += 1\r\n\r\n # gets the current value\r\n fib = self._current\r\n\r\n # gets next values in sequence\r\n self._current, self._next = self._next, self._current + self._next\r\n return fib\r\n\r\n # raises the StopIteration error which Python uses to end e.g. for loops\r\n else:\r\n raise StopIteration<\/pre>\nHere we see all the essential Iterable methods: __iter__<\/code> and __next__<\/code> implemented. During each call to __next__<\/code> (made during a for<\/code> loop iteration for example) the next digit in the Fibonacci sequence is calculated until we reach the value of n<\/code> at which point the StopIteration<\/code> exception is raised — signaling Python’s Iterator Protocol that the iterator has been exhausted. Let’s see how this can be used:<\/p>\nApproach 1: Looping<\/h3>\n# loops\r\nfor digit in Fibonacci(10):\r\n print(digit)\r\n\r\n# output\r\n0\r\n1\r\n1\r\n2\r\n3\r\n5\r\n8\r\n13\r\n21\r\n34\r\n<\/pre>\nApproach 2: Explicit List Conversion<\/h3>\n
Our Fibonacci<\/code> sequence can be converted to a list explicitly by using Python’s list<\/code> builtin:<\/p>\n# explicitly converted to a list\r\ndigits = list(Fibonacci(10))\r\n\r\n# output\r\n[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]\r\n<\/pre>\nApproach 3: List Comprehension<\/h3>\n
Implementing Fibonacci using Python’s Iterator Protocol means we can also leverage Python’s list comprehension<\/strong>:<\/p>\n# via list comprehension\r\ndigits = [d for d in Fibonacci(10)]\r\n\r\n# output\r\n[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]\r\n<\/pre>\nApproach 4: Wrapped in a Function<\/h3>\n
For cases where\u00a0only<\/em> the n-th digit is needed, the Fibbonacci<\/code> class can be wrapped into a custom function:<\/p>\n# via custom function\r\ndef get_nth_fibonacci(n: int) -> int:\r\n \"\"\"Returns the nth digit in the Fibonacci sequence\"\"\"\r\n return list(Fibonacci(n))[-1]\r\n\r\n# get a digit\r\nprint(get_nth_fibonacci(n=15))\r\n\r\n>>> 377<\/pre>\nThe headline here is: anything one can do with an Iterable<\/code> in Python one can now do with our custom Fibbonacci<\/code> class!<\/p>\nSummary<\/h2>\n
Python’s Iterator Protocol is a powerful framework whereby developers can implement custom classes that are able to leverage the full power of the Python programming language. As described by the documentation: “This style of access is clear, concise, and convenient” — the three C’s.\u00a0<\/em>To quote again from the documentation: “The use of iterators pervades and unifies Python.” As such, making use of the Iterator Protocol helps ensure that one’s custom classes are Pythonic and more easily understood by other developers. For those readers yearning for more: check out the documentation for Python’s Data Model<\/a> for an even deeper dive into iterators.<\/p>\n","protected":false},"excerpt":{"rendered":"Python has a novel framework whereby objects can define how to iterate over the data they encapsulate. This framework, known as the Iterator Protocol, empowers developers to abstract the process of how an object returns members individually. Here we take a dive into understanding Python’s Iterator Protocol, the “magic” methods used, and how to tap […]<\/p>\n","protected":false},"author":2,"featured_media":3451,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[186,655],"tags":[369,740,143],"acf":[],"_links":{"self":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/3448"}],"collection":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/comments?post=3448"}],"version-history":[{"count":7,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/3448\/revisions"}],"predecessor-version":[{"id":3458,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/posts\/3448\/revisions\/3458"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media\/3451"}],"wp:attachment":[{"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/media?parent=3448"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/categories?post=3448"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpharithms.com\/wp-json\/wp\/v2\/tags?post=3448"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}