In practice, what are the main uses for the yield from syntax in Python 3.3?
Let's get one thing out of the way first. The explanation that yield from g
is equivalent to for v in g: yield v
does not even begin to do justice to what yield from
is all about. Because, let's face it, if all yield from
does is expand the for
loop, then it does not warrant adding yield from
to the language and preclude a whole bunch of new features from being implemented in Python 2.x.
What yield from
does is it establishes a transparent bidirectional connection between the caller and the sub-generator:
The connection is "transparent" in the sense that it will propagate everything correctly too, not just the elements being generated (e.g. exceptions are propagated).
The connection is "bidirectional" in the sense that data can be both sent from and to a generator.
(If we were talking about TCP, yield from g
might mean "now temporarily disconnect my client's socket and reconnect it to this other server socket".)
BTW, if you are not sure what sending data to a generator even means, you need to drop everything and read about coroutines first—they're very useful (contrast them with subroutines), but unfortunately lesser-known in Python. Dave Beazley's Curious Course on Coroutines is an excellent start. Read slides 24-33 for a quick primer.
Reading data from a generator using yield from
def reader():
"""A generator that fakes a read from a file, socket, etc."""
for i in range(4):
yield '<< %s' % i
def reader_wrapper(g):
# Manually iterate over data produced by reader
for v in g:
yield v
wrap = reader_wrapper(reader())
for i in wrap:
print(i)
# Result
<< 0
<< 1
<< 2
<< 3
Instead of manually iterating over reader()
, we can just yield from
it.
def reader_wrapper(g):
yield from g
That works, and we eliminated one line of code. And probably the intent is a little bit clearer (or not). But nothing life changing.
Sending data to a generator (coroutine) using yield from - Part 1
Now let's do something more interesting. Let's create a coroutine called writer
that accepts data sent to it and writes to a socket, fd, etc.
def writer():
"""A coroutine that writes data *sent* to it to fd, socket, etc."""
while True:
w = (yield)
print('>> ', w)
Now the question is, how should the wrapper function handle sending data to the writer, so that any data that is sent to the wrapper is transparently sent to the writer()
?
def writer_wrapper(coro):
# TBD
pass
w = writer()
wrap = writer_wrapper(w)
wrap.send(None) # "prime" the coroutine
for i in range(4):
wrap.send(i)
# Expected result
>> 0
>> 1
>> 2
>> 3
The wrapper needs to accept the data that is sent to it (obviously) and should also handle the StopIteration
when the for loop is exhausted. Evidently just doing for x in coro: yield x
won't do. Here is a version that works.
def writer_wrapper(coro):
coro.send(None) # prime the coro
while True:
try:
x = (yield) # Capture the value that's sent
coro.send(x) # and pass it to the writer
except StopIteration:
pass
Or, we could do this.
def writer_wrapper(coro):
yield from coro
That saves 6 lines of code, make it much much more readable and it just works. Magic!
Sending data to a generator yield from - Part 2 - Exception handling
Let's make it more complicated. What if our writer needs to handle exceptions? Let's say the writer
handles a SpamException
and it prints ***
if it encounters one.
class SpamException(Exception):
pass
def writer():
while True:
try:
w = (yield)
except SpamException:
print('***')
else:
print('>> ', w)
What if we don't change writer_wrapper
? Does it work? Let's try
# writer_wrapper same as above
w = writer()
wrap = writer_wrapper(w)
wrap.send(None) # "prime" the coroutine
for i in [0, 1, 2, 'spam', 4]:
if i == 'spam':
wrap.throw(SpamException)
else:
wrap.send(i)
# Expected Result
>> 0
>> 1
>> 2
***
>> 4
# Actual Result
>> 0
>> 1
>> 2
Traceback (most recent call last):
... redacted ...
File ... in writer_wrapper
x = (yield)
__main__.SpamException
Um, it's not working because x = (yield)
just raises the exception and everything comes to a crashing halt. Let's make it work, but manually handling exceptions and sending them or throwing them into the sub-generator (writer
)
def writer_wrapper(coro):
"""Works. Manually catches exceptions and throws them"""
coro.send(None) # prime the coro
while True:
try:
try:
x = (yield)
except Exception as e: # This catches the SpamException
coro.throw(e)
else:
coro.send(x)
except StopIteration:
pass
This works.
# Result
>> 0
>> 1
>> 2
***
>> 4
But so does this!
def writer_wrapper(coro):
yield from coro
The yield from
transparently handles sending the values or throwing values into the sub-generator.
This still does not cover all the corner cases though. What happens if the outer generator is closed? What about the case when the sub-generator returns a value (yes, in Python 3.3+, generators can return values), how should the return value be propagated? That yield from
transparently handles all the corner cases is really impressive. yield from
just magically works and handles all those cases.
I personally feel yield from
is a poor keyword choice because it does not make the two-way nature apparent. There were other keywords proposed (like delegate
but were rejected because adding a new keyword to the language is much more difficult than combining existing ones.
In summary, it's best to think of yield from
as a transparent two way channel
between the caller and the sub-generator.
References:
- PEP 380 - Syntax for delegating to a sub-generator (Ewing) [v3.3, 2009-02-13]
- PEP 342 -
Coroutines via Enhanced Generators (GvR, Eby) [v2.5, 2005-05-10]
What does the yield from syntax do in asyncio and how is it different from await
Short answer:
yield from
is an old way to wait for asyncio's coroutine.
await
is a modern way to wait for asyncio's coroutine.
Detailed answer:
Python has generators - special kind of functions that produces a sequence of results instead of a single value. Starting with Python 3.3 yield from
expression was added. It allows one generator to delegate part of its operations to another generator.
Starting with Python 3.4 asyncio
module was added to standard library. It allow us to write clear and understandable asynchronous code. While technically asyncio's coroutines could be implemented different ways, in asyncio
they were implemented using generators (you can watch for excellent video where shown how generators can be used to implement coroutines). @asyncio.coroutine
was a way to make coroutine from generator and yield from
was a way to await for coroutine - just details of implementation.
That's how happened that yield from
started to be used for two "different things".
Starting with Python 3.5 (see PEP 492) coroutines got new syntax. Now you can define coroutine with async def
and await for it using await
expression. It's not only shorter to write, but also makes clearer to understand that we work with asyncio's coroutines.
If you're using Python 3.5+ you can forget about using yield from
for asyncio's coroutines and use await
for it.
Having trouble understanding the yield part of this code
while True:
The while
statement checks whether the right-hand value is true. If yes, it performs another loop, and if no, it breaks out of the loop. This statement loops indefinitely.
yield a
In a generator, this is sort of like a return
statement, except that it lets the generator function continue executing. This statement enables the program to execute so that the generator gives values to the caller in between execution and does not have to run all the way to end (which in this case is infinity).
a, b=b, a+b
This is simply a Pythonic way of assigning values to variables. It is called multiple assignment and it works as follows:
variable1, variable2, ... varaibleN = value1, value2, ... valueN
If a
starts at 1
, the next time a
is referred to is in yield a
, and it will yield 1
.
a
is assigned a new value at a, b = b, a + b
.
Python 3 generator expression
No need for nested functions and your base case can be even simpler:
def gen_letters(l):
if not l:
yield ""
else:
for c in 'abcdefghijklmnopqrstuvwxyz ':
for combo in gen_letters(l-1):
yield c + combo
[*gen_letters(2)]
# ['aa', 'ab', ..., ' x', ' y', ' z', ' ']
Or, closer to your original:
def gen_letters(s,l):
if l:
for c in 'abcdefghijklmnopqrstuvwxyz ':
yield from gen_letters(s+c,l-1)
else:
yield s
[*gen_letters('', 2)]
# ['aa', 'ab', ..., ' x', ' y', ' z', ' ']
Notice the yield from
line. This is where you did the correct recursive call, but did not do anything with the result.
Related Topics
Binning Data in Python with Scipy/Numpy
How to Write Output in Same Place on the Console
How to Check Whether a File Is Empty or Not
Interprocess Communication in Python
Split a Large Pandas Dataframe
How to One-Hot-Encode from a Pandas Column Containing a List
How to Split a Column of Tuples in a Pandas Dataframe
How to Test the Membership of Multiple Values in a List
How Do Chained Assignments Work
Circular List Iterator in Python
Permissionerror: [Errno 13] in Python
Subprocess.Popen: Cloning Stdout and Stderr Both to Terminal and Variables
Python List Doesn't Reflect Variable Change
Tkinter: "Python May Not Be Configured for Tk"
Python Matplotlib Update Scatter Plot from a Function
Why Are Packages Installed Rather Than Just Linked to a Specific Environment