In Python, Is It Better to Use List Comprehensions or For-Each Loops

In Python, is it better to use list comprehensions or for-each loops?

If the iteration is being done for its side effect ( as it is in your "print" example ), then a loop is clearer.

If the iteration is executed in order to build a composite value, then list comprehensions are usually more readable.

Is it more efficient to use list comprehension or nested loops?

As with any question about efficiency, the only real answer is to profile your code, because it always depends.

That said, there should be no noticeable difference between for loops and list comprehensions (they most likely compile to the same byte code) and you should use what is most readable. In this case, I'd say the nested for loops are far more readable than nested comprehensions.

When to use 'For Loop vs List Comprehension' to create New Lists?

What are the advantages of using List Comprehensions? First of all, you’re reducing 3 lines of code into one, which will be instantly recognizable to anyone who understands list comprehensions. Secondly, the second code is faster, as Python will allocate the list’s memory first, before adding the elements to it, instead of having to resize on runtime. It’ll also avoid having to make calls to ‘append’, which may be cheap but add up. Lastly, code using comprehensions is considered more ‘Pythonic’ — better fitting Python’s style guidelines. Python’s List Comprehensions: Uses and Advantages, Luciano Strika

List comprehension:

Easier to read
Quicker, because of prior memory allocation

For loop:

More flexible

What is the difference between normal loops and list comprehension?

A list comprehension is used to comprehend (Make) a list. It is useful only when making lists. However, here you are not making a list, so it is not recommended to use list comprehension. You only print the value and not store it as a list. Here, use a for a loop.

The reason you get None is - the list comprehension basically becomes a list of print() functions like [print(...),print(...)....]

So when you call them it becomes like - print(print(...)), which, if you try this code, will return a None along with the output.

So, do not use list comprehension unless you are using it to build a list.

References - This and That

Comparing list comprehensions and explicit loops (3 array generators faster than 1 for loop)

Let's define the functions we will need to answer the question and timeit them:

In [18]: def iter():
    l = [x for x in range(100) if x > 10]
   ....:

In [19]: %timeit iter()
100000 loops, best of 3: 7.92 µs per loop

In [20]: def loop():
    l = []
    for x in range(100):
        if x > 10:
            l.append(x)
   ....:

In [21]: %timeit loop()
10000 loops, best of 3: 20 µs per loop

In [22]: def loop_fast():
    l = []
    for x in range(100):
        if x > 10:
            pass
   ....:

In [23]: %timeit loop_fast()
100000 loops, best of 3: 4.69 µs per loop

we can see that the for loops without the append command is as fast as the list comprehension. In fact, if we have a look at the bytecode we can see that in the case of the list comprehension python is able to use a built-in bytecode command called LIST_APPEND instead of:

Load the list: 40 LOAD_FAST
Load the attribute: 43 LOAD_ATTRIBUTE
Call the loaded function: 49 CALL_FUNCTION
Unload the list(?): 52 POP_TOP

As you can see from the output below the previous bytecode are missing with list comprehension and with the "loop_fast" function. Comparing the timeit of the three function is clear that those are responsible for the different timing of the three methods.

In [27]: dis.dis(iter)
  2          0 BUILD_LIST             0
             3 LOAD_GLOBAL            0 (range)
             6 LOAD_CONST             1 (1)
             9 LOAD_CONST             2 (100)
            12 CALL_FUNCTION          2
            15 GET_ITER
       >>   16 FOR_ITER              24 (to 43)
            19 STORE_FAST             0 (x)
            22 LOAD_FAST              0 (x)
            25 LOAD_CONST             2 (100)
            28 COMPARE_OP             4 (>)
            31 POP_JUMP_IF_FALSE     16
            34 LOAD_FAST              0 (x)
            37 LIST_APPEND            2
            40 JUMP_ABSOLUTE         16
       >>   43 STORE_FAST             1 (l)
            46 LOAD_CONST             0 (None)
            49 RETURN_VALUE

In [28]: dis.dis(loop)
  2          0 BUILD_LIST             0
             3 STORE_FAST             0 (1)

  3          6 SETUP_LOOP            51 (to 60)
             9 LOAD_GLOBAL            0 (range)
            12 LOAD_CONST             1 (1)
            15 LOAD_CONST             2 (100)
            18 CALL_FUNCTION          2
            21 GET_ITER
       >>   22 FOR_ITER              34 (to 59)
            25 STORE_FAST             1 (x)

  4         28 LOAD_FAST              1 (x)
            31 LOAD_CONST             3 (10)
            34 COMPARE_OP             4 (>)
            37 POP_JUMP_IF_FALSE     22

  5         40 LOAD_FAST              0 (l)
            43 LOAD_ATTR              1 (append)
            46 LOAD_FAST              1 (x)
            49 CALL_FUNCTION          1
            52 POP_TOP
            53 JUMP_ABSOLUTE         22
            56 JUMP_ABSOLUTE         22
       >>   59 POP_BLOCK
       >>   60 LOAD_CONST             0 (None)
            63 RETURN_VALUE

In [29]: dis.dis(loop_fast)
  2          0 BUILD_LIST             0
             3 STORE_FAST             0 (1)

  3          6 SETUP_LOOP            38 (to 47)
             9 LOAD_GLOBAL            0 (range)
            12 LOAD_CONST             1 (1)
            15 LOAD_CONST             2 (100)
            18 CALL_FUNCTION          2
            21 GET_ITER
       >>   22 FOR_ITER              21 (to 46)
            25 STORE_FAST             1 (x)

  4         28 LOAD_FAST              1 (x)
            31 LOAD_CONST             3 (10)
            34 COMPARE_OP             4 (>)
            37 POP_JUMP_IF_FALSE     22

  5         40 JUMP_ABSOLUTE         22
            43 JUMP_ABSOLUTE         22
       >>   46 POP_BLOCK
       >>   47 LOAD_CONST             0 (None)
            50 RETURN_VALUE

Are list-comprehensions and functional functions faster than for loops?

The following are rough guidelines and educated guesses based on experience. You should timeit or profile your concrete use case to get hard numbers, and those numbers may occasionally disagree with the below.

A list comprehension is usually a tiny bit faster than the precisely equivalent for loop (that actually builds a list), most likely because it doesn't have to look up the list and its append method on every iteration. However, a list comprehension still does a bytecode-level loop:

>>> dis.dis(<the code object for `[x for x in range(10)]`>)
 1           0 BUILD_LIST               0
             3 LOAD_FAST                0 (.0)
       >>    6 FOR_ITER                12 (to 21)
             9 STORE_FAST               1 (x)
            12 LOAD_FAST                1 (x)
            15 LIST_APPEND              2
            18 JUMP_ABSOLUTE            6
       >>   21 RETURN_VALUE

Using a list comprehension in place of a loop that doesn't build a list, nonsensically accumulating a list of meaningless values and then throwing the list away, is often slower because of the overhead of creating and extending the list. List comprehensions aren't magic that is inherently faster than a good old loop.

As for functional list processing functions: While these are written in C and probably outperform equivalent functions written in Python, they are not necessarily the fastest option. Some speed up is expected if the function is written in C too. But most cases using a lambda (or other Python function), the overhead of repeatedly setting up Python stack frames etc. eats up any savings. Simply doing the same work in-line, without function calls (e.g. a list comprehension instead of map or filter) is often slightly faster.

Suppose that in a game that I'm developing I need to draw complex and huge maps using for loops. This question would be definitely relevant, for if a list-comprehension, for example, is indeed faster, it would be a much better option in order to avoid lags (Despite the visual complexity of the code).

Chances are, if code like this isn't already fast enough when written in good non-"optimized" Python, no amount of Python level micro optimization is going to make it fast enough and you should start thinking about dropping to C. While extensive micro optimizations can often speed up Python code considerably, there is a low (in absolute terms) limit to this. Moreover, even before you hit that ceiling, it becomes simply more cost efficient (15% speedup vs. 300% speed up with the same effort) to bite the bullet and write some C.

When to use a list comprehension vs a for loop

The answer depends on your opinion. However, since I do remember a specific piece of advice of this from an author of a book that is well regarded in the Python community, I can share the following excerpt from the book 'Fluent Python' by Luciano Ramalho:

A for loop may be used to do lots of different things: scanning a sequence to count or pick items, computing aggregates (sums, averages), or any number of other processing tasks. [...] In contrast, a listcomp is meant to do one thing only: to build a new list.

Of course, it is possible to abuse list comprehensions to write truly incomprehensible code. I’ve seen Python code with listcomps used just to repeat a block of code for its
side effects.

If you are not doing something with the produced list, you should not use that syntax.

Also, try to keep it short. If the list comprehension spans more than two lines, it is probably best to break it apart or rewrite as a plain old for loop. Use your best judgment: for Python as for English, there are no hard-and-fast rules for clear writing.

Multiple list comprehension vs single for loop

I would try it without a loop using np.where clauses for the if-elif-else combinations. That's usually pretty fast.

import numpy as np

# dataframe is a DataFrame containing data
# Now this:

dataframe["Price"] = np.where(dataframe["Price_Dummy"] == "0",0,1)

# String operations work on whole string columns as well
unit_of_measure = dataframe["Size"].str.split(" ", expand=True)[1].lower()

size = dataframe["Size"].str.split(" ", expand=True)[0].astype("float")

kb_case = np.where(unit_of_measure =="kb", size/1000, size)
dataframe["Size"] = np.where(unit_of_measure =="gb", size*1000, kb_case)

Notice that I replaced the [-1] in the unit_of_measure = line with [1] as the expand=True option does not support the -1 indexing. So you would have to know at which position your unit ends up.

Information on splitting strings in DataFrames can be found here.

In the last two lines, I reproduced the if-elif-else combination which you kind of have to create from the bottom up: Your final result dataframe["Size"] equals size*1000 if the unit is gb. If not, it equals the kb_case which includes the case where the unit is kb as well as all other cases.

What is the advantage of a list comprehension over a for loop?

List comprehensions are more compact and faster than an explicit for loop building a list:

def slower():
    result = []
    for elem in some_iterable:
        result.append(elem)
    return result

def faster():
    return [elem for elem in some_iterable]

This is because calling .append() on a list causes the list object to grow (in chunks) to make space for new elements individually, while the list comprehension gathers all elements first before creating the list to fit the elements in one go:

>>> some_iterable = range(1000)
>>> import timeit
>>> timeit.timeit('f()', 'from __main__ import slower as f', number=10000)
1.4456570148468018
>>> timeit.timeit('f()', 'from __main__ import faster as f', number=10000)
0.49323201179504395

However, this does not mean you should start using list comprehensions for everything! A list comprehension will still build a list object; if you are using a list comprehension just because it gives you a one-line loop, think again. You are probably wasting cycles building a list object that you then discard again. Just stick to a normal for loop in that case.

Python List Comprehension vs For

Essentially, list comprehension and for loops does pretty similar things, with list comprehension doing away some overheads and making it look pretty.
To understand why this is faster, you should look in Efficiency of list comprehensions and to quote the relevant part for your problem:

List comprehensions perform better here because you don’t need to load
the append attribute off of the list (loop program, bytecode 28) and
call it as a function (loop program, bytecode 38). Instead, in a
comprehension, a specialized LIST_APPEND bytecode is generated for a
fast append onto the result list (comprehension program, bytecode 33).
In the loop_faster program, you avoid the overhead of the append
attribute lookup by hoisting it out of the loop and placing the result
in a fastlocal (bytecode 9-12), so it loops more quickly; however, the
comprehension uses a specialized LIST_APPEND bytecode instead of
incurring the overhead of a function call, so it still trumps.

The link also details some of the possible pitfalls associated with lc and I would recommend you to go through it once.

In Python, Is It Better to Use List Comprehensions or For-Each Loops