Python: Editing List While Iterating Over It

Modifying list while iterating

I've been bitten before by (someone else's) "clever" code that tries to modify a list while iterating over it. I resolved that I would never do it under any circumstance.

You can use the slice operator mylist[::3] to skip across to every third item in your list.

mylist = [i for i in range(100)]
for i in mylist[::3]:
    print(i)

Other points about my example relate to new syntax in python 3.0.

• I use a list comprehension to define mylist because it works in Python 3.0 (see below)
• print is a function in python 3.0

Python 3.0 range() now behaves like xrange() used to behave, except it works with values of arbitrary size. The latter no longer exists.

Modify a list while iterating

You are not modifying the list, so to speak. You are simply modifying the elements in the list. I don't believe this is a problem.

To answer your second question, both ways are indeed allowed (as you know, since you ran the code), but it would depend on the situation. Are the contents mutable or immutable?

For example, if you want to add one to every element in a list of integers, this would not work:

>>> x = [1, 2, 3, 4, 5]
>>> for i in x:
...     i += 1
... 
>>> x
[1, 2, 3, 4, 5] 

Indeed, ints are immutable objects. Instead, you'd need to iterate over the indices and change the element at each index, like this:

>>> for i in range(len(x)):
...     x[i] += 1
...
>>> x
[2, 3, 4, 5, 6]

If your items are mutable, then the first method (of directly iterating over the elements rather than the indices) is more efficient without a doubt, because the extra step of indexing is an overhead that can be avoided since those elements are mutable.

Modifying a list while iterating over it - why not?

while len(mylist) > 0:
    print mylist.pop()

You are not iterating over the list. You are each time checking an atomic condition.

Also:

while len(mylist) > 0:

can be rewritten as:

while len(mylist):

which can be rewritten as:

while mylist:

How to modify list entries during for loop?

It's considered poor form. Use a list comprehension instead, with slice assignment if you need to retain existing references to the list.

a = [1, 3, 5]
b = a
a[:] = [x + 2 for x in a]
print(b)

How to remove items from a list while iterating?

You can use a list comprehension to create a new list containing only the elements you don't want to remove:

somelist = [x for x in somelist if not determine(x)]

Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want:

somelist[:] = [x for x in somelist if not determine(x)]

This approach could be useful if there are other references to somelist that need to reflect the changes.

Instead of a comprehension, you could also use itertools. In Python 2:

from itertools import ifilterfalse
somelist[:] = ifilterfalse(determine, somelist)

Or in Python 3:

from itertools import filterfalse
somelist[:] = filterfalse(determine, somelist)

Updating a set while iterating over its elements

It's actually very easy, sets are implemented in CPython as hash - item table:

  hash  |  item  
-----------------
    -   |    -
-----------------
    -   |    -
       ...

CPython uses open-addressing so not all rows in the table are filled and it determines the position of an element based on the (truncated) hash of the item with a "pseudo-randomized" position determination in case of collisions. I will ignore truncated-hash-collisions in this answer.

I'll also ignore the specifics of the hash-truncation and just work with integers, which all (with some exceptions) map their hash to the actual value:

>>> hash(1)
1
>>> hash(2)
2
>>> hash(20)
20

So when you create a set with the values 1, 2 and 3 you'll have (roughly) the following table:

  hash  |  item  
-----------------
    -   |    -
-----------------
    1   |    1
-----------------
    2   |    2
-----------------
    3   |    3
       ...

The set is iterated from the top of the table to the end of the table and empty "rows" are ignored. So if you would iterate over that set without modifying it you would get the numbers 1, 2 and 3:

>>> for item in {1, 2, 3}: print(item)
1
2
3

Basically the iterator starts in row 0 and sees that the row is empty and goes to row 1 which contains the item 1. This item is returned by the iterator. The next iteration it's in row 2 and returns the value there, which is 2, then it goes to row 3 and returns the 3 that is stored there. The following iteration the iterator is in row 4 which is empty, so it goes to row 5 which is also empty, then to row 6, .... until it reaches the end of the table and throws a StopIteration exception, which signals that the iterator finished.

However if you would change the set while iterating over it the current position (row) where the iterator is is remembered. That means if you add an item in a previous row the iterator won't return it, if it's added in a later row it will be returned (at least if it's not removed again before the iterator is there).

Assume, you always remove the current item and add an integer which is item + 1 to the set. Something like this:

s = {1}
for item in s: 
    print(item)
    s.discard(item)
    s.add(item+1)

The set before the iteration looks like this:

  hash  |  item  
-----------------
    -   |    -
-----------------
    1   |    1
-----------------
    -   |    -
       ...

The iterator will start in row 0 find it is empty and goes to row 1 which contains the value 1 which is then returned and printed. If the arrow would indicate the position of the iterator it would look like this in the first iteration:

  hash  |  item  
-----------------
    -   |    -
-----------------
    1   |    1      <----------
-----------------
    -   |    -

Then the 1 is removed and the 2 is added:

  hash  |  item  
-----------------
    -   |    -
-----------------
    -   |    -      <----------
-----------------
    2   |    2

So in the next iteration the iterator finds the value 2 and returns it. Then the two is added and a 3 is added:

  hash  |  item  
-----------------
    -   |    -
-----------------
    -   |    -
-----------------
    -   |    -      <----------
-----------------
    3   |    3

And so on.

Until it reaches 7. At that point something interesting happens: The truncated hash of 8 means that the 8 will be put in row 0, however row 0 has already been iterated over so it will stop with 7. The actual value might depend on Python version and the add/removal history of the set, for example just changing the order of the set.add and set.discard operations will give a different result (goes up to 15 because the set is resized).

For the same reason the iterator would only display 1 if one would add the item - 1 in each iteration, because 0 would be (because of hash 0) to the first row:

s = {1}
for item in s: 
    print(item)
    s.discard(item)
    s.add(item-1)

  hash  |  item  
-----------------
    -   |    -
-----------------
    1   |    1      <----------
-----------------
    -   |    -

  hash  |  item  
-----------------
    0   |    0
-----------------
    -   |    -
-----------------
    -   |    -      <----------

Visualized with a simple GIF:

Note that these examples are very simplistic, if the set resizes during the iteration it will re-distribute the stored items based on the "new" truncated hash and it will also remove dummies that are left behind when you remove an item from a set. In that case you still can (roughly) predict what will happen but it will become a lot more convoluted.

One additional but very important fact is that Python (since Python 3.4) randomizes the hash value of strings for each interpreter. That means that each Python session will produce different hash-values for strings. So if you add/remove strings while iterating the behavior will also be random.

Suppose you have this script:

s = {'a'}
for item in s: 
    print(item)
    s.discard(item)
    s.add(item*2)

and you run it multiple times from the command line the result will be different.

For example my first run:

'a'
'aa'

My second / third / fourth run:

'a'

My fifth run:

'a'
'aa'

That's because scripts from the command line always start a new interpreter session. If you run the script multiple times in the same session the results won't differ. For example:

>>> def fun():
...     s = {'a'}
...     for item in s: 
...         print(item)
...         s.discard(item)
...         s.add(item*2)

>>> for i in range(10):
...     fun()

produces:

a
aa
a
aa
a
aa
a
aa
a
aa
a
aa
a
aa
a
aa
a
aa
a
aa

But it could also have given 10 times a or 10 times a, aa and aaaa, ...

---

To summarize it:

• A value added to a set during iteration will be displayed if the item is put in a position that hasn't been iterated over. The position depends on the truncated hash of the item and the collision strategy.

• The truncation of the hash depends on the size of the set and that size depends on the add/removal history of the set.

• With strings one cannot predict the position because their hash values are randomized on a per-session-basis in recent Python versions.

• And most importantly: Avoid modifying the set / list / dict / ... while iterating over it. It almost always leads to problems and even if it doesn't it will confuse anyone reading it! Although there are a few, very rare cases where adding elements to a list while iterating over it makes sense. That would require very specific comments alongside it, otherwise it would look like a mistake! Especially with sets and dicts you will rely on implementation details that might change at any point!

---

Just in case you're curious I inspected the internals of the set using (somewhat fragile and probably only works on Python 3.6 and definitely not usable in production-code) Cython introspection in a Jupyter Notebook:

%load_ext Cython

%%cython

from cpython cimport PyObject, PyTypeObject
cimport cython

cdef extern from "Python.h":
    ctypedef Py_ssize_t Py_hash_t

    struct setentry:
        PyObject *key
        Py_hash_t hash

    ctypedef struct PySetObject:
        Py_ssize_t ob_refcnt
        PyTypeObject *ob_type
        Py_ssize_t fill
        Py_ssize_t used
        Py_ssize_t mask
        setentry *table
        Py_hash_t hash
        Py_ssize_t finger

        setentry smalltable[8]
        PyObject *weakreflist

cpdef print_set_table(set inp):
    cdef PySetObject* innerset = <PySetObject *>inp
    for idx in range(innerset.mask+1):
        if (innerset.table[idx].key == NULL):
            print(idx, '<EMPTY>')
        else:
            print(idx, innerset.table[idx].hash, <object>innerset.table[idx].key)

Which prints the key-value table inside the set:

a = {1}
print_set_table(a)

for idx, item in enumerate(a):
    print('\nidx', idx)
    a.discard(item)
    a.add(item+1)
    print_set_table(a)

Note that the output will contain dummys (left-overs from deleted set-items) and they will sometimes disappear (when the set get's too full or resized).

Modifying a list while iterating when programming with python

This should work

AvailibilityLine1 = 1.9
AvailibilityLine2 = 45.9
PerformanceLine1 = 76.5
PerformanceLine2 = 99.9
QualityLine1 = 100.0
QualityLine2 = 0.0

#these come in as floats with a single decimal space, ex:99.9
val = [AvailibilityLine1, AvailibilityLine2, PerformanceLine1, 
       PerformanceLine2, QualityLine1, QualityLine2] 

j = 0
for i in val:
    if i >= float(100):
        val[j] = "100 "
    elif i >= 10 and i < 100:
        val[j] = str(val[j])
    elif i > 0 and  i < 10:
        val[j] = " " + str(val[j])
    elif i <= 0:
        val[j] = "    "    #insert 4 spaces
    else:
        val[j] = "    "    #insert 4 spaces if all else fails
    j = j + 1

print val

Output

>>> [' 1.9', '45.9', '76.5', '99.9', '100 ', '    ']

problems in your code:

1. Initialization of counter J should be outside the for loop
2. You are assigning whole list (val) to list index in 2nd and 3rd condition of if-else

Python: How to remove elements from list while iterating through it without skipping future iterations

You are right. You need an additional list. But there is an easier solution.

def print_numTXTs(fileList):

    counter = 0
    for file in list(fileList):
        if file.name[-4:] == ".txt":
            counter +=1
            if file.name == "a.txt":
                fileList.remove(file)
   

The secret is "list(fileList)". You creating an additional list and iterates over this.

Just as powerful are list compressions. In your example it should work like this. I have not tried now...only quickly written here.

fileList = [ file for file in fileList if file.name != "a.txt" ]

---

Related Topics