Strange Result When Removing Item from a List While Iterating Over It

Strange result when removing item from a list while iterating over it

You're modifying the list while you iterate over it. That means that the first time through the loop, i == 1, so 1 is removed from the list. Then the for loop goes to the second item in the list, which is not 2, but 3! Then that's removed from the list, and then the for loop goes on to the third item in the list, which is now 5. And so on. Perhaps it's easier to visualize like so, with a ^ pointing to the value of i:

[1, 2, 3, 4, 5, 6...]
 ^

That's the state of the list initially; then 1 is removed and the loop goes to the second item in the list:

[2, 3, 4, 5, 6...]
    ^
[2, 4, 5, 6...]
       ^

And so on.

There's no good way to alter a list's length while iterating over it. The best you can do is something like this:

numbers = [n for n in numbers if n >= 20]

or this, for in-place alteration (the thing in parens is a generator expression, which is implicitly converted into a tuple before slice-assignment):

numbers[:] = (n for in in numbers if n >= 20)

If you want to perform an operation on n before removing it, one trick you could try is this:

for i, n in enumerate(numbers):
    if n < 20 :
        print("do something")
        numbers[i] = None
numbers = [n for n in numbers if n is not None]

How to remove items from a list while iterating?

You can use a list comprehension to create a new list containing only the elements you don't want to remove:

somelist = [x for x in somelist if not determine(x)]

Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want:

somelist[:] = [x for x in somelist if not determine(x)]

This approach could be useful if there are other references to somelist that need to reflect the changes.

Instead of a comprehension, you could also use itertools. In Python 2:

from itertools import ifilterfalse
somelist[:] = ifilterfalse(determine, somelist)

Or in Python 3:

from itertools import filterfalse
somelist[:] = filterfalse(determine, somelist)

Removing Item From List - during iteration - what's wrong with this idiom?

Some answers explain why this happens and some explain what you should've done. I'll shamelessly put the pieces together.

---

What's the reason for this?

Because the Python language is designed to handle this use case differently. The documentation makes it clear:

It is not safe to modify the sequence being iterated over in the loop (this can only happen for mutable sequence types, such as lists). If you need to modify the list you are iterating over (for example, to duplicate selected items) you must iterate over a copy.

_{Emphasis mine. See the linked page for more -- the documentation is copyrighted and all rights are reserved.}

You could easily understand why you got what you got, but it's basically undefined behavior that can easily change with no warning from build to build. Just don't do it.

It's like wondering why i += i++ + ++i does whatever the hell it is it that line does on your architecture on your specific build of your compiler for your language -- including but not limited to trashing your computer and making demons fly out of your nose :)

---

How it could this be re-written to remove every item?

• del letters[:] (if you need to change all references to this object)
• letters[:] = [] (if you need to change all references to this object)
• letters = [] (if you just want to work with a new object)

Maybe you just want to remove some items based on a condition? In that case, you should iterate over a copy of the list. The easiest way to make a copy is to make a slice containing the whole list with the [:] syntax, like so:

#remove unsafe commands
commands = ["ls", "cd", "rm -rf /"]
for cmd in commands[:]:
  if "rm " in cmd:
    commands.remove(cmd)

If your check is not particularly complicated, you can (and probably should) filter instead:

commands = [cmd for cmd in commands if not is_malicious(cmd)]

Removing from a list while iterating over it

I debated answering this for a while, because similar questions have been asked many times here. But it's just unique enough to be given the benefit of the doubt. (Still, I won't object if others vote to close.) Here's a visual explanation of what is happening.

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]       <-  b = 0; remove? no
 ^
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]       <-  b = 1; remove? yes
    ^
[0, 2, 3, 4, 5, 6, 7, 8, 9]          <-  b = 3; remove? no
       ^
[0, 2, 3, 4, 5, 6, 7, 8, 9]          <-  b = 4; remove? yes
          ^
[0, 2, 3, 5, 6, 7, 8, 9]             <-  b = 6; remove? no
             ^
[0, 2, 3, 5, 6, 7, 8, 9]             <-  b = 7; remove? yes
                ^
[0, 2, 3, 5, 6, 8, 9]                <-  b = 9; remove? no
                   ^

Since no one else has, I'll attempt to answer your other questions:

Why is no error given to indicate that underlying iterator is being modified?

To throw an error without prohibiting many perfectly valid loop constructions, Python would have to know a lot about what's going on, and it would probably have to get that information at runtime. All that information would take time to process. It would make Python a lot slower, in just the place where speed really counts -- a loop.

Have the mechanics changed from earlier versions of Python with respect to this behaviour?

In short, no. Or at least I highly doubt it, and certainly it has behaved this way since I learned Python (2.4). Frankly I would expect any straightforward implementation of a mutable sequence to behave in just this way. Anyone who knows better, please correct me. (Actually, a quick doc lookup confirms that the text that Mikola cited has been in the tutorial since version 1.4!)

python, removing items from list

There’s a simple fix, just do this

l = [1,2,3]
for item in list(l): 
    l.remove(item)
    
print(l) 

The reason your method doesn’t work is that you are iterating over the list while removing items on the list. However, if you use the list method on l, you are passing a temporary copy of the list into the for loop and iterating over that. The Length will remain constant while you remove the items.

Removing item in list in Python3 as a for loop - weird behavior

You are iterating over the list [1,2,3,4,5] and modifying its length.

Writing

for i in lst:
    print(lst)
    time.sleep(1)
    lst.remove(i)

is the same as writing

lst_iter = iter(lst):
for i in lst_iter:
    print(lst)
    time.sleep(1)
    lst.remove(i)

Internally iterator holds a pointer aka index aka position of current item in the list. The loop directive calls i=next(lst_iter) internally, until StopIteration exception is raised.

In your case calling next(lst_iter) for the 1st time returns first element of list [1,2,3,4,5], which is 1. Calling next(lst_iter) for the 2nd time returns second element of list [2,3,4,5], which is 3.

Remove elements from list if appears more than a specific value in python

When you are using a for loop to iterate through a list, it is very important that you understand how the iteration changes as you modify the list.

The problem with this code is that once you are removing elements as you iterate, you are skipping elements and thus your loop will end early.

We can fix this problem by simply iterating through a copy of our list, but applying the changes to our original list:

list1 = [1,1,1,1]
x = 1
def delete(list1,x):
     for i in list1.copy(): #change is made here
        if list1.count(i) > x:
            list1.remove(i)

delete(list1,x)
print(list1)

Output:

[1]

I hope this helped! Let me know if you need any further help or clarification :)

why my for loop is skipping elements in list even if not manipulating the iterating list?

arr2=lst1 doesn't make a copy, it just creates a reference to (the same) list lst1.

Thus, by changing arr2 in arr2.remove(i), you're also altering lst1.

Use arr2 = lst1[:] (or arr2 = lst1.copy), but people tend to use the first version) to make a copy instead, that you can freely alter without harming lst1.

This mutability issue applies to both lists and dicts, but not to strings, tuples and "simple" variables such as ints, floats or bools.

---

Related Topics