Eliminate Consecutive Duplicates of List Elements

Removing elements that have consecutive duplicates

>>> L = [1,1,1,1,1,1,2,3,4,4,5,1,2]
>>> from itertools import groupby
>>> [key for key, _group in groupby(L)]
[1, 2, 3, 4, 5, 1, 2]

---

For the second part

>>> [k for k, g in groupby(L) if len(list(g)) < 2]
[2, 3, 5, 1, 2]

If you don't want to create the temporary list just to take the length, you can use sum over a generator expression

>>> [k for k, g in groupby(L) if sum(1 for i in g) < 2]
[2, 3, 5, 1, 2]

Eliminate consecutive duplicates of list elements with prolog

We can solve this problem by one iteration along the list. At any point in the list we check the current element and the next element, if they are the same then we ignore the current element, else if they are different we take the current element.

rm_dup([], []).
rm_dup([X], [X]).
rm_dup([X1, X2 | Xs], [X1 | Ys]) :-
    dif(X1, X2), rm_dup([X2|Xs], Ys).
rm_dup([X, X | Xs], Ys) :-
    rm_dup([X | Xs], Ys).

The first and second clauses are base clauses in which there are no duplicate elements. The third and fourth clauses are recursive rules.

In third clause we state that if the input list has two values X1 and X2 and they are different dif(X1, X2), then keep the current value.

In fourth clause if we have same consecutive values then we ignore the current value.

The third and fourth clauses are mutually exclusive and hence to make the predicate deterministic it is better to combine them as follows

rm_dup([X], [X]) :- !.
rm_dup([X1, X2 | Xs], Ys) :-
    dif(X1, X2) -> (rm_dup([X2 | Xs], Ys1), Ys = [X1 | Ys1]);
    rm_dup([X2 | Xs], Ys).

Even better is to just use equality as a condition and flip the then and else clauses.

rm_dup([X], [X]) :- !.
rm_dup([X1, X2 | Xs], Ys) :-
    X1 = X2 ->  rm_dup([X2 | Xs], Ys);
    rm_dup([X2 | Xs], Ys1), Ys = [X1 | Ys1].

How do I remove consecutive duplicates from a list?

itertools.groupby() is your solution.

newlst = [k for k, g in itertools.groupby(lst)]

---

If you wish to group and limit the group size by the item's value, meaning 8 4's will be [4,4], and 9 3's will be [3,3,3] here are 2 options that does it:

import itertools

def special_groupby(iterable):
    last_element = 0
    count = 0
    state = False
    def key_func(x):
        nonlocal last_element
        nonlocal count
        nonlocal state
        if last_element != x or x >= count:
            last_element = x
            count = 1
            state = not state
        else:
            count += 1
        return state
    return [next(g) for k, g in itertools.groupby(iterable, key=key_func)]

special_groupby(lst)

OR

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return itertools.zip_longest(*args, fillvalue=fillvalue)

newlst = list(itertools.chain.from_iterable(next(zip(*grouper(g, k))) for k, g in itertools.groupby(lst)))

Choose whichever you deem appropriate. Both methods are for numbers > 0.

Python remove N consecutive duplicates from the list

def remove_consecutive(l, length):
    amount = len(l)
    count = 1
    start = 0
    current = l[0]
    i = 1
    while i < len(l):
        if l[i] == current:
            count += 1
        else:
            if count >= length:
                for i in range(count):
                    l.pop(start)
                start = 0
                i = 0
                current = l[0]
            else:
                start = i         
            current = l[i]
            count = 1
        i+=1
    if count >= length:
        for i in range(count):
            l.pop(start)
    return amount - len(l)

Wuff, i got it. My brain is kinda stinky lately so it took so long.

Eliminate consecutive duplicates of list elements

A simple and very readable solution:

List<string> results = new List<string>();
foreach (var element in array)
{
    if(results.Count == 0 || results.Last() != element)
        results.Add(element);
}

Eliminate consecutive duplicates of list elements ocaml

The variable smaller is an 'a list. It matches anything that doesn't match the earlier branch, i.e., a list with one element or the empty list.

AutoLISP: Removing consecutive duplicates in list

Here's an iterative method:

(defun remcondupes ( l / r )
    (while l
        (if (not (equal (car l) (cadr l) 1e-8))
            (setq r (cons (car l) r))
        )
        (setq l (cdr l))
    )
    (reverse r)
)

And here's a recursive method:

(defun remcondupes ( l )
    (if l
        (if (equal (car l) (cadr l) 1e-8)
            (remcondupes (cdr l))
            (cons (car l) (remcondupes (cdr l)))
        )
    )
)

In both of the above, the first element in the list is compared to the second using the equal function with a tolerance of 1e-8 (since we're comparing points), with the first element discarded if this test is validated.

Testing:

_$ (setq p1 '(1.2 2.3) p2 '(3.4 4.5) p3 '(5.6 6.7) p4 '(7.8 8.9))
(7.8 8.9)
_$ (setq lst (list p1 p2 p3 p3 p3 p2 p2 p4))
((1.2 2.3) (3.4 4.5) (5.6 6.7) (5.6 6.7) (5.6 6.7) (3.4 4.5) (3.4 4.5) (7.8 8.9))
_$ (remcondupes lst)
((1.2 2.3) (3.4 4.5) (5.6 6.7) (3.4 4.5) (7.8 8.9))

EDIT:

Alternatively, to account for consecutive points each successively within the comparison tolerance (per Will's comments below), you might consider the following variations:

(defun remcondupes ( l / r )
    (while l
        (if (equal (car l) (cadr l) 1e-8)
            (setq l (cons (car l) (cddr l)))
            (setq r (cons (car l) r)
                  l (cdr l)
            )
        )
    )
    (reverse r)
)

(defun remcondupes ( l )
    (if l
        (if (equal (car l) (cadr l) 1e-8)
            (remcondupes (cons (car l) (cddr l)))
            (cons (car l) (remcondupes (cdr  l)))
        )
    )
)

Removing consecutive duplicates from a list in Prolog

In your predicate, the second argument should always represent the result of duplicates being removed from the first argument. That leads to the following clauses when broken down into each case:

remove_dups([], []).   % Empty list is empty list with dups removed
remove_dups([X], [X]). % Single element list is itself with dups removed

% The result of removing duplicates from `[X,X|T]` should be the same
%   as the result of removing duplicates from `[X|T]`
remove_dups([X,X|T], [X|R]) :-
    remove_dups([X|T], [X|R]).

% The result of removing duplicates from `[X,Y|T]` where X and Y are different
%   should be the result [X|R] where R is the result of removing duplicates
%   from [Y|T]
remove_dups([X,Y|T], [X|R]) :-
    X \== Y,
    remove_dups([Y|T], R).

The 3rd and 4th clauses could be replaced with:

remove_dups([X,Y|T], [X|R]) :-
    (   X == Y
    ->  remove_dups([Y|T], [X|R])
    ;   remove_dups([Y|T], R)
    ).

But then it will limit solutions where the first argument is variable.

Remove consecutive duplicates from nested list in Python?

A solution with itertools.groupby:

from itertools import groupby

mylist = [['A', 'Car', '15'], ['A', 'Car', '15'], ['A', 'Plane', '16'], ['A', 'Bike', '20'], ['A', 'Car', '16'], ['A', 'Boat', '16']]

out = [next(g) for _, g in groupby(mylist, lambda k: k[2])]

print(out)

Prints:

[['A', 'Car', '15'], ['A', 'Plane', '16'], ['A', 'Bike', '20'], ['A', 'Car', '16']]

Benchmark (with 10_000_000 item list):

from timeit import timeit
from random import randint
from itertools import groupby

mylist = []
for i in range(10_000_000):
    mylist.append(['X', 'X', str(randint(0, 20))])

def f1():
    out = [next(g) for _, g in groupby(mylist, lambda k: k[2])]
    return out

t1 = timeit(lambda: f1(), number=1)

print(t1)

This prints on my machine (AMD 2400G, Python 3.8):

2.408908904006239

---

Related Topics