How to Iterate Over a List in Chunks

How to iterate over a list in chunks

Modified from the Recipes section of Python's itertools docs:

from itertools import zip_longest

def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

Example

grouper('ABCDEFG', 3, 'x')  # --> 'ABC' 'DEF' 'Gxx'

Note: on Python 2 use izip_longest instead of zip_longest.

How to loop through a python list in batch?

You are close.

chunks = (total_count - 1) // 50 + 1
for i in range(chunks):
     batch = data[i*50:(i+1)*50]

Looping through a list in chunks

Here's a possible quick'n'dirty solution:

nSym <- nrow(symbols)
chunkSize <- 200
for(i in 1:ceiling(nSym / chunkSize)){  
  shortsymb<-symbols[((i-1)*chunkSize+1):min(nSym,(i*chunkSize)),]
  # do what you need with shortsymb
}

Description of the code:

• compute the number of chunks by simply dividing: nSym / chunkSize (we take the ceiling since there can be a remainder, if nSym is not multiple of chunkSize)
• for each chunk (1,2,...,n) we compute the corresponding start and end row indexes, as start = ((i-1)*chunkSize+1) and end = min(nSym,(i*chunkSize)) (min function is necessary because last chunk might be smaller than chunkSize)
• we use the indexes to subset the original data.frame

Iterate an iterator by chunks (of n) in Python?

The grouper() recipe from the itertools documentation's recipes comes close to what you want:

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

It will fill up the last chunk with a fill value, though.

A less general solution that only works on sequences but does handle the last chunk as desired is

[my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)]

Finally, a solution that works on general iterators and behaves as desired is

def grouper(n, iterable):
    it = iter(iterable)
    while True:
        chunk = tuple(itertools.islice(it, n))
        if not chunk:
            return
        yield chunk

How to create a loop in vba iterating over a list in chunks?

Option 1 - using a jagged array

CreateSubLists() returns a jagged array Array(Array(), Array(), ...) from the sliced elements of the original array.

Option Explicit

Function CreateSubLists(itemlist, count)
    Dim amount, chunks, i, j, retval, a
    amount = UBound(itemlist) - LBound(itemlist) + 1        'get the amount of the elements
    chunks = WorksheetFunction.RoundUp(amount / count, 0)   'calculate the number of chunks (e.g. if 2.5 then 3)
    
    ReDim retval(0 To chunks - 1)   'make the outer array
    For i = 0 To UBound(retval)
        'make the inner array. If the rest of the itemlist is less then chunk then get the size of the rest
        ReDim a(0 To WorksheetFunction.Min(count - 1, amount - i * count - 1))
        For j = 0 To UBound(a)
            a(j) = itemlist(i * count + j)  'fill the inner array
        Next
        retval(i) = a   'place the inner array into the outer array
    Next
    CreateSubLists = retval 'return the jagged array
End Function

Sub Example()
    Dim itemlist, chunk
    itemlist = Array("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11")
    For Each chunk In CreateSubLists(itemlist, 4)  ' works with any positive integer number
        Debug.Print "[" & Join(chunk, ",") & "]"
    Next
End Sub

Option 2 - using Collection

Function CreateSubLists2(itemlist, count) As Collection
    Dim amount, chunks, i, j, a
    amount = UBound(itemlist) - LBound(itemlist) + 1        'get the amount of the elements
    chunks = WorksheetFunction.RoundUp(amount / count, 0)   'calculate the number of chunks (e.g. if 2.5 then 3)
    
    Dim retval As New Collection   'make the outer collection
    For i = 0 To chunks - 1
        'make the inner array. If the rest of the itemlist is less then chunk then get the size of the rest
        Set a = New Collection
        For j = 0 To WorksheetFunction.Min(count - 1, amount - i * count - 1)
            a.Add itemlist(i * count + j)  'fill the inner collection
        Next
        retval.Add a   'place the inner collection into the outer collection
    Next
    Set CreateSubLists2 = retval 'return the collection of collections
End Function

Sub Example2()
    Dim itemlist, chunk, el, s
    itemlist = Array("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11")
    For Each chunk In CreateSubLists2(itemlist, 4)  ' works with any positive integer number
        s = "["
        For Each el In chunk
            s = s & el & ","
        Next
        s = s & "]"
        Debug.Print Replace(s, ",]", "]")
    Next
End Sub

Prints:

[1,2,3,4]
[5,6,7,8]
[9,10,11]

How to run a function in loop over the chunks of a list?

Loop over the chunks, use a list comprehension to multiple all the chunk elements, and use extend to append the result to the new list.

def function(new_list, list_of_lists):
    for l in list_of_lists:
        new_list.extend([2 * x for x in l])

function(new_list, list_with_chunks)
print(new_list)

How do I split a list into equally-sized chunks?

Here's a generator that yields evenly-sized chunks:

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

import pprint
pprint.pprint(list(chunks(range(10, 75), 10)))
[[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
 [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
 [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
 [50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
 [60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
 [70, 71, 72, 73, 74]]

For Python 2, using xrange instead of range:

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in xrange(0, len(lst), n):
        yield lst[i:i + n]

---

Below is a list comprehension one-liner. The method above is preferable, though, since using named functions makes code easier to understand. For Python 3:

[lst[i:i + n] for i in range(0, len(lst), n)]

For Python 2:

[lst[i:i + n] for i in xrange(0, len(lst), n)]

---

Related Topics