How to Use Filter, Map, and Reduce in Python 3

How to use filter, map, and reduce in Python 3

You can read about the changes in What's New In Python 3.0. You should read it thoroughly when you move from 2.x to 3.x since a lot has been changed.

The whole answer here are quotes from the documentation.

Views And Iterators Instead Of Lists

Some well-known APIs no longer return lists:

• [...]
• map() and filter() return iterators. If you really need a list, a quick fix is e.g. list(map(...)), but a better fix is often to use a list comprehension (especially when the original code uses lambda), or rewriting the code so it doesn’t need a list at all. Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).
• [...]

Builtins

• [...]
• Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
• [...]

Efficient filter, map, and reduce operation on a long list in Python

Answer

The Numpy is fast in the calculation but it is inefficient in converting the string list to an int list

Tips:

1. Use the Numpay array with NumPy functions only, using the Numpay data structure with pythonic functins is inefficient and vice-versa
2. Counter in some cases (when doing the conversion inside the counter) is the fastest among the Pythonic methods
3. To get the benefit of Numpy efficiency do the list conversion with python.

To better compare the results, the running time can be divided into three categories for this problem:

1. Time for reading from file (same for all tries)
2. Time to convert the string list into an int list (different for python list and numpy array)
3. Time to calculation (varies for different methods)

This separation is required to clearly show the efficiency of python list vs NumPy array.
Having a look to the code:

import time
import numpy as np
from functools import reduce
from collections import Counter

def print_result(total_price, duration, method):
    print('{method:<35s} {0:>20.3f} (sec), total_price={1:<10.3f}'
          .format(duration, total_price, method=method))

start = time.time()
with open('price_list.txt') as f:
    price_list = f.read().split('\n')
print('Duration for reading from file: {0:<50.3f}'.format(time.time() - start))

start = time.time()
np_price_array = np.array(price_list).astype(int)  # uing numpy array
print('Duration for converting to numpy int array: {0:<50.3f}'.format(
    time.time() - start))

start = time.time()
price_list = list(map(int, price_list))
print('Duration for converting to python int list: {0:<50.3f}'.format(
    time.time() - start))

start = time.time()    
total_price = np.sum(np_price_array[np_price_array < 30] * 1.05)
print_result(total_price, time.time() - start, "NumPy function (numpy array)")

start = time.time()
total_price = sum([x for x in np_price_array if x < 30])*1.05
print_result(total_price, time.time() - start,
             "List comprehension (numpy array)")

start = time.time()
total_price = sum((x for x in price_list if x < 30))*1.05
print_result(total_price, time.time() - start, "List generator (Python list)")

start = time.time()
total_price = sum([x for x in price_list if x < 30])*1.05
print_result(total_price, time.time() - start,
             "List comprehension (Python list)")

start = time.time()
total_price = sum([float(x[0]) * float(x[1])
                  for x in Counter(price_list).items() if x[0] < 30]) * 1.05
print_result(total_price, time.time() - start, "Counter (Python list)")

start = time.time()
total_price = reduce(
    lambda x, y: x+y, filter(lambda x: x < 30, price_list)) * 1.05
print_result(total_price, time.time() - start, "functions (Python list)")

Following chart is as the result of the code output:

Passing functions without () into map/filter/reduce

This is confusing at first, especially when you come from other programming languages. In Python the functions are objects and you can pass them along without invoking them. Check below example:

def func():  # function definition
    print("func was invoked")

a = func  # assign func object (without invoking it)
print(a, type(a))

a()  # invoking the function

map will use this function object to invoke the function for each item in the iterable.

how to write map, filter and spread in python3?

Okay, lemme just state that python syntax is different from JS. That said the functionality is quite the same barring the data structures used in them.

Converting, mapping and filtering:

---

Steps to do the above task, as best i can comprehend.

1. Convert JSON to a python iterable or sequence. some info here

json_data_dict = json.loads(data.data.result)

2. Filter the data. filter(filter_func, seq) is an inbuilt function that lets filter a sequence.

filtered_iter = filter(func_name, json_data_dict)

3. Map the data. writes similar to filter map(map_func, seq)

mapped_iter = map(func_name, iterable)

if you are looking for chaining, functools may have some help for you. Otherwise you can compose them

filtered_then_mapped = map(map_func, filter(func_name, iterable));

more info here

Spread operator:

---

There is no special spread operator in python, you can just destructure lists and tuples in python like

x, y = (10, 20);
# or
my_tupl = (10, 20);
x, y = my_tuple;

the rest operator may be used like such

start, num_2, *end_nums = [0, 1, 2, 3, 4, 5]
print(start) # 0
print(num_2) # 1
print(end_nums) # [2, 3, 4, 5]

objects will be destructured using itemgetter and attrgetter from operator module

Clean code for sequence of map/filter/reduce functions

PyFunctional lets you do exactly that after installing via pip install PyFunctional

from functional import seq

seq(some_list).map(lambda x: x -1, a).filter(lambda x: x>0).reduce(lambda x, y: x*y)

The package can do much more than that, which you can checkout at
github.com/EntilZha/PyFunctional

What are Python's equivalent of Javascript's reduce(), map(), and filter()?

They are all similar, Lamdba functions are often passed as a parameter to these functions in python.

Reduce:

 >>> from functools import reduce
 >>> reduce( (lambda x, y: x + y), [1, 2, 3, 4]
 10

Filter:

>>> list( filter((lambda x: x < 0), range(-10,5)))
[-10, -9, -8, -7, - 6, -5, -4, -3, -2, -1]

Map:

>>> list(map((lambda x: x **2), [1,2,3,4]))
[1,4,9,16]

Docs

Python difference between filter() and map()

They both work a little bit differently but you've got the right idea.

Map takes all objects in a list and allows you to apply a function to it
Filter takes all objects in a list and runs that through a function to create a new list with all objects that return True in that function.

Here's an example

def square(num):
    return num * num

nums = [1, 2, 3, 4, 5]
mapped = map(square, nums)

print(*nums)
print(*mapped)

The output of this is

1 2 3 4 5
1 4 9 16 25

Here's an example of filter

def is_even(num):
    return num % 2 == 0

nums = [2, 4, 6, 7, 8]
filtered = filter(is_even, nums)

print(*nums)
print(*filtered)

The output of this would be

2 4 6 7 8
2 4 6 8

Does filter,map, and reduce in Python create a new copy of list?

list_of_nums[:] = filter(lambda x: x != 2, list_of_nums)

and

list_of_nums = filter(lambda x: x != 2, list_of_nums)

are two different operations that end up with mostly the same result.

In both cases,

filter(lambda x: x != 2, list_of_nums)

returns either a new list containing items that match the filter (in Python 2), or an iterable over list_of_nums that returns the same items (in Python 3).

The first case,

list_of_nums[:] = filter(lambda x: x != 2, list_of_nums)

then deletes all items from list_of_nums and replaces them with the items from the new list or iterable.

The second case,

list_of_nums = filter(lambda x: x != 2, list_of_nums)

assigns the new list to the variable list_of_nums.

The time when this makes a difference is this:

def processItemsNotTwo_case1(list_of_nums):
    list_of_nums[:] = filter(lambda x: x != 2, list_of_nums)
    # do stuff here
    # return something

def processItemsNotTwo_case2(list_of_nums):
    list_of_nums = filter(lambda x: x != 2, list_of_nums)
    # do stuff here
    # return something

list1 = [1,2,2,3,4,5]
processItemsNotTwo_case1(list1)
list2 = [1,2,2,3,4,5]
processItemsNotTwo_case2(list2)

With this code, list1 ends up with the new contents [1,3,4,5], whereas list2 ends up with the original contents [1,2,2,3,4,5].

---

Related Topics