Notes on Python multiprocessing

March 22, 2018 2 minutes read

Notes • Python

multiprocessing • ParallelProgramming • python

multiprocessing is a package that supports spawning processes using an API similar to the threading module.[1]
The Pool object which offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism).[1]
One should protect the “entry point” of the program by using if __name__ == '__main__' this line is mandatory.

One of the simplest ways of show the use of multi processing is,

import multiprocessing as mp

def fx(x):
    print("hello for "+str(x)+"th time!")

if __name__=="__main__":
    with mp.Pool(10) as p:
        p.map(fx,[1,2,3,4,5,6,7,8,9])

It doesn’t look much of multiprocessed, right?. After adding a bit of complexity as follows,

import multiprocessing as mp

def fx(x):
    print("hello for "+str(x)+"th time!")

if __name__=="__main__":
    numbers = list()
    for n in range(0,100):
       numbers.append(n)

    with mp.Pool(10) as p:
        p.map(fx,numbers)

You will see that the processes execute in non ordered way!,

...
hello for 15th time!
hello for 9th time!
hello for 6th time!
hello for 10th time!
hello for 11th time!
hello for 7th time!
hello for 8th time!
hello for 13th time!
hello for 24th time!
...

Neat hah?

By the time of this writting multiprocessing doesn’t allow to use multiple-argument functions in its implementation of multiprocessing.Pool.map(). But, it is possible to iterate a multiple argument case by using functors(a C++ concept)[3] as follows,

def Multi(x,y,z):
    return z*(x*z + y)
class Functor(object):
    def __init__(self,y,z):
        self.y = y
        self.z = z
    def __call__(self,x):
        return Multi(x,self.y,self.z)
Instance = Functor(2,3) # instance of the functor
Instance(0) # Should return 6
Instance(1) # Should return 15
import multiprocessing as mp
p = mp.Pool() # instance of pool
p.map(Instance, [1, 2, 3, 4, 5]) # returns [15, 24, 33, 42, 51]

Observations

On the use of `with ... as ...` statement.

Consider the following class in Python,

    class sample
        def __enter__(self):
		# some code ..
        def __exit__(self, type, value, traceback):
		# some code ..

The keyword with sample as ... is used to access the sample’s method __enter__(self) and executes it and assings whatever it returns to the variable after the as. Once the code following the with statement is executed and no matter what happens in there, the with statement calls the __exit__(self) method of the object [2]. The __exit__(self, type, value, traceback) also can handle exceptions for wich you have to define the behaviour once a particular exception is raised.
For example,

with open("x.txt") as f:
    data = f.read()
    # do something with data

Carefully closes the file descriptor f is something bad happens within the scope of the with statement.

Observations

On the use of with ... as ... statement.

Resources

On the use of `with ... as ...` statement.