Python Multiprocessing

In this python tutorial, you will learn about Python Multiprocessing and also we will check:

• What is multiprocessing?
• Multiprocessing examples in Python
• The different process running of the same python script
• Python Multiprocessing Pool Class
• Example using multiprocessing
• Python multiprocessing queue
• Python multiprocessing Lock Class
• Python multiprocessing process class

What is multiprocessing?

The multiprocessing is a process in which two or more processors in computer simultaneously process two or more different portion of the same program.

Multiprocessing example

Here, we can see an example to find the cube of a number using multiprocessing in python

• In this example, I have imported a module called multiprocessing.
• The module multiprocessing is a package that supports the swapping process using an API.
• The function is defined as a def cube(num).
• The (num * num * num) is used to find the cube of the number.
• The if __name__ == “__main__” is used to run the code directly when the file is not imported.
• The p1.start() function is used to start the process and call p1.join will terminate the process.

Example:

import multiprocessing 
def cube(num): 
	print("Cube: {}".format(num * num * num)) 
if __name__ == "__main__":  
	p1 = multiprocessing.Process(target=cube, args=(5,)) 
	p1.start() 
	p1.join() 
	print("complete") 

We can see the cube of 5 is 125 as the output. You can refer to the below screenshot for the output.

Also read, How to Print Python Fibonacci series.

The different process running of the same python script

Now, we can see how different process running of the same python script in python

• In this example, I have imported a module called multiprocessing and os.
• A function is defined as def worker1() and to get the present process ID, I have used os.getpid(). for both functions.
• The if __name__ == “__main__” is used to run the code directly when the file is not imported.
• The p1.start() ,p2.start() function is used to start the process and call p1.join() and p2.join() will terminate the process.
• The p1.pid is used to get the present process Id.
• Once the execution is finished. The .is_alive is used to check whether the process is present or not.

Example:

import multiprocessing 
import os 
def worker1(): 
	print("ID of worker1: {}".format(os.getpid())) 
def worker2(): 
	print("ID of worker2: {}".format(os.getpid())) 
if __name__ == "__main__": 
	print("ID of main process: {}".format(os.getpid())) 
	p1 = multiprocessing.Process(target=worker1) 
	p2 = multiprocessing.Process(target=worker2) 
	p1.start() 
	p2.start() 
	print("ID of process p1: {}".format(p1.pid)) 
	print("ID of process p2: {}".format(p2.pid))  
	p1.join() 
	p2.join() 
	print("processes finished execution!") 
	print("Process p1 is alive: {}".format(p1.is_alive())) 
	print("Process p2 is alive: {}".format(p2.is_alive())) 

As the execution is completed, we can see that process not alive so the false is returned as the output. You can refer to the below screenshot for the output.

Python Multiprocessing Pool Class

Now, we can see an example on multiprocessing pool class in python

• In this example, I have imported a module called pool from multiprocessing.
• The pool module is used for the parallel execution of a function across multiple input values.
• The function is defined as def num(n) then the function is returned as n*4.
• The if __name__ == “__main__” is used to run the code directly when file is not imported.
• The numbers which are to multiply with the function are specified in the list as numbers=[3,6,9].
• Here, we are using the pool to increase the performance in the execution of the program.
• The pool.map() takes the function that we want to parallelize and an iterable as an argument

Example:

from multiprocessing import Pool
def num(n):
   return n*4
if __name__=='__main__':
   numbers=[3,6,9]
   pool=Pool(processes=1)
   print(pool.map(num,numbers))

We can see the numbers are multplied with the function as the output. You can refer to the below screenshot for the output.

Example using multiprocessing

Now, we can see an example on multiprocessing in python

• In this example, I have imported a module called multiprocessing.
• The function is defined as def worker() and then the function is returned.
• The if__name__==’__main__’ is used to execute directly when the file is not imported.
• The range 6 is used to print the statement 6 times.
• An empty queue is declared then for loop is used for iteration and after iteration, the statement is appended into the queue by using task.append().
• The target is used to pass the argument. The p.start is used to start the process.

Example:

import multiprocessing
def worker():
    print('multiprocessing')
    return
if __name__ == '__main__':
    task = []
    for i in range(6):
        p = multiprocessing.Process(target=worker)
        task.append(p)
        p.start()

The multiprocessing statement is printed for 6 times as the output. You can refer to the below screenshot for the output.

Python multiprocessing queue

Now, we can see multiprocessing queue in python

• In this example, I have imported a module called process from multiprocessing.
• I have defined a function called print_func and a parameter is passed as (fruit=’Mango’).
• The if __name__ == “__main__” is used to run the code directly when the file is not imported.
• The list is created with items in it as names = [‘apple’, ‘orange’, ‘pineapple’].
• An queue is created and the items in the list are appended into the queue.
• An argument in the function is passed by using target and .start () function is used to start the process.
• The .join is used to complete the process.

Example:

from multiprocessing import Process
def print_func(fruit='Mango'):
    print('The fruits is : ', fruit)
if __name__ == "__main__":  
    names = ['apple', 'orange', 'pineapple']
    procs = []
    proc = Process(target=print_func)
    procs.append(proc)
    proc.start()
    for name in names:
        proc = Process(target=print_func, args=(name,))
        procs.append(proc)
        proc.start()
    for proc in procs:
        proc.join()

The items in the list are appended in to the queue as the output. You can refer to the below screenshot for the output.

Python multiprocessing Queue class

Here, we can see multiprocessing Queue class in python.

• In this example, I have imported a module called Queue from multiprocessing.
• The list is defined and it contains items in it. To assign the index to the items to the queue, I have used index = 0.
• The queue is a data structure used to store the items from the list.
• The for loop is used for iteration of the put() is used to block when the max size of the queue is filled.
• The index number for the item is assigned as index += 1.
• The items from the queue are popped. the while condition is execeuted and the items are again pushed by using a queue.get() from the index value +=1.

Example:

from multiprocessing import Queue
fruits = ['Apple', 'Mango', 'orange']
index = 0
queue = Queue()
print('pushing fruits to queue:')
for fruit in fruits:
    print('fruit no: ', index, ' ', fruit)
    queue.put(fruit)
    index += 1
print('\npopping fruits from queue:')
index = 1
while not queue.empty():
    print('fruit no: ', index, ' ', queue.get())
    index += 1

We can see pushing and poping of an item into the queue as the output. You can referto the below screenshort for the output.

Python multiprocessing Lock Class

Now, we can see multiprocessing Lock Class in python

• In this example, I have imported a module called Lock, Process, Queue, current_process from multiprocessing.
• The function job is defined and the parameter(tasks_to_accomplish, tasks_that_are_completed) is passed.
• The while condition is used the try block is used for an exception.
• The .get_nowait() returns an item if an item is immediately available, the Exception queue.Empty is used for further execution of the program.
• Else. Put an item into the queue. If the queue is full, wait until a free slot is available before adding the item.
• The time.sleep is used to add a delay to the execution of the program.
• The function is defined as def main().
• The number of tasks and number of processes is assigned as 5 and3.
• The Queue() returns true if the queue is empty and false if the queue is not empty.
• The for loop is used for iteration.
• The p = Process(target=job, args=(tasks_to_accomplish, tasks_that_are_completed)) is used to create the process.
• The p.join is used to complete the processes.
• To get the output the (while not tasks_that_are_completed.empty()) is used.
• The print(tasks_that_are_completed.get()) is used to get the output.
• The .empty() returns true if the queue is empty.
• The .get() is used to remove and return the elements from the queue.

Example:

from multiprocessing import Lock, Process, Queue, current_process
import time
import queue
def job(tasks_to_accomplish, tasks_that_are_completed):
    while True:
        try:
            task = tasks_to_accomplish.get_nowait()
        except queue.Empty:
            break
        else:
            print(task)
            tasks_that_are_completed.put(task + ' is done by ' + current_process().name)
            time.sleep(.5)
    return True
def main():
    number_of_task = 5
    number_of_processes = 3
    tasks_to_accomplish = Queue()
    tasks_that_are_completed = Queue()
    processes = []
    for i in range(number_of_task):
        tasks_to_accomplish.put("Task no " + str(i))
    for w in range(number_of_processes):
        p = Process(target=job, args=(tasks_to_accomplish, tasks_that_are_completed))
        processes.append(p)
        p.start()
    for p in processes:
        p.join()
    while not tasks_that_are_completed.empty():
        print(tasks_that_are_completed.get())
    return True
if __name__ == '__main__':
    main()

We can see the number of tasks done by the which processor as the output. You can refer to the below screenshot for the output.

Python multiprocessing process class

Here, we can see multiprocessing process class in python

• In this example, I have imported a module called Process from multiprocessing.
• I have defined a function called fun and passed a parameter as fruit=’custarsapple’.
• The if __name__ == “__main__” is used to execute directly when file is not imported.
• To fill the queue the procs.append is used.
• The start() function is used to start processing then the process will run and return the result.
• The args is the keyword used to pass the arguments through the process.
• The p.join() is used to complete the process.

Example:

from multiprocessing import Process
def fun(fruit='custardapple'):
    print('The name of the fruit is : ', fruit)
if __name__ == "__main__":  
    names = ['Apple', 'Mango', 'Orange']
    procs = []
    proc = Process(target=fun)
    procs.append(proc)
    proc.start()
    for name in names:
        proc = Process(target=fun, args=(name,))
        procs.append(proc)
        proc.start()
    for proc in procs:
        proc.join()

The below screenshot shows the output.

You may like the following Python tutorials:

• Os change directory Python
• Python program to reverse a string with examples
• Missing Data in Pandas in Python
• How to concatenate strings in python
• Python Concatenate Dictionary + Examples
• Python Tkinter to Display Data in Textboxes

In this Python tutorial, we have learned about Python Multiprocessing. Also, we covered these below topics:

• What is multiprocessing?
• Multiprocessing example
• The different process running of the same python script
• Python Multiprocessing Pool Class
• Example using multiprocessing
• Python multiprocessing queue
• Python multiprocessing Lock Class
• Python multiprocessing process class