Today I had the requirement to achieve a task by using parallel processing in order to save time.
The task to be achieved
For this demonstration, I have a list of people and each task needs to lookup its pet name and return to stdout. I want to spawn a task for each persons pet name lookup and run the tasks in parallel so that all the results can be returned back at once, instead of sequential.
This is a basic task, but you could have a CPU intensive job, where it will shine better.
When using multiple processes, one generally uses message passing for communication between processes and avoids having to use any synchronization primitives like locks.
The Queue type is a multi producer, multi consumer FIFO queues modelled on the queue.Queue class in the standard library. You can read more up on it here
Our multiprocessing workflow will look like this:
- We will define our data, which will be a dictionary of people and their pet names
- We will define an output queue
- Create a example function that will produce each task to the queue
- Then we will setup a lost of processes that we want to run
- From the list of processes that we defined, we will run each process, then wait and exit the completed processes
- We will then consume from the queue. For each process in our processes list
Note that I also added a delay of 2 seconds, so that you can see that the tasks are run in parallel, so the delay will only be 2 seconds.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Running the example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.