Ruan Bekker's Blog

From a Curious mind to Posts on Github

Sharing Global Variables in Python Using Multiprocessing

While I was using multiprocessing, I found out that global variables are not shared between processes.

ruanbekker-cheatsheets

Example of the Issue

Let me first provide an example of the issue that I was facing.

I have 2 input lists, which 2 processes wil read from and append them to the final list and print the aggregated list to stdout

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import multiprocessing
final_list = []

input_list_one = ['one', 'two', 'three', 'four', 'five']
input_list_two = ['six', 'seven', 'eight', 'nine', 'ten']

def worker(data):
    for item in data:
        final_list.append(item)

process1 = multiprocessing.Process(target=worker, args=[final_list_one])
process2 = multiprocessing.Process(target=worker, args=[final_list_two])

process1.start()
process2.start()
process1.join()
process2.join()

print(final_list)

When running the example:

1
2

$ python3 mp_list_issue.py
[]

As you can see the response from the list is still empty.

Resolution

We need to use multiprocessing.Manager.List.

From Python’s Documentation:

“The multiprocessing.Manager returns a started SyncManager object which can be used for sharing objects between processes. The returned manager object corresponds to a spawned child process and has methods which will create shared objects and return corresponding proxies.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import multiprocessing
manager = multiprocessing.Manager()
final_list = manager.list()

input_list_one = ['one', 'two', 'three', 'four', 'five']
input_list_two = ['six', 'seven', 'eight', 'nine', 'ten']

def worker(data):
    for item in data:
        final_list.append(item)

process1 = multiprocessing.Process(target=worker, args=[final_list_one])
process2 = multiprocessing.Process(target=worker, args=[final_list_two])

process1.start()
process2.start()
process1.join()
process2.join()

print(final_list)

Now when we run our script, we can see that our processes are aware of our defined list:

1
2

$ python3 mp_list.py
['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten']

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.