Finally, we start a process pool, and log a message in one of the child subprocesses.Another thread writes a continuous stream of log messages.In the parent process, log messages are routed to a queue, and a thread reads from the queue and writes those messages to a log file.While True : with Pool () as pool : pool. We do this in a loop to make race condition more error ( "hello, I did something" ) print ( ".logged" ) if _name_ = '_main_' : setup_logging () # Meanwhile, we start a process pool that writes some start () def runs_in_subprocess (): print ( "About to log." ) logging. error ( "hello, I just did something" ) Thread ( target = write_logs ). addHandler ( QueueHandler ( _log_queue )) # Our parent process is running a thread thatĭef write_logs (): while True : logging. _log_queue = Queue () QueueListener ( _log_queue, logging. # from that queue and writes messages to a file: Import logging from threading import Thread from queue import Queue from logging.handlers import QueueListener, QueueHandler from multiprocessing import Pool def setup_logging (): # Logs get written to a queue, and then a thread reads (Note that none of these examples were tested on Windows I’m focusing on the *nix platform here.) Python provides a handy module that allows you to run tasks in a pool of processes, a great way to improve the parallelism of your program. Let’s begin! Introducing multiprocessing.Pool The solution that will keep your code from being eaten by sharks.Some bandaids that won’t stop the bleeding.A conundrum wherein fork() copying everything is a problem, and fork() not copying everything is also a problem.A mysterious failure wherein Python’s multiprocessing.Pool deadlocks, mysteriously.In this journey through space and time you will encounter: (The sharks are a metaphor for processes.)īlood starts seeping out, the sharks start circling, and pretty soon you find yourself-dead(locked) in the water! Let’s set the metaphorical scene: you’re swimming in a pool full of sharks. In many cases you can fix this with a single line of code-skip to the end to try it out-but first, it’s time for a deep-dive into Python brokenness and the pain that is POSIX system programming, using exciting and not very convincing shark-themed metaphors! You check CPU usage-nothing happening, it’s not doing any work. You’re using multiprocessing to run some code across multiple processes, and it just-sits there.
0 kommentar(er)
