且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何在运行脚本时将更多项目添加到多处理队列中

更新时间:2023-02-01 21:22:31

让我们弄清楚:

  • 目标函数worker(q)在上述方案中仅被调用一次.在第一次调用时,该函数将暂停等待阻塞操作q.get()的结果.它从queue获取实例MyFancyClass('Fancy Dan'),调用其do_something方法并完成操作.
  • MyFancyClass('Frankie')将被放入队列中,但不会进入Process,因为该进程的目标功能已完成.
  • 其中一种方法是从队列中读取并等待一个信号/标记的项目,该信号/信号表明已停止使用队列.假设None值.
  • the target function worker(q) will be called just once in the above scheme. At that first call the function will suspend waiting the result from blocking operation q.get(). It gets the instance MyFancyClass('Fancy Dan') from the queue, invokes its do_something method and get finished.
  • MyFancyClass('Frankie') will be put into the queue but won't go to the Process cause the process' target function is done.
  • one of the ways is to read from the queue and wait for a signal/marked item which signals that queue usage is stopped. Let's say None value.
import multiprocessing


class MyFancyClass:

    def __init__(self, name):
        self.name = name

    def do_something(self):
        proc_name = multiprocessing.current_process().name
        print('Doing something fancy in {} for {}!'.format(proc_name, self.name))


def worker(q):
    while True:
        obj = q.get()
        if obj is None:
            break
        obj.do_something()


if __name__ == '__main__':
    queue = multiprocessing.Queue()

    p = multiprocessing.Process(target=worker, args=(queue,))
    p.start()

    queue.put(MyFancyClass('Fancy Dan'))
    queue.put(MyFancyClass('Frankie'))
    # print(queue.qsize())
    queue.put(None)

    # Wait for the worker to finish
    queue.close()
    queue.join_thread()
    p.join()

输出:

Doing something fancy in Process-1 for Fancy Dan!
Doing something fancy in Process-1 for Frankie!