且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

'multiprocessing.resource_sharer'中的AttributeError'DupFd'| Python多处理+线程

更新时间:2022-06-05 22:44:53

您正在此处分叉已经是多线程的主进程.一般而言,这是有问题的.

You are forking an already multi-threaded main-process here. That is known to be problematic in general.

这实际上很容易出现问题(而不仅仅是在Python中).规则是分叉之后而不是之前的线程".否则,线程执行器使用的锁将在各个进程之间重复.如果其中一个进程在拥有该锁的同时死亡,则使用该锁的所有其他进程将死锁- Raymond Hettinger .

引发错误的原因显然是在子进程中管道的文件描述符复制失败.

Trigger for the error you get is apparantly that the duplication of the file-descriptor for the pipe fails in the child process.

要解决此问题,请在主进程仍为单线程的情况下创建您的子进程,或者使用另一个start_method来创建新的进程,例如"spawn"(在Windows上为默认值)或"forkserver",如果有的话.

To resolve this issue, either create your child-processes as long as your main-process is still single-threaded or use another start_method for creating new processes like 'spawn' (default on Windows) or 'forkserver', if available.

forkserver

forkserver

当程序启动并选择forkserver start方法时,服务器进程将启动.从那时起,每当需要一个新进程时,父进程就连接到服务器并请求它派生一个新进程. fork服务器进程是单线程的,因此使用os.fork()是安全的.没有多余的资源被继承.

When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.

在Unix平台上可用,该平台支持通过Unix管道传递文件描述符. 文档

Available on Unix platforms which support passing file descriptors over Unix pipes. docs

您可以使用以下方法指定另一种start_method:

You can specify another start_method with:

multiprocessing.set_start_method(方法) 设置用于启动子进程的方法.方法可以是"fork","spawn"或"forkserver".

multiprocessing.set_start_method(method) Set the method which should be used to start child processes. method can be 'fork', 'spawn' or 'forkserver'.

请注意,此方法最多应调用一次,并且应在主模块的if name ==' main '子句中进行保护. 文档

Note that this should be called at most once, and it should be protected inside the if name == 'main' clause of the main module. docs

有关特定start_method(在Ubuntu 18.04上)的基准,请参见此处.

For a benchmark of the specific start_methods (on Ubuntu 18.04) look here.