且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

导入scipy破坏了Python中的多处理支持

更新时间:2023-09-18 12:08:58

经过大量挖掘并发布问题在Scipy GitHub网站上,我找到了一个解决方案.

After much digging around and posting an issue on the Scipy GitHub site, I've found a solution.

开始之前,此处 -我只作一个概述.

Before I start, this is documented very well here - I'll just give an overview.

此问题与我使用的Scipy或Numpy版本无关.它起源于Numpy和Scipy用于各种线性代数例程的系统BLAS库.您可以通过运行来确定Numpy链接到哪些库

This problem is not related to the version of Scipy, or Numpy that I was using. It originates in the system BLAS libraries that Numpy and Scipy use for various linear algebra routines. You can tell which libraries Numpy is linked to by running

python -c 'import numpy; numpy.show_config()'

如果在Linux中使用OpenBLAS,则可能会发现CPU关联性设置为1,这意味着一旦这些算法(通过Numpy/Scipy)以Python形式导入,您最多可以访问CPU的一个核心.要对此进行测试,请在Python终端中运行

If you are using OpenBLAS in Linux, you may find that the CPU affinity is set to 1, meaning that once these algorithms are imported in Python (via Numpy/Scipy), you can access at most one core of the CPU. To test this, within a Python terminal run

import os
os.system('taskset -p %s' %os.getpid())

如果以fff的形式返回CPU亲缘关系,则可以访问多个内核.就我而言,它是这样开始的,但是在导入numpy或scipy.any_module时,它将切换到1,因此是我的问题.

If the CPU affinity is returned as f, of ff, you can access multiple cores. In my case it would start like that, but upon importing numpy or scipy.any_module, it would switch to 1, hence my problem.

我找到了两种解决方案:

I've found two solutions:

更改CPU关联性

您可以在主要功能的顶部手动设置主进程的CPU关联性,以便代码如下所示:

You can manually set the CPU affinity of the master process at the top of the main function so that the code looks like this:

import multiprocessing
import numpy as np
import math
import time
import os

def compute_something(t):
    a = 0.
    for i in range(10000000):
        a = math.sqrt(t)
    return a

if __name__ == '__main__':

    pool_size = multiprocessing.cpu_count()
    os.system('taskset -cp 0-%d %s' % (pool_size, os.getpid()))

    print "Pool size:", pool_size
    pool = multiprocessing.Pool(processes=pool_size)

    inputs = range(10)

    tic = time.time()
    builtin_outputs = map(compute_something, inputs)
    print 'Built-in:', time.time() - tic

    tic = time.time()
    pool_outputs = pool.map(compute_something, inputs)
    print 'Pool    :', time.time() - tic

请注意,选择比taskset的内核数高的值似乎无关紧要-它仅使用最大可能数.

Note that selecting a value higher than the number of cores for taskset doesn't seem to matter - it just uses the maximum possible number.

切换BLAS库

解决方案在上面链接的网站中记录.基本上:安装libatlas并运行update-alternatives将numpy指向ATLAS,而不是OpenBLAS.

Solution documented at the site linked above. Basically: install libatlas and run update-alternatives to point numpy to ATLAS rather than OpenBLAS.