且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

确保混合MPI/OpenMP在不同的内核上运行每个OpenMP线程

更新时间:2022-12-26 23:24:05

实际上,我希望您的第一个示例能够奏效.在此处设置OMP_PROC_BIND=true很重要,这样,在钉住线程的线程时,OpenMP会保留在MPI进程与CPU的绑定中.

Actually, I'd expect your first example to work. Setting the OMP_PROC_BIND=true here is important, so that OpenMP stays within the CPU binding from the MPI process when pinning it's threads.

取决于批处理系统和MPI实施,可能会有非常单独的方法来设置这些内容.

Depending on the batch system and MPI implementation, there might be very individual ways to set these things up.

在Linux中,所有超线程或通常每个内核有多个硬件线程都显示为内核"可能是问题的一部分,因为当两个进程在两个超线程上运行时,您将永远看不到200%一个核心.

Also Hyperthreading, or in general multiple hardware threads per core, that all show up as "cores" in your Linux, might be part of the problem as you'll never see 200% when two processes run on the two Hyperthreads of one cores.

这是一个通用的解决方案,我在为某些MPI和某些系统上的某些OpenMP实现确定这些东西时会用到. Cray提供了一个文档,其中包含一个非常有用的程序,可以快速找出这些问题,称为xthi.c此处(不确定将其粘贴在此处是否合法...).编译:

Here is a generic solution, I use when figuring these things for some MPI and some OpenMP implementation on some system. There's documentation from Cray which contains a very helpful program to figure these things out quickly, it's called xthi.c, google the filename or paste it from here (not sure if it's legal to paste it here...). Compile with:

mpicc xthi.c -fopenmp -o xthi

现在我们可以看到发生了什么,例如,在具有超线程和Intel MPI(基于MPICH)的2x 8 Core Xeon上,我们得到了:

Now we can see what exactly is going on, for instance on a 2x 8 Core Xeon with Hyperthreading and Intel MPI (MPICH-based) we get:

$ OMP_PROC_BIND=true OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi

Hello from rank 0, thread 0, on localhost. (core affinity = 0,16)
Hello from rank 0, thread 1, on localhost. (core affinity = 1,17)
Hello from rank 1, thread 0, on localhost. (core affinity = 8,24)
Hello from rank 1, thread 1, on localhost. (core affinity = 9,25)

如您所见,核心是指核心的所有超线程.请注意,默认情况下,mpirun也是如何将其固定在不同的插座上的.而使用OMP_PLACES=threads,每个内核可以获得一个线程:

As you can see, core means, all the Hyperthreads of a core. Note how mpirun pins it different sockets, too by default. And With OMP_PLACES=threads you get one thread per core:

$ OMP_PROC_BIND=true OMP_PLACES=threads OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi
Hello from rank 0, thread 0, on localhost. (core affinity = 0)
Hello from rank 0, thread 1, on localhost. (core affinity = 1)
Hello from rank 1, thread 0, on localhost. (core affinity = 8)
Hello from rank 1, thread 1, on localhost. (core affinity = 9)

使用OMP_PROC_BIND=false(第二个示例),我得到:

With OMP_PROC_BIND=false (your second example), I get:

$ OMP_PROC_BIND=false OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi
Hello from rank 0, thread 0, on localhost. (core affinity = 0-7,16-23)
Hello from rank 0, thread 1, on localhost. (core affinity = 0-7,16-23)
Hello from rank 1, thread 0, on localhost. (core affinity = 8-15,24-31)
Hello from rank 1, thread 1, on localhost. (core affinity = 8-15,24-31)

在这里,每个OpenMP线程都有一个完整的套接字,因此MPI等级仍在不同的资源上运行.但是,操作系统可以在所有内核中疯狂地调度一个进程中的OpenMP线程.就像在测试系统上设置OMP_NUM_THREADS=2一样.

Here, each OpenMP thread gets a full socket, so the MPI ranks still operate on distinct resources. However, the OpenMP threads, within one process could be scheduled wildly by the OS across all cores. It's the same as just setting OMP_NUM_THREADS=2 on my test system.

同样,这可能取决于特定的OpenMP和MPI实现和版本,但是我认为您可以很容易理解上面的描述.

Again, this might depend on specific OpenMP and MPI implementations and versions, but I think you'll easily figure out what's going on with the description above.

希望有帮助.