且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

使用块的多处理不适用于 predict_proba

更新时间:2023-12-02 14:52:46

Pool 在内部使用 Queue,任何进入那里的东西都需要被腌制.该错误告诉您 PicklingError: Can't pickle 不能被pickle.

Pool internally uses Queue and anything that goes there needs to be pickled. The error tells you that PicklingError: Can't pickle <function OneVsRestClassifier.predict_proba cannot be pickled.

您有多种选择,其中一些在 这篇 SO 帖子中进行了描述.另一种选择是使用 joblibloky 后端.后者使用 cloudpickle 允许序列化默认 pickle 不支持的结构.

You have several options, some are described in this SO post. Another option is to use joblib with loky backend. The latter uses cloudpickle that allows for serialisation of constructs not supported by default pickle.

代码看起来或多或少是这样的:

The code will look more or less like this:

from joblib import Parallel, delayed

Parallel(n_jobs=4, backend='loky')(delayed(model.predict_proba)(dataFrame=dataFrame) for chunk in chunks)

请注意,对对象进行经典酸洗这种方法通常不是健康的想法.dill 在这里可以很好地工作.

Mind that classic pickling such methods on objects is in general not healthy idea. dill could work here well.