更新时间:2023-12-02 14:52:46
Pool
在内部使用 Queue,任何进入那里的东西都需要被腌制.该错误告诉您 PicklingError: Can't pickle
Pool
internally uses Queue and anything that goes there needs to be pickled. The error tells you that PicklingError: Can't pickle <function OneVsRestClassifier.predict_proba
cannot be pickled.
您有多种选择,其中一些在 这篇 SO 帖子中进行了描述.另一种选择是使用 joblib 和 loky 后端.后者使用 cloudpickle 允许序列化默认 pickle 不支持的结构.
You have several options, some are described in this SO post. Another option is to use joblib with loky backend. The latter uses cloudpickle that allows for serialisation of constructs not supported by default pickle.
代码看起来或多或少是这样的:
The code will look more or less like this:
from joblib import Parallel, delayed
Parallel(n_jobs=4, backend='loky')(delayed(model.predict_proba)(dataFrame=dataFrame) for chunk in chunks)
请注意,对对象进行经典酸洗这种方法通常不是健康的想法.dill 在这里可以很好地工作.
Mind that classic pickling such methods on objects is in general not healthy idea. dill could work here well.