且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

pandas 用 NaN 描述 vs scipy.stats percentileofscore?

更新时间:2021-10-26 22:44:36

scipy.stats.percentileofscore 不会忽略 nan,也不会检查值和句柄它以某种特殊的方式.它只是数据中的另一个浮点值.这意味着 percentileofscore 与包含 nan 的数据的行为是未定义的,因为 nan 在比较中的行为:

scipy.stats.percentileofscore does not ignore nan, nor does it check for the value and handle it in some special way. It is just another floating point value in your data. This means the behavior of percentileofscore with data containing nan is undefined, because of the behavior of nan in comparisons:

In [44]: np.nan > 0
Out[44]: False

In [45]: np.nan < 0
Out[45]: False

In [46]: np.nan == 0
Out[46]: False

In [47]: np.nan == np.nan
Out[47]: False

那些结果都是正确的——这就是 nan 应该表现的方式.但这意味着,为了知道 percentileofscore 如何处理 nan,您必须知道代码如何进行比较.这是一个您不应该知道的实现细节,并且您不能依赖于在 scipy 的未来版本中保持相同.

Those results are all correct--that is how nan is supposed to behave. But that means, in order to know how percentileofscore handles nan, you have to know how the code does comparisons. And that is an implementation detail that you shouldn't have to know, and that you can't rely on to be the same in future versions of scipy.

如果您调查 percentfileofscore 的行为,您会发现它的行为好像 nan 是无限的.例如,如果您将 nan 替换为大于输入中任何其他值的值,您将得到相同的结果:

If you investigate the behavior of percentfileofscore, you'll find that it behaves as if nan was infinite. For example, if you replace nan with a value larger than any other value in the input, you'll get the same results:

In [53]: percentileofscore([10, 20, 25, 30, np.nan, np.nan], 18)
Out[53]: 16.666666666666664

In [54]: percentileofscore([10, 20, 25, 30, 999, 999], 18)
Out[54]: 16.666666666666664

不幸的是,您不能依赖这种行为.如果未来的实现发生变化,nan 可能最终表现得像负无穷大,或者有一些其他未指定的行为.

Unfortunately, you can't rely on this behavior. If the implementation changes in the future, nan might end up behaving like negative infinity, or have some other unspecified behavior.

解决这个问题"很简单:不要给 percentileofscore 任何 nan 值.您必须先清理数据.请注意,这可以很简单:

The solution to this "problem" is simple: don't give percentileofscore any nan values. You'll have to clean up your data first. Note that this can be as simple as:

result = percentileofscore(a[~np.isnan(a)], score)