且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何有效地比较pandas DataFrame中的行?

更新时间:2023-11-28 18:16:46

此答案可能不是很有效.我面临着一个非常相似的问题,并且目前正在寻找一种比我的方法更有效的方法,因为在我的数据帧(60万行)上计算仍需要一个小时.

This answer might not be very efficient. I'm facing a very similar problem and am currently looking for something more efficient than what I do because it still takes one hour to compute on my dataframe (600k rows).

我首先建议您不要像您一样考虑使用for循环.您可能无法避免一个(这是我使用apply所做的事情),但是第二个可以(必须)被矢量化.

I first suggest you don't even think about using for loops like you do. You might not be able to avoid one (which is what I do using apply), but the second can (must) be vectorized.

该技术的想法是在数据框中创建一个新列,以存储附近是否在附近(临时和空间上)罢工.

The idea of this technique is to create a new column in the dataframe storing whether there is another strike nearby (temporarly and spatially).

首先,让我们创建一个函数(使用numpy包)来计算一个打击(reference)与所有其他打击之间的距离:

First let's create a function calculating (with numpy package) the distances between one strike (reference) and all the others:

def get_distance(reference,other_strikes):

    radius = 6371.00085 #radius of the earth
    # Get lats and longs in radians, then compute deltas:
    lat1 = np.radians(other_strikes.Lat)
    lat2 = np.radians(reference[0])
    dLat = lat2-lat1
    dLon = np.radians(reference[1]) - np.radians(other_strikes.Lon)
    # And compute the distance (in km)
    a = np.sin(dLat / 2.0) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(dLon / 2.0) ** 2
    return 2 * np.arcsin(np.minimum(1, np.sqrt(a))) * radius

然后创建一个函数,该函数将检查一个给定的打击是否在附近至少存在另一个:

Then create a function that will check whether, for one given strike, there is at least another nearby:

def is_there_a_strike_nearby(date_ref, lat_ref, long_ref, delta_t, delta_d, other_strikes):
    dmin = date_ref - np.timedelta64(delta_t,'D')
    dmax = date_ref + np.timedelta64(delta_t,'D')

    #Let's first find all strikes within a temporal range
    ind = other_strikes.Date.searchsorted([date_ref-delta_t,date_ref+delta_t])
    nearby_strikes = other_strikes.loc[ind[0]:ind[1]-1].copy()

    if len(nearby_strikes) == 0:
        return False

    #Let's compute spatial distance now:
    nearby_strikes['distance'] = get_distance([lat_ref,long_ref], nearby_strikes[['Lat','Lon']])

    nearby_strikes = nearby_strikes[nearby_strikes['distance']<=delta_d]

    return (len(nearbystrikes)>0)

现在您已经准备好所有功能,可以在数据框上使用apply:

Now that all your functions are ready, you can use apply on your dataframe:

data['presence of nearby strike'] = data[['Date','Lat','Lon']].apply(lambda x: is_there_a_strike_nearby(x['Date'],x['Lat'],x['Long'], delta_t, delta_d,data)

就是这样,您现在已经在数据框中创建了一个新列,该行指示您的罢工是隔离的(False)还是非隔离的(True),从中轻松创建新的数据框.

And that's it, you have now created a new column in your dataframe that indicates whether your strike is isolated (False) or not (True), creating your new dataframe from this is easy.

此方法的问题是它仍然需要很长时间才能打开.有多种方法可以使速度更快,例如,将is_there_a_strike_nearby更改为data按lat和long排序的其他参数,并在计算距离之前使用其他searchsorted过滤LatLong(例如,如果您希望打击范围在10公里之内,则可以将delta_Lat设为0.09)进行过滤.

The problem of this method is that it still is long to turn. There are ways to make it faster, for instance change is_there_a_strike_nearby to take as other arguments your data sorted by lat and long, and using other searchsorted to filter over Lat and Long before computing the distance (for instance if you want the strikes within a range of 10km, you can filter with a delta_Lat of 0.09).

对此方法的任何反馈都非常受欢迎!

Any feedback over this method is more than welcome!