Not a real answer but too long for a comment:

Maybe you could use the prominence of the peaks. You can use find_peaks with the prominence and width parameters and try and tweak other parameters like window size for prominence calculation (wlen).

The following quick example only illustrates the usage. It just finds peaks with a minimum prominence of arbitrarily 3 times the median:

from scipy.signal import find_peaks
df = pd.read_csv('https://raw.githubusercontent.com/MigasTigas/peak_removal/master/dataset_simple_example.csv')
peaks,_ = find_peaks(df.value, prominence=df.value.median()*3, width=(1,2))
ax = df.plot()
df.iloc[peaks.tolist()].plot(style=['x'], ax=ax)