Hard to say without a working example, but I think your problem is that in your add function you are adding "p.sumIndex2 += v.h" while in your remove function you are subtracting "p.sumIndex2 -= v.g". So your sumIndex2 isn't really tracking any specific value. You should add and subtract the same thing from it so that adding a record and then removing it results in no change.

Additionally, you should recalculate your average value when you remove records, not just when you add them. Your average is going to be wrong after filters are applied.

I'd also recommend just creating all three groups you need and the bar chart with the default group. Then switch out the group on the bar chart and rerender it when the group you want to display changes.