From ?geom_segment we see the requried aesthetics are x, xend, y, yend, so we can use

ggplot(data=mydf[mydf$groups==1,],aes(x=X,y=Y))+
  geom_point(size=2)+
  theme_bw() +
  geom_segment(aes(x=X, y=-Inf, xend=X, yend = Y))