There is a vectorized if, called ifelse. It also seems you are missing an extra minus.

super_trans <- function() {
     trans_new('super', 
               function(x) ifelse(x>0, x^0.5, -(-x)^0.5), 
               function(x) ifelse(x>0, x^2, -(-x)^2))
}