The row click event uses a function called 'openGridRow'

Include some javascript with your grid, and add this function with some custom code to cancel the event if certain conditions are met. Also then set the checkbox back to checked.

Example will be

function openGridRow(grid, event){
                var element = Event.findElement(event, 'tr');
                //alert(Event.element(event).tagName.toLowerCase());
                if(Event.element(event).type != 'checkbox'){
                  if(['img', 'a', 'input', 'select', 'option', 'img'].indexOf(Event.element(event).tagName.toLowerCase())!=-1) {
                      // re-enable the checkbox
                      var checkbox = Element.select(element, 'input');
                      if(checkbox[0] && !checkbox[0].disabled){
                          grid.setCheckboxChecked(checkbox[0], true);
                      }
                      return;
                  }
                }
                if(element.title){
                    setLocation(element.title);
                }
    }    

The above example will do nothing, if the element type clicked is a, input, select or option Anything else will continue as per normal.