Call the marker.setIcon('newImage.png')... Look here for the docs.

Are you asking about the actual way to do it? You could just create each div, and a add a mouseover and mouseout listener that would change the icon and back for the markers.