this.hash will return "#foo" which is also a valid ID selector ^[docs]. Hence $(this.hash) is the same as $("#foo") and will select the element with ID foo.
In your second example, you don't have an element with ID bar. Thus $(this.hash) won't select any element and target.length will be 0.

Maybe you are confused by the fact that the browser still jumps to <a name="bar"></a>, although the alert is not shown. The browser does not behave the same as jQuery.

From the HTML specification about the name attribute:

This attribute names the current anchor so that it may be the destination of another link. The value of this attribute must be a unique anchor name. The scope of this name is the current document. Note that this attribute shares the same name space as the id attribute.

So if the browser recognises a hash (fragment identifier) in the URL, it tries to find the first element with this ID or the first a element with that name.

In contrast, CSS ID selectors (that's what jQuery is using) only search for elements with that ID, not for (a) elements with that name. Internally, jQuery is using document.getElemenById.

If the hash value is always referring to either an ID or a name, you can use the multiple selector to just make one selection:

$(this.hash + ', a[name="' + this.hash.substr(1) + '"]')

In case there would be an element with this ID and an anchor with this name, you'd select all of them though.