Right so if we go check out the Default Template and at the very top of the first code example we see;

<!--Style and template for the button in the upper left corner of the DataGrid.-->

With the declared style template of:

<Style TargetType="{x:Type Button}"
       x:Key="{ComponentResourceKey ResourceId=DataGridSelectAllButtonStyle, 
               TypeInTargetAssembly={x:Type DataGrid}}">

We now know what/where it is. After that it's just a matter of editing said style part to our hearts content and overriding at the instance level or at the template etc.

As for your bonus question, if we go check out the same style templates at the <Style TargetType="{x:Type DataGridRowHeader}"> we'll see hard-set BorderThickness on the x:Name="rowHeaderBorder" that we'll just change to whatever. Wherein the same applies to the <Style TargetType="{x:Type DataGridColumnHeader}"> template as there's also another hard-set BorderThickness of "1" on the x:Name="columnHeaderBorder"