Have you checked the surface hardness of headset contact surfaces compared to ball bearings?
there is more to it than that alone; when balls indent into surfaces at high loads, yield occurs subsurface. This means that a ball that appears to be softer than a substrate surface can indent into that substrate if the hardness is not uniform with depth.
This illustration isn't exactly correct for several reasons but it shows more or less what is going on
You can see that the material is stressed and may yield to a depth (and radius) beyond the indenter that is comparable to the diameter of the indenter. One effect of this is that when doing hardness testing, you need to space the indents apart using a pitch that is at least x3 the diameter of the indents, else the yielded regions can overlap and this can affect the results. It also means that indenting into hardened surfaces (like ballraces) can produce some weird effects; for example the 'pile up' region can be larger in diameter and less high that you might expect, because the plastic zone beneath the indenter is the shape of an upside down mushroom.
Bearing surfaces are commonly graded in hardness with depth and (depending on the diameter of the indenter (ball)) the surface stress at which deformation occurs will vary, largely because the stresses can easily extend downwards into considerably softer material, and cause subsurface yield.
IIRC Brandt's argument was more or less that it couldn't be brinelling because the loads per ball were not high enough (about which he was mistaken on several counts) the marking on the raceways was the wrong geometry (again a flawed supposition for several reasons) and he therefore inferred that fretting must be occurring by a process of elimination. The fact that the number of cycles for true fretting to occur was usually completely different, and there were typically none of the usual signs of fretting (such as wear debris) in most cases didn't seem to worry him in the slightest.
So yes fretting is certainly possible (on the turbo, or a roofrack for example) but on the road, meh, the arguments are a lot less convincing.
cheers