The hat I use has a sliding catch at the back so it's easy to set it for the ideal size/snugness.
I've never lost a hat due to pure speed yet, they even stay on at +50 mph.
But I do tend to make sure it's well down on my head if I think I'm going to do a silly speed.
It tends to be a sudden very strong cross draft that lifts the hat off my head.
I've also made the mistake once of getting a hat where the inside edge of the brim wasn't hard against my forehead.
It did make it very comfortable to ride in as it felt very soft.
That was until I got up to a silly speed.
At which point the slight cloth filled gap let the brim flutter badly.
This caused my glasses to vibrate and I suddenly couldn't see where I was going ......