The traditional winter cap, with a peak and knitted elasticated material which you can fold up or pull down to cover the ears, is supposed to be tight/close fitting, it certainly does not flap around.
A cotton cap is supposed to be fairly loose fitting, it's the elastic which keeps it on.
Both do not come off however windy it is.
Well, on a recumbent I suppose because you are looking up, the peak will catch on any wind if the cap is not tight.