Birds aren’t a significant threat until they are the size of a cassowary at least. I don’t see why people freak out about birds smaller than that.
Traumatic childhood encounters? (See above.)
It's relative size that matters. I present this typical example of a stand-off with a Gt Yarmouth Shitehawk (wheelie-bin for scale):
I'd classify that as 'no big deal, just keep an eye on your chips'. But then a family with a toddler appeared, and it started pestering them. Toddler, quite reasonably, was having none of being strapped to a chair while a flappy dinosaur half its size with at least one sharp end hopped around just out of arm's reach. I thought the parents' approach of laughing at the distressed child was a bit unfair, but I suppose that's what my parents did in similar circumstances, and it hasn't done me any harm...
And that's just a shitehawk, which is ultimately only interested in theft. Geese have terrible attitude and will attack just because they're moody bastards. And swans are basically geese with white privilege and the ability to break your arm.