I have been a teacher, and yes - it does spook the kids. Someone did a study a year or so back - it concluded something that every experienced teacher knew already!
I also used to work with bulls - they get more difficult when it's windy in a noticeable way. Friends with horses tell me the same.
I wonder if it might be to do with a primeval self-preservation thing, and it being hard to process what we hear in terms of picking up clues about danger? Just a thought.