People don't like silence between the question and answer, but you need a second or two to gather your thoughts. This is especially true in a group setting where someone else might jump in if you haven't started saying something. So filler words are used, like "Well, " or "Right, " or nowadays, "So, ".
It's mildly irritating, but not as irritating as the Q&A session I attended recently where the person preceded every single answer with "That's a really good question, ". Every single answer, for an hour.