I don't mind someone sitting on my wheel - makes no odds to me really.
I sometimes find though that I very slowly catch someone and need a slight rest before I eventually overtake, so I'll sit there behind them for a little while. Last thing I want is to over take, then be knackered and passed again within a mile!
Most of the time though, if I do drift up behind someone, I'm pretty concious of not drafting. I tend to sit far enough off the back wheel not to gain any significant effect. Horses for courses though - if I'm riding with some people I know, or in a group, I do try to do my bit at the front.
As previous posters have wisely pointed out, the only thing that really annoys me on an audax is me. My lack of strength, stamina, hill climbing ability, whatever. And that is for me to deal with.