I used to get a lot of that type of comment when I rode my Carlton. It was 21", soo, technically, too small, and I got a lot of adverse comments about the amount of seatpost I had showing. Of course, now compact frames are the thing, and there's loads more seatpost visible on the average bike.
And why did I ride the Carlton? Because it was the one I found in the cellar. Why didn't I buy a 'better' bike? Because I couldn't afford one, and, besides, the Carlton carried my toolbox over Sheffield's hills, and took me out to the Peak District, including up Winnats Pass. Did it matter that it wasn't a flash bike? Not one bit.
Moose, if you're happy, just get out & ride it. Have fun. That's what it's all about.