Well, Hmm, yebbut.
I'd not expect a member of the general public to recognise mathematical notation.
It's probably reasonable to flag a concern on the flimsy basis that someone's behaviour seems 'not quite right', however wrong you may be.
But the response needs to be proportionate to the flimsiness of the concern.
Perhaps if he'd just responded along the lines of 'Oh, no. I'm just going to a conference, and I've got these really hard maths problems to solve before I get there.' would have worked better for all concerned.