Whilst it is true that mounting a horse from the right whilst wearing a sword on the left would be distinctly awkward, invoking swords just moves the "because most people are right handed" conclusion back a step, since swords are mostly worn on the left because most people are right-handed.
Yes, I see that - but when civilians were just using their horses to go to the market, visit friends, etc - it probably didn't matter which side of the road they used, similar to how it doesn't really matter nowadays on cycle paths. It was only when there were battles on that there was a sense of urgency, and thus there had to be a convention. So it was the knights with their swords that initially defined this convention.
Also even though I accept it's
fairly rare amongst civilians to mount horses from the right, it was probably even rarer amongst knights.
If they'd needed a convention without swords, it probably would have been to ride on the left, but they didn't need one.