I think the saying "fits like a glove" probably dates from the age when all clothes where handmade to measure. There might be fewer sizing options for gloves but perhaps more importantly, gloves need to fit more precisely, because hands are our most precision-acting part of our body.
This doesn't really help with the too small or too large dilemma, unless, perhaps, the gloves have a bit of stretch in them.