I googled after I answered.
My brother lived in Lincolnshire for a few years after he left University. It was he who told me about the expression "Yellow Belly" and the reason behind it. Having read the Wiki entry, I'd say the military explanation sounds the least far-fetched.