Being stored in the stuff sack probably doesn't help, though I think that's more of a problem for down than synthetic.
How much sleeping bag is enough is going to vary from person to person. I wouldn't want less than a 3-season bag for BRITISH could-get-down-to-zero-with-a-couple-of-days-notice conditions, though I do sometimes sleep with my feet sticking out the bottom, or with the whole thing unfolded into an awkwardly-shaped quilt if it's warm. It's much easier to let heat out of a warm sleeping bag than keep heat in a cold one!
I think comfort can vary with age. I camped happily with a skinny synthetic bag and a closed-cell foam mat when I was a teen, but as an adult I feel the cold. I also went from being borderline anorexic to the lardy side of healthy in my early 20s, so it's hard to say what had the most effect.