I started this time last year. It was sunny but cold, and we did capsize drill. It was cold.
I had surface sport wetsuit, neoprene booties (note to self, must buy a pair of proper kayaking shoes), basic semi-dry cag top and buoyancy aid. The buoyancy aid will afford some comfort to your chest and back, as the foam is just another insulating layer. I'm so-so about the gloves, tried it with and without, but we had good weather early last year, so I didn't really use them until the autumn - I used my Answer racing gloves, which don't mind a bit of water, and they were fine. When you're moving you'll be warm. If you're wet, or sitting around for a long time, you'll start to get cold.
What kind of kayak? short=bouncy and unstable, long=more stable/manageable if you're a novice. If somebody points you at a Pyranha G3 I bet you'll be fine. I don't think you'd be happy in an InaZone, which is where I started.