I fitted a rack recently. Ive never had one on my bike before.
No handling difference from when it was on to when it was off.
I strapped my lock on top of it the other day(it normally goes in my back pocket, was so much nicer to not have it digging in, pulling the back of my jersey down) and this was when i noticed a difference.
You could feel more weight was on the bike, and not low down. This was noticeable more when i stood up. You could feel more of a sway.
Another thing which was noticed was, obviously, more weight on the bike. When i leg braked at first you could tell there was more momentum with the bike. I seemed to brake a normal amount with my legs, then have to up the pressure being applied. Never noticed that when i had the lock in my back pocket. Might have just been me though. I soon adjusted though.
Im yet to ride it with a pannier on. I might strap my bag to the rack tomorrow when im on my way over to my GF's, or might just keep it on my back.
But to answer your question, the rack itself didnt really change the handling of my bike at all. Its quite a light rack anyway, much lighter then what i would have thought anyway.
If you are going to use the rack then fit one. Mine is going to be left on all year round(but then i plan to attach and antenna to it for a radio, i do amateur radio) and will be used all year round.
I think the time when my rack and pannier(should hopefully have it by then) will be a god send will be the next Saturday(the 20th) and just about any other time i ride to my GFs after work.
I have clothes for the weekend in my bag, plus work clothes, plus wash bag, plus trainers and my bag on my back is full to overflowing. So being able to put most of my stuff in the pannier will be great!