The short answer is just do it. The long answer is below
We cycle-camped around south england for two and a half weeks last summer on our (quite old) Dawes Supergalaxy tandem. We took four full-size panniers, a bar bag and carried the tent and sleeping mats on top of the rear rack.
The bar bag just had the things we needed often (camera, money, food) and the full size panniers at the front had (relatively) light but bulky stuff in (sleeping bags, stove, plastic cooking things). We used the rear panniers for clothing and other heavy stuff. We did try to pack light but I don't think we succeeded - we certainly didn't find that we were missing things to make life comfortable.*
The tandem wobbled alarmingly when we weren't riding it - stood in line with it you could see the front like / and the back like \ - but when we were on it, it didn't feel too bad. We certainly didn't feel unsafe and we didn't have any problems with handling. The downhills were definitely fast though!
You're right that the pilot (me in this case) has to work a bit harder to steer but I got used to it after a couple of days and by the end I barely noticed.** I'm a 10st weakling and I didn't end up with tired arms and shoulders although I admit that we didn't always cover a huge distance (30-40miles most days, a couple of days of 70-80miles at either end).
Hope that helps,
Dave
* We have quite a light tent for its size (3.5kgs for a 3 man tent with big porch), a trangia for cooking, but not particularly lightweight anything else. We quite a few spare clothes and bits and pieces too.
** That is, until I got home and rode my town bike with wide handlebars instead of narrow drops. I almost fell off!