I just leave the stickers on, adds a patina of character to my passport. It's scruffy enough with stains and watermarks. They're amazingly resistant to abuse are passports.
Isopropyl alcohol will remove most adhesive residues. Alternatively, if you don't have a handy light-fingered organic chemist, you can buy commercial stain and gum removers (De-Solv-It seems to be the brand) that work very well (can't remember what they contain, probably perchloroethylene). I'd avoid oily, it'll soak into paper and won't evaporate. None of these should remove ink, as Jurek says, modern printing uses UV curing, especially for permanence on documents like passports, but as ever, test on a small area.