AFAIK the email address book facility simply allows you to contact people from your address book. If you choose not to, they can only do likewise if you are already in their address book. And, even if they do, simply ignore them.
I have resisted FaceAche for a very long time but I've found that it is a very useful way to keep in touch with my extended family.