I think if your photos are 'public', then the whole world can see them.
The sharing link just provides a front door into your account, where they can browse around as they see fit.
Anyone else can too, if they browse around in flickr.
I think you'd need to set all your pics to 'private', then sharing using a Guest Pass would only show those images.
Or something like that.