I get Teh Fear too but it never seems as bad when I'm actually out. If I don't feel lucky, I take a slower and safer route. There are things I'd do 20 or 30 years ago, like riding on DCs and major trunk roads, that I wouldn't do now. Most of this is about worsening traffic levels and anti-cyclist attitudes among drivers than me losing my nerve, though. I still don't wear a helmet because I don't believe it will help much if I'm mown down by a texting moron.
We all make our own risk assessments and they should be flexible: day or night, traffic volumes and speeds, number of other cyclists, driver familiarity with cyclists (good in Oxford or London, appalling in Swindon), wet/dry roads etc. Sometimes it just feels wrong.