My worst was in Suffolk. I found a nice pub, with a nice quiet field about 1/4 mile up a dead end road to lay out my bivvy bag in, checked it out, and went to the pub.
Unfortunately, when inside, I wasn't paying attention, and at closing time, I went out to find that it as (a) very dark, and (b) absolutely tipping down. It turns out it's not possible to get your bivvy bag out, get you sleeping bag out and into the bivvy bag, and me undresed and into the sleeping bag without getting fairly soaked (sleeping bag too). The rain (and associated thunder) stopped at about 2 am, and I gave up, packed up, and left at about 5.
The moral is to always ensure that you've some sort of shelter in addition to the bivvy bag, such as a small tarp.
Others include...
bivvy bag: in the A65 underpass at Clapham. It rained, and it turns out the underpass does double duty as a flood relief channel. I was woken up by cold feet, and found that the river had overflowed out of the adjacent tunnel, and was now flowing round my feet.
bivvy bag: poking my nose out in the morning to find a very big bull grazing about 5 yards off (I retreated back inside and waited until it was the other end of the field, then threw everything over the wall and followed it, smartish).
tent: spending half the night listening to the wolves howling just outside the tent, definitely not getting up for a pee. (Probably coyotes, and maybe 1/4 mile off, but someone's only got to suggest wolves to keep you inside.)
I've had worse by way of noisy neighbours on a campside than with wild camping - a village festival with fireworks etc 10 pm-2 am (France), people playing the bongos until 3 am (Barge, Honeystreet)