Depends a lot on the bike WRT how gracefully it handles gusting sidewind. Anything with a large front wheel or that isn't built for stability is going to be scary if there are sudden gusts.
I did Leek-Buxton-Whaley Bridge on the otherwise quite wind-tolerant Streetmachine in similar conditions once. A 30mph tailwind helped me on the way up, but once I got to the top it and turned around a bit, it became a struggle to adjust for every change in wind intensity in order to keep between the ditch and oncoming traffic. The glorious descent was performed at below 12mph in order to maintain control.
On the other hand, I've done a couple of semi-technical mountain bike rides in worse wind, and it wasn't anywhere near as scary, due to the much lower bike speeds.
On the gripping hand, after my recent off, anything that feels like a loss of steering control terrifies me. I chickened out of riding to the shops in 17mph gusts that I'd normally class as 'annoying' last week.
Of course if the wind's going to be in your face the whole way, it's an excellent time to chicken out on safety grounds.