There are so many factors that can conspire to make a ride hard. Fitness (or lack of), preparation (or lack of), weather, company, experience (yes especially experience), psychological reasons, the list goes on. Some of my hardest Audaxes were when I was inexperienced and made bad decisions to do with food or hydration. I can remember my second 200km Audax, the Start of Summertime Special and being so badly dehydrated that I could not work out what to do with a R@T instruction!
On a good day a 'hard' ride can be easy, and on a bad day even the 'easiest' of rides can be bloody hard. I have done about 40 audaxes and the only audax that I have done that most experienced audaxers people would consider 'hard' was the Elenith, and for some strange reason that went so well that it would not make it onto the list of my top ten hardest.