There are a lot of variables other than fitness that determine whether a ride is hard. Maybe you went into the red more than you think you did in the opening stages. If so, you would tire in later sections. Did you drink enough? Did you eat the right food regularly enough? When you stopped did you stop for too long? etc etc. All these things are learned through experience.
To put this into perspective, I rode more of less the same route as you, minus about 35 miles on the Saturday. I got to Dolgellau at 2pm. No sit down rests until 160 miles, just brief petrol station and supermarket raids. On the saturday and the Sunday, at no point did I feel tired at all. Average moving speed was 16mph. Heart rate zone was almost entirely Zone 2.
Often on really long rides you have to ride well within your ability, not pushing it at all, and you end up finishing faster than if you don't. You can smash yourself to bits and only gain 10 minutes per stage on somebody who is taking it easy....That 10 minute advantage is lost when you sit down for 30 minutes in a cafe because you are knackered.
If you are in group dominated by one individual you can end up riding at their rhythm and not yours...trying too hard on hills, wasting speed on descents etc
