I found that endurance increased with age. I put this down to a couple of things: just having more miles in the legs and also learning to slow down a bit such that I wasn't expending all my energy in the first couple of hours. These are both as much about headology as physical constraints.
Once endurance increases one has the basic stamina to do more 'speed' work with less danger of burning ones self out too early in the ride so speed should go up a bit.
But my cycling was complicated in my 40s as I changed from mainly countryside longer rides that had few stops and starts but plenty of rolling hills, to urban riding - lots of stops and starts, short sprints, not a lot of long hills. This had the obvious effect of making my flat cruising speed come down but my sprints a bit better (but not that much). And reducing climbing ability. Also the strange side effect: I'd do (miles wise) in a whole day of urban riding in what I used to do in a morning in the countryside....which meant that when I went back to the countryside riding I could go further but had to do it at a slower pace....refer back to the above I guess. I could also do long rides with much less preparation than I used to need.
Then it all went pear shaped last October with a fractured knee cap and off the bike for four months....when I got out of the cast my left leg had virtually no muscle left. I was surprised by how fast I got back to a usable fitness. Indeed, the other week I did my first 50miler (having only got up to 30 since the off) with no real noticeable extra effort, not to mention finding myself doing 18mph into the head wind chasing down some roadies at one point! Thus, not only does recovery at this age seem quicker than it used to (e.g. when I was off with a knackered shoulder for a month or two in the 1990s), but getting up to the 50mile ride milestone seemed easier despite the injury than when I used to get slack over Christmas and then start rebuilding in the spring 20 years back with no injury to overcome. This would seem contra to Ian H's statement....but I again I think that Ian is correct in terms of the physical recovery, but what I am experiencing is the headology of age.....I now know more about cycling, I am more confident, I worry less about being unfit, and thus the recovery 'seems' easy and because it feels like progress is being made faster it does actually get made faster.
Of course, climbing is still a pain in the knee. I was always the kind of climber (when fit) who did most of a climb out of the saddle and only sat down to accelerate. This is now much harder as the knee doesn't like taking the extra weight, thus to an extent I'm having to learn to climb again. Not even going to look at the fixed or the SS for another year!
Anyway that's my experience of fitness vs ageing: physically you might be declining but you can make up for this with increased mental strength and confidence as you become more at ease in your own skin as it were. New-age hairy fairy bunkum? Maybe, but I seem to be happier in my cycling now than 20 years ago, plus I can do stuff with far less preparation....so it works for me.