I'd expect riding solo with no time goal is probably going to tend to the same speed number whatever your fitness level.
This is true, I think. A thin Wowbagger is still a Wowbagger and behaves like a Wowbagger.
I find there's a bike factor, too. ICE Sprint and Streetmachine are more or less the same weight, more or less the same tyres, and both a bit more aerodynamic than a standard upwrong. Conventional wisdom is that the trike would be about 12% slower because of the extra wheel. If I give it full beans, I climb at the same speed on either, and the Streetmachine rolls a bit better downhill. However, if I'm just aimlessly pootling along, I go a *lot* slower on the trike. I think it's psychological, because it feels faster, being low down and more inclined to vibrations.
The Red Baron on the other hand is also low down, and extremely inclined to vibrations. It's not a bike you pootle aimlessly on, though. The handling encourages speed, and riding slowly requires an element of concentration.
Similarly, I'm always surprised by my relative speed on tarmac and off-road on the mountain bike. On tarmac it's a slog, as you'd expect for an upright riding position and knobbly tyres. But on decent singletrack, or disintegrating towpaths I'll blat along, because it's fun to ride.