I have no problem at all with the floating, but don't have very good technique, am not particularly fit and am most definitely not streamlined. I can comfortably do a mile (64 lengths of the very same pool as you were probably using, mrcharly!) in under an hour but struggle to go faster.
When I swam several times a week (and did a lot of aquafit classes, plus a handful of reasonably challenging gym sessions most weeks) I was a fair bit fitter and a bit more streamlined and managed to comfortably do 64 lengths of truly shockingly bad breaststroke in the Barbican, which I reckon was a slightly 'slower' pool, in just under 40 minutes. 128, which I aimed for once a week, was about 1 hour 25 minutes of steady away plodding.
Must get back to the pool, haven't been for weeks.....