Being a rubbish cyclist is fine. I am one. The best I ever managed was 14.7 mph avg over about 30 miles in a group and 14 on my own. Since then my lungs have declined due to cystic fibrosis and I am now only able to cycle indoors. I would be happy to potter around, getting off to push up hill, and generally being rubbish. Except...
I've found the problem is that others won't let you be a rubbish cyclist. When out, puffing my way up hills with my 40% lung capacity, others pass me and offer me advice about trying to get fit, about working harder, trying harder, doing intervals. It actually isn't OK to be a rubbish cyclist at all. If you are, it is because you aren't trying hard enough, at least according to people who breeze past you.
So I would say if you can ignore everything that people throw at you, people laughing at you, people giving you advice about how to get fit, then it is OK.
As for fit. How many people can actually ride for hours with an oxygen saturation in the 70%s. I was always actually much fitter than all of them, riding closer to my maximum all the time than they could ever do. Now, I just cycle indoors. I can't get up the hill to my house any more.