I'd agree with that, and I've been guilty of passing competent-looking[1] cyclists a bit close myself on occasion - at least before the lockdown. The important thing is not to startle them; I don't think passing someone cycling in left-car-wheeltrack by overtaking in right-car-wheeltrack (so as to avoid dealing with oncoming traffic) is usually a problem. It's blasting through with inches of clearance, or (pet hate) sneaking into their blind spot and then making some noise that's the problem.
Children, BMX bandits, BSOists, the poorly secured luggage brigade, anyone negotiating crap road surfaces and obvious wobblers I'll give a wide berth. They could do anything.
[1] Which isn't the same thing as fast. Plenty of slower non-lycra-wearing cyclists ride predictably, and I rarely get to overtake the fast ones.