My local pool has one wide lane for lap swimmers, generally there's plenty of room for people to safely overtake as it's effectively double width. There's a free swimming lane for slower people, though it can get clogged with kids, and for some reason it only one family to block the entire lane and make it unusable for anyone trying to swim, parents rarely try to corral their kids. So the slower swimmers sometimes end up the lap lane.
Anyway, I confess it's always blokes of my age or less who give way and take care to swim towards the side. Women and older men, just push out in front and amble down the middle of the pool as though they're on a mission to obstruct as many people as possible.
(My wife always swims in the slow lane [when there's one available] and she's not a slow swimmer, though she doesn't believe me despite often being told to move to the medium lane.)