I suspect it all has to do with our (human) generally shit risk assessment ability when faced with pitting our ability against statistics. Because of course, that's not what we do. We just ride and expect to stay alive. In reality, there are situations that are more dangerous than others and in order to maximise your chance of survival, you avoid putting yourself into that position.
The easiest explanation for this is the danger of lorries in cities. If you don't ever drive up the left hand side, you reduce (not remove, note) the danger from that source.
Any preventable accident needs the coincidence of a minimum of two low frequency events. For example, nipping the wrong way down a one way street that you've used thousands of times, at the same time as a boy racer comes the other way. You might have a reasonable expectation that there shouldn't be a boy racer on the road, but if the two of you are in the same place at the same time, you got trouble.
Main roads and side roads just have different risks attaching to them - neither are actually "safe" and if you rode around some parts of Essex, you would realise that. Me, I'm happy (for variable quantities of happy) on any road, but I try to make sensible decisions based on conditions I perceive at the time. The main roads have a much more clear and present danger, were I to use the side I would modify my speed and look so that I wouldn't be forced to wobble in the road.
As to what other riders do, often not the right thing. Whether in a car or on a bike. The way forward is for more people to cycle and to introduce strict liability. Then, after some years, it might begin to approach the level of cycling comfort in Catalonia, where drivers would occasionally give a sociable toot as the passed safely on the other side of the road, and I never felt actually threatened even on a single carriageway main road in the driving rain. OK That did get a bit scary with the sidewind.