There's obviously a duty of care on organisers to avoid roads that are hazardous. That's clearest on rides such as LEL.
I rode LEL 2009 as a filming exercise, and rode straight up the A10 to Royston, Ermine Street to Godmanchester, then to Stilton and through Peterborough to the A15 and on to the first control. That's an extreme example of not following the course, but it hit the controls, and was fast.
I came back a different way, as Gamlingay made a diversion inevitable. There are riders who like to follow major roads, as they're fast, and it gets points logged with less effort. I'd seen the contraflow on the A10 on my way out, but it frightened some of the people using that route on the way back.
There has been discussion on here about fast-route hacks for LEL, those routes tend to be a product of local knowledge. The routing is easy to adapt to GPS, but traffic conditions at various times won't be as obvious.
So it's possible for any amount of due diligence on routing by an organiser to be subverted by participants, given that riders are on a private excursion, taking in a number of defined points, which happen to be open between certain times. But the route provided shouldn't present undue hazards for any of the riders, who will be unfamiliar with local conditions.
Audax has become more difficult, and easier at the same time, depending on whether you follow lanes or fast main roads. You could make 'safer' routes mandatory rather than advisory, but that removes the element of free agency that comes with a 'private excursion'. It also requires enforcement, and penalties.
PBP is obviously the most extreme example of a mandatory route, with monitoring, and time penalties for deviating from the set course. That requires a certain prestige from the event, so that participants see transgressions as 'cheating', which compromise their own rules-bound achievement.
I don't know enough about adventure racing to make any comparisons.