Excessive time at controls can be a sign something else is wrong - either your mind is no long on the ride or you're suffering physically one way or another. The fiddling with bag/phone/etc is just something to do with your hands.
I can't imagine many riders get anything like 5-6 hours sleep on a typical BRM 600.
A couple of models
Rider A moves at 25 km/h and spends 30 minutes at each of the 8 controls, it will take him/her 24 hours moving + 4 hours at controls, so in a BRM that's 12 hours to play with, obviously not 12 hours sleep, because he/she would probably be out of time at the following control, but still a lot of time.
Should the same rider spend 1 hours at each control, then it's "only" 8 hours to play with
Rider B moves at 20 km/h, using the same model, it's 30 hours on the road + 4 at controls, still a 6 hours contingency, which means possibly up to 4 hours sleep at the night contro, not bad.
With 1 hour at each control, then it's 38 hours, which means only 2 hours contingency, a lot less at the time of the night control, and basically zero time to sleep or he/she will be out of time at the following control.
If moving at less than 20km/h in a non AAA event, one should question whether BRM is suitable.
Ultimately, an Audax is a "race" against time, faffing with smart phones and taking 15 minutes from wanting to leave the control to actually leaving the control needs to be fixed