There's recovery time, rest and sleep.
Recovery is doing light exercise (such as gentle cycling, walking, shopping, etc).
Rest is being awake but doing nothing (sitting on the sofa watching TV, emails, uploading stuff to Strava, etc).
Obvious sleep is obvious.
Riding fast allows you to have enough of all 3 and time spent shopping won't be "wasted", it's part of active recovery and helps wind down from the physical part of the attempt whilst also providing some useful mental stimulation and keeping touch with reality.
The problem with riding slowly is that you lose out on time for some of these and so something is going to give at some point. If you don't get enough recovery then your legs are going to get heavier and heavier, we all know about sleep deprivation and if you don't get enough rest you'll be mentally frazzled soon enough.
Angst about where to sleep certainly won't help though.