When I first started riding with a club and doing cols in the Vosges I needed 4 days to recover. My usual schedule was Sunday 100k + cols and Thursday 40k coffee trog. Usually I would hardly be able to walk on Mondays, still feel stiff on Tuesdays then pee like a world champ on Wednesday night. As I got fitter I kept to the same cycle but the stiffness and pain receded.
4 days seemed to be the magic number, though.