I know that route, ridden it twice, once in the dry on a tourer and wished I hadn't, once on a MTB in the deep mud. There are stones, rough surfaces, steep offroad-esq climbs - fantastic fun unless you've brought the wrong bike.
I don't recommend it.
Rather than head up to White Cross and down Sand Hill (which is a tough ask), how about going a little further through Castleton and taking the right to Commondale Moor. That road surface is lovely and the route is very pretty. You have to get out of Castleton, but the climb is much easier than the climb to White Cross.