Surely navvying is a lot more anaerobic than marathon running or riding a long way on a bicycle? Lots of digging, heavy lifting and hitting things with BFO hammers...
They had to move the digging, lifting and hitting apparatus around (they'd probably already eaten the horses of they had them) , they also had to walk to work sites and back often several miles from accommodation, and walk around the site and of course the most important of navvy activity, fighting.
Setting up the site would be the work of day-labourers. Navvies involved in excavation and muck-shifting were paid by the wagon-load, and there was no mechanism for paying them to do general labouring. It's the same on many building sites today. Tradesmen are on piecework, and general labourers do the connecting bits, which aren't amenable to pricing.
Similarly, you wouldn't expect racing cyclists to be involved in shifting the barriers, and putting up signs. It's interesting to consider 'organised' cycling as work. The output is miles ridden within a given framework. The framework varies, and some 'workers' are highly paid. They can be considered as the successors of manual workers with a continuous heavy workload. The bulk of participants are imitating the level of activity that was commonplace in industry and agriculture in the past.
The division between amateur and professional sport originated from the fitness and stamina of manual workers, and their need to be paid. The schism between the two codes of rugby was the prime example. That division between 'Gentlemen', and 'Players' has narrowed as the amount of physical work in manual trades has diminished.
The last bastion of hard manual work is in upland agriculture, and is celebrated in competitions in sheep-shearing and dry-stone walling. The Basques have a site of 18 officially sanctioned endurance and strength sports, scything is probably the most aerobically demanding.
https://en.wikipedia.org/wiki/Basque_rural_sportsThat was the culture that gave us Miguel Indurain.