I usually chip in when golf gets a bashing on YACF so here goes.
Being an occasional golfer, I can attest that, certainly for amateurs carrying their own clubs, lugging a full set around a long & hilly course is far from sedentary. Then there is the psychological stress of screwing up repeatedly, unlike cycling which is a generally therapeutic activity, albeit with a certain amount of anxiety to be had on really testing rides. Also, golf balls don't generally travel 300 yards on their own without a fair amount of persuasion.
I've often thought that long audax rides are a rarified form of self abuse but nothing even approaches the inner turmoil and self-loathing incurred by slicing a ball into the trees or taking three out of a steep-sided bunker.
Those who scoff at golf have almost certainly never tried it and if they have gave it up because it was too difficult. Often when cycling past a golf course with friends I will say "Now that's a hard sport .... only partly joking"!