Not Robinswood Hill.
The bit of Gloucester I live in, that would be in the wrong direction too.
It's steep enough that the chap in red, and the other one you probably can't make out behind the trees, had to walk up.
To recap the clues so far.
It's north of Gloucester, within an evening's ride
in/on the edge of a town
There's a steep hill and a cafe