From about 1967 to 1990 I rode Brooks, mainly Professional. I had an original Pro Select which was excellent. When the Colt was introduced (early 90s?) I tried that, but it quite quickly collapsed on one side. I gave in in disgust and took to using Rolls, like you. They were good enough in that I barely noticed them. Then, about 2004, I decided to try Brooks again. I had B17 which were OK, a Pro that collapsed on one side, and two Swifts of which neither lasted more than a few thousand miles before becoming very saggy. Apart from the Pro, all were comfortable enough bearing in mind I date from days before huge great nappy pads in shorts. In 2012 I decided to try a Gilles Berthoud Aspin, and found it suited me straight out of the box, so much so that I now have 5 of them. I still have a 1960s Brooks Pro which is also very nice.
I think the change in leather (this century) has been problematic, not to mention snapped rails (B17 titanium) and snapped nose bolt (B17 steel) and the aforementioned leather distortion.
The short answer is - yes, I made the move back to leather (if not Brooks) with no regrets. Your forthcoming Swift may well be a good one. I certainly found them comfortable before they sagged.