Caveats: I am not a doctor, haven't had the health issues, and am a chap. In fact the only qualification I have for any kind of opinion is that I ride some audax.
A history of cancer and chemo didn't seem to do Lance Armstrong's cycling much harm.
My BiL had radio therapy for cancer, and that seems to have made him more "fragile" - more aches and pains - but he still cycles.
I'd have though you'll have a highly tuned sense of self-awareness about your health now, and will be able to tell if something is doing you harm or not.
Try some 100's, see how you get on, and assuming all is well, start to work up the distances. Eat well, with lots of anti-oxidants (exercise is a form of oxidative stress - endurance exercises even more so). Relax in between long rides.
What is it about your fitness that makes you frustrated? If you can ride 50 miles in one go, and average 100 miles a week, there's a good base level of fitness by the sound of it.
Motivation is a whole other thing, but if you enjoy it, you'll want to do it more - and that will drive your motivation, at least to a point.
Edit: BTW - "50" is not "late in life". I'm 48 and only started randonneering a couple of years ago. Over 40's have better endurance and stamina than those whipper-snapper youngsters who blow themselves up in the first 50km
.
