I rode both LVRC and BC(F) races into my early 50s. As you say, the LVRCs were harder, but you can ease your way back by riding in a category or two higher (i.e. with the over 60s or over 70s) by arrangement with the organiser - just don't take the prizes!
You can't be a 4th cat if you've ever been a 3rd or higher, so if you've been a 2nd or 1st in the past you can only be a 3rd. Actually, that's OK because some events are 3/4, and 4th cats only can be a total casino, with bodies flying everywhere!
Personally, I'd turn up to a few circuit races and accept that for the first few I'd find it hard, but that I'd soon get back to it.
Don't forget - age, experience and cunning will always defeat youth and fitness!