The quiet way from Abergavenny to Danywenallt hostel is along the Brecon & Monmouth canal.
Get onto NCN 46 in Abergavenny, leave it for the canal either up a steep zigzag shortly after crossing over the B4246 or at the canal, and stay on the canal until Talybont on Usk. You can either get onto an
old tramway up to the hostel at the White Hart, or carry on past the Star Inn to the road bridge and left up the road and across the dam.
To get to Pen y Fan, I'd carry on up the valley, and over the top. There are tracks on the right that take you to a col between Cribyn and Fan y Big. Dump the bike at the col and start walking (if you don't go over Cribyn, it looks like it would be mostly rideable on a mountain bike).
Did phil w get the wrong hostel?