I disagree. It matters a lot, because it's the Op's "muscle memory" (I really, really dislike that term) that will enable him to get back on his bike after a layoff & be able to ride it straight away.
What's open to discussion is -
1) how much of it will have been lost, to which the answer is almost certainly "not enough to worry about". It is, after all, [like] riding a bike, & we all know how that's used.
2) physical condition. As I know very well, that deteriorates, but as you said, someone who's done a lot of cycling should find it easier to regain bike fitness than someone who's never done it. Certainly true for me. From having to stop & rest half way when walking a mile, to 100 km randonnees plus riding to & from the start, in six months.