Surely the question here is how you balance the wants and needs of various family members at various times
Are you talking to your wife and kids about it?
My situation is 2 kids, one disabled the other "normal", one wife. Daughter has a busy life - now 14, son also has a busy life with carers. Our challenge is my work taking me away a lot, including some weekends, dealing with our son and making sure he's a full part of the family etc. I'm Type 1 diabetic, so regular excercise is fundamental to managing that, which is understood, and my wife needs to exercise to help manage back problems, but as usual it's a balance. I'm the only cyclist.
We've come to an understanding that one audax a month is not unreasonable if balanced with family time as well, and I try and manage training to be as non-disruptive as possible including being out before first light if that's what it takes - which wasn't a bad thing last summer tbh. Holidays, I hate sitting on a beach - last year and this, I've gone on cycling holidays in the Scottish isles, wife and daughter went elsewhere with friends, this year on a cruise with family. The other of us was then at home to help with son who is now far too big and unweildy for family holidays. Next year will be a family holiday with the three of us and overnight care at home for him. Not saying this is perfect, but it sort of works