In the kids TV room, I have a generic Amazon sourced 3-way HDMI switch and it Just Works.
It will switch to the most-recently-used device, and if it gets it wrong, you just prod the button on it to cycle between the inputs.
So no, I don't think there are any pitfalls that I can think of.