It's not so much the bandwidth (as long as it's sufficient) as the QOS: VOIP (be it Skype, SIP/RTP or some proprietary thing) needs consistent latency and low packet loss, which means it'll start to go wrong if the link is being intermittently saturated by some other data.
Ideally you'd have some traffic shaping at both ends to ensure that the VOIP packets have priority and that the link never saturates (or have a dedicated line for the VOIP data). Failing that, you throw bandwidth at the problem. Failing that, you try not to go torrenting or let Windows 10 download itself or whatever while you're making a phone call.
Unsurprisingly, things get worse as the number of simultaneous calls increases. Fortunately, that's unlikely to be more than about 2 in a domestic environment.