It was the cricket boycott that really hurt them though.
That and being beaten by the Lions in the 1970s tour. I was against that tour taking place initially, but watching Gareth Edwards, JPR et al giving the Springboks a roasting was wonderful. From memory the lions played 22 matches and won 21, drawing the last. The SA team was known for its violence and dirty play so the Lions had their "99" call, in which they just waded in when things got tough, in the knowledge that the ref wouldn't discipline the SA players properly.
What really rubbed the salt into the wounds for the SA rugby hierarchy was that the crowds were segregated and the blacks were all cheering on the British Lions!