The Colour Of Magic (and the second half of the story, The Light Fantastic) was an antidote to fantasy novels when it first came out, back before the rest of the Discworld novels had been written. It's not a terrible place to start, as long as you do so with the understanding that if you don't enjoy it, you shouldn't dismiss the entire Discworld because of it. If nothing else, it would be a shame to miss out on The Luggage.
There are several series-within-a-series, chiefly the City Watch (start with Guards Guards), The Witches (start with Equal Rites) and Death (start with Mort), which are all much stronger stories than TCOM/TLF. The various new-technology-comes-to-the-Discworld stories could also be considered as a series, though I reckon you need some prior Discworld experience to fully appreciate them.