I would say that the intended meaning of "less/fewer clothes", in this context, is the same as saying "less clothing".
For example, a shorter skirt, or a thinner jumper, is technically the same number of garments - but would fit the "everyday usage" meaning of "less clothes".
Hence "less" is more correct in a practical sense! (if that concept actually exists ...)