if the road is billiard table smooth, 23s are faster. They are more aerodynamic, too. But in the real world, on normal roads, at modest speeds, good 28s (which unlike a couple of decades ago, you can buy) roll more easily and you will be both comfier and faster.
The rougher the surface, the fatter the tyres that are likely to suit best, so plenty of folk in the UK who have 'third world' standard tarmac locally have taken to riding on 38mm tyres as standard. I think they are perhaps slower during fierce out of the saddle efforts, but I'm less inclined to those these days so if it is the case, it matters less....
cheers