My advice is that you carry out your own tests -on your bike, on your roads- to see what is going on and whether it might be significant or not. My take on it is that provided you are not demonstrably slower using the wider tyres (eg during coastdown and rollout tests), you are probably better off overall in any kind of distance event. This relies on another imponderable (which may vary greatly from one rider to another) which is that after a long day in the saddle "more comfort = more speed".
This, in spades.
I switched to really wide, 37mm (but thin) tyres because I needed to run lowish pressures to cushion a damaged wrist. I also noticed that the wider tyres didn't tramline, rolled straight over broken surfaces without deviating and laughed at most potholes.
A guy on a sportif in the USA died because his (narrow) tyres got caught in the cracks inbetween concrete slabs on the road. His family sued the race organisers and lost.
Unless the wide tyres are costing you speed, why would you ride on skinnies?