In my experience you spend more time going uphill than downhill so fixed bikes are very efficient machines
That's physics innit ... but the speed differential between fixed and free is greater downhill than uphill is also a factor.
Let's assume that geared bikes descend 8 x faster than they climb, while fixed descend only 5 x their uphill speed, and let's suppose the fixed rider is 20% faster up the hills. Assume too that on the flat both ride at the same speed. For a ride with 400m of climbing over 20 km (ie pretty damned hilly) where the fixed rider averages 12 km/h, and 400m of descending also over 20 km the numbers pan out as follows:
Before reading on ... do these assumptions seem realistic to you?
Fixed uphill: 20 km at 12 km/h = 1h40m
Fixed downhill: 20 km at 60 km/h = 0h20m
Total climbing+descending time = 2 hrs
Geared uphill: 20 km at 10 km/h = 2h0m
(you can see which way this is going!)
Geared downhill: 20 km at 80 km/h (wow!!!) = 0h15m
Total climbing+descending time = 2h15m
It turns out that the fixed bike really is more efficient on this scenario! Small gains on the climbs outweigh big gains on the descents. While the geared bike descends 80/60 faster than the fixie (+33%) it climbs at 10/12 of the fixie speed (-17%) and all the extra time spent climbing means the fixie wins.
OK I picked the numbers partly to make the arithmetic easy, but are they really that far from reality? It suggests to me that the downhill disadvantage of fixed wheel, while real, also has a significant psychological element that amplifies the feeling of being slower. The old cycling folklore that "fixed or gears ... there's not really that much difference" has more than a grain of truth.
Personally, I really enjoy the even, steady pace when riding fixed which seems to me a very efficient way to cover long distances. Just don't try too hard to keep up on the descents!