When I was building up a fixer the other year, after an evening of fettling, I'd often go to bed with paranoid thoughts of "What if I can't do it?". "What if I crash?" etc etc... It really isn't like that. You don't have to be super human to ride fixed.
The other week, I hammered down a local hill on a geared bike - bunged it in 53x12 and watched my speedo rack up the miles per hour. At the bottom I realised that after probably hundreds of descents of said hill, my fastest speed achieved down that hill was actually on fixed. Sure, my legs were spinning like crazy, but when you have no choice as to spin like mad or grind out a comedically slow cadence to get up a hill, you just do it!
As others have said up thread - if the terrain isn't violently vertiguous, I reckon I'm probably a little faster on fixed than I would be on gears.