As many have noted previously, the physical fitness is only one part of the equation. Mental fortitude is also very important, as the ability to cope with sleep deprivation and the discomfort related to spend long hours in the saddle. Efficiency at controls is also paramount.
I think that mental fortitude, as well as control efficiency is largely a function of experience. I started my Audaxing life in the summer of 2014, and during the dozen or so 200k rides, I had the feeling that I was learning something new on every ride. Until today, the same is still true for 400+ km rides.
The really long rides are particularly helpful in learning that your body is able to overcome really bad patches. Even the strongest riders have their dark moments where you feel really rotten and will wonder how on earth you will be able to complete this. I hit this point usually at around the 350 to 450 km mark. As somebody here once said very aptly: "Always remember: no matter how bad you are felling, it will get better if you continue." That's really true, but you will only get this experience on really long rides.
Another important bit to learn is pacing yourself properly. Many people start too fast (for me) on Audaxes, and it is so easy to get sucked into this and pay the price later. Happened to me more than once, until I started to keep a close eye on my heart rate. I'm trying to keep it below 125 in the first hours of any ride longer than 100km.
Speed is a completely different matter. I'm not particularly fast, and long rides don't really help improving your pace. Prior to PBP, @Frank9755 advised me to do some shorter, high intensity rides. Starting in April 2015, I at least one a week did an hour of Regent Park laps at full throttle. Over several months, my pace on the laps as well as on long rides increased.
My preparation for PBP was starting a RRtY in Sept. 2014 plus the qualifiers. It worked very well - I came around with three hours time in hand. Will stick to this diet as LEL prep.