Wrap up warmer than you would normally, maybe ride near the back (so you keep moving, albeit slowly) and enjoy the journey.
If "not serious cyclists" frustrate you, if non-technical cyclists frustrate you or just slow cyclists frustrate you, this may not be the ride for you.
Treat it as an opportunity to help people become, more serious, more technical and quicker.
I did a Brighton FNRTTC about 6 years ago, it was November, very wet, very slow and very cold..... oh yes.. and great fun.