This is essentially what people do on ultra-races such as TCR
Only time I've slept during an audax is also on Flatlands. In a bus shelter, I got about 90 minutes in an Alpkit Hunka bivvy with a slik liner and a foil blanket as a grounsheet (concrete floor, no bench). Also had ultra-light down jacket.
Normally, ie on TCR, IndyPac, etc, I'd have a sleeping pad as well (instead of foil blanket), but didn't bring it that time as I'd originally planned to ride through. I've used it a couple of times to sleep before an audax, when I've ridden out overnight from London for Welsh rides.
SOL bivvy is on my list of things to try as would save space vs Alpkit.
Also should try a modern inflatable mat as upgrade from my £5-eBay-blue-foil-backed-foam-mat-cut-in-half. Have used the foam mat as puncture proof and to avoid extra faff of morning deflation but people tell me that is not a big issue and they certainly do pack smaller.
Would need more warmth, eg sleeping bag, on colder nights, eg May in Wales.