There is absolutely a happy medium and service providers absolutely have to meet it, that's the law.
The law is anticipatory (in goods and services), which means service providers have to consider how they would include disabled people with different needs in advance of them turning up to use the service.
That could be via a Risk Assessment process, but that should be well published and its purpose made clear and access explained. The process could have different common conditions and categories of impairment e.g. visual impairment and conditions like Down Syndrome. Known risk concerns could be researched and listed to discuss. Document the discussion and agreed outcomes. The process should be made as not-onerous for the disabled person as possible, which includes keeping evidence demands to a minimum and working out if likely-existing evidence or a statement and signing it is sufficient.
If I use a non disabled changing area in my sports centre and missed an evacuation, I'd have no rights to complain or claim on insurance because I agreed I would always use those changing spaces on my PEEP. If I hadn't disclosed my disabilities when it was clear on the registration forms that I would need to and do a PEEP, then I'd also have no rights under insurance etc.
We also need orgs who have no rights to disability info to NOT demand it, I've been topping up a spreadsheet I publish about disability discrimination of all kinds (
https://docs.google.com/spreadsheets/d/1U5w4Zgdz-UgTtTSZaRmGSelTMxbnIptKW7ryUMPD5sM/edit# for anyone bored enough to look). Refusal of entry to everywhere from pubs to leisure places is not uncommon for people with physical and cognitive impairments and often staff just make shit up cos they're badly trained and supported by the organisations' owners.
For everyone with a visible condition like Down syndrome there's going to be others whose condition or impairment is less visible. If there's significant safety risks, then there is a question of whether all new members/visitors should have to have a basic safety/health check to ask specific questions (which is how my sports centre do it, they have a section on the membership form asking about mobility, sight, hearing and other likely impairments).
And given how fucking awful so many organisations are, I know many disabled people don't disclose cos they don't want the hassle and stigmatisation.