After a bout of head scratching/swearing I replaced the chain and the cogs* and that fixed it and it changes beautifully again.
Can anyone give a good mechanical reason
why that would work?
Having had the same problem, I found this (very old) thread, followed up the links and suggestions and so on, and found it very helpful. I'd already stripped off and cleaned the sprockets (and cleaned the chain, obviously), and stripped and cleaned the chain tensioner, derailleur etc. Following this thread, I had a fiddle with the cable in the shifter. That seemed to help, and for a week or so it's been OK. Now it's sticking again.
Logic says change the cable, so I popped into a shop, but they stuck a gauge on my chain and said change the chain and sprockets. The chain's probably 6-8 months old; the sprockets came with the (lightly used) bike nearly a year ago.
The Brompton replaced the Dahon I broke. That was getting new chains every 18 months maybe, doing the exact same commuting trips. It even had a new cassette once in three and a half years! The Brompton uses basically a similar chain - none of this 11-speed stuff that wears out as soon as you walk out of the shop. On my old 5-speeds, 40 years ago, I'd have expected 3-5000 miles at least before changing a chain. I'll allow that stop-start commuting probably causes more wear than sport riding, mind you.
But the killer is that worn chains and sprockets, if anything, cause a derailleur to change too often. They don't stop it changing. So what's going on? Just to confirm, the issue is the chain not dropping onto the small sprocket.
I'll measure it myself tomorrow, and it may well need a change. But I'm not sure I won't still need a cable too...