Commercial products to do exactly this exist: Google for "power failure alarm".
Or roll your own: Long-life battery, suitable noise-making-device, normally-closed relay. Power the coil of the relay from a wall-wart.
Just checking I understand how this works: Mains power keeps the relay open (not completed). When mains power fails, the relay closes and completes the circuit, sending power from batter to alarm. At least that's the only way it makes sense to me, but the term "normally-closed" is not entirely clear to me.
I think you have it right.
Relay contacts can be "normally-open" where they are "open", or not conducting, "normally", or when there isn't power on the coil.
Or they can be "normally-closed" where they are "closed", or conducting, "normally", or when there isn't power on the coil.
Kim has suggested the latter, normally-closed, so it will conduct and sound the alarm when there isn't power on the coil.
For details, use a wall-wart that supplies 12 V. Any old and unused 12 V router power supply will do, or one of these:-
https://www.ebay.co.uk/itm/AC-100-240V-to-DC-12V-Power-Supply-Converter-Adapter-Charger-Transformer-UK-Plug/392189151864?hash=item5b504b8a78:g:ocIAAOSwUP5cCzlf:rk:12:pf:0 if you can't find one in your dead router box.
Wire that to a 5 pin automotive relay, such as this:-
https://www.ebay.co.uk/itm/12V-DC-40A-30A-5-Pin-Relay-Automotive-Changeover-Car-Bike-Boat-Switch/201944673743?hash=item2f04d72dcf:m:mVa8o5Q27MeNkmKe5spoq0A:rk:8:pf:0Those have 5 terminals numbered 85, 86, 87, 30 and 87a. Wire the 12 V wall wart to terminals 85 and 86, either way round. That will click when you plug in the power supply, and will click more quietly, a short time after you unplug.
Now wire a battery to a buzzer and put the relay in to break the circuit, wiring to terminals 20 and 87a, either way round. Make no connection to terminal 87. It should buzz when the power supply is unplugged, and be silent when the power supply is plugged in.