The
SPOF that is the main homeserver, which runs the virtual firewall and the DNS/DHCP servers on the network hosed its root partition. When I
fscked it, every single inode was orphaned
This of course happened mid-morning on Thursday, at which point we were working from home.
Server duly rebuilt, re-configured (albeit by manually applying settings from ansible scripts) and bind zone files, samba database etc. restored from backups.
I've got a second DNS server running and I'm just applying some changes that will DHCP into a failover pair. That leaves baking in some magic to backup up the firewall virtual machine and restore it onto the backup server as part of the daily backup activity, plus another script that runs periodically on the backup server and fires up the restored firewall VM if the main home server is not reachable. So a repeat instance will mean moving a couple of network cables and turning on the backup server, instead of taking a day off work.