So, I moved my nextcloud directory from a local SATA drive to a NFS mount from a nvme array on a 10G network
“I just need to change /docker/nextcloud to /mnt/nfs/nextcloud in the docker-compose.yml, what’s the issue, i do it live” - i tell myself
So i stop the container, copy /docker/nextcloud to /mnt/nfs/nextcloud, then edit the docker-compose.yml… and… because I’m doing it during a phone call without paying too much attention i change the main directory to /docker
I rebuild the container and I immediately hear a flood of telegram notifications from my uptime-kuma bot… oh oh…
Looks like the nextcloud docker image has an initialization script that if it doesn’t find the files in the directory, it will delete everything and install a fresh copy of nextcloud… so it deleted everything on my server
Luckily i had a very recent full borg backup and i’m restoring it (i kinda love-hate borg, i always forget the restore commands when in panic and the docs are a bit cryptic for me)
Lessons learned:
-
always double check everything
-
offsite backups are a must (if i accidentally wrote
/as path, i would have lost also the borg backups!) -
offsite backups should not be permanently mounted, otherwise they would have been wiped as well
-
learn how to use and schedule filesystem snapshots, so the recovery wouldn’t take ages like it’s taking right now (2+ hours and i’m not even half way…)
You can label your devices. When formatting, do
mkfs.ext4 -l my-descriptive-name /dev/whatever. Now, refer to it exclusively by/dev/disk/by-label/my-descriptive-name. Much harder to mix uphomeandswapthansdc2andsdc3(or, for that matter, two UUIDs).