Copy a drive on a running machine

I was close to running out of disk space on the homelab machine I use for running services with personal data that I run in my home (I feel more comfortable having physical control). I was too lazy to connect the machine to a monitor and keyboard and reboot from a thumbdrive so I tried to copy the old drive to a new drive while the machine was running. Here's what I did:

1. Stop running processes that have open files. For me it was stopping Docker containers, some services (I have most things running in containers so most of the services are ones that come with a standard OS installation).

   docker container ls | awk 'NR>1 {print $1}' | xargs docker stop
   sudo systemctl stop fwupd.service systemd-timesyncd.service docker.service docker.socket containerd.service cron.service systemd-journald.service systemd-journald.socket systemd-journald-dev-log.socket
   

2. cd to a directory on a different drive to avoid keeping the path open (cd /tmp). Remount all partitions as read-only: sudo mount --all -o remount,ro -t vfat,btrfs,ext4. This may fail and you will see an error with a path that couldn't be remounted due to open files. Run lsof /path/that/has/open/files, find the processes that still have open files and either stop that service or kill the process (stopping the service is better as it won't get restarted). Repeat until the partitions are mounted read-only. Verify with the mount command. Run sudo sync and now you should be able to copy the drive as there won't writes to it while copying and the copy will be consistent.

3. Start a session on the host using tmux and copy dd if=/dev/sda of=/dev/nvme0n1. Now we wait.

4. Once the copy is complete, run sudo sync again and echo 1 | sudo tee /sys/block/nvme0n1/device/rescan. Now we need to resize the data partition, sudo parted /dev/nvme0n1 --script resizepart 3 100% (in my case the data partition was the 3rd partition after the EFI and root partitions). Open the encrypted partition sudo cryptsetup open /dev/nvme0n1p3 _dev_nvme0n1p2 and resize the encrypted partition sudo cryptsetup resize _dev_nvme0n1p3. Last resize is the filesystem in the encrypted partition, we mount it sudo mount /dev/mapper/_dev_nvme0n1p3 /mnt and resize it sudo btrfs filesystem resize max /mnt. Umount the data partition sudo umount /mnt.

5. Let's reinstall the boot loader.

   sudo mount /dev/nvme0n1p2 /mnt
   sudo mount /dev/nvme0n1p1 /mnt/boot/efi
   for dir in dev proc sys; do sudo mount --bind "/${dir}" "/mnt/${dir}"; done
   sudo chroot /mnt
   grub-install /dev/nvme0n1
   exit
   for dir in dev proc sys; do sudo umount "/mnt/${dir}"; done
   umount /mnt/boot/efi
   umount /mnt
   

6. Because we copied the existing partitions, their UUIDs will remain the same so there's no need to update mounts. Stop the machine, remove the old drive and boot. With a little luck the UEFI BIOS will not need configuration to use the new drive and the machine will boot happily.