Setting up my ns4 instance

Published

This is one of the posts that is more of me writing what I did so I can do it again later and isn't geared toward the general public, you've been warned. A little bit of background. The instance in question is my bare-metal Dedibox instance. I use it for the following:

  1. Secondary DNS instance
  2. Docker registry
  3. Some static web sites.
  4. GitLab CI runner
  5. Remote Workbench instance.

A bare-metal instance is very performant, allows me to run KVM and VirtualBox VMs and I feel a little better on the security front. Although I can due a remote KVM install of Debian, so far I've been using the online installation. This makes me wonder how stock is the installation and what I'll encounter if I do an in-place upgrade of Debian to a new release. The only data I care about are the images in the registry and they are backed up weekly to a machine in my house that I have off-site backups for. /var and /home are subvolumes in a btrfs volume in a LUKS encrypted partition. I enter the encryption key using KVM over IP. I have some private information but nothing I can't just git clone back to existence. I acknowledge that it's not as secure as can be but it's good enough for me and I judge the tradeoffs to be valid.

Everything listed above is deployed using GitLab CI pipelines but there's a chicken and egg situation here that some of the services depend on each other and at the very least I can't deploy the GitLab runner using GitLab pipelines. So here's what I did.

OS installation

OS installation takes ~15 minutes but afterwards KVM over IP is disabled for another hour. I work around this limitation by opening the KVM session first and then go ahead with the OS installation. In the online.net console, install Debian using the wizard. It's pretty standard stuff but the partitioning default to a RAID-1 setup which I don't want. Instead have /boot/efi at 512 MB, /boot at 2048 MB and / using the rest on the first drive (the order is important as changes don't boot for some reason I'm not interested enough to find). Because I must have a partition on each drive, keep /data on the second using all of the space.

Bootstrapping

Remove the existing SSH host key:

ssh-keygen -R ns4.shore.co.il
ssh-keygen -R "$(dig ns4.shore.co.il)"

Connect using KVM as root and setup sudo.

apt update
apt install -y sudo
echo 'nimrod ALL=(ALL) NOPASSWD: ALL' | tee /etc/sudoers.d/nimrod

Now SSH as normal and check sudo (sudo whoami). Now remove the passwords and the SSH public key from the root account.

sudo passwd --delete nimrod
sudo passwd --delete root
sudo rm -r /root/.ssh

Setting up the encrypted partition.

sudo umount /data
sudo rm --dir /data
sudo sed -i '#/data#d' /etc/fstab
sudo apt install -y btrfs-progs cryptsetup parted
sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100%
sudo cryptsetup luksFormat /dev/sdb1
sudo cryptsetup luksOpen /dev/sdb1 data
sudo mkfs -t btrfs /dev/mapper/data
sudo mount /dev/mapper/data /mnt
sudo btrfs subvolume create /mnt/builds
sudo btrfs subvolume create /mnt/home
sudo btrfs subvolume create /mnt/var
sudo mkdir /builds
UUID="$(lsblk --output UUID --list --noheadings --nodeps /dev/sdb1)"
echo "data UUID=$UUID none luks,discard" | sudo tee -a /etc/crypttab
echo "/dev/mapper/data /builds btrfs defaults,compress=zstd,discard,relatime,subvol=builds" | sudo tee -a /etc/fstab
echo "/dev/mapper/data /home btrfs defaults,compress=zstd,discard,relatime,subvol=home" | sudo tee -a /etc/fstab
echo "/dev/mapper/data /var btrfs defaults,compress=zstd,discard,relatime,subvol=var" | sudo tee -a /etc/fstab
sudo cp --archive /home/* /mnt/home/
sudo cp --archive /var/* /mnt/var/
sudo umount /mnt
sudo cryptsetup luksClose data
sudo systemc reboot --now

The instance will now reboot, enter the encryption key in the KVM console. From the homelab repository:

cd ./Ansible
ansible-playbook bootstrap.yaml -l ns4
ansible-playbook debian_server.yaml -l ns4

Services

From the secondary-dns-docker repository:

eval $(docker remote ns4.shore.co.il)
docker-compose build
docker-compose pull
docker-compose up -d

Same as above, from the web-proxy-docker repository, branch ns4. Some secret environment variables are set in the GitLab pipeline, so for them to exist re-run the deploy job in GitLab afterwards.

From the homelab repository:

cd ./Ansible
ansible-playbook renew-certs.yaml -t ns4

Setup the Docker registry, but the cron container uses an image from the registry. Start just the registry container and afterwards run the GitLab deploy job. From the registry-docker repository:

eval $(docker remote ns4.shore.co.il)
docker-compose pull registry
docker-compose up -d registry
docker build -t registry.shore.co.il/registry-backup ./backup

Copying the registry backup to ns4 (it's going to take a long while).

scp -3Cr host01.shore.co.il:/var/backups/registry ns4.shore.co.il:/home/nimrod/

On ns4:

docker run --rm -u nobody -v /home/nimrod/registry:/registry:ro registry.shore.co.il/registry-backup restore /registry registry.shore.co.il
rm -r /home/nimrod/registry

From the gitlab-runner-docker repository (delete the old runner in the GitLab admin area):

eval $(docker remote ns4.shore.co.il)
docker-compose pull
./deploy ns4

From here on out it's just running GitLab jobs and pipelines for the static web sites. etc. Remember to run the deploy jobs for web-proxy-docker and registry-docker.

Remote workbench

Follow the instructions in the rcfiles repository and finish it off with wb -u.

Closing thoughts

Along with my previous post on my OpenBSD machine covers the more involved parts of my infrastructure. Overall, I think that I'm in a pretty good state here. My setup is cost conscious, relies very little on external services and is more pets than cattle. The coverage of infrastructure code is high and the parts that aren't covered are done rarely (like reinstalling OpenBSD which I do about once a year or reinstalling Debian which I do about every 2 years) and are either trivial or nicely documented.

Going forward, I'm in the process of merging the different *-docker repositories in to the homelab repository. Once done, I may copy the blog posts there and maintain them ongoing as changes are done.