infrastruktur/tf

Terraform'ed Infrastructure for Chaos-West TV.

Not included

This repository just sets up all of our services on Docker Swarm. It does not take care of setting up the underlying bare-metal server, or configuring Docker Swarm itself. This is done by hand. The server is running Arch Linux and was ordered, installed and configured initially by hand.

Maintenance (updates, SSH key management) is still done by hand.

Structure

.
├── README.md         # You are here
├── modules           # Terraform modules
│   ├── swarm         # Swarm Stacks
│   │   ├── traefik   # Traefik Ingress
├── stacks            # Terraform stacks - combinations of modules
│   ├── ax41-1        # Stack for ax41-1, big 'ol bare-metal server at Hetzner

The Terraform code is split up into modules so that they are small and easy to understand. The modules are then combined into stacks, which are the actual Terraform configurations that are applied. Currently, there is only one stack, ax41-1, which is the stack for the big 'ol bare-metal server at Hetzner.

The Terraform state is stored in a versioned S3 bucket. The bucket is located in thunfisch's private AWS account. A dedicated AWS credential pair with minimal permissions is saved in the aws_key.enc file. This file is encrypted using sops and age.

Usage

Prerequisites

• Terraform 1.5.5 (not newer because Hashicorp changed to non-free licensing)
• age (for encrypting & decrypting secrets)
• sops (for encrypting & decrypting secrets)
• pre-commit (only required when commiting changes)

There is a asdf configuration in the repository, so you can run asdf install to install all the required tool versions. You might need to install the asdf plugins first.

Deploying

export SOPS_AGE_KEY_FILE=/path/to/your/private/key
eval $(sops -d aws_key.enc) # Sets the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
cd stacks/ax41-1
terraform init
terraform apply

Check, double-check and triple-check the changes that Terraform wants to apply. If everything looks good, type yes and hit enter. Terraform will then apply the changes.

Commiting changes

This project uses pre-commit to automatically run some checks before commiting changes. Run pre-commit install to install the git hook. Now, pre-commit will run automatically before every commit, hopefully preventing you from commiting stupid things.

Secrets

Secrets are encrypted using sops and age. The public keys for the age encryption are stored in the repository, so that anyone can encrypt secrets for the repository. Your private key is stored in your password manager and is only available to you.

If you want to roll out changes to the actual infrastructure, you need to be able to decrypt the secrets. To do so, your private key needs to be used as a recipient for the age encryption by someone that previously had access to the secrets. If you don't have access to the secrets, ask someone who does. If you don't know who that is, you probably shouldn't be rolling out changes to the infrastructure.

Decrypting Secrets

To be able to decrypt secrets, sops needs to know where your private key is stored. This is done by setting the SOPS_AGE_KEY_FILE environment variable to the path of your private key or passing it directly by setting the SOPS_AGE_KEY environment variable.

You can manually decrypt a file using sops:

sops --decrypt stuff.enc.yaml > stuff.yaml

Terraform is using the carlpett/sops provider to decrypt secrets. This provider is configured to use these environment variables, so you don't need to do anything else.

Encrypting Secrets

To encrypt secrets, you need to have the public keys of the recipients. These are stored in the repository, so you can just use them. The public keys are stored in the sops-age-recipients.txt file. To encrypt a secret, load the public keys from this file to your SOPS_AGE_RECIPIENTS environment variable and then use sops to encrypt the secret.

export SOPS_AGE_RECIPIENTS="$(cat sops-age-recipients.txt)"
sops --encrypt stuff.yaml > stuff.enc.yaml

Modifying Secrets

To modify secrets, run sops stuff.enc.yaml and edit the file with your default $EDITOR. When you save the file, sops will automatically decrypt and re-encrypt the file. Alternatively, you can also use sops --decrypt stuff.enc.yaml > stuff.yaml to decrypt the file and then edit it. When you're done, use sops --encrypt stuff.yaml > stuff.enc.yaml to re-encrypt the file. Make sure to remove the unencrypted file afterwards.

Onboarding new people

Onboarding new people mostly requires setting up access to the encrypted files and the server. Before onboarding new people, make sure that they are kind of trustworthy and that they know what they are doing. The tools used here and Linux systems in general should not be anything new to them.

Secrets access

To onboard a new person, you need to add their public age key to the repository. To generate one, the person needs to run age-keygen -o keys.txt to generate a new key pair. The keys.txt file needs to be placed in the age configuration directory for the used system, e.g. $XDG_CONFIG_HOME/sops/age/keys.txt, $HOME/Library/Application Support/sops/age/keys.txt or %AppData%\sops\age\keys.txt. Then, add the public key to the sops-age-recipients.txt file in this repository. Commit the changes and push them to the repository.

Now, some person that already has had access to all the secrets needs to re-encrypt them with the updated sop-age-recipients. To do so, run sops --rotate --in-place stuff.enc.yaml for every encrypted file in the repository. Commit the changes and push them to the repository.

Server access

Server access is done using SSH keys. To onboard a new person, a person that already has access needs to add the new person's public SSH key to the server. The public key needs to be added to the ~/.ssh/authorized_keys file of the hausmeister user on the server. The new person can then log in using their private SSH key.