infrastruktur/tf

Terraform'ed Infrastructure for Chaos-West TV.

Not included

This repository just sets up all of our services on Docker Swarm. It does not take care of setting up the underlying bare-metal server, or configuring Docker Swarm itself. This is done by hand. The server is running Arch Linux and was ordered, installed and configured initially by hand.

Maintenance (updates, SSH key management) is still done by hand.

Structure

.
├── README.md         # You are here
├── modules           # Terraform modules
│   ├── swarm         # Swarm Stacks
│   │   ├── traefik   # Traefik Ingress
├── stacks            # Terraform stacks - combinations of modules
│   ├── ax41-1        # Stack for ax41-1, big 'ol bare-metal server at Hetzner

The Terraform code is split up into modules so that they are small and easy to understand. The modules are then combined into stacks, which are the actual Terraform configurations that are applied. Currently, there is only one stack, ax41-1, which is the stack for the big 'ol bare-metal server at Hetzner.

The Terraform state is stored in a versioned S3 bucket. The bucket is located in thunfisch's private AWS account. A dedicated AWS credential pair with minimal permissions is saved in the aws_key.enc file. This file is encrypted using sops and age.

Usage

Prerequisites

• Terraform 1.5.5 (not newer because Hashicorp changed to non-free licensing)
• age (for encrypting & decrypting secrets)
• sops (for encrypting & decrypting secrets)
• pre-commit (only required when commiting changes)

There is a asdf configuration in the repository, so you can run asdf install to install all the required tool versions. You might need to install the asdf plugins first.

Deploying

export SOPS_AGE_KEY_FILE=/path/to/your/private/key
eval $(sops -d aws_key.enc) # Sets the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
cd stacks/ax41-1
terraform init
terraform apply

Check, double-check and triple-check the changes that Terraform wants to apply. If everything looks good, type yes and hit enter. Terraform will then apply the changes.

Commiting changes

This project uses pre-commit to automatically run some checks before commiting changes. Run pre-commit install to install the git hook. Now, pre-commit will run automatically before every commit, hopefully preventing you from commiting stupid things.

Secrets

Secrets are encrypted using sops and age. The public keys for the age encryption are stored in the repository, so that anyone can encrypt secrets for the repository. Your private key is stored in your password manager and is only available to you.

If you want to roll out changes to the actual infrastructure, you need to be able to decrypt the secrets. To do so, your private key needs to be used as a recipient for the age encryption by someone that previously had access to the secrets. If you don't have access to the secrets, ask someone who does. If you don't know who that is, you probably shouldn't be rolling out changes to the infrastructure.

Decrypting Secrets

To be able to decrypt secrets, sops needs to know where your private key is stored. This is done by setting the SOPS_AGE_KEY_FILE environment variable to the path of your private key or passing it directly by setting the SOPS_AGE_KEY environment variable.

You can manually decrypt a file using sops:

sops --decrypt stuff.enc.yaml > stuff.yaml

Terraform is using the carlpett/sops provider to decrypt secrets. This provider is configured to use these environment variables, so you don't need to do anything else.

Encrypting Secrets

To encrypt secrets, you need to have the public keys of the recipients. These are stored in the repository, so you can just use them. The public keys are stored in the sops-age-recipients.txt file. To encrypt a secret, load the public keys from this file to your SOPS_AGE_RECIPIENTS environment variable and then use sops to encrypt the secret.

export SOPS_AGE_RECIPIENTS="$(cat sops-age-recipients.txt)"
sops --encrypt stuff.yaml > stuff.enc.yaml

Modifying Secrets

To modify secrets, run sops stuff.enc.yaml and edit the file with your default $EDITOR. When you save the file, sops will automatically decrypt and re-encrypt the file. Alternatively, you can also use sops --decrypt stuff.enc.yaml > stuff.yaml to decrypt the file and then edit it. When you're done, use sops --encrypt stuff.yaml > stuff.enc.yaml to re-encrypt the file. Make sure to remove the unencrypted file afterwards.