Unit 5.2: Security Automation
Introduction
TaskBoard now has a CI user with a Cloud API token, a pull token for the registry, Object Storage keys, two database passwords, and SSH access to the worker VM. Each of these was created once, by hand, and nobody remembers when. That is how credentials leak: a token from a departed colleague, a password in a .tfvars file, a firewall rule someone clicked into the DCD during an incident.
In this unit you turn security into code that runs on a schedule. You rotate API tokens with the Auth API, model access with Terraform-managed users and groups, feed Kubernetes Secrets from Terraform outputs, keep NSG rules in the pipeline, and handle SSH keys the way IONOS CLOUD injects them. In the Code Lab you rotate TaskBoard's CI token, add an NSG rule through Terraform, and wire the PostgreSQL credentials into a Kubernetes Secret.
1. API Token Lifecycle Automation
1.1 Inventory Before Rotation
curl --silent "https://api.ionos.com/auth/v1/tokens" \
--header "Authorization: Bearer $IONOS_TOKEN" \
| jq -r '.tokens[] | "\(.id) created \(.createdDate) expires \(.expirationDate)"'
GET /tokens lists the tokens of the authenticated user with id, createdDate and expirationDate. Users with several contracts add the X-Contract-Number header. The token value itself never appears again: the Token Manager shows it only once, on generation. A user can hold up to 100 tokens, so a rotation job that never deletes old tokens eventually fails.
1.2 A Rotation Job
Each service gets its own IONOS CLOUD user (Section 2), and the rotation job runs under that user's current token:
import base64
import json
import os
import subprocess
import requests
AUTH = "https://api.ionos.com/auth/v1"
TTL = 30 * 24 * 3600 # seconds; the API accepts 60 to 31536000
def key_id(jwt: str) -> str:
header = jwt.split(".")[0]
header += "=" * (-len(header) % 4)
return json.loads(base64.urlsafe_b64decode(header))["kid"]
def main() -> None:
old = os.environ["IONOS_TOKEN"]
resp = requests.get(f"{AUTH}/tokens/generate", params={"ttl": TTL},
headers={"Authorization": f"Bearer {old}"}, timeout=30)
resp.raise_for_status()
new = resp.json()["token"]
auth_new = {"Authorization": f"Bearer {new}"}
# 1. prove the new token works before anything depends on it
requests.get("https://api.ionos.com/cloudapi/v6/datacenters", params={"depth": 0},
headers=auth_new, timeout=30).raise_for_status()
# 2. store it (stdin, so the value never appears in the process list)
subprocess.run(["gh", "secret", "set", "IONOS_TOKEN"], input=new, text=True, check=True)
# 3. revoke the old token by its key ID, then remove expired tokens
requests.delete(f"{AUTH}/tokens/{key_id(old)}", headers=auth_new, timeout=30).raise_for_status()
requests.delete(f"{AUTH}/tokens", params={"criteria": "EXPIRED"},
headers=auth_new, timeout=30).raise_for_status()
print("rotated, new token expires in", TTL // 86400, "days")
if __name__ == "__main__":
main()
The path parameter of DELETE /tokens/{tokenId} is the Key ID from the token's header section. Deleting a token deactivates it immediately, even before it expires, so the order matters: generate, verify, store, then revoke. Users with 2-Factor Authentication must use 2FA-secured tokens to delete tokens; run rotation under dedicated service users.
1.3 Scheduling and Cleanup
Save the job as .github/workflows/rotate-token.yml:
on:
schedule:
- cron: "0 3 * * 1"
jobs:
rotate:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- run: pip install requests && python ops/rotate_token.py
env:
IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
GH_TOKEN: ${{ secrets.SECRETS_ADMIN_PAT }}
A weekly job with a 30-day ttl leaves enough margin for a failed run. The script's last call uses criteria=EXPIRED, which deletes expired tokens for the contract. The other values are ALL and CURRENT, so never build that parameter from input. GH_TOKEN must be allowed to write repository secrets; the default workflow token is not.
2. IAM as Code
2.1 Users and Groups per Service
Rights in IONOS CLOUD are assigned through groups. A group carries privileges, and a user inherits them from every group it belongs to:
resource "ionoscloud_group" "taskboard_ci" {
name = "taskboard-ci"
create_datacenter = false
create_k8s_cluster = true
create_network_security_groups = true
access_and_manage_monitoring = true
access_and_manage_logging = true
manage_dbaas = true
}
resource "ionoscloud_user" "taskboard_ci" {
first_name = "taskboard"
last_name = "ci"
email = "taskboard-ci@example.com"
password_wo = random_password.ci.result
password_wo_version = "1"
administrator = false
force_sec_auth = false
active = true
group_ids = [ionoscloud_group.taskboard_ci.id]
}
resource "random_password" "ci" {
length = 24
special = true
}
password_wo is write-only: it is not saved in state, requires Terraform 1.11 or later, and is updated when you raise password_wo_version. Keep administrator = false for every service user. Administrators do not need to be managed in groups, because they automatically have access to all resources of the contract, so an administrator token defeats every other control in this unit.
2.2 Sharing Resources
Privileges let a group create resources. Access to existing resources comes from shares:
resource "ionoscloud_share" "ci_datacenter" {
group_id = ionoscloud_group.taskboard_ci.id
resource_id = ionoscloud_datacenter.taskboard.id
edit_privilege = true
share_privilege = false
}
resource "ionoscloud_share" "ci_cluster" {
group_id = ionoscloud_group.taskboard_ci.id
resource_id = ionoscloud_k8s_cluster.taskboard.id
edit_privilege = true
share_privilege = false
depends_on = [ionoscloud_share.ci_datacenter]
}
Shares control access to data centers, snapshots, images, IP blocks, Cross Connects, backup units and Kubernetes clusters, with three authorization levels: Read, Edit and Sharing. Read is implicit as soon as a group is assigned to a resource. Creating several shares at the same time leads to an error, which is why the second share uses depends_on; terraform apply -parallelism=1 is the alternative.
For TaskBoard, use one group per service and environment: taskboard-ci-dev with edit rights on the dev data center only, taskboard-ci-prod on production, and a read-only taskboard-monitor group for scripts such as the Activity Log fetcher from Unit 5.1, which also needs access_activity_log = true.
3. Kubernetes Secrets from Terraform Outputs
3.1 The Pipeline Pattern
Unit 2.5 kept the PostgreSQL password out of Terraform state with a write-only argument and exported only the host. The deploy job combines both into a Secret:
PGHOST="$(terraform -chdir=infra output -raw pg_host)"
kubectl -n taskboard create secret generic taskboard-db \
--from-literal=PGHOST="$PGHOST" \
--from-literal=PGUSER=taskboard \
--from-literal=PGPASSWORD="$TF_VAR_pg_password" \
--from-literal=PGSSLMODE=verify-full \
--dry-run=client -o yaml | kubectl apply -f -
--dry-run=client -o yaml | kubectl apply makes the step idempotent, so the same job creates or updates the Secret. Nothing touches disk or Git. Avoid the Kubernetes Terraform provider for credentials you deliberately kept out of state: a Secret resource stores its data in the state file.
3.2 External Secrets and Rotation
When the password lives in a secret store, let the cluster pull it instead of pushing it from CI:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: taskboard-db
namespace: taskboard
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: vault
target:
name: taskboard-db
data:
- secretKey: PGPASSWORD
remoteRef: { key: secret/taskboard/prod, property: pg_password }
A rotation then has three steps: write the new password to the store, apply Terraform with password_version raised by one, and restart the pods with kubectl rollout restart deploy/taskboard-api after the operator refreshed the Secret. Environment variables are read only at container start. On MKS, encryption of Secret data at rest can be enabled and configured.
4. NSG Automation in the Pipeline
4.1 Rules as Data
Unit 2.3 introduced ionoscloud_nsg and ionoscloud_nsg_firewallrule. In production, describe the rules as data so that a pull request shows exactly which port opens for which source:
variable "app_ingress" {
type = map(object({ source_ip = string, port = number }))
default = {
http-from-alb = { source_ip = "10.10.1.6", port = 80 }
}
}
resource "ionoscloud_nsg_firewallrule" "app" {
for_each = var.app_ingress
datacenter_id = ionoscloud_datacenter.taskboard.id
nsg_id = ionoscloud_nsg.app.id
name = each.key
type = "INGRESS"
protocol = "TCP"
source_ip = each.value.source_ip
port_range_start = each.value.port
port_range_end = each.value.port
}
The CI user's group needs create_network_security_groups. protocol cannot be modified after creation; changing it forces a new rule.
4.2 Drift Detection
A rule clicked into the DCD during an incident survives until someone notices. A scheduled plan notices for you:
terraform plan -input=false -detailed-exitcode -lock=false > drift.txt
case $? in
0) echo "no drift" ;;
2) echo "drift detected"; gh issue create --title "NSG drift" --body-file drift.txt ;;
*) echo "plan failed"; exit 1 ;;
esac
Exit code 2 means the real configuration differs from code. Resolve it by changing code and applying, never by leaving the manual change in place.
4.3 Where NSGs Do Not Reach
NSG members are VMs and NICs. The ALB and NLB are not members, and Managed Kubernetes node pool nodes are excluded from NSGs. For pod-to-pod traffic on MKS, use Kubernetes NetworkPolicies, which the Calico CNI plugin on MKS supports:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-from-ingress-only
namespace: taskboard
spec:
podSelector: { matchLabels: { app: taskboard-api } }
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress } }
ports:
- { port: 8000, protocol: TCP }
5. SSH Keys
5.1 Injection at Provisioning
variable "team_ssh_keys" {
type = list(string) # public keys, one per team member
}
resource "ionoscloud_vcpu_server" "worker" {
# ... arguments and volume from Unit 2.1 ...
image_name = data.ionoscloud_image.ubuntu.id
ssh_keys = var.team_ssh_keys
}
The Cloud API injects public keys when a volume is created from an IONOS CLOUD Linux image; Windows images, snapshots, private images and MS SQL images do not support injection. ssh_keys is immutable and may only be set at creation. The keys saved in the DCD's SSH Key Manager (up to 100 per user) are a DCD convenience; the Cloud API does not manage that store, so keep the team key list in code.
5.2 Rotation
Because injected keys cannot change on a running server, manage authorized_keys with cloud-init (Unit 2.1) for new servers and with your configuration management for running ones:
#cloud-config
users:
- name: deploy
groups: [sudo]
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 AAAA... alice@taskboard
- ssh-ed25519 AAAA... bob@taskboard
To remove a departed colleague's key, delete it from the variable, roll new worker VMs or run the configuration job, and confirm with ssh-keygen -lf on the server.
API Reference Quick Card
Key API endpoints for this unit's topic:
| Method | Endpoint | Description |
|---|---|---|
GET |
https://api.ionos.com/auth/v1/tokens/generate?ttl=<seconds> |
Generate a new token |
GET |
https://api.ionos.com/auth/v1/tokens |
List the user's tokens |
DELETE |
https://api.ionos.com/auth/v1/tokens/{tokenId} |
Revoke a token by Key ID |
DELETE |
https://api.ionos.com/auth/v1/tokens?criteria=EXPIRED |
Remove expired tokens |
POST |
/datacenters/{datacenterId}/securitygroups/{securityGroupId}/rules |
Add an NSG rule |
Base URL: https://api.ionos.com/cloudapi/v6 (Auth API as listed)
Authentication: Authorization: Bearer <token>
Code Lab
Objective: Rotate TaskBoard's CI token, add an NSG rule through Terraform, and wire the PostgreSQL credentials into a Kubernetes Secret.
Prerequisites:
- A dedicated CI user and its token in
IONOS_TOKEN - The Terraform project with the NSG from Unit 2.3 and PostgreSQL from Unit 2.5
kubectlaccess to the TaskBoard namespace,jq,ghand Python 3 withrequests
Step 1: List the current tokens
curl --silent "https://api.ionos.com/auth/v1/tokens" -H "Authorization: Bearer $IONOS_TOKEN" | jq '.tokens | length'
Expected output:
3
Step 2: Rotate the token
python ops/rotate_token.py
Expected output:
✓ Set Actions secret IONOS_TOKEN for <org>/taskboard
rotated, new token expires in 30 days
Step 3: Confirm the old token is revoked
curl --silent -o /dev/null -w "%{http_code}\n" \
"https://api.ionos.com/cloudapi/v6/datacenters" -H "Authorization: Bearer $IONOS_TOKEN"
Expected output:
401
Load the new value from your secret store into IONOS_TOKEN before you continue.
Step 4: Add an NSG rule through a pull request
Add ssh-from-vpn = { source_ip = "10.10.9.10", port = 22 } to var.app_ingress (Section 4.1), push a branch and open a pull request.
terraform plan
Expected output:
# ionoscloud_nsg_firewallrule.app["ssh-from-vpn"] will be created
Plan: 1 to add, 0 to change, 0 to destroy.
Step 5: Check for drift
After the merge and apply, run the script from Section 4.2. Expected output:
no drift
Step 6: Create the database Secret
export TF_VAR_pg_password="$(vault kv get -field=pg_password secret/taskboard/dev)"
bash ops/db-secret.sh # the commands from Section 3.1
kubectl -n taskboard get secret taskboard-db -o jsonpath='{.data.PGHOST}' | base64 -d
Expected output:
secret/taskboard-db configured
<cluster-dns-name>
Validation Checklist:
- [ ] The old token returns
401, the new one works from CI - [ ] The NSG rule exists only because of a merged pull request
- [ ] The drift check reports no drift
- [ ]
taskboard-dbexists in the cluster and in no file in Git
Cleanup:
unset TF_VAR_pg_password
history -c
Keep the rotation workflow; remove the ssh-from-vpn rule with another pull request if you do not use the VPN.
Common Pitfalls
Developer mistakes to avoid with this unit's topic:
-
Revoking before storing
- Problem: The rotation job deletes the old token, then fails to write the new one, and every pipeline returns
401. - Why it happens: Deletion takes effect immediately, and the new token value is shown only once.
- Fix: Generate, verify and store first, delete last (Section 1.2); keep a break-glass procedure to mint a token with the user's password.
- Problem: The rotation job deletes the old token, then fails to write the new one, and every pipeline returns
-
Service tokens from an administrator account
- Problem: A leaked CI token deletes resources in a contract-wide scope.
- Why it happens: Administrators have access to all resources of the contract, regardless of groups and shares.
- Fix: Create a non-administrator user per service and environment, grant privileges through a group and resource access through
ionoscloud_share.
-
Expecting NSGs to isolate pods
- Problem: You add an NSG rule for the database port, but pods on MKS still reach each other on every port.
- Why it happens: Managed Kubernetes node pool nodes are excluded from NSGs.
- Fix: Use Kubernetes NetworkPolicies for pod traffic (Section 4.3) and NSGs for VMs and their NICs.
Summary
You can now run TaskBoard's credentials and access rules as code. Tokens rotate on a schedule with a short ttl, service users get only the privileges and shares they need, database passwords flow from the secret store into Kubernetes without touching Git or state, NSG rules change only through pull requests with drift detection, and SSH access follows the team key list in code.
Key Points:
- Rotate tokens in the order generate, verify, store, revoke; deletion is immediate
- One non-administrator user per service and environment, rights through groups, access through shares
- Feed Kubernetes Secrets from Terraform outputs and the secret store, never from files in Git
- NSGs cover VMs and NICs; use NetworkPolicies for pods on MKS
- Injected SSH keys are immutable; manage rotation with cloud-init and configuration management
Important Terminology:
- Key ID: The token identifier in the JWT header, used by
DELETE /tokens/{tokenId} - Privilege: A group right such as
create_k8s_clusterthat allows creating a resource type - Share: A group's
Read,EditorSharingaccess to an existing resource - Write-only argument: A Terraform argument such as
password_wothat is never stored in state - Drift: A difference between the real configuration and the code, detected with
plan -detailed-exitcode
Next Steps
Continue Learning: Unit 5.3: GitOps and Deployment Operations
Related Topics: