13 min read

Learning Objectives

By the end of this module, you will be able to:

  • Automate API token rotation with the Auth API: mint, verify, store and revoke tokens per service and environment
  • Manage TaskBoard's users, groups, privileges and resource shares with `ionoscloud_user`, `ionoscloud_group` and `ionoscloud_share`
  • Deliver database credentials from Terraform outputs into Kubernetes Secrets without committing them anywhere
  • Run NSG rules through the infrastructure pipeline with drift detection, and protect MKS workloads where NSGs do not apply
  • Distribute and rotate SSH keys for Linux servers with Terraform and cloud-init

Unit 5.2: Security Automation

Introduction

TaskBoard now has a CI user with a Cloud API token, a pull token for the registry, Object Storage keys, two database passwords, and SSH access to the worker VM. Each of these was created once, by hand, and nobody remembers when. That is how credentials leak: a token from a departed colleague, a password in a .tfvars file, a firewall rule someone clicked into the DCD during an incident.

In this unit you turn security into code that runs on a schedule. You rotate API tokens with the Auth API, model access with Terraform-managed users and groups, feed Kubernetes Secrets from Terraform outputs, keep NSG rules in the pipeline, and handle SSH keys the way IONOS CLOUD injects them. In the Code Lab you rotate TaskBoard's CI token, add an NSG rule through Terraform, and wire the PostgreSQL credentials into a Kubernetes Secret.

1. API Token Lifecycle Automation

1.1 Inventory Before Rotation

curl --silent "https://api.ionos.com/auth/v1/tokens" \
  --header "Authorization: Bearer $IONOS_TOKEN" \
  | jq -r '.tokens[] | "\(.id)  created \(.createdDate)  expires \(.expirationDate)"'

GET /tokens lists the tokens of the authenticated user with id, createdDate and expirationDate. Users with several contracts add the X-Contract-Number header. The token value itself never appears again: the Token Manager shows it only once, on generation. A user can hold up to 100 tokens, so a rotation job that never deletes old tokens eventually fails.

1.2 A Rotation Job

Each service gets its own IONOS CLOUD user (Section 2), and the rotation job runs under that user's current token:

import base64
import json
import os
import subprocess

import requests

AUTH = "https://api.ionos.com/auth/v1"
TTL = 30 * 24 * 3600  # seconds; the API accepts 60 to 31536000


def key_id(jwt: str) -> str:
    header = jwt.split(".")[0]
    header += "=" * (-len(header) % 4)
    return json.loads(base64.urlsafe_b64decode(header))["kid"]


def main() -> None:
    old = os.environ["IONOS_TOKEN"]
    resp = requests.get(f"{AUTH}/tokens/generate", params={"ttl": TTL},
                        headers={"Authorization": f"Bearer {old}"}, timeout=30)
    resp.raise_for_status()
    new = resp.json()["token"]
    auth_new = {"Authorization": f"Bearer {new}"}

    # 1. prove the new token works before anything depends on it
    requests.get("https://api.ionos.com/cloudapi/v6/datacenters", params={"depth": 0},
                 headers=auth_new, timeout=30).raise_for_status()

    # 2. store it (stdin, so the value never appears in the process list)
    subprocess.run(["gh", "secret", "set", "IONOS_TOKEN"], input=new, text=True, check=True)

    # 3. revoke the old token by its key ID, then remove expired tokens
    requests.delete(f"{AUTH}/tokens/{key_id(old)}", headers=auth_new, timeout=30).raise_for_status()
    requests.delete(f"{AUTH}/tokens", params={"criteria": "EXPIRED"},
                    headers=auth_new, timeout=30).raise_for_status()
    print("rotated, new token expires in", TTL // 86400, "days")


if __name__ == "__main__":
    main()

The path parameter of DELETE /tokens/{tokenId} is the Key ID from the token's header section. Deleting a token deactivates it immediately, even before it expires, so the order matters: generate, verify, store, then revoke. Users with 2-Factor Authentication must use 2FA-secured tokens to delete tokens; run rotation under dedicated service users.

1.3 Scheduling and Cleanup

Save the job as .github/workflows/rotate-token.yml:

on:
  schedule:
    - cron: "0 3 * * 1"
jobs:
  rotate:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: pip install requests && python ops/rotate_token.py
        env:
          IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
          GH_TOKEN: ${{ secrets.SECRETS_ADMIN_PAT }}

A weekly job with a 30-day ttl leaves enough margin for a failed run. The script's last call uses criteria=EXPIRED, which deletes expired tokens for the contract. The other values are ALL and CURRENT, so never build that parameter from input. GH_TOKEN must be allowed to write repository secrets; the default workflow token is not.

2. IAM as Code

2.1 Users and Groups per Service

Rights in IONOS CLOUD are assigned through groups. A group carries privileges, and a user inherits them from every group it belongs to:

resource "ionoscloud_group" "taskboard_ci" {
  name                           = "taskboard-ci"
  create_datacenter              = false
  create_k8s_cluster             = true
  create_network_security_groups = true
  access_and_manage_monitoring   = true
  access_and_manage_logging      = true
  manage_dbaas                   = true
}

resource "ionoscloud_user" "taskboard_ci" {
  first_name          = "taskboard"
  last_name           = "ci"
  email               = "taskboard-ci@example.com"
  password_wo         = random_password.ci.result
  password_wo_version = "1"
  administrator       = false
  force_sec_auth      = false
  active              = true
  group_ids           = [ionoscloud_group.taskboard_ci.id]
}

resource "random_password" "ci" {
  length  = 24
  special = true
}

password_wo is write-only: it is not saved in state, requires Terraform 1.11 or later, and is updated when you raise password_wo_version. Keep administrator = false for every service user. Administrators do not need to be managed in groups, because they automatically have access to all resources of the contract, so an administrator token defeats every other control in this unit.

2.2 Sharing Resources

Privileges let a group create resources. Access to existing resources comes from shares:

resource "ionoscloud_share" "ci_datacenter" {
  group_id        = ionoscloud_group.taskboard_ci.id
  resource_id     = ionoscloud_datacenter.taskboard.id
  edit_privilege  = true
  share_privilege = false
}

resource "ionoscloud_share" "ci_cluster" {
  group_id        = ionoscloud_group.taskboard_ci.id
  resource_id     = ionoscloud_k8s_cluster.taskboard.id
  edit_privilege  = true
  share_privilege = false
  depends_on      = [ionoscloud_share.ci_datacenter]
}

Shares control access to data centers, snapshots, images, IP blocks, Cross Connects, backup units and Kubernetes clusters, with three authorization levels: Read, Edit and Sharing. Read is implicit as soon as a group is assigned to a resource. Creating several shares at the same time leads to an error, which is why the second share uses depends_on; terraform apply -parallelism=1 is the alternative.

For TaskBoard, use one group per service and environment: taskboard-ci-dev with edit rights on the dev data center only, taskboard-ci-prod on production, and a read-only taskboard-monitor group for scripts such as the Activity Log fetcher from Unit 5.1, which also needs access_activity_log = true.

3. Kubernetes Secrets from Terraform Outputs

3.1 The Pipeline Pattern

Unit 2.5 kept the PostgreSQL password out of Terraform state with a write-only argument and exported only the host. The deploy job combines both into a Secret:

PGHOST="$(terraform -chdir=infra output -raw pg_host)"

kubectl -n taskboard create secret generic taskboard-db \
  --from-literal=PGHOST="$PGHOST" \
  --from-literal=PGUSER=taskboard \
  --from-literal=PGPASSWORD="$TF_VAR_pg_password" \
  --from-literal=PGSSLMODE=verify-full \
  --dry-run=client -o yaml | kubectl apply -f -

--dry-run=client -o yaml | kubectl apply makes the step idempotent, so the same job creates or updates the Secret. Nothing touches disk or Git. Avoid the Kubernetes Terraform provider for credentials you deliberately kept out of state: a Secret resource stores its data in the state file.

3.2 External Secrets and Rotation

When the password lives in a secret store, let the cluster pull it instead of pushing it from CI:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: taskboard-db
  namespace: taskboard
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: taskboard-db
  data:
    - secretKey: PGPASSWORD
      remoteRef: { key: secret/taskboard/prod, property: pg_password }

A rotation then has three steps: write the new password to the store, apply Terraform with password_version raised by one, and restart the pods with kubectl rollout restart deploy/taskboard-api after the operator refreshed the Secret. Environment variables are read only at container start. On MKS, encryption of Secret data at rest can be enabled and configured.

4. NSG Automation in the Pipeline

4.1 Rules as Data

Unit 2.3 introduced ionoscloud_nsg and ionoscloud_nsg_firewallrule. In production, describe the rules as data so that a pull request shows exactly which port opens for which source:

variable "app_ingress" {
  type = map(object({ source_ip = string, port = number }))
  default = {
    http-from-alb = { source_ip = "10.10.1.6", port = 80 }
  }
}

resource "ionoscloud_nsg_firewallrule" "app" {
  for_each         = var.app_ingress
  datacenter_id    = ionoscloud_datacenter.taskboard.id
  nsg_id           = ionoscloud_nsg.app.id
  name             = each.key
  type             = "INGRESS"
  protocol         = "TCP"
  source_ip        = each.value.source_ip
  port_range_start = each.value.port
  port_range_end   = each.value.port
}

The CI user's group needs create_network_security_groups. protocol cannot be modified after creation; changing it forces a new rule.

4.2 Drift Detection

A rule clicked into the DCD during an incident survives until someone notices. A scheduled plan notices for you:

terraform plan -input=false -detailed-exitcode -lock=false > drift.txt
case $? in
  0) echo "no drift" ;;
  2) echo "drift detected"; gh issue create --title "NSG drift" --body-file drift.txt ;;
  *) echo "plan failed"; exit 1 ;;
esac

Exit code 2 means the real configuration differs from code. Resolve it by changing code and applying, never by leaving the manual change in place.

4.3 Where NSGs Do Not Reach

NSG members are VMs and NICs. The ALB and NLB are not members, and Managed Kubernetes node pool nodes are excluded from NSGs. For pod-to-pod traffic on MKS, use Kubernetes NetworkPolicies, which the Calico CNI plugin on MKS supports:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-ingress-only
  namespace: taskboard
spec:
  podSelector: { matchLabels: { app: taskboard-api } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress } }
      ports:
        - { port: 8000, protocol: TCP }

5. SSH Keys

5.1 Injection at Provisioning

variable "team_ssh_keys" {
  type = list(string)   # public keys, one per team member
}

resource "ionoscloud_vcpu_server" "worker" {
  # ... arguments and volume from Unit 2.1 ...
  image_name = data.ionoscloud_image.ubuntu.id
  ssh_keys   = var.team_ssh_keys
}

The Cloud API injects public keys when a volume is created from an IONOS CLOUD Linux image; Windows images, snapshots, private images and MS SQL images do not support injection. ssh_keys is immutable and may only be set at creation. The keys saved in the DCD's SSH Key Manager (up to 100 per user) are a DCD convenience; the Cloud API does not manage that store, so keep the team key list in code.

5.2 Rotation

Because injected keys cannot change on a running server, manage authorized_keys with cloud-init (Unit 2.1) for new servers and with your configuration management for running ones:

#cloud-config
users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... alice@taskboard
      - ssh-ed25519 AAAA... bob@taskboard

To remove a departed colleague's key, delete it from the variable, roll new worker VMs or run the configuration job, and confirm with ssh-keygen -lf on the server.

API Reference Quick Card

Key API endpoints for this unit's topic:

Method Endpoint Description
GET https://api.ionos.com/auth/v1/tokens/generate?ttl=<seconds> Generate a new token
GET https://api.ionos.com/auth/v1/tokens List the user's tokens
DELETE https://api.ionos.com/auth/v1/tokens/{tokenId} Revoke a token by Key ID
DELETE https://api.ionos.com/auth/v1/tokens?criteria=EXPIRED Remove expired tokens
POST /datacenters/{datacenterId}/securitygroups/{securityGroupId}/rules Add an NSG rule

Base URL: https://api.ionos.com/cloudapi/v6 (Auth API as listed) Authentication: Authorization: Bearer <token>

Code Lab

Objective: Rotate TaskBoard's CI token, add an NSG rule through Terraform, and wire the PostgreSQL credentials into a Kubernetes Secret.

Prerequisites:

  • A dedicated CI user and its token in IONOS_TOKEN
  • The Terraform project with the NSG from Unit 2.3 and PostgreSQL from Unit 2.5
  • kubectl access to the TaskBoard namespace, jq, gh and Python 3 with requests

Step 1: List the current tokens

curl --silent "https://api.ionos.com/auth/v1/tokens" -H "Authorization: Bearer $IONOS_TOKEN" | jq '.tokens | length'

Expected output:

3

Step 2: Rotate the token

python ops/rotate_token.py

Expected output:

✓ Set Actions secret IONOS_TOKEN for <org>/taskboard
rotated, new token expires in 30 days

Step 3: Confirm the old token is revoked

curl --silent -o /dev/null -w "%{http_code}\n" \
  "https://api.ionos.com/cloudapi/v6/datacenters" -H "Authorization: Bearer $IONOS_TOKEN"

Expected output:

401

Load the new value from your secret store into IONOS_TOKEN before you continue.

Step 4: Add an NSG rule through a pull request

Add ssh-from-vpn = { source_ip = "10.10.9.10", port = 22 } to var.app_ingress (Section 4.1), push a branch and open a pull request.

terraform plan

Expected output:

  # ionoscloud_nsg_firewallrule.app["ssh-from-vpn"] will be created
Plan: 1 to add, 0 to change, 0 to destroy.

Step 5: Check for drift

After the merge and apply, run the script from Section 4.2. Expected output:

no drift

Step 6: Create the database Secret

export TF_VAR_pg_password="$(vault kv get -field=pg_password secret/taskboard/dev)"
bash ops/db-secret.sh    # the commands from Section 3.1
kubectl -n taskboard get secret taskboard-db -o jsonpath='{.data.PGHOST}' | base64 -d

Expected output:

secret/taskboard-db configured
<cluster-dns-name>

Validation Checklist:

  • [ ] The old token returns 401, the new one works from CI
  • [ ] The NSG rule exists only because of a merged pull request
  • [ ] The drift check reports no drift
  • [ ] taskboard-db exists in the cluster and in no file in Git

Cleanup:

unset TF_VAR_pg_password
history -c

Keep the rotation workflow; remove the ssh-from-vpn rule with another pull request if you do not use the VPN.

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. Revoking before storing

    • Problem: The rotation job deletes the old token, then fails to write the new one, and every pipeline returns 401.
    • Why it happens: Deletion takes effect immediately, and the new token value is shown only once.
    • Fix: Generate, verify and store first, delete last (Section 1.2); keep a break-glass procedure to mint a token with the user's password.
  2. Service tokens from an administrator account

    • Problem: A leaked CI token deletes resources in a contract-wide scope.
    • Why it happens: Administrators have access to all resources of the contract, regardless of groups and shares.
    • Fix: Create a non-administrator user per service and environment, grant privileges through a group and resource access through ionoscloud_share.
  3. Expecting NSGs to isolate pods

    • Problem: You add an NSG rule for the database port, but pods on MKS still reach each other on every port.
    • Why it happens: Managed Kubernetes node pool nodes are excluded from NSGs.
    • Fix: Use Kubernetes NetworkPolicies for pod traffic (Section 4.3) and NSGs for VMs and their NICs.

Summary

You can now run TaskBoard's credentials and access rules as code. Tokens rotate on a schedule with a short ttl, service users get only the privileges and shares they need, database passwords flow from the secret store into Kubernetes without touching Git or state, NSG rules change only through pull requests with drift detection, and SSH access follows the team key list in code.

Key Points:

  • Rotate tokens in the order generate, verify, store, revoke; deletion is immediate
  • One non-administrator user per service and environment, rights through groups, access through shares
  • Feed Kubernetes Secrets from Terraform outputs and the secret store, never from files in Git
  • NSGs cover VMs and NICs; use NetworkPolicies for pods on MKS
  • Injected SSH keys are immutable; manage rotation with cloud-init and configuration management

Important Terminology:

  • Key ID: The token identifier in the JWT header, used by DELETE /tokens/{tokenId}
  • Privilege: A group right such as create_k8s_cluster that allows creating a resource type
  • Share: A group's Read, Edit or Sharing access to an existing resource
  • Write-only argument: A Terraform argument such as password_wo that is never stored in state
  • Drift: A difference between the real configuration and the code, detected with plan -detailed-exitcode

Next Steps

Continue Learning: Unit 5.3: GitOps and Deployment Operations

Related Topics: