Unit 3.3: CI/CD Pipelines for IONOS CLOUD
Introduction
So far you have run every step by hand: terraform apply from your laptop, docker push from your shell, kubectl apply with an admin kubeconfig. That works for one developer. For a team, every change to main must go through the same reviewed, repeatable path, and no laptop should hold production credentials.
In this unit you automate both halves of TaskBoard. The application pipeline builds, tests, pushes and deploys. The infrastructure pipeline plans on pull requests and applies on merge. You store each credential in the CI system's secret store, choose a deployment strategy, practise rollbacks, and turn TaskBoard's Terraform into modules your team can reuse. In the Code Lab you build a GitHub Actions pipeline that deploys TaskBoard on every push to main.
1. Pipeline Design
TaskBoard uses two pipelines with different triggers and credentials:
| Pipeline | Trigger | Steps | Credentials |
|---|---|---|---|
| Application | Push to main |
test, build, push image, deploy | Registry push token, deployer kubeconfig |
| Infrastructure | Pull request / merge | terraform plan, terraform apply |
IONOS_TOKEN, Object Storage keys for state |
Keep them apart. The application pipeline never needs the Cloud API token, so a compromised build step cannot create or delete resources. The infrastructure pipeline never needs the kubeconfig.
1.1 Secrets Inventory
IONOS_TOKEN Cloud API bearer token (infrastructure pipeline only)
AWS_ACCESS_KEY_ID Object Storage Access Key for the Terraform state bucket
AWS_SECRET_ACCESS_KEY Object Storage Secret Key for the Terraform state bucket
CR_HOST Registry hostname (Unit 3.1)
CR_USERNAME Push token username
CR_PASSWORD Push token password
KUBECONFIG_DEPLOYER Kubeconfig of the taskboard-deployer service account (Unit 3.2)
Generate IONOS_TOKEN for a dedicated CI user with only the privileges the infrastructure needs, and set an explicit ttl, as in Unit 1.1. The registry push token from Unit 3.1 carries pull and push. Object Storage authenticates with an Access Key and a Secret Key, not with the bearer token, so the state backend needs its own pair.
1.2 A Kubeconfig for the Pipeline
The kubeconfig from the API gives full cluster access. For CI, build one for the taskboard-deployer service account from Unit 3.2:
kubectl -n taskboard apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: taskboard-deployer-token
annotations:
kubernetes.io/service-account.name: taskboard-deployer
type: kubernetes.io/service-account-token
EOF
SA_TOKEN="$(kubectl -n taskboard get secret taskboard-deployer-token -o jsonpath='{.data.token}' | base64 -d)"
SERVER="$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}')"
CA="$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')"
kubectl config --kubeconfig=deployer.yaml set-cluster taskboard --server="$SERVER"
kubectl config --kubeconfig=deployer.yaml set clusters.taskboard.certificate-authority-data "$CA"
kubectl config --kubeconfig=deployer.yaml set-credentials deployer --token="$SA_TOKEN"
kubectl config --kubeconfig=deployer.yaml set-context taskboard --cluster=taskboard --user=deployer --namespace=taskboard
kubectl config --kubeconfig=deployer.yaml use-context taskboard
Store deployer.yaml base64-encoded as KUBECONFIG_DEPLOYER and delete the local file. If you set api_subnet_allow_list on the cluster, the runner's egress address must be in it, or kubectl cannot reach the API server.
2. GitHub Actions: The Application Pipeline
# .github/workflows/deploy.yml
name: deploy
on:
push:
branches: [main]
concurrency:
group: deploy-production
cancel-in-progress: false
jobs:
build-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Test
run: |
pip install -r api/requirements.txt -r api/requirements-dev.txt
pytest api/tests
- name: Log in to the registry
env:
CR_HOST: ${{ secrets.CR_HOST }}
CR_USERNAME: ${{ secrets.CR_USERNAME }}
CR_PASSWORD: ${{ secrets.CR_PASSWORD }}
run: echo "$CR_PASSWORD" | docker login "$CR_HOST" --username "$CR_USERNAME" --password-stdin
- name: Build and push
env:
CR_HOST: ${{ secrets.CR_HOST }}
run: |
SHA="${GITHUB_SHA::12}"
for svc in api web; do
docker build -t "$CR_HOST/taskboard/$svc:$SHA" "$svc/"
docker push "$CR_HOST/taskboard/$svc:$SHA"
done
- name: Deploy to MKS
env:
CR_HOST: ${{ secrets.CR_HOST }}
KUBECONFIG_DEPLOYER: ${{ secrets.KUBECONFIG_DEPLOYER }}
run: |
echo "$KUBECONFIG_DEPLOYER" | base64 -d > "$RUNNER_TEMP/kubeconfig"
export KUBECONFIG="$RUNNER_TEMP/kubeconfig"
SHA="${GITHUB_SHA::12}"
kubectl set image deploy/taskboard-api api="$CR_HOST/taskboard/api:$SHA"
kubectl set image deploy/taskboard-web web="$CR_HOST/taskboard/web:$SHA"
kubectl rollout status deploy/taskboard-api --timeout=5m
kubectl rollout status deploy/taskboard-web --timeout=5m
The concurrency group lets only one deployment run at a time, and rollout status fails the job when the new pods do not become ready. kubectl set image changes only the image; the manifests themselves stay in Git and are applied when they change. Pushing both images in sequence keeps a single runner well inside the registry's per-client limits of 100 concurrent connections and 60 requests per second.
3. GitLab CI: The Equivalent Pipeline
# .gitlab-ci.yml
stages: [test, build, deploy]
variables:
SHA: $CI_COMMIT_SHORT_SHA
test:
stage: test
image: python:3.12-slim
script:
- pip install -r api/requirements.txt -r api/requirements-dev.txt
- pytest api/tests
build:
stage: build
image: docker:27
services: [docker:27-dind]
before_script:
- echo "$CR_PASSWORD" | docker login "$CR_HOST" --username "$CR_USERNAME" --password-stdin
script:
- for svc in api web; do docker build -t "$CR_HOST/taskboard/$svc:$SHA" "$svc/"; docker push "$CR_HOST/taskboard/$svc:$SHA"; done
rules:
- if: $CI_COMMIT_BRANCH == "main"
deploy:
stage: deploy
image: alpine:3.20
resource_group: production
before_script:
- apk add --no-cache curl
- curl -sLo /usr/local/bin/kubectl "https://dl.k8s.io/release/v1.34.0/bin/linux/amd64/kubectl" && chmod +x /usr/local/bin/kubectl
- echo "$KUBECONFIG_DEPLOYER" | base64 -d > /tmp/kubeconfig
script:
- export KUBECONFIG=/tmp/kubeconfig
- kubectl set image deploy/taskboard-api api="$CR_HOST/taskboard/api:$SHA"
- kubectl rollout status deploy/taskboard-api --timeout=5m
rules:
- if: $CI_COMMIT_BRANCH == "main"
resource_group is GitLab's equivalent of the GitHub concurrency group. Define CR_PASSWORD and KUBECONFIG_DEPLOYER as masked, protected variables. Keep the kubectl version within one minor version of your cluster.
4. Terraform in CI/CD
4.1 Plan on Pull Request, Apply on Merge
# .github/workflows/infra.yml
name: infra
on:
pull_request:
paths: ["infra/**"]
push:
branches: [main]
paths: ["infra/**"]
concurrency:
group: terraform-production
cancel-in-progress: false
permissions:
contents: read
pull-requests: write
jobs:
terraform:
runs-on: ubuntu-latest
defaults:
run:
working-directory: infra
env:
IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- run: terraform init -input=false
- run: terraform plan -input=false -no-color -out=tfplan
- name: Comment the plan
if: github.event_name == 'pull_request'
env:
GH_TOKEN: ${{ github.token }}
run: |
terraform show -no-color tfplan > plan.txt
gh pr comment ${{ github.event.pull_request.number }} --body-file plan.txt
- name: Apply
if: github.event_name == 'push'
run: terraform apply -input=false tfplan
The provider reads IONOS_TOKEN from the environment, and the s3 backend from Unit 1.2 reads the Object Storage keys from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The terraform-production concurrency group serializes runs, which is the safe default until you have verified state locking against your bucket. Keep the provider pinned to 6.7.39 and commit .terraform.lock.hcl, so that the plan on the pull request and the apply on main use the same release.
Each apply waits for the asynchronous IONOS CLOUD requests. A cluster or node pool can take a long time, so do not lower the provider timeouts for CI. If a run fails halfway, the state already holds the finished resources; rerun the pipeline after you fix the cause.
4.2 Managing Several Environments
Give each environment its own state key and its own workflow job, for example dev/terraform.tfstate and prod/terraform.tfstate. Protect the production job with a GitHub environment that requires a reviewer, so a merge alone cannot change production.
5. Deployment Strategies and Rollback
5.1 Rolling Update and Recreate
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
RollingUpdate is the Kubernetes default: new pods start before old ones stop, and with maxUnavailable: 0 capacity never drops. Recreate stops all old pods first, which you need only when two versions cannot run side by side, for example during a database schema change that breaks the old version. Canary and blue-green releases need either a second Deployment behind the same Service with fewer replicas, or a controller such as Argo Rollouts; Unit 5.3 covers GitOps with ArgoCD.
5.2 Rolling Back the Application
kubectl rollout history deploy/taskboard-api
kubectl rollout undo deploy/taskboard-api
kubectl rollout undo deploy/taskboard-api --to-revision=3
Because every image has a Git SHA tag, a rollback restores a known build. A moving tag such as latest would make the previous revision point at the same, broken image.
5.3 Recovering Terraform State
Unit 2.4 enabled versioning on the state bucket. A corrupted or wrongly overwritten state can be restored from an older version:
EP="https://s3.eu-central-3.ionoscloud.com"
aws s3api list-object-versions --endpoint-url "$EP" \
--bucket taskboard-tfstate-7f3a --prefix prod/terraform.tfstate
aws s3api get-object --endpoint-url "$EP" --bucket taskboard-tfstate-7f3a \
--key prod/terraform.tfstate --version-id "<version-id>" restored.tfstate
terraform state push restored.tfstate
terraform plan
Stop all pipelines first. After the push, terraform plan shows the difference between the restored state and the real infrastructure; resolve it with imports (Unit 1.2) before the next apply.
6. Terraform Modules for TaskBoard
As TaskBoard grows, the same blocks repeat per environment. Extract them into modules:
infra/
modules/
mks-cluster/
main.tf # ionoscloud_k8s_cluster, ionoscloud_k8s_node_pool
variables.tf
outputs.tf
envs/
dev/main.tf
prod/main.tf
# modules/mks-cluster/variables.tf
variable "name" { type = string }
variable "datacenter_id" { type = string }
variable "lan_id" { type = string }
variable "k8s_version" { type = string }
variable "node_count" {
type = number
default = 2
}
# modules/mks-cluster/outputs.tf
output "cluster_id" { value = ionoscloud_k8s_cluster.this.id }
# envs/prod/main.tf
module "mks" {
source = "git::https://git.example.com/platform/tf-modules.git//mks-cluster?ref=v1.2.0"
name = "taskboard-prod"
datacenter_id = ionoscloud_datacenter.taskboard.id
lan_id = ionoscloud_lan.app.id
k8s_version = var.k8s_version
node_count = 3
}
A module declares its inputs as variables and exposes only the outputs other code needs. Pin modules to a Git tag with ?ref=, exactly as you pin the provider, and release changes as new tags. Put the required_providers block with the ionoscloud source in each module so that it states the provider it was tested with.
API Reference Quick Card
Key API endpoints for this unit's topic:
| Method | Endpoint | Description |
|---|---|---|
GET |
https://api.ionos.com/auth/v1/tokens/generate?ttl=<seconds> |
Mint the CI access token |
GET |
/k8s/{k8sClusterId}/kubeconfig |
Download the admin kubeconfig for setup |
GET |
/requests/{requestId}/status |
Poll an asynchronous request |
POST |
https://api.ionos.com/containerregistries/registries/{registryId}/tokens |
Create a registry token for a pipeline |
Base URL: https://api.ionos.com/cloudapi/v6
Authentication: Authorization: Bearer <token>
Code Lab
Objective: Build a GitHub Actions pipeline that tests TaskBoard, pushes Git-SHA images to the registry and deploys them to MKS on every push to main, then roll back a broken release.
Prerequisites:
- The registry and push token from Unit 3.1
- The MKS cluster,
taskboardnamespace, deployer service account and running Deployments from Unit 3.2 - A GitHub repository with TaskBoard and the
ghCLI
Step 1: Create the deployer kubeconfig
Run the commands from Section 1.2 with the admin kubeconfig.
KUBECONFIG=deployer.yaml kubectl auth can-i update deployments
KUBECONFIG=deployer.yaml kubectl auth can-i create namespaces
Expected output:
yes
no
Step 2: Store the secrets
gh secret set CR_HOST --body "$CR_HOST"
gh secret set CR_USERNAME --body "taskboard-ci-push"
gh secret set CR_PASSWORD < ci-push.pass
gh secret set KUBECONFIG_DEPLOYER --body "$(base64 -w0 deployer.yaml)"
rm deployer.yaml
gh secret list
Expected output:
CR_HOST Updated ...
CR_PASSWORD Updated ...
CR_USERNAME Updated ...
KUBECONFIG_DEPLOYER Updated ...
Step 3: Add the workflow
Save the workflow from Section 2 as .github/workflows/deploy.yml, then commit and push.
git add .github/workflows/deploy.yml
git commit -m "ci: deploy TaskBoard to MKS"
git push origin main
gh run watch
Expected output:
✓ build-deploy
✓ Test
✓ Build and push
✓ Deploy to MKS
Step 4: Verify the running image
kubectl -n taskboard get deploy taskboard-api -o jsonpath='{.spec.template.spec.containers[0].image}'
git rev-parse --short=12 HEAD
Expected output:
<registry-hostname>/taskboard/api:<sha>
<sha>
Step 5: Ship a broken release
Change the readiness probe path in the API to return 500, commit and push.
git commit -am "break health check" && git push origin main
gh run watch
Expected output:
X Deploy to MKS
error: timed out waiting for the condition
Step 6: Roll back
kubectl -n taskboard rollout undo deploy/taskboard-api
kubectl -n taskboard rollout status deploy/taskboard-api
Expected output:
deployment.apps/taskboard-api rolled back
deployment "taskboard-api" successfully rolled out
Validation Checklist:
- [ ] A push to
mainruns tests, pushes both images and updates the Deployments - [ ] The running image tag matches the commit SHA
- [ ] A failed rollout fails the pipeline, and
rollout undorestores service - [ ] No credential is stored in the repository
Cleanup:
git revert --no-edit HEAD && git push origin main
Keep the cluster for Unit 3.4 and Module 5, or run terraform destroy from Unit 3.2.
Common Pitfalls
Developer mistakes to avoid with this unit's topic:
-
Two Terraform runs at the same time
- Problem: Two merges in quick succession start two applies, and one of them fails or leaves the state inconsistent.
- Why it happens: Both runs read and write the same state object in the bucket.
- Fix: Serialize runs with a
concurrencygroup (GitHub) orresource_group(GitLab) per environment, and apply only the reviewedtfplan.
-
Credentials that expire silently
- Problem: The pipeline worked for weeks, then every run fails with
401on the Cloud API or ondocker login. - Why it happens: The access token reached its
ttl, or the registry token reached its expiry date and was deleted. - Fix: Record each expiry date next to the secret, rotate before it, and create the replacement registry token before you remove the old one.
- Problem: The pipeline worked for weeks, then every run fails with
-
Admin kubeconfig in CI
- Problem: A compromised build step deletes namespaces or reads every secret in the cluster.
- Why it happens: The pipeline uses the kubeconfig from the API, which has full cluster access.
- Fix: Use the namespace-scoped
taskboard-deployerkubeconfig from Section 1.2, and keep the admin kubeconfig out of CI.
Summary
You can now ship TaskBoard through code review only. The application pipeline tests, builds Git-SHA images, pushes them to the Private Container Registry and rolls them out to MKS with a namespace-scoped account. The infrastructure pipeline posts the Terraform plan to every pull request and applies it on merge, one run at a time. You can roll back a release, restore an older state version, and share infrastructure as versioned modules.
Key Points:
- Split application and infrastructure pipelines so that each holds only the credentials it needs
- Use
IONOS_TOKENfor Terraform, an Access Key and Secret Key pair for the state backend, and a registry token for Docker - Serialize Terraform runs and deployments per environment with
concurrencyorresource_group - Deploy Git-SHA tags and wait for
rollout status; roll back withkubectl rollout undo - Pin the provider and your modules, and restore state from bucket versions only with all pipelines stopped
Important Terminology:
- Plan on PR: Running
terraform planfor every pull request and posting the result for review - Concurrency group: A CI setting that allows only one run per group at a time
- Service account kubeconfig: A kubeconfig with a Kubernetes service account token limited by RBAC
- Rolling update: The default Deployment strategy that replaces pods gradually
- Terraform module: A reusable directory of resources with input variables and outputs
Next Steps
Continue Learning: Unit 3.4: Knowledge Check - Containers and CI/CD
Related Topics: