Unit 5.3: GitOps and Deployment Operations
Introduction
TaskBoard's pipeline from Unit 3.3 pushes changes into the cluster: CI holds a kubeconfig and runs kubectl set image. That works, but the cluster state is whatever the last job did, a manual kubectl edit survives unnoticed, and every pipeline needs credentials for the cluster.
GitOps turns this around. Git holds the desired state of the application, and an agent inside the cluster pulls it and reconciles continuously. If someone changes a Deployment by hand, the agent puts it back. If you want to roll back, you revert a commit. In this unit you install ArgoCD on MKS, connect it to TaskBoard's manifests, structure the repositories, promote releases across environments, and tear down branch environments automatically. You also decide which tool owns which object, so Terraform and ArgoCD do not undo each other's work.
1. GitOps Principles in Practice
1.1 Push vs Pull
| Aspect | Push (Unit 3.3) | Pull (GitOps) |
|---|---|---|
| Who changes the cluster | CI job with a kubeconfig | Agent inside the cluster |
| Source of truth | Last successful job | Git branch or tag |
| Manual change in the cluster | Stays until the next deploy | Detected and reverted |
| Rollback | kubectl rollout undo |
git revert |
| Cluster credentials in CI | Required | Not required |
The pull model also simplifies access to the Kubernetes API. If you restricted it with api_subnet_allow_list (Unit 3.2), every external CI runner must be in that list. ArgoCD runs inside the cluster and talks to the API server through https://kubernetes.default.svc, and cluster-internal traffic is not affected by the allow list.
1.2 Drift Detection
Drift is any difference between Git and the live state. You already detect infrastructure drift with terraform plan -detailed-exitcode (Unit 5.2). ArgoCD does the same for manifests:
argocd app diff taskboard-dev # exit code 1 when live state differs from Git
argocd app get taskboard-dev -o json | jq -r '.status.sync.status, .status.health.status'
Synced and Healthy mean the cluster matches Git and every resource passes its health checks, including the readiness probes from Unit 5.1.
2. ArgoCD on MKS
2.1 Installation
kubectl create namespace argocd
kubectl apply -n argocd \
-f "https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml"
kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d
Pin ARGOCD_VERSION to a release tag, exactly as you pin the Terraform provider, and upgrade through a pull request. Reach the UI with kubectl -n argocd port-forward svc/argocd-server 8080:443. Do not expose it with type: LoadBalancer: on MKS that Service gets a static IP address on one node, not a load balancer (Unit 3.2). If the team needs permanent access, route it through the ingress controller behind the ALB.
2.2 Connecting the Application Repository
apiVersion: v1
kind: Secret
metadata:
name: taskboard-deploy-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: git
url: https://git.example.com/taskboard/deploy.git
username: argocd
password: <read-only access token>
Create this Secret from your secret store, as in Unit 5.2, never from a file in Git. ArgoCD needs read access only; it never pushes.
2.3 The Application CRD
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: taskboard-dev
namespace: argocd
spec:
project: default
source:
repoURL: https://git.example.com/taskboard/deploy.git
targetRevision: main
path: overlays/dev
destination:
server: https://kubernetes.default.svc
namespace: taskboard
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
automated syncs every new commit on main. prune deletes objects that you removed from Git, and selfHeal reverts manual changes in the cluster. ArgoCD applies manifests but does not supply registry credentials: the pods still need the imagePullSecrets entry and the pull Secret from Unit 3.2, or they stay in ImagePullBackOff.
3. Repository Topology and Promotion
3.1 Two Repositories, Two Owners
taskboard-infra/ taskboard-deploy/
modules/ base/
mks-cluster/ api-deployment.yaml
postgres/ web-deployment.yaml
envs/ worker-deployment.yaml
dev/main.tf kustomization.yaml
staging/main.tf overlays/
prod/main.tf dev/kustomization.yaml
.github/workflows/infra.yml staging/kustomization.yaml
prod/kustomization.yaml
taskboard-infra is the infrastructure repository from Unit 3.3: plan on pull request, apply on merge, with IONOS_TOKEN in its secrets. taskboard-deploy holds only Kubernetes manifests and has no cloud credentials at all. The application source code stays in a third repository whose CI builds and pushes images.
3.2 Overlays and Image Tags
The production overlay, overlays/prod/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: taskboard
resources:
- ../../base
images:
- name: taskboard-api
newName: <registry-hostname>/taskboard/api
newTag: 3f9c2a1b7d04
replicas:
- name: taskboard-api
count: 3
The base manifests reference the image as taskboard-api, and each overlay pins the Git SHA tag from Unit 3.1. Promotion is a commit that copies a tested tag from one overlay to the next.
3.3 Promotion from CI
The application CI no longer touches the cluster. After it pushes the image, it opens a pull request against taskboard-deploy:
SHA="${GITHUB_SHA::12}"
git clone "https://x-access-token:${DEPLOY_REPO_TOKEN}@git.example.com/taskboard/deploy.git"
cd deploy/overlays/dev
kustomize edit set image "taskboard-api=${CR_HOST}/taskboard/api:${SHA}"
git checkout -b "promote-dev-${SHA}"
git commit -am "dev: taskboard-api ${SHA}"
git push origin "promote-dev-${SHA}"
gh pr create --title "dev: taskboard-api ${SHA}" --body "Automated promotion" --base main
For dev you can merge automatically once checks pass. For staging and production, require a review, so the pull request history is your release log. A rollback is git revert of the promotion commit; ArgoCD syncs the previous tag back.
4. Branch Environments and Automated Teardown
4.1 One State per Branch
The workflow .github/workflows/branch-env.yml in taskboard-infra:
on:
pull_request:
types: [opened, synchronize, closed]
jobs:
env:
runs-on: ubuntu-latest
concurrency: branch-${{ github.head_ref }}
defaults:
run:
working-directory: envs/branch
env:
IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
TF_VAR_branch: ${{ github.head_ref }}
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- run: terraform init -input=false -backend-config="key=branches/${TF_VAR_branch}/terraform.tfstate"
- if: github.event.action != 'closed'
run: terraform apply -input=false -auto-approve
- if: github.event.action == 'closed'
run: terraform destroy -input=false -auto-approve
Every pull request gets its own state object in the state bucket, so a branch environment can never touch the state of dev or production. Name every resource after var.branch and keep the branch stack small: an Object Storage bucket, a database and a namespace on the shared dev cluster, not a cluster of its own.
4.2 Teardown That Actually Runs
Resources incur charges from the moment they exist, and a forgotten branch environment costs money every hour. The nightly safety net destroys branch states whose pull request is closed:
for key in $(aws s3 ls --endpoint-url "$EP" "s3://taskboard-tfstate-7f3a/branches/" | awk '{print $2}' | tr -d /); do
state=$(gh pr list --head "$key" --state open --json number --jq 'length')
if [ "$state" = "0" ]; then
terraform init -reconfigure -input=false -backend-config="key=branches/${key}/terraform.tfstate"
TF_VAR_branch="$key" terraform destroy -input=false -auto-approve
fi
done
That is one of three safeguards. First, the closed event destroys the stack when the pull request is merged or closed. Second, the nightly job catches runs that failed; it assumes branch names without /, so use a slug if yours contain one. Third, tag branch resources with the branch name and a creation date where the resource supports it, so a review of the contract shows stragglers. If a Managed Kubernetes cluster is part of a branch stack, its node pools must be deleted before the cluster; Terraform orders this through the dependency between ionoscloud_k8s_node_pool and ionoscloud_k8s_cluster.
The ArgoCD side cleans up too. With prune: true and an ApplicationSet that generates one Application per open pull request, closing the pull request deletes the generated Application and its namespace resources.
5. Terraform and ArgoCD: Who Owns What
5.1 The Ownership Line
| Object | Owner | Why |
|---|---|---|
| Data center, LANs, NSGs, ALB, target groups | Terraform | IONOS CLOUD resources, need IONOS_TOKEN |
| MKS cluster, node pools, kubeconfig | Terraform | IONOS CLOUD resources |
| DBaaS, In-Memory DB, Kafka, Object Storage buckets | Terraform | IONOS CLOUD resources |
| Deployments, Services, Ingress, ConfigMaps, NetworkPolicies | ArgoCD | Kubernetes objects, versioned with the app |
| Secrets with credentials | External Secrets Operator, synced by ArgoCD | Values stay in the secret store |
ArgoCD itself and its Application objects |
Bootstrap once, then ArgoCD | Avoids a chicken-and-egg loop |
One object, one owner. The bridge between the two worlds is data, not shared resources: Terraform outputs such as pg_host go to the secret store, and the manifests reference the resulting Kubernetes Secret (Unit 5.2).
5.2 Avoiding Fight Loops
A fight loop happens when two controllers keep changing the same field. To let a HorizontalPodAutoscaler own the replica count, add this to the Application:
spec:
ignoreDifferences:
- group: apps
kind: Deployment
name: taskboard-api
jsonPointers:
- /spec/replicas
Typical TaskBoard cases:
- HPA vs Git: If an HPA scales
taskboard-api, removereplicasfrom the overlay or ignore the field as shown, orselfHealresets the count on every sync. - Autoscaler vs Terraform: With
auto_scalingon a node pool, keeplifecycle { ignore_changes = [node_count] }from Unit 3.2, or every apply resets the pool. - CI vs ArgoCD: Remove
kubectl set imagefrom the Unit 3.3 workflow when ArgoCD takes over. WithselfHeal, ArgoCD reverts it within minutes, and the deployment appears to roll back by itself. - Terraform vs MKS: The servers, NICs and IPs that Managed Kubernetes creates in your data center are shown read-only in the DCD. Never import them into Terraform; manage them through the node pool.
API Reference Quick Card
Key API endpoints for this unit's topic:
| Method | Endpoint | Description |
|---|---|---|
GET |
/k8s/{k8sClusterId}/kubeconfig |
Download the kubeconfig to bootstrap ArgoCD |
GET |
/k8s/{k8sClusterId}/nodepools |
Check node pools before a branch teardown |
DELETE |
/k8s/{k8sClusterId}/nodepools/{nodepoolId} |
Delete a node pool (before its cluster) |
DELETE |
/k8s/{k8sClusterId} |
Delete a cluster without node pools |
GET |
/requests/{requestId}/status |
Poll an asynchronous delete |
Base URL: https://api.ionos.com/cloudapi/v6
Authentication: Authorization: Bearer <token>
Code Lab
Objective: Set up ArgoCD auto-sync for TaskBoard's manifests, push a manifest change and watch reconciliation, revert a manual change through self-heal, and tear down a branch environment.
Prerequisites:
- The MKS cluster and TaskBoard namespace from Unit 3.2, with the admin kubeconfig
- A
taskboard-deployrepository withbase/andoverlays/dev/as in Section 3 - The
argocdCLI,kustomize,ghand Terraform with the state bucket from Unit 2.4
Step 1: Install ArgoCD
export ARGOCD_VERSION="<release tag>"
kubectl create namespace argocd
kubectl apply -n argocd -f "https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml"
kubectl -n argocd rollout status deploy/argocd-server
Expected output:
deployment "argocd-server" successfully rolled out
Step 2: Log in
kubectl -n argocd port-forward svc/argocd-server 8080:443 &
argocd login localhost:8080 --username admin --insecure \
--password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)"
Expected output:
'admin:login' logged in successfully
Step 3: Register the repository and the Application
Apply the repository Secret (Section 2.2) and the Application (Section 2.3).
kubectl apply -f argocd/repo-secret.yaml -f argocd/taskboard-dev.yaml
argocd app wait taskboard-dev --health --timeout 300
Expected output:
Name: argocd/taskboard-dev
Sync Status: Synced to main (<sha>)
Health Status: Healthy
Step 4: Push a manifest change
cd taskboard-deploy/overlays/dev
kustomize edit set image "taskboard-api=${CR_HOST}/taskboard/api:<new-sha>"
git commit -am "dev: taskboard-api <new-sha>" && git push
argocd app get taskboard-dev --refresh | grep -E "Sync Status|Health"
Expected output:
Sync Status: Synced to main (<new commit>)
Health Status: Healthy
Step 5: Watch self-heal
kubectl -n taskboard scale deploy/taskboard-web --replicas=0
sleep 30
kubectl -n taskboard get deploy taskboard-web
Expected output:
NAME READY UP-TO-DATE AVAILABLE
taskboard-web 2/2 2 2
Step 6: Create and destroy a branch environment
Open a pull request in taskboard-infra with the workflow from Section 4.1, wait for the apply, then close the pull request.
gh pr close <number>
gh run watch
Expected output:
✓ env
✓ terraform destroy -input=false -auto-approve
Destroy complete! Resources: <n> destroyed.
Validation Checklist:
- [ ]
taskboard-devisSyncedandHealthy - [ ] A commit to
overlays/devchanges the running image without any CI access to the cluster - [ ] A manual scale-down is reverted by self-heal
- [ ] Closing the pull request destroys the branch stack and leaves no state behind except the empty state object
Cleanup:
argocd app delete taskboard-dev --yes
kubectl delete namespace argocd
kill %1
Common Pitfalls
Developer mistakes to avoid with this unit's topic:
-
The deployment that rolls itself back
- Problem: CI deploys a new image, and a few minutes later the old image is running again.
- Why it happens: The old
kubectl set imagestep is still in the pipeline, and ArgoCD'sselfHealrestores the tag from Git. - Fix: Change images only through commits to the overlay (Section 3.3) and remove cluster credentials from the application CI.
-
Branch environments nobody destroys
- Problem: The bill grows every month, and the contract contains buckets and databases named after long-merged branches.
- Why it happens: The
closedjob failed once, or ran beforeterraform initpointed at the branch's state key. - Fix: Use one state key per branch, run destroy on
closed, and add the nightly safety net from Section 4.2.
-
Terraform and ArgoCD managing the same object
- Problem: Every
terraform applychanges a Deployment, and every ArgoCD sync changes it back. - Why it happens: The Deployment was created with the Kubernetes Terraform provider and is also in the deploy repository.
- Fix: Pick one owner per object (Section 5.1); remove the resource from Terraform state with
terraform state rmand let ArgoCD own it.
- Problem: Every
Summary
You can now operate TaskBoard declaratively. Terraform owns everything that is an IONOS CLOUD resource and applies on merge; ArgoCD owns everything that is a Kubernetes object and syncs it from Git with pruning and self-heal. Releases move between environments as reviewed commits to Kustomize overlays, and branch environments are created and destroyed by the pull request lifecycle.
Key Points:
- Git is the source of truth; ArgoCD pulls and reconciles, so CI needs no cluster credentials
- ArgoCD uses
kubernetes.default.svc, so the API allow list does not block it - Promote by committing image tags to overlays; roll back with
git revert - One state key per branch, destroy on pull request close, plus a nightly safety net
- One owner per object: Terraform for IONOS CLOUD resources, ArgoCD for Kubernetes objects
Important Terminology:
- GitOps: Operating a system by declaring its desired state in Git and reconciling it automatically
- Application: The ArgoCD custom resource that maps a Git path to a cluster namespace
- Self-heal: ArgoCD's automatic reversal of changes made directly in the cluster
- Overlay: A Kustomize directory that patches the shared base for one environment
- Fight loop: Two controllers repeatedly overwriting the same field of the same object
Next Steps
Continue Learning: Unit 5.4: Knowledge Check - Production Operations
Related Topics: