13 min read

Learning Objectives

By the end of this module, you will be able to:

  • Install ArgoCD on Managed Kubernetes and deploy TaskBoard from Git with an `Application` that auto-syncs and self-heals
  • Split TaskBoard into an infrastructure repository (Terraform, apply on merge) and an application repository (manifests, ArgoCD sync)
  • Promote image tags from dev to staging to production with Kustomize overlays and pull requests
  • Create and destroy branch environments automatically from CI to keep costs under control
  • Draw a clear ownership line between Terraform and ArgoCD so the two tools never fight over the same object

Unit 5.3: GitOps and Deployment Operations

Introduction

TaskBoard's pipeline from Unit 3.3 pushes changes into the cluster: CI holds a kubeconfig and runs kubectl set image. That works, but the cluster state is whatever the last job did, a manual kubectl edit survives unnoticed, and every pipeline needs credentials for the cluster.

GitOps turns this around. Git holds the desired state of the application, and an agent inside the cluster pulls it and reconciles continuously. If someone changes a Deployment by hand, the agent puts it back. If you want to roll back, you revert a commit. In this unit you install ArgoCD on MKS, connect it to TaskBoard's manifests, structure the repositories, promote releases across environments, and tear down branch environments automatically. You also decide which tool owns which object, so Terraform and ArgoCD do not undo each other's work.

1. GitOps Principles in Practice

1.1 Push vs Pull

Aspect Push (Unit 3.3) Pull (GitOps)
Who changes the cluster CI job with a kubeconfig Agent inside the cluster
Source of truth Last successful job Git branch or tag
Manual change in the cluster Stays until the next deploy Detected and reverted
Rollback kubectl rollout undo git revert
Cluster credentials in CI Required Not required

The pull model also simplifies access to the Kubernetes API. If you restricted it with api_subnet_allow_list (Unit 3.2), every external CI runner must be in that list. ArgoCD runs inside the cluster and talks to the API server through https://kubernetes.default.svc, and cluster-internal traffic is not affected by the allow list.

1.2 Drift Detection

Drift is any difference between Git and the live state. You already detect infrastructure drift with terraform plan -detailed-exitcode (Unit 5.2). ArgoCD does the same for manifests:

argocd app diff taskboard-dev        # exit code 1 when live state differs from Git
argocd app get taskboard-dev -o json | jq -r '.status.sync.status, .status.health.status'

Synced and Healthy mean the cluster matches Git and every resource passes its health checks, including the readiness probes from Unit 5.1.

2. ArgoCD on MKS

2.1 Installation

kubectl create namespace argocd
kubectl apply -n argocd \
  -f "https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml"
kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d

Pin ARGOCD_VERSION to a release tag, exactly as you pin the Terraform provider, and upgrade through a pull request. Reach the UI with kubectl -n argocd port-forward svc/argocd-server 8080:443. Do not expose it with type: LoadBalancer: on MKS that Service gets a static IP address on one node, not a load balancer (Unit 3.2). If the team needs permanent access, route it through the ingress controller behind the ALB.

2.2 Connecting the Application Repository

apiVersion: v1
kind: Secret
metadata:
  name: taskboard-deploy-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://git.example.com/taskboard/deploy.git
  username: argocd
  password: <read-only access token>

Create this Secret from your secret store, as in Unit 5.2, never from a file in Git. ArgoCD needs read access only; it never pushes.

2.3 The Application CRD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: taskboard-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/taskboard/deploy.git
    targetRevision: main
    path: overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: taskboard
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

automated syncs every new commit on main. prune deletes objects that you removed from Git, and selfHeal reverts manual changes in the cluster. ArgoCD applies manifests but does not supply registry credentials: the pods still need the imagePullSecrets entry and the pull Secret from Unit 3.2, or they stay in ImagePullBackOff.

3. Repository Topology and Promotion

3.1 Two Repositories, Two Owners

taskboard-infra/                 taskboard-deploy/
  modules/                         base/
    mks-cluster/                     api-deployment.yaml
    postgres/                        web-deployment.yaml
  envs/                              worker-deployment.yaml
    dev/main.tf                      kustomization.yaml
    staging/main.tf                overlays/
    prod/main.tf                     dev/kustomization.yaml
  .github/workflows/infra.yml        staging/kustomization.yaml
                                     prod/kustomization.yaml

taskboard-infra is the infrastructure repository from Unit 3.3: plan on pull request, apply on merge, with IONOS_TOKEN in its secrets. taskboard-deploy holds only Kubernetes manifests and has no cloud credentials at all. The application source code stays in a third repository whose CI builds and pushes images.

3.2 Overlays and Image Tags

The production overlay, overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: taskboard
resources:
  - ../../base
images:
  - name: taskboard-api
    newName: <registry-hostname>/taskboard/api
    newTag: 3f9c2a1b7d04
replicas:
  - name: taskboard-api
    count: 3

The base manifests reference the image as taskboard-api, and each overlay pins the Git SHA tag from Unit 3.1. Promotion is a commit that copies a tested tag from one overlay to the next.

3.3 Promotion from CI

The application CI no longer touches the cluster. After it pushes the image, it opens a pull request against taskboard-deploy:

SHA="${GITHUB_SHA::12}"
git clone "https://x-access-token:${DEPLOY_REPO_TOKEN}@git.example.com/taskboard/deploy.git"
cd deploy/overlays/dev
kustomize edit set image "taskboard-api=${CR_HOST}/taskboard/api:${SHA}"
git checkout -b "promote-dev-${SHA}"
git commit -am "dev: taskboard-api ${SHA}"
git push origin "promote-dev-${SHA}"
gh pr create --title "dev: taskboard-api ${SHA}" --body "Automated promotion" --base main

For dev you can merge automatically once checks pass. For staging and production, require a review, so the pull request history is your release log. A rollback is git revert of the promotion commit; ArgoCD syncs the previous tag back.

4. Branch Environments and Automated Teardown

4.1 One State per Branch

The workflow .github/workflows/branch-env.yml in taskboard-infra:

on:
  pull_request:
    types: [opened, synchronize, closed]
jobs:
  env:
    runs-on: ubuntu-latest
    concurrency: branch-${{ github.head_ref }}
    defaults:
      run:
        working-directory: envs/branch
    env:
      IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      TF_VAR_branch: ${{ github.head_ref }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -input=false -backend-config="key=branches/${TF_VAR_branch}/terraform.tfstate"
      - if: github.event.action != 'closed'
        run: terraform apply -input=false -auto-approve
      - if: github.event.action == 'closed'
        run: terraform destroy -input=false -auto-approve

Every pull request gets its own state object in the state bucket, so a branch environment can never touch the state of dev or production. Name every resource after var.branch and keep the branch stack small: an Object Storage bucket, a database and a namespace on the shared dev cluster, not a cluster of its own.

4.2 Teardown That Actually Runs

Resources incur charges from the moment they exist, and a forgotten branch environment costs money every hour. The nightly safety net destroys branch states whose pull request is closed:

for key in $(aws s3 ls --endpoint-url "$EP" "s3://taskboard-tfstate-7f3a/branches/" | awk '{print $2}' | tr -d /); do
  state=$(gh pr list --head "$key" --state open --json number --jq 'length')
  if [ "$state" = "0" ]; then
    terraform init -reconfigure -input=false -backend-config="key=branches/${key}/terraform.tfstate"
    TF_VAR_branch="$key" terraform destroy -input=false -auto-approve
  fi
done

That is one of three safeguards. First, the closed event destroys the stack when the pull request is merged or closed. Second, the nightly job catches runs that failed; it assumes branch names without /, so use a slug if yours contain one. Third, tag branch resources with the branch name and a creation date where the resource supports it, so a review of the contract shows stragglers. If a Managed Kubernetes cluster is part of a branch stack, its node pools must be deleted before the cluster; Terraform orders this through the dependency between ionoscloud_k8s_node_pool and ionoscloud_k8s_cluster.

The ArgoCD side cleans up too. With prune: true and an ApplicationSet that generates one Application per open pull request, closing the pull request deletes the generated Application and its namespace resources.

5. Terraform and ArgoCD: Who Owns What

5.1 The Ownership Line

Object Owner Why
Data center, LANs, NSGs, ALB, target groups Terraform IONOS CLOUD resources, need IONOS_TOKEN
MKS cluster, node pools, kubeconfig Terraform IONOS CLOUD resources
DBaaS, In-Memory DB, Kafka, Object Storage buckets Terraform IONOS CLOUD resources
Deployments, Services, Ingress, ConfigMaps, NetworkPolicies ArgoCD Kubernetes objects, versioned with the app
Secrets with credentials External Secrets Operator, synced by ArgoCD Values stay in the secret store
ArgoCD itself and its Application objects Bootstrap once, then ArgoCD Avoids a chicken-and-egg loop

One object, one owner. The bridge between the two worlds is data, not shared resources: Terraform outputs such as pg_host go to the secret store, and the manifests reference the resulting Kubernetes Secret (Unit 5.2).

5.2 Avoiding Fight Loops

A fight loop happens when two controllers keep changing the same field. To let a HorizontalPodAutoscaler own the replica count, add this to the Application:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: taskboard-api
      jsonPointers:
        - /spec/replicas

Typical TaskBoard cases:

  • HPA vs Git: If an HPA scales taskboard-api, remove replicas from the overlay or ignore the field as shown, or selfHeal resets the count on every sync.
  • Autoscaler vs Terraform: With auto_scaling on a node pool, keep lifecycle { ignore_changes = [node_count] } from Unit 3.2, or every apply resets the pool.
  • CI vs ArgoCD: Remove kubectl set image from the Unit 3.3 workflow when ArgoCD takes over. With selfHeal, ArgoCD reverts it within minutes, and the deployment appears to roll back by itself.
  • Terraform vs MKS: The servers, NICs and IPs that Managed Kubernetes creates in your data center are shown read-only in the DCD. Never import them into Terraform; manage them through the node pool.

API Reference Quick Card

Key API endpoints for this unit's topic:

Method Endpoint Description
GET /k8s/{k8sClusterId}/kubeconfig Download the kubeconfig to bootstrap ArgoCD
GET /k8s/{k8sClusterId}/nodepools Check node pools before a branch teardown
DELETE /k8s/{k8sClusterId}/nodepools/{nodepoolId} Delete a node pool (before its cluster)
DELETE /k8s/{k8sClusterId} Delete a cluster without node pools
GET /requests/{requestId}/status Poll an asynchronous delete

Base URL: https://api.ionos.com/cloudapi/v6 Authentication: Authorization: Bearer <token>

Code Lab

Objective: Set up ArgoCD auto-sync for TaskBoard's manifests, push a manifest change and watch reconciliation, revert a manual change through self-heal, and tear down a branch environment.

Prerequisites:

  • The MKS cluster and TaskBoard namespace from Unit 3.2, with the admin kubeconfig
  • A taskboard-deploy repository with base/ and overlays/dev/ as in Section 3
  • The argocd CLI, kustomize, gh and Terraform with the state bucket from Unit 2.4

Step 1: Install ArgoCD

export ARGOCD_VERSION="<release tag>"
kubectl create namespace argocd
kubectl apply -n argocd -f "https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml"
kubectl -n argocd rollout status deploy/argocd-server

Expected output:

deployment "argocd-server" successfully rolled out

Step 2: Log in

kubectl -n argocd port-forward svc/argocd-server 8080:443 &
argocd login localhost:8080 --username admin --insecure \
  --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)"

Expected output:

'admin:login' logged in successfully

Step 3: Register the repository and the Application

Apply the repository Secret (Section 2.2) and the Application (Section 2.3).

kubectl apply -f argocd/repo-secret.yaml -f argocd/taskboard-dev.yaml
argocd app wait taskboard-dev --health --timeout 300

Expected output:

Name:     argocd/taskboard-dev
Sync Status:   Synced to main (<sha>)
Health Status: Healthy

Step 4: Push a manifest change

cd taskboard-deploy/overlays/dev
kustomize edit set image "taskboard-api=${CR_HOST}/taskboard/api:<new-sha>"
git commit -am "dev: taskboard-api <new-sha>" && git push
argocd app get taskboard-dev --refresh | grep -E "Sync Status|Health"

Expected output:

Sync Status:   Synced to main (<new commit>)
Health Status: Healthy

Step 5: Watch self-heal

kubectl -n taskboard scale deploy/taskboard-web --replicas=0
sleep 30
kubectl -n taskboard get deploy taskboard-web

Expected output:

NAME            READY   UP-TO-DATE   AVAILABLE
taskboard-web   2/2     2            2

Step 6: Create and destroy a branch environment

Open a pull request in taskboard-infra with the workflow from Section 4.1, wait for the apply, then close the pull request.

gh pr close <number>
gh run watch

Expected output:

✓ env
  ✓ terraform destroy -input=false -auto-approve
Destroy complete! Resources: <n> destroyed.

Validation Checklist:

  • [ ] taskboard-dev is Synced and Healthy
  • [ ] A commit to overlays/dev changes the running image without any CI access to the cluster
  • [ ] A manual scale-down is reverted by self-heal
  • [ ] Closing the pull request destroys the branch stack and leaves no state behind except the empty state object

Cleanup:

argocd app delete taskboard-dev --yes
kubectl delete namespace argocd
kill %1

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. The deployment that rolls itself back

    • Problem: CI deploys a new image, and a few minutes later the old image is running again.
    • Why it happens: The old kubectl set image step is still in the pipeline, and ArgoCD's selfHeal restores the tag from Git.
    • Fix: Change images only through commits to the overlay (Section 3.3) and remove cluster credentials from the application CI.
  2. Branch environments nobody destroys

    • Problem: The bill grows every month, and the contract contains buckets and databases named after long-merged branches.
    • Why it happens: The closed job failed once, or ran before terraform init pointed at the branch's state key.
    • Fix: Use one state key per branch, run destroy on closed, and add the nightly safety net from Section 4.2.
  3. Terraform and ArgoCD managing the same object

    • Problem: Every terraform apply changes a Deployment, and every ArgoCD sync changes it back.
    • Why it happens: The Deployment was created with the Kubernetes Terraform provider and is also in the deploy repository.
    • Fix: Pick one owner per object (Section 5.1); remove the resource from Terraform state with terraform state rm and let ArgoCD own it.

Summary

You can now operate TaskBoard declaratively. Terraform owns everything that is an IONOS CLOUD resource and applies on merge; ArgoCD owns everything that is a Kubernetes object and syncs it from Git with pruning and self-heal. Releases move between environments as reviewed commits to Kustomize overlays, and branch environments are created and destroyed by the pull request lifecycle.

Key Points:

  • Git is the source of truth; ArgoCD pulls and reconciles, so CI needs no cluster credentials
  • ArgoCD uses kubernetes.default.svc, so the API allow list does not block it
  • Promote by committing image tags to overlays; roll back with git revert
  • One state key per branch, destroy on pull request close, plus a nightly safety net
  • One owner per object: Terraform for IONOS CLOUD resources, ArgoCD for Kubernetes objects

Important Terminology:

  • GitOps: Operating a system by declaring its desired state in Git and reconciling it automatically
  • Application: The ArgoCD custom resource that maps a Git path to a cluster namespace
  • Self-heal: ArgoCD's automatic reversal of changes made directly in the cluster
  • Overlay: A Kustomize directory that patches the shared base for one environment
  • Fight loop: Two controllers repeatedly overwriting the same field of the same object

Next Steps

Continue Learning: Unit 5.4: Knowledge Check - Production Operations

Related Topics: