14 min read

Learning Objectives

By the end of this module, you will be able to:

  • Provision Monitoring Service and Logging Service pipelines with Terraform and push TaskBoard's metrics and logs to them
  • Forward Managed Kubernetes workload logs to the Logging Service with Fluent Bit and search them with LogQL in Grafana
  • Send TaskBoard traces to the Tracing Service over OTLP and correlate a slow span with its log lines
  • Audit infrastructure changes with the Activity Log API and debug network paths with Flow Logs
  • Debug a failing deployment on MKS step by step, from `kubectl` to metrics and log search

Unit 5.1: Observability and Debugging

Introduction

TaskBoard runs on Managed Kubernetes, ships through CI/CD and talks to PostgreSQL, In-Memory DB, Kafka and Object Storage. Then a user reports that the task list is slow, and the worker stopped sending notifications. You need answers fast: which service is slow, what changed, and where the packets go.

The Monitoring, Logging and Tracing Services store metrics, logs and traces, all visible in a managed Grafana. They work by push: you create a pipeline, then point an agent or SDK at it. The Activity Log API records who changed what, and Flow Logs record network flows. In this unit you build that setup for TaskBoard as code, and in the Code Lab you use it to debug a simulated outage.

1. Metrics with the Monitoring Service

1.1 Provisioning a Pipeline

resource "ionoscloud_monitoring_pipeline" "taskboard" {
  location = "de/txl"
  name     = "taskboard-metrics"
}

output "metrics_push_url" {
  value = "${ionoscloud_monitoring_pipeline.taskboard.http_endpoint}/api/v1/push"
}

output "metrics_key" {
  value     = ionoscloud_monitoring_pipeline.taskboard.key
  sensitive = true
}

output "grafana_url" {
  value = ionoscloud_monitoring_pipeline.taskboard.grafana_endpoint
}

The resource accepts only tokens for authorization, so set IONOS_TOKEN. The same pipeline over the REST API is a POST to the regional host:

curl --request POST "https://monitoring.de-txl.ionos.com/pipelines" \
  --header "Authorization: Bearer $IONOS_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"metadata": {}, "properties": {"name": "taskboard-metrics"}}'

The pipeline key is returned only in the create response and by key rotation (POST /pipelines/{pipelineID}/key), which invalidates the old key. Store it in your secret store immediately. A contract can have up to 10 pipelines by default; IONOS CLOUD Support can raise the limit. Poll GET /pipelines/{pipelineID} until metadata.state is AVAILABLE before you configure agents.

1.2 Pushing TaskBoard Metrics

The Monitoring Service is push-based: Prometheus, Grafana Agent, OpenTelemetry and Fluent Bit send Prometheus-format metrics to https://<HTTP_ENDPOINT>/api/v1/push over outbound port 443. For a Prometheus server in the cluster, add a remote_write block to prometheus.yml:

remote_write:
  - url: "https://<pipeline-id>-metrics.<tenant>.monitoring.de-txl.ionos.com/api/v1/push"
    headers:
      APIKEY: "<pipeline key>"

The agent sends the key in the APIKEY header. The default push interval is one minute, and you can change it. OpenTelemetry Collectors can also push natively in OTLP format to <httpEndpoint>/otlp/v1/metrics. Every pushed data point counts toward billing, so drop series you do not need before they leave the cluster.

1.3 Alerts

Custom alerts live in Grafana: you define the alert condition, threshold and notification settings there. For TaskBoard's API latency, alert on the 95th percentile of the request duration histogram that the API exports:

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="taskboard-api"}[5m]))
) > 0.5

Keep the expression in Git next to the service. For server-level checks without any agent, query the Telemetry API. It is Prometheus-compatible, read-only, uses your Cloud API token and becomes available once your contract has at least one observability pipeline:

curl --silent --get "https://api.ionos.com/telemetry/api/v1/query" \
  --header "Authorization: Bearer $IONOS_TOKEN" \
  --data-urlencode "query=instance_cpu_utilization_average > 90"

instance_cpu_utilization_average is a gauge; the other eight Telemetry API metrics are counters, so wrap them in rate(). Metrics carry the server uuid, not its name, so resolve names through the Cloud API.

2. Logs with the Logging Service

2.1 Provisioning a Pipeline

resource "ionoscloud_logging_pipeline" "taskboard" {
  location = "de/txl"
  name     = "taskboard-logs"

  log {
    source   = "kubernetes"
    tag      = "k8s"
    protocol = "tcp"
    destinations {
      type              = "loki"
      retention_in_days = 14
    }
  }
}

output "logging_tcp_host" {
  value = split(":", ionoscloud_logging_pipeline.taskboard.tcp_address)[0]
}

output "logging_key" {
  value     = ionoscloud_logging_pipeline.taskboard.key
  sensitive = true
}

Each log block is a log stream with a unique tag. Retention is 7, 14 or 30 days per stream, with 30 as the default. The key follows a share-once policy: it cannot be retrieved again, and generating a new one with POST /pipelines/{pipelineID}/key revokes the previous key instantly.

The following table lists the Logging Service limits:

Aspect Description Limit
HTTP Rate Limit Default rate limit for HTTP requests per pipeline during log ingestion. 50 requests/second
TCP Bandwidth Default TCP bandwidth limit per pipeline, approximately in terms of logs per second. ~10,000 logs/second
Maximum Pipelines The maximum number of pipelines allowed per contract. 10 pipelines
Log Streams per Pipeline The maximum number of log streams allowed per pipeline. 5 log streams/pipeline

Records dropped by rate limiting are not stored.

2.2 Forwarding Cluster Logs from MKS

Kubernetes API audit logs go to the Object Storage bucket you configure on the cluster (Unit 3.2). To get pod logs into the Logging Service, run an agent in the cluster, usually Fluent Bit as a DaemonSet:

[OUTPUT]
    Name        forward
    Match       kube.*
    Host        ${LOGGING_TCP_HOST}
    Port        9000
    Tag         k8s
    tls         on
    Shared_Key  ${LOGGING_KEY}

The TCP stream requires port 9000 with TLS, and Shared_Key carries the pipeline key. HTTP streams accept only JSON, with the key in the APIKEY header.

2.3 Searching Logs

The Logging Service stores logs in Loki, and you search them with LogQL in Grafana Explore. Querying does not count toward billing.

{tag_name="k8s"} |= "taskboard-api" | json | level="error"
{tag_name="k8s"} |= "taskboard-worker" |= "KafkaError"

The stream's tag becomes the tag_name label. Further labels depend on the log source and your custom labels, so check the label browser in Explore before you save a query in a dashboard.

3. Traces with the Tracing Service

3.1 Creating an OTLP Pipeline

curl --request POST "https://tracing.de-txl.ionos.com/pipelines" \
  --header "Authorization: Bearer $IONOS_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"metadata": {}, "properties": {"name": "taskboard-traces", "protocol": "otlp-http"}}'

protocol is otlp-http or otlp-grpc. Once the pipeline is AVAILABLE, metadata.otlpEndpoint holds the ingestion host; it is a unique hostname per pipeline, not the tracing.<region>.ionos.com management host. OTLP/HTTP exporters append /v1/traces and send the key in the apikey header; OTLP/gRPC exporters send it as apikey metadata on port 443.

3.2 Instrumenting the TaskBoard API

The exporter reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://<tracing-ingestion-endpoint>/v1/traces", OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" and OTEL_EXPORTER_OTLP_HEADERS="apikey=<API_KEY>" from the environment:

import json

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "taskboard-api"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("taskboard")

def log_json(level, msg):
    ctx = trace.get_current_span().get_span_context()
    print(json.dumps({"level": level, "msg": msg, "trace_id": format(ctx.trace_id, "032x")}))

with tracer.start_as_current_span("list_tasks"):
    log_json("info", "cache miss, reading tasks from PostgreSQL")

Put the headers value in a Kubernetes Secret. The trace_id in every JSON log line makes trace-to-log correlation work: in the same Grafana workspace you navigate from a span to the matching Loki logs and related Mimir metrics.

A single trace cannot exceed 5 MB of combined span data. Ingestion is limited to 15 MB/s sustained with a 20 MB burst; excess traffic gets HTTP 429 (OTLP/HTTP) or RESOURCE_EXHAUSTED (OTLP/gRPC). Traces are kept for 30 days. Configure sampling to reduce the volume you send.

4. Auditing Changes and Network Flows

4.1 The Activity Log API

The Activity Log API is read-only (GET only) at https://api.ionos.com/activitylog/v1, accepts Basic auth or a Bearer token, and returns JSON with the entries under hits:

import os
import requests

BASE = "https://api.ionos.com/activitylog/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['IONOS_TOKEN']}",
           "Content-Type": "application/json"}

def activity(contract, date_start, date_end, page=500):
    offset = 0
    while True:
        params = {"dateStart": date_start, "dateEnd": date_end, "offset": offset, "limit": page}
        r = requests.get(f"{BASE}/contracts/{contract}", headers=HEADERS, params=params, timeout=30)
        r.raise_for_status()
        hits = r.json()["hits"]["hits"]
        for hit in hits:
            yield hit["_source"]
        if len(hits) < page:
            return
        offset += page

for e in activity(31721060, "2026-11-02T00:00:00Z", "2026-11-03T00:00:00Z"):
    actions = [a for res in e["event"]["resources"] for a in res["action"]]
    print(e["meta"]["time"], e["principal"]["identity"]["username"], e["event"]["type"], actions)

dateStart is inclusive and dateEnd exclusive; limit accepts up to 10000. GET /contracts lists the contracts your credentials can access. By default only the contract owner and administrators can call the API; other users need a group with the Access Activity Log privilege. Entries are kept for 35 days, so export them to Object Storage or to the Logging Service if you need a longer history. The Activity Log covers IaaS products, not PaaS products.

4.2 Flow Logs for Connectivity Issues

resource "ionoscloud_nic" "worker" {
  datacenter_id = ionoscloud_datacenter.taskboard.id
  server_id     = ionoscloud_vcpu_server.worker.id
  lan           = ionoscloud_lan.app.id
  dhcp          = true

  flowlog {
    name      = "worker-rejected"
    action    = "REJECTED"
    direction = "BIDIRECTIONAL"
    bucket    = "taskboard-flowlogs-7f3a"
  }
}

Flow Logs are supported on VM network interfaces, the Managed Network Load Balancer, the Managed Application Load Balancer and the Managed NAT Gateway, with one flow log per resource. The bucket must already exist and be user-owned. action, direction and bucket are immutable; removing the flowlog block forces re-creation of the NIC. Records arrive as .log.gz objects every 10 minutes, and no file is written for an interval without flows:

2 12345678 7ffd6527-ce80-4e57-a949-f9a45824ebe2 172.17.1.100 172.17.1.101 59113 20756 6 17 1325 1587983051 1587983052 REJECT OK

The fields are version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status. A REJECT on the destination port of PostgreSQL or Kafka points at a firewall or NSG rule, not at the application. Set a Lifecycle Policy on the bucket, because flow logs stay until you delete them.

5. The Kubernetes Debugging Workflow

5.1 From Pod to Platform

Work from the narrowest view outwards:

kubectl -n taskboard get pods -o wide                        # 1. status, restarts, node
kubectl -n taskboard describe pod <pod>                      # 2. probe failures, pull errors
kubectl -n taskboard get events --sort-by=.lastTimestamp     # 3. scheduling, evictions
kubectl -n taskboard logs deploy/taskboard-api --previous    # 4. the crashed container's output

Then widen to the platform: (5) the latency and error-rate panels from the Monitoring Service, (6) a LogQL search across all pods and the worker VM, (7) the trace of a slow request, and (8) the Activity Log for changes in the same time window. Kubernetes keeps events for a limited time only, so save the output of kubectl get events during the incident.

5.2 Health Checks

livenessProbe:
  httpGet: { path: /api/health/live, port: 8000 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /api/health/ready, port: 8000 }
  periodSeconds: 5

Liveness restarts a stuck container; readiness removes a pod from its Service while dependencies such as PostgreSQL are unreachable. At the ALB, point the http_health_check of the target group from Units 2.3 and 3.2 at the same readiness path. Its health_check block sets check_interval and check_timeout in milliseconds; the default interval is 2000.

API Reference Quick Card

Key API endpoints for this unit's topic:

Method Endpoint Description
POST https://monitoring.<region>.ionos.com/pipelines Create a Monitoring Service pipeline
POST https://logging.<region>.ionos.com/pipelines/{pipelineID}/key Rotate a Logging Service pipeline key
POST https://tracing.<region>.ionos.com/pipelines Create a Tracing Service pipeline
GET https://api.ionos.com/telemetry/api/v1/query Query server metrics with PromQL
GET https://api.ionos.com/activitylog/v1/contracts/{contractNumber} Read Activity Log entries

Base URL: regional per service, as listed above Authentication: Authorization: Bearer <token>

Code Lab

Objective: Provision metrics and log pipelines for TaskBoard, forward MKS logs with Fluent Bit, then debug a simulated outage with kubectl, LogQL and the Activity Log.

Prerequisites:

  • IONOS CLOUD account with an API token in IONOS_TOKEN
  • The MKS cluster and TaskBoard Deployments from Unit 3.2
  • Terraform, kubectl, Helm and Python 3 with requests

Step 1: Provision the pipelines

Add the resources and outputs from Sections 1.1 and 2.1.

terraform apply -target=ionoscloud_monitoring_pipeline.taskboard -target=ionoscloud_logging_pipeline.taskboard
terraform output grafana_url

Expected output:

ionoscloud_monitoring_pipeline.taskboard: Creation complete after ...
ionoscloud_logging_pipeline.taskboard: Creation complete after ...
"https://grafana.<...>.ionos.com"

Step 2: Store the logging credentials in the cluster

kubectl create namespace logging
kubectl -n logging create secret generic logging-pipeline \
  --from-literal=LOGGING_TCP_HOST="$(terraform output -raw logging_tcp_host)" \
  --from-literal=LOGGING_KEY="$(terraform output -raw logging_key)"

Expected output:

namespace/logging created
secret/logging-pipeline created

Step 3: Deploy Fluent Bit

Save fluent-bit-values.yaml with envFrom: [{secretRef: {name: logging-pipeline}}] and the [OUTPUT] section from Section 2.2 under config.outputs.

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml
kubectl -n logging get pods

Expected output:

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-<id>    1/1     Running   0          30s

Step 4: Find TaskBoard logs in Grafana

Open the Grafana URL, sign in with your IONOS CLOUD credentials, choose Explore and run:

{tag_name="k8s"} |= "taskboard-api"

Expected output:

<timestamp>  {"level": "info", "msg": "GET /api/health 200", ...}

Step 5: Simulate an outage and find the cause

kubectl -n taskboard set env deploy/taskboard-api DATABASE_URL="postgresql://wrong-host:5432/taskboard"
kubectl -n taskboard describe pod <new-pod> | grep Readiness

Then search Grafana Explore with {tag_name="k8s"} |= "taskboard-api" |= "wrong-host". Expected output:

Readiness probe failed: HTTP probe failed with statuscode: 503
{"level": "error", "msg": "could not translate host name \"wrong-host\" ..."}

Step 6: Rule out infrastructure changes and recover

Run the script from Section 4.1 for today. A kubectl change is not a Cloud API call, so look only for unexpected infrastructure entries; then roll back.

python activity.py
kubectl -n taskboard rollout undo deploy/taskboard-api
kubectl -n taskboard rollout status deploy/taskboard-api

Expected output:

<time> <user> <event type> [<actions>]
deployment "taskboard-api" successfully rolled out

Validation Checklist:

  • [ ] Both pipelines exist and their keys are only in Terraform state and the Kubernetes Secret
  • [ ] TaskBoard pod logs are searchable in Grafana with LogQL
  • [ ] The broken rollout shows failing readiness probes and the error in the logs
  • [ ] rollout undo restores a ready API

Cleanup:

helm uninstall fluent-bit -n logging
kubectl delete namespace logging
terraform destroy -target=ionoscloud_logging_pipeline.taskboard -target=ionoscloud_monitoring_pipeline.taskboard

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. Waiting for cluster logs that never arrive

    • Problem: The Logging Service pipeline is AVAILABLE, but Grafana shows no Kubernetes logs or events.
    • Why it happens: Control-plane events do not flow through the Logging Service, audit logs go to the cluster's Object Storage bucket, and pod logs need an agent that you deploy.
    • Fix: Run Fluent Bit (or an OTLP exporter) in the cluster with the pipeline's TCP host, port 9000, tls on and Shared_Key (Section 2.2).
  2. Losing the pipeline key

    • Problem: A new agent needs the key, but GET /pipelines/{pipelineID} does not return it.
    • Why it happens: Monitoring and Logging keys are shown once; rotation revokes the old key at once.
    • Fix: Mark the Terraform output sensitive, store the key in your secret store, and update every agent in the same change when you rotate.
  3. Flow log changes that recreate a NIC

    • Problem: Changing action or removing the flowlog block plans a replacement of the worker's NIC.
    • Why it happens: Flow log settings are immutable, and deleting a flow log forces NIC re-creation.
    • Fix: Decide the capture settings once, change only name in place, and plan flow log removals as a NIC replacement in a maintenance window.

Summary

You can now give TaskBoard observability as code: Terraform creates the pipelines, Prometheus and Fluent Bit push to them, and OpenTelemetry traces link to log lines through the trace ID. When something breaks, you work from kubectl outwards to metrics, logs, traces, the Activity Log and Flow Logs.

Key Points:

  • Every observability service is push-based: create a pipeline, then point an agent or SDK at its endpoint with the pipeline key
  • Pipeline keys are shown once; rotating a key revokes the old one
  • MKS API-server audit logs go to an Object Storage bucket you name; ship pod logs with an agent you deploy
  • The Activity Log API is read-only, keeps 35 days, and covers IaaS products
  • Flow log settings are immutable; records land in your own Object Storage bucket every 10 minutes

Important Terminology:

  • Pipeline: A Monitoring, Logging or Tracing Service instance with its own ingestion endpoint and key
  • Remote write: The Prometheus protocol for pushing samples to /api/v1/push
  • LogQL: The Loki query language you use in Grafana Explore to search logs
  • OTLP: The OpenTelemetry Protocol, used over HTTP or gRPC to send traces
  • Flow log record: A space-separated line describing one network flow, its byte count and its ACCEPT or REJECT action

Next Steps

Continue Learning: Unit 5.2: Security Automation

Related Topics: