13 min read

Learning Objectives

By the end of this module, you will be able to:

  • Map the five telemetry planes (metrics, logs, audit, network flow and traces) to the services that carry them, with their fixed scopes, retention and limits
  • Design around the gaps between the planes: per-contract audit, the 35-day audit window, cluster audit logs in a bucket, and logs that only arrive when you ship them
  • Use Central Logging and Central Monitoring for product-initiated forwarding, and explain the two switches that must both be on
  • Place Private Cloud and vCenter observability beside the platform planes in a hybrid estate
  • Design the fan-in to an external SIEM, then build a metrics pipeline with an alert and a flow log in the Data Center Designer (DCD)

Unit 7.2: Observability and Operations

Introduction

Every earlier module produced telemetry in a different place: activity log entries per contract (Unit 2.3), flow logs in a bucket (Unit 3.2), database backups and metrics (Unit 5.3), Kubernetes audit logs in another bucket (Unit 6.1). An operations team cannot work from six places. It needs to know what each plane can see, how long it keeps it, and where the planes meet.

This unit maps the planes, names the gaps that a design must close, and builds FinCorp's first operational layer: product-initiated forwarding for the databases, a metrics pipeline for the VMs with an alert, and a flow log on the egress path.

1. The Five Telemetry Planes

1.1 What Each Plane Carries

The platform splits telemetry by kind, and each plane has its own scope and retention:

Plane Service How data arrives Scope Retention
Metrics Monitoring Service Pushed by your agents (Prometheus, Grafana Agent, OpenTelemetry, FluentBit), or forwarded by integrated products Regional pipelines, up to 10 per contract Storage billed per GB per 30 days, growing with your retention settings
Logs Logging Service Fluent Bit or HTTP from your sources (Kubernetes, Docker, Systemd, Generic, OTLP), or forwarded by integrated products Regional pipelines, up to 10 per contract, 5 log streams each 7, 14 or 30 days per log stream (default 30), or unlimited
Audit Activity Logs; Kubernetes audit logging Activity Logs: you pull through a read-only, GET-only API. Cluster audit logs: written to a bucket you name Activity Logs: one contract, IaaS activity Activity Logs: 35 days
Network flow Flow Logs Written to a user-owned Object Storage bucket in 10-minute intervals One flow log per VM NIC, NLB, ALB or NAT Gateway Until your bucket's lifecycle policy deletes it
Traces Tracing Service OTLP/HTTP or OTLP/gRPC on port 443 Regional pipelines, up to 10 per contract 30 days, then deleted automatically

Three planes share one lens. Metrics, logs and traces are visualised in a managed Grafana instance per contract and region, backed by Grafana Mimir for metrics, Loki for logs and Tempo for traces. From a slow span you can move to the logs and metrics of the same request. Audit and flow data do not arrive in Grafana by default: they land in an API and in buckets.

1.2 Traces, the Fifth Plane

The Tracing Service is OpenTelemetry-native: it accepts OTLP without proprietary agents, so the instrumentation stays portable. Its limits are design inputs rather than footnotes:

  • Maximum trace size: 5 MB. If the spans of one trace ID add up to more, the trace is rejected at ingestion and never appears in Grafana. Long batch jobs with thousands of spans must be split into separate traces.
  • Ingestion rate: 15 MB/s sustained, with a 20 MB burst. Above it the platform answers HTTP 429 for OTLP/HTTP or RESOURCE_EXHAUSTED for OTLP/gRPC; exporters retry with backoff, and sustained excess is discarded.
  • Live traces: 10,000 concurrent in-progress traces. Above that, new traces are refused until open ones are flushed.
  • Late spans arriving after their trace has been flushed are stored as a separate, partial trace, so keep clock skew small and export promptly.

FinCorp uses traces for the portal request path and for its AI assistant, since the service supports the OpenTelemetry semantic conventions for generative AI, including model latency and token consumption. Thirty days covers incident analysis; anything a regulator asks for later belongs in the audit plane, not in traces.

1.3 The Logging Limits

Log ingestion has fixed default limits per pipeline:

Aspect Description Limit
HTTP Rate Limit Default rate limit for HTTP requests per pipeline during log ingestion. 50 requests/second
TCP Bandwidth Default TCP bandwidth limit per pipeline, approximately in terms of logs per second. ~10,000 logs/second
Maximum Pipelines The maximum number of pipelines allowed per contract. 10 pipelines
Log Streams per Pipeline The maximum number of log streams allowed per pipeline. 5 log streams/pipeline

As shown above, the HTTP path suits low-volume senders; high-volume sources such as a busy Kubernetes cluster ship through Fluent Bit over TCP. With five streams per pipeline and ten pipelines per contract, plan streams by source type and retention class, not one per server. Data masking is your job: sanitise personal data in the agent before it leaves the source.

2. Product-Initiated Forwarding: Central Logging and Monitoring

Some managed products forward their own logs and metrics, so you run no agent for them. The switch for this is Central Logging and Central Monitoring: a contract-level flag per region that lets integrated products ingest on your behalf. It is activated in the DCD with a per-region toggle, or through the regional API with a PUT to /central. To find out which products integrate, ask your account manager or IONOS CLOUD Support.

Two layers must both be on before any data appears:

  1. Contract and region: Central Logging and Central Monitoring activated for the region where the product runs.
  2. Product: the product's own toggle, such as Enable logs and Enable metrics in a DBaaS cluster's Observability settings (Unit 5.3).

The order does not block you: a cluster accepts the toggles while the region is still inactive, but no data flows until the regional switch is on. That is also the trap: a cluster can show both toggles on and still send nothing. Once both layers are on, Grafana shows six predefined dashboards under Dashboards > IONOS CLOUD, a metrics and a logs dashboard each for DBaaS for PostgreSQL, DBaaS for MariaDB and DBaaS for In-Memory DB, including replication and WAL archiving for PostgreSQL. Activation is regional and billable, so activate it only in the regions where FinCorp runs products: Frankfurt and Berlin.

3. Grafana Access Is Governed by IAM

Grafana access is tied to the pipelines and to IAM single sign-on (SSO). Login works as soon as one pipeline or a Central Logging or Monitoring instance exists; before that, it fails with "Sign up is disabled". If all of them are deleted, access remains for 30 days and is then removed, service accounts are disabled and their tokens become invalid. Keep Central Monitoring active in production so that Grafana never depends on one pipeline.

Revoking access needs two steps: remove or restrict the user in IAM for the contract, then remove the user from the Grafana organization. Changing privileges in the DCD alone may not immediately stop Grafana SSO. Add both steps to the leaver runbook of Unit 2.2. Creating pipelines needs the Access and manage Monitoring, Access and manage Logging Service or Access and manage Tracing privilege; grant them to the platform team's group, not to individuals.

4. The Gaps to Design Around

Each plane is complete within its scope. The design work is at the edges:

Gap Consequence Native pattern
Activity Logs keep 35 days, per contract Older evidence is gone; no view across FinCorp's two contracts Scheduled export of every contract into a locked bucket (Unit 2.3)
Activity Logs trace records are limited to IaaS products Activity inside PaaS products needs another record Use each service's own logs, through Central Logging where it integrates
Kubernetes audit logs land in a bucket Cluster audit events sit in objects, not in a log stream Read the audit bucket from the SIEM, or forward it with a collector
Application and node logs arrive only when shipped A cluster without an agent has no logs in Grafana Deploy Fluent Bit in every cluster to a Kubernetes log stream (Unit 6.1)
Flow logs land in a bucket Network evidence is not in Grafana Lifecycle policy for retention; SIEM reads the bucket
A TCP health check proves only that a port answers A broken service can pass the NLB check (Unit 3.4) Application metrics and alerts on the service itself

5. Hybrid Estates: vCenter Beside the Platform Planes

In Private Cloud, virtual machine monitoring is a customer responsibility: IONOS CLOUD support covers the hardware and the VMware stack, not the monitoring of your VMs (Unit 4.4). The vSphere Client and vCenter therefore stay the operational console for hosts, VMs and vSAN capacity, including the 30% slack rule.

The two worlds meet in Grafana, not in vCenter. Monitoring Service accepts Prometheus-compatible metrics from anywhere that can reach the regional endpoint over HTTPS on port 443, and Fluent Bit can send logs from any source that speaks TCP or HTTP. FinCorp therefore runs collectors on VMs inside the Private Cloud and pushes guest and application metrics and logs to the same Frankfurt pipelines as the VDC estate. Enabling the management level in the Cloud Panel creates an NSX segment from which such tools can reach the vSphere management plane. vSAN capacity alerts stay in vCenter, and the runbook names the person who watches them.

6. Fan-In to an External SIEM

FinCorp's security operations run a Security Information and Event Management (SIEM) system outside the platform. Every plane can feed it, through a different path:

Source Path to the SIEM
Activity Logs The exporter from Unit 2.3 writes to the locked archive and sends the same entries to the SIEM
Flow logs and Kubernetes audit logs The SIEM reads the user-owned buckets through the S3-compatible API
VM and container logs The Fluent Bit agent sends to the Logging Service and, through a second output, to the SIEM
Metrics An external Grafana queries the IONOS CLOUD Grafana through a Prometheus data source with a Grafana service account

The pattern keeps Grafana as the engineers' working view and the SIEM as the security record, without making either depend on the other. The sending side is always the agent or exporter you operate, so the SIEM path survives a change of pipeline.

DCD Implementation Walkthrough

You will switch on product-initiated forwarding for FinCorp's databases in Frankfurt, create a metrics pipeline for the VMs, add an alert, and enable the egress flow log that Unit 3.2 planned. This realises Sections 2 and 4: both forwarding layers on, and network evidence on the path that leaves the VDC.

Prerequisites: the Access and manage Monitoring, Access and manage Logging Service and Create Flow logs privileges, or the Administrator role; fc-pg-portal with Enable logs and Enable metrics turned on (Unit 5.3); the NAT Gateway fc-nat-app (Unit 3.6); a user-owned Object Storage bucket for flow logs, here fc-flowlogs-fra; outbound HTTPS from the VMs through the NAT Gateway.

Build goal: Create a metrics pipeline with an alert and enable flow logs.

Steps (in the Data Center Designer):

  1. Go to Menu > Observability > Logging Service, open the Central Logging drop-down list, toggle the STATE for the Frankfurt region and select Confirm. Connection Details lists the forwarding products.
  2. Go to Menu > Observability > Monitoring Service, open the Central Monitoring drop-down list, toggle the STATE for the Frankfurt region and select Confirm. Repeat both steps for Berlin, where the recovery cluster of Unit 7.1 runs.
  3. On the Monitoring Service overview page, select Create monitoring pipeline. Enter a Name such as fc-vm-metrics-fra, select the Frankfurt Location, and select Create. Copy the API key into FinCorp's secret store at once: it is shown only on creation.
  4. Configure the metrics agent on fc-app-01 and fc-app-02 with the pipeline's endpoint, adding /api/v1/push, and the API key in the APIKEY header. The default push interval is one minute.
  5. Go to Menu > Databases > PostgreSQL, open fc-pg-portal, and in the Observability section select Open in Grafana. Log in with IONOS CLOUD SSO. Initial database metrics can take 10 minutes to 1 hour to appear.
  6. In Grafana, open Dashboards > IONOS CLOUD and check the PostgreSQL metrics dashboard for fc-pg-portal. Create a Grafana alert rule on storage usage and one on replication, with notifications to the operations on-call channel. Add a CPU alert on the portal VMs from the pipeline of step 3.
  7. Open fc-prod-fra-core, select fc-nat-app and its Settings tab. Enter a flow log Name, choose Direction Bidirectional and Action Any, enter fc-flowlogs-fra as Target Object Storage bucket, and select Add flow log.
  8. Select Provision changes. After the next 10-minute interval, confirm that .log.gz objects appear in the bucket, and set the bucket's lifecycle policy to the retention period the security team signed off.

Common mistakes:

  • Turning on Enable logs and Enable metrics on a cluster without activating the region. The cluster accepts it and sends nothing.
  • Losing a pipeline's API key. It is displayed once and cannot be retrieved; you must issue a new one.
  • Revoking a leaver's DCD privileges but not their IAM access and Grafana membership.
  • Exporting very long batch jobs as one trace. Above 5 MB, the trace is rejected.
  • Expecting cluster audit events or flow logs in Grafana. They land in buckets.
  • Editing a flow log after creation. Delete it and create a new one.

Summary

Telemetry on IONOS CLOUD comes in five planes with fixed scopes: metrics, logs and traces in regional pipelines viewed in one Grafana instance per contract and region, audit in a per-contract API with a 35-day window and in cluster audit buckets, and network flows in user-owned buckets. Central Logging and Central Monitoring forward integrated products' data once both the regional and the product switch are on. The design closes the gaps with exports, agents and bucket readers, and fans everything into the SIEM through agents and exporters that FinCorp operates.

Key Points:

  • Pipelines are regional, up to 10 per contract for each of metrics, logs and traces; Grafana is per contract and region.
  • Logs keep 7, 14 or 30 days (default 30) or unlimited; traces 30 days; Activity Logs 35 days.
  • Traces are capped at 5 MB each and 15 MB/s sustained ingestion.
  • Product-initiated forwarding needs Central Logging or Monitoring for the region and the product's own toggle.
  • Audit and flow data land in an API and buckets; the SIEM reads them there.

Important Terminology:

  • Central Logging / Central Monitoring: a per-region, contract-level switch that lets integrated products forward their logs or metrics to your observability services.
  • Telemetry plane: one kind of operational signal with its own service, scope and retention.
  • SIEM fan-in: collecting every plane into one external security system through the agents and exporters you operate.