Unit 7.2: Observability and Operations
Introduction
Every earlier module produced telemetry in a different place: activity log entries per contract (Unit 2.3), flow logs in a bucket (Unit 3.2), database backups and metrics (Unit 5.3), Kubernetes audit logs in another bucket (Unit 6.1). An operations team cannot work from six places. It needs to know what each plane can see, how long it keeps it, and where the planes meet.
This unit maps the planes, names the gaps that a design must close, and builds FinCorp's first operational layer: product-initiated forwarding for the databases, a metrics pipeline for the VMs with an alert, and a flow log on the egress path.
1. The Five Telemetry Planes
1.1 What Each Plane Carries
The platform splits telemetry by kind, and each plane has its own scope and retention:
| Plane | Service | How data arrives | Scope | Retention |
|---|---|---|---|---|
| Metrics | Monitoring Service | Pushed by your agents (Prometheus, Grafana Agent, OpenTelemetry, FluentBit), or forwarded by integrated products | Regional pipelines, up to 10 per contract | Storage billed per GB per 30 days, growing with your retention settings |
| Logs | Logging Service | Fluent Bit or HTTP from your sources (Kubernetes, Docker, Systemd, Generic, OTLP), or forwarded by integrated products | Regional pipelines, up to 10 per contract, 5 log streams each | 7, 14 or 30 days per log stream (default 30), or unlimited |
| Audit | Activity Logs; Kubernetes audit logging | Activity Logs: you pull through a read-only, GET-only API. Cluster audit logs: written to a bucket you name | Activity Logs: one contract, IaaS activity | Activity Logs: 35 days |
| Network flow | Flow Logs | Written to a user-owned Object Storage bucket in 10-minute intervals | One flow log per VM NIC, NLB, ALB or NAT Gateway | Until your bucket's lifecycle policy deletes it |
| Traces | Tracing Service | OTLP/HTTP or OTLP/gRPC on port 443 | Regional pipelines, up to 10 per contract | 30 days, then deleted automatically |
Three planes share one lens. Metrics, logs and traces are visualised in a managed Grafana instance per contract and region, backed by Grafana Mimir for metrics, Loki for logs and Tempo for traces. From a slow span you can move to the logs and metrics of the same request. Audit and flow data do not arrive in Grafana by default: they land in an API and in buckets.
1.2 Traces, the Fifth Plane
The Tracing Service is OpenTelemetry-native: it accepts OTLP without proprietary agents, so the instrumentation stays portable. Its limits are design inputs rather than footnotes:
- Maximum trace size: 5 MB. If the spans of one trace ID add up to more, the trace is rejected at ingestion and never appears in Grafana. Long batch jobs with thousands of spans must be split into separate traces.
- Ingestion rate: 15 MB/s sustained, with a 20 MB burst. Above it the platform answers HTTP 429 for OTLP/HTTP or
RESOURCE_EXHAUSTEDfor OTLP/gRPC; exporters retry with backoff, and sustained excess is discarded. - Live traces: 10,000 concurrent in-progress traces. Above that, new traces are refused until open ones are flushed.
- Late spans arriving after their trace has been flushed are stored as a separate, partial trace, so keep clock skew small and export promptly.
FinCorp uses traces for the portal request path and for its AI assistant, since the service supports the OpenTelemetry semantic conventions for generative AI, including model latency and token consumption. Thirty days covers incident analysis; anything a regulator asks for later belongs in the audit plane, not in traces.
1.3 The Logging Limits
Log ingestion has fixed default limits per pipeline:
| Aspect | Description | Limit |
|---|---|---|
| HTTP Rate Limit | Default rate limit for HTTP requests per pipeline during log ingestion. | 50 requests/second |
| TCP Bandwidth | Default TCP bandwidth limit per pipeline, approximately in terms of logs per second. | ~10,000 logs/second |
| Maximum Pipelines | The maximum number of pipelines allowed per contract. | 10 pipelines |
| Log Streams per Pipeline | The maximum number of log streams allowed per pipeline. | 5 log streams/pipeline |
As shown above, the HTTP path suits low-volume senders; high-volume sources such as a busy Kubernetes cluster ship through Fluent Bit over TCP. With five streams per pipeline and ten pipelines per contract, plan streams by source type and retention class, not one per server. Data masking is your job: sanitise personal data in the agent before it leaves the source.
2. Product-Initiated Forwarding: Central Logging and Monitoring
Some managed products forward their own logs and metrics, so you run no agent for them. The switch for this is Central Logging and Central Monitoring: a contract-level flag per region that lets integrated products ingest on your behalf. It is activated in the DCD with a per-region toggle, or through the regional API with a PUT to /central. To find out which products integrate, ask your account manager or IONOS CLOUD Support.
Two layers must both be on before any data appears:
- Contract and region: Central Logging and Central Monitoring activated for the region where the product runs.
- Product: the product's own toggle, such as Enable logs and Enable metrics in a DBaaS cluster's Observability settings (Unit 5.3).
The order does not block you: a cluster accepts the toggles while the region is still inactive, but no data flows until the regional switch is on. That is also the trap: a cluster can show both toggles on and still send nothing. Once both layers are on, Grafana shows six predefined dashboards under Dashboards > IONOS CLOUD, a metrics and a logs dashboard each for DBaaS for PostgreSQL, DBaaS for MariaDB and DBaaS for In-Memory DB, including replication and WAL archiving for PostgreSQL. Activation is regional and billable, so activate it only in the regions where FinCorp runs products: Frankfurt and Berlin.
3. Grafana Access Is Governed by IAM
Grafana access is tied to the pipelines and to IAM single sign-on (SSO). Login works as soon as one pipeline or a Central Logging or Monitoring instance exists; before that, it fails with "Sign up is disabled". If all of them are deleted, access remains for 30 days and is then removed, service accounts are disabled and their tokens become invalid. Keep Central Monitoring active in production so that Grafana never depends on one pipeline.
Revoking access needs two steps: remove or restrict the user in IAM for the contract, then remove the user from the Grafana organization. Changing privileges in the DCD alone may not immediately stop Grafana SSO. Add both steps to the leaver runbook of Unit 2.2. Creating pipelines needs the Access and manage Monitoring, Access and manage Logging Service or Access and manage Tracing privilege; grant them to the platform team's group, not to individuals.
4. The Gaps to Design Around
Each plane is complete within its scope. The design work is at the edges:
| Gap | Consequence | Native pattern |
|---|---|---|
| Activity Logs keep 35 days, per contract | Older evidence is gone; no view across FinCorp's two contracts | Scheduled export of every contract into a locked bucket (Unit 2.3) |
| Activity Logs trace records are limited to IaaS products | Activity inside PaaS products needs another record | Use each service's own logs, through Central Logging where it integrates |
| Kubernetes audit logs land in a bucket | Cluster audit events sit in objects, not in a log stream | Read the audit bucket from the SIEM, or forward it with a collector |
| Application and node logs arrive only when shipped | A cluster without an agent has no logs in Grafana | Deploy Fluent Bit in every cluster to a Kubernetes log stream (Unit 6.1) |
| Flow logs land in a bucket | Network evidence is not in Grafana | Lifecycle policy for retention; SIEM reads the bucket |
| A TCP health check proves only that a port answers | A broken service can pass the NLB check (Unit 3.4) | Application metrics and alerts on the service itself |
5. Hybrid Estates: vCenter Beside the Platform Planes
In Private Cloud, virtual machine monitoring is a customer responsibility: IONOS CLOUD support covers the hardware and the VMware stack, not the monitoring of your VMs (Unit 4.4). The vSphere Client and vCenter therefore stay the operational console for hosts, VMs and vSAN capacity, including the 30% slack rule.
The two worlds meet in Grafana, not in vCenter. Monitoring Service accepts Prometheus-compatible metrics from anywhere that can reach the regional endpoint over HTTPS on port 443, and Fluent Bit can send logs from any source that speaks TCP or HTTP. FinCorp therefore runs collectors on VMs inside the Private Cloud and pushes guest and application metrics and logs to the same Frankfurt pipelines as the VDC estate. Enabling the management level in the Cloud Panel creates an NSX segment from which such tools can reach the vSphere management plane. vSAN capacity alerts stay in vCenter, and the runbook names the person who watches them.
6. Fan-In to an External SIEM
FinCorp's security operations run a Security Information and Event Management (SIEM) system outside the platform. Every plane can feed it, through a different path:
| Source | Path to the SIEM |
|---|---|
| Activity Logs | The exporter from Unit 2.3 writes to the locked archive and sends the same entries to the SIEM |
| Flow logs and Kubernetes audit logs | The SIEM reads the user-owned buckets through the S3-compatible API |
| VM and container logs | The Fluent Bit agent sends to the Logging Service and, through a second output, to the SIEM |
| Metrics | An external Grafana queries the IONOS CLOUD Grafana through a Prometheus data source with a Grafana service account |
The pattern keeps Grafana as the engineers' working view and the SIEM as the security record, without making either depend on the other. The sending side is always the agent or exporter you operate, so the SIEM path survives a change of pipeline.
DCD Implementation Walkthrough
You will switch on product-initiated forwarding for FinCorp's databases in Frankfurt, create a metrics pipeline for the VMs, add an alert, and enable the egress flow log that Unit 3.2 planned. This realises Sections 2 and 4: both forwarding layers on, and network evidence on the path that leaves the VDC.
Prerequisites: the Access and manage Monitoring, Access and manage Logging Service and Create Flow logs privileges, or the Administrator role; fc-pg-portal with Enable logs and Enable metrics turned on (Unit 5.3); the NAT Gateway fc-nat-app (Unit 3.6); a user-owned Object Storage bucket for flow logs, here fc-flowlogs-fra; outbound HTTPS from the VMs through the NAT Gateway.
Build goal: Create a metrics pipeline with an alert and enable flow logs.
Steps (in the Data Center Designer):
- Go to Menu > Observability > Logging Service, open the Central Logging drop-down list, toggle the STATE for the Frankfurt region and select Confirm. Connection Details lists the forwarding products.
- Go to Menu > Observability > Monitoring Service, open the Central Monitoring drop-down list, toggle the STATE for the Frankfurt region and select Confirm. Repeat both steps for Berlin, where the recovery cluster of Unit 7.1 runs.
- On the Monitoring Service overview page, select Create monitoring pipeline. Enter a Name such as
fc-vm-metrics-fra, select the Frankfurt Location, and select Create. Copy the API key into FinCorp's secret store at once: it is shown only on creation. - Configure the metrics agent on
fc-app-01andfc-app-02with the pipeline's endpoint, adding/api/v1/push, and the API key in theAPIKEYheader. The default push interval is one minute. - Go to Menu > Databases > PostgreSQL, open
fc-pg-portal, and in the Observability section select Open in Grafana. Log in with IONOS CLOUD SSO. Initial database metrics can take 10 minutes to 1 hour to appear. - In Grafana, open Dashboards > IONOS CLOUD and check the PostgreSQL metrics dashboard for
fc-pg-portal. Create a Grafana alert rule on storage usage and one on replication, with notifications to the operations on-call channel. Add a CPU alert on the portal VMs from the pipeline of step 3. - Open
fc-prod-fra-core, selectfc-nat-appand its Settings tab. Enter a flow log Name, choose DirectionBidirectionaland ActionAny, enterfc-flowlogs-fraas Target Object Storage bucket, and select Add flow log. - Select Provision changes. After the next 10-minute interval, confirm that
.log.gzobjects appear in the bucket, and set the bucket's lifecycle policy to the retention period the security team signed off.
Common mistakes:
- Turning on Enable logs and Enable metrics on a cluster without activating the region. The cluster accepts it and sends nothing.
- Losing a pipeline's API key. It is displayed once and cannot be retrieved; you must issue a new one.
- Revoking a leaver's DCD privileges but not their IAM access and Grafana membership.
- Exporting very long batch jobs as one trace. Above 5 MB, the trace is rejected.
- Expecting cluster audit events or flow logs in Grafana. They land in buckets.
- Editing a flow log after creation. Delete it and create a new one.
Summary
Telemetry on IONOS CLOUD comes in five planes with fixed scopes: metrics, logs and traces in regional pipelines viewed in one Grafana instance per contract and region, audit in a per-contract API with a 35-day window and in cluster audit buckets, and network flows in user-owned buckets. Central Logging and Central Monitoring forward integrated products' data once both the regional and the product switch are on. The design closes the gaps with exports, agents and bucket readers, and fans everything into the SIEM through agents and exporters that FinCorp operates.
Key Points:
- Pipelines are regional, up to 10 per contract for each of metrics, logs and traces; Grafana is per contract and region.
- Logs keep 7, 14 or 30 days (default 30) or unlimited; traces 30 days; Activity Logs 35 days.
- Traces are capped at 5 MB each and 15 MB/s sustained ingestion.
- Product-initiated forwarding needs Central Logging or Monitoring for the region and the product's own toggle.
- Audit and flow data land in an API and buckets; the SIEM reads them there.
Important Terminology:
- Central Logging / Central Monitoring: a per-region, contract-level switch that lets integrated products forward their logs or metrics to your observability services.
- Telemetry plane: one kind of operational signal with its own service, scope and retention.
- SIEM fan-in: collecting every plane into one external security system through the agents and exporters you operate.