9 min read

Learning Objectives

By the end of this module, you will be able to:

  • Design topics around the partition as the unit of ordering and consumer parallelism, with partition counts that fit the broker count
  • Use replication factor, retention and cluster size as reliability and cost levers, and size storage from throughput and retention
  • Design application-level event capture as the substitute for database change-data-capture
  • Place Object Storage as the archive and dead-letter tail of a stream, and secure clients with mutual TLS
  • Create an Event Streams for Apache Kafka cluster and a well-shaped topic in the Data Center Designer (DCD)

Unit 5.6: Event Streaming (Managed Kafka)

Introduction

FinCorp's fraud models, its notification service and its reporting warehouse all need to know when a payment happens. Polling the payment database from three consumers would load the primary that Unit 5.3 worked to protect. The architecture therefore needs a log of business events, published by the application, that any number of consumers can read at their own pace.

Event Streams for Apache Kafka is a fully managed Apache Kafka service with automatic failover and TLS-secured communication, attached to a private LAN. The cluster is quick to create; the decisions that last are the topic shapes, because the partition count and the key decide ordering and parallelism for the life of the topic. This unit designs those decisions and ends by building FinCorp's event cluster and its first topic in the DCD.

1. Partitions: Ordering and Parallelism

1.1 What a Partition Guarantees

Kafka maintains the order of messages within a partition, not across a topic. A producer that sends messages with a key places all messages with the same key in the same partition, so ordering holds per key. For FinCorp, the key of a payment event is the account ID: every event of one account arrives in order, while events of different accounts are processed in parallel.

Partitions are also the unit of parallel processing. Within a consumer group, each partition is read by one consumer at a time, and each group tracks its own position, the offset, per partition. Three consumer groups, for fraud, notification and reporting, therefore read the same topic independently, and the partition count caps how many consumers in one group do useful work.

1.2 Partition Counts and the Broker Count

Every cluster size runs three brokers. The partition count should be equal to or greater than the number of brokers, and 2x or 3x the broker count or more when throughput is a priority. Use only multiples of the broker count, such as 3, 6, 9 or 12, to avoid an uneven distribution of partitions. Choose the count for the peak consumer parallelism you expect, because adding partitions later moves keys to different partitions and breaks per-key ordering across the change.

2. Reliability and Cost Levers

2.1 Replication Factor and Retention

The replication factor sets how many copies of each message are kept on different brokers; a replication factor of 3 is recommended, and with three brokers it puts a copy on each. Retention time decides how long messages stay in a topic. It defaults to 604800000 ms (7 days), and -1 removes the time limit. The segment size, at which log segments are rolled over, defaults to 1073741824 bytes (1 GB) and must be at least 14 bytes.

Both levers multiply storage. The final storage consumption can be several times the incoming data volume, depending on the replication factor.

2.2 Cluster Sizes

The following table lists the cluster sizes:

Size Brokers Cores per broker RAM per broker Storage per broker Total storage
XS 3 1 2 GB 195 GB 585 GB
S 3 2 4 GB 250 GB 750 GB
M 3 2 8 GB 400 GB 1200 GB
L 3 4 16 GB 800 GB 2400 GB
XL 3 8 32 GB 1500 GB 4500 GB

As shown above, every size adds capacity per broker, not brokers. Size from throughput and retention. FinCorp expects about 100 KB per second of payment events and 50 KB per second of customer events, kept for seven days: 150 KB/s x 604,800 s is about 91 GB of incoming data, about 272 GB with a replication factor of 3. The S size, with 750 GB, leaves headroom for growth and for a longer retention on the dead-letter topic.

2.3 Security and Placement

Communication with the cluster is TLS-secured, and authentication uses mutual TLS (mTLS): the client verifies the server's certificate against the cluster's certificate authority, because the cluster has no publicly signed certificates, and the server verifies the client's certificate. Each certificate is valid for 365 days, and a renewed certificate is available from the API 30 days before expiry. Certificate rotation is therefore a yearly runbook item, with a 30-day window to roll it out to every producer and consumer.

The cluster connects to a LAN in your VDC, and the broker addresses must belong to that LAN's subnet. Kafka is available in Berlin (de-txl) and Frankfurt (de-fra, de-fra-1 and de-fra-2), among other locations.

3. The Stream in the Data Architecture

3.1 Application-Level Event Capture

In the native-substitution model, change-data-capture is composed at the application level (Unit 1.3): the application publishes each business change as an event. The risk is a write that reaches the database while its event never reaches Kafka, or the reverse. The standard remedy is the transactional outbox: the payment service writes the payment and an outbox row in the same PostgreSQL transaction, and a relay process reads new outbox rows, publishes them to Kafka and marks them sent. A crash then causes at most a duplicate, never a lost event, so consumers must be idempotent, for example by tracking the event ID. The outbox lives in the Strictly Synchronous cluster of Unit 5.3, so a database failover does not lose events either.

3.2 Archive and Dead-Letter Tail

Retention is not an archive. A consumer job writes each topic's events to IONOS Cloud Object Storage in time-partitioned objects, under a lifecycle rule (Unit 5.2), so that reporting and model training can replay history beyond seven days. Deleting a topic permanently removes its data, which is one more reason to archive first.

Messages that a consumer cannot process go to a dead-letter topic, payments.dlq, instead of blocking the partition. A small consumer moves dead letters to an Object Storage prefix for investigation, and the lifecycle rule on that prefix expires them after the investigation window.

3.3 The Ingestion Spine for AI

The same topic feeds FinCorp's AI tier. Consumers in fc-prod-fra-ai read over the Cross Connect of Unit 3.6, which must therefore join the fc-data LAN, as their own consumer group, so model features update from the event stream without touching the payment database. Unit 6.5 builds the inference side.

DCD Implementation Walkthrough

You will create FinCorp's event cluster fc-kafka-events on fc-data and its first topic, payments.events, shaped by Sections 1 and 2. This realises the event log that the outbox relay writes and three consumer groups read.

Prerequisites: the VDC fc-prod-fra-core with at least one VM on the fc-data LAN; three free addresses on fc-data (FinCorp uses .7 to .9, which fills the managed-service range of that LAN; the next managed service needs a new LAN); the IONOS CLOUD API access of the platform team to retrieve credentials.

Build goal: Create a cluster and a well-shaped topic.

Steps (in the Data Center Designer):

  1. Go to Menu > Data Analytics > Event Streams for Apache Kafka and select Create cluster.
  2. Enter the Cluster Name fc-kafka-events, select the Kafka Version, choose the Cluster Size S, and select the Frankfurt Location.
  3. Select fc-prod-fra-core as the Datacenter and fc-data as the Datacenter LAN.
  4. Enter the three broker addresses .7, .8 and .9 of the fc-data subnet. They must belong to that LAN's subnet.
  5. Select Save and wait until the cluster is in the Available state.
  6. Select the cluster, open the Topics tab and select Create topic.
  7. Enter the Name payments.events, a Replication Factor of 3, and a Number of Partitions of 6, twice the broker count.
  8. Keep Retention Time (ms) at 604800000 for seven days, and keep the default Retention Segment Size (B).
  9. Select Create. Repeat for customer.events and for payments.dlq, giving the dead-letter topic a longer retention time.
  10. Retrieve the client credentials, the certificate authority, certificate and private key, through the API endpoint GET /clusters/{clusterId}/users/{userId}/access, and store them in the secret store used by the outbox relay and the consumers.

Common mistakes:

  • Choosing a partition count that is not a multiple of three. Partitions then spread unevenly across the brokers.
  • Keying payment events randomly. Ordering holds only per key within a partition; key by account ID.
  • Planning to add partitions later "when needed". It moves keys and breaks per-key ordering.
  • Treating seven days of retention as the archive. Archive to Object Storage before data ages out or a topic is deleted.
  • Letting client certificates expire. They are valid for 365 days; schedule the renewal in the 30-day window.

Summary

A Kafka topic's partition count and key are its lasting design: ordering holds per key within a partition, parallelism is capped by the partition count, and the count should be a multiple of the three brokers. Replication factor and retention multiply storage, so size the cluster from throughput times retention times replication. The stream substitutes for change-data-capture through the transactional outbox, hands its history to Object Storage, and feeds the AI tier as one more consumer group.

Key Points:

  • Every cluster size has three brokers; sizes differ in cores, RAM and storage per broker.
  • Use partition counts in multiples of the broker count, at least equal to it.
  • Replication factor 3 is recommended; default retention is seven days.
  • Clients authenticate with mTLS; certificates last 365 days, with renewal available 30 days ahead.
  • The outbox pattern makes application-level event capture safe against crashes.

Important Terminology:

  • Partition: an ordered log within a topic, the unit of ordering and of consumer parallelism.
  • Consumer group: a set of consumers that share a topic's partitions and track their offsets together.
  • Transactional outbox: a table written in the same database transaction as the business change, from which a relay publishes events.
  • Dead-letter topic: a topic that receives messages a consumer could not process.