12 min read

Learning Objectives

By the end of this module, you will be able to:

  • Connect a confluent-kafka producer and consumer to Event Streams for Apache Kafka with mutual TLS, using credentials and broker addresses from Terraform
  • Configure an idempotent producer with `acks=all` and tuned batching for TaskBoard's task-change events
  • Implement a consumer group with manual offset commits and at-least-once delivery
  • Build retry and dead-letter topics that isolate poison messages without blocking a partition
  • Scale consumers on Managed Kubernetes within the partition-count ceiling

Unit 4.3: Event Streaming Integration

Introduction

When a user completes a task, TaskBoard should notify the task's watchers. Sending those notifications inside the API request makes the request slow and fragile. Instead, the API publishes a task.completed event, and the worker service consumes it and sends the notifications at its own pace.

In Unit 2.5 you provisioned an Event Streams for Apache Kafka cluster and the task-events topic with Terraform. Now you write the client side. You configure mutual TLS (mTLS) with the cluster's certificates, write a reliable producer, run a consumer group with explicit offset commits, and route failing messages to retry and dead-letter topics. You also define the event contract and scale the worker on Kubernetes. In the Code Lab you produce and consume TaskBoard events end to end.

1. Connecting with mTLS

1.1 Credentials and Brokers from Terraform

Communication with the cluster is TLS secured, and both sides authenticate each other through mTLS. The cluster does not have publicly signed certificates, so your client validates the server with the cluster's certificate authority (CA). The provider reads the CA, the client certificate and the key for a user:

data "ionoscloud_kafka_users" "events" {
  cluster_id = ionoscloud_kafka_cluster.events.id
  location   = ionoscloud_kafka_cluster.events.location
}

data "ionoscloud_kafka_user_credentials" "admin" {
  cluster_id = ionoscloud_kafka_cluster.events.id
  location   = ionoscloud_kafka_cluster.events.location
  username   = "admin"
}

output "kafka_bootstrap" {
  value = join(",", ionoscloud_kafka_cluster.events.broker_addresses)
}

output "kafka_ca"   {
  value = data.ionoscloud_kafka_user_credentials.admin.certificate_authority
  sensitive = true
}
output "kafka_cert" {
  value = data.ionoscloud_kafka_user_credentials.admin.certificate
  sensitive = true
}
output "kafka_key"  {
  value = data.ionoscloud_kafka_user_credentials.admin.private_key
  sensitive = true
}

The computed broker_addresses attribute holds the IP address and port of each broker, for example 10.10.2.2:9093. The credentials belong to the Kafka administrator user, so treat them as admin secrets. The data source stores them in state; the ionoscloud_kafka_user_credentials ephemeral resource reads the same values without storing them.

Without Terraform, the same values come from the API:

curl -s https://kafka.de-txl.ionos.com/clusters/$CLUSTER_ID/users/$USER_ID/access \
  -H "Authorization: Bearer $IONOS_TOKEN" \
  | jq -r '.metadata.certificateAuthority' > ca-cert.pem

The .metadata.certificate and .metadata.privateKey fields hold the client certificate and key. Get the user ID from GET /clusters/{clusterId}/users.

1.2 Client Configuration

import os
from confluent_kafka import Producer

KAFKA_TLS = {
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "security.protocol": "SSL",
    "ssl.ca.location": "/etc/kafka/ca-cert.pem",
    "ssl.certificate.location": "/etc/kafka/client-cert.pem",
    "ssl.key.location": "/etc/kafka/client-key.pem",
    "ssl.endpoint.identification.algorithm": "none",
}

The client configuration for the cluster uses security.protocol=SSL and an empty ssl.endpoint.identification.algorithm, which turns off the host name check while the CA still verifies the certificate chain. In librdkafka, the value for "off" is none. The brokers live in your private DB LAN, so the API and worker must run on servers or node pools attached to that LAN (Unit 3.2).

Each client certificate is valid for 365 days. Thirty days before it expires, a renewed certificate is available from the user credentials API. Re-read the credentials on a schedule and roll your pods, or the clients stop connecting on the expiry date.

2. Producing Task Events Reliably

2.1 Topic Design

Use one topic per event family and key each message by the entity whose order matters. TaskBoard keys task-events by task ID, so all events of one task land in one partition, in order. The topic from Unit 2.5 has 6 partitions and replication factor 3.

Create the retry and dead-letter topics the same way:

resource "ionoscloud_kafka_cluster_topic" "task_events_retry" {
  cluster_id           = ionoscloud_kafka_cluster.events.id
  location             = ionoscloud_kafka_cluster.events.location
  name                 = "task-events-retry"
  replication_factor   = 3
  number_of_partitions = 6
}

resource "ionoscloud_kafka_cluster_topic" "task_events_dlq" {
  cluster_id           = ionoscloud_kafka_cluster.events.id
  location             = ionoscloud_kafka_cluster.events.location
  name                 = "task-events-dlq"
  replication_factor   = 3
  number_of_partitions = 3
  retention_time       = 2592000000   # 30 days to investigate failures
}

Each cluster has three brokers, so use partition counts that are multiples of 3 for an even distribution. Without retention_time, a topic keeps messages for 604800000 ms (7 days); -1 removes the time limit.

2.2 The Producer

import json, uuid, datetime

producer = Producer({
    **KAFKA_TLS,
    "acks": "all",
    "enable.idempotence": True,
    "linger.ms": 20,
    "batch.size": 65536,
    "compression.type": "lz4",
})

def publish_task_event(task_id: int, event_type: str, data: dict) -> None:
    event = {
        "event_id": str(uuid.uuid4()),
        "type": event_type,                 # "task.completed"
        "schema_version": 1,
        "occurred_at": datetime.datetime.now(datetime.UTC).isoformat(),
        "task_id": task_id,
        "data": data,
    }
    producer.produce("task-events", key=str(task_id), value=json.dumps(event),
                     on_delivery=_report)
    producer.poll(0)

def _report(err, msg):
    if err is not None:
        log.error("delivery failed: %s", err)   # alert; the event is not stored
  • acks=all waits until all in-sync replicas have the write, so a single broker failure loses nothing.
  • enable.idempotence stops retries from creating duplicates and keeps per-partition order.
  • linger.ms and batch.size trade a few milliseconds of latency for fewer, larger requests.

Call producer.flush(10) on shutdown, for example in your FastAPI lifespan handler, or the last batch is lost when Kubernetes stops the pod.

3. Consuming with Offset Control

3.1 Consumer Group and Commits

from confluent_kafka import Consumer, KafkaException

consumer = Consumer({
    **KAFKA_TLS,
    "group.id": "taskboard-notifier",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "max.poll.interval.ms": 300000,
})
consumer.subscribe(["task-events", "task-events-retry"])

while running:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        raise KafkaException(msg.error())
    handle_with_retry(msg)                       # Section 4
    consumer.commit(message=msg, asynchronous=False)
consumer.close()

All worker pods share the group.id, so Kafka divides the partitions among them. With enable.auto.commit off, the worker commits an offset only after it has handled the message.

3.2 Delivery Semantics

Commit point Delivery Failure effect
After processing (this code) At-least-once A crash between processing and commit replays the message
Before processing At-most-once A crash after commit loses the message

As the table shows, at-least-once can deliver a message twice, so make the handler idempotent. The notifier records each event_id in PostgreSQL with a unique constraint and skips events it has already sent.

def already_sent(event_id: str) -> bool:
    with engine.begin() as c:
        r = c.execute(text("insert into sent_events (event_id) values (:e) on conflict do nothing"),
                      {"e": event_id})
        return r.rowcount == 0

4. Retries, Dead Letters and Event Contracts

4.1 Retry and Dead-Letter Routing

A message that always fails, a poison message, must not block its partition. The worker retries a few times through the retry topic and then parks the message in the dead-letter topic:

from confluent_kafka import Message
from jsonschema import ValidationError

MAX_ATTEMPTS = 3

def handle_with_retry(msg: Message) -> None:
    headers = dict(msg.headers() or [])
    attempt = int(headers.get("attempt", b"0"))
    try:
        event = json.loads(msg.value())
        validate_event(event)                     # Section 4.2
        if not already_sent(event["event_id"]):
            send_notifications(event)
    except (json.JSONDecodeError, ValidationError) as exc:
        forward(msg, "task-events-dlq", attempt, exc)   # a retry cannot fix bad data
    except Exception as exc:
        target = "task-events-retry" if attempt + 1 < MAX_ATTEMPTS else "task-events-dlq"
        forward(msg, target, attempt, exc)

def forward(msg: Message, target: str, attempt: int, exc: Exception) -> None:
    producer.produce(target, key=msg.key(), value=msg.value(), headers={
        "attempt": str(attempt + 1),
        "error": type(exc).__name__,
        "source": f"{msg.topic()}/{msg.partition()}/{msg.offset()}",
    })
    producer.flush(5)

The original offset is committed after the message is safely in the next topic, so the partition moves on. Invalid JSON or a failed schema check goes to the dead-letter topic at once, because a retry cannot fix it. A small tool can read the dead-letter topic, fix the cause and produce the messages back to task-events.

4.2 Schema Management

Kafka stores bytes; the contract between API and worker is your responsibility. Define it as a JSON Schema in a shared package and validate on both sides:

from jsonschema import validate

TASK_EVENT_V1 = {
    "type": "object",
    "required": ["event_id", "type", "schema_version", "task_id", "occurred_at"],
    "properties": {
        "event_id": {"type": "string"},
        "type": {"enum": ["task.created", "task.updated", "task.completed"]},
        "schema_version": {"const": 1},
        "task_id": {"type": "integer"},
    },
}

def validate_event(event: dict) -> None:
    validate(instance=event, schema=TASK_EVENT_V1)

Evolve the schema additively: add optional fields, never rename or remove one. For a breaking change, raise schema_version and keep the consumer reading both versions until no old producer remains. Avro with a schema registry gives the same discipline with compiler support.

5. Scaling Consumers on Kubernetes

The partition count is the parallelism ceiling of a consumer group: each partition is read by at most one consumer in the group. With 6 partitions, a seventh worker pod stays idle.

apiVersion: apps/v1
kind: Deployment
metadata: { name: taskboard-worker }
spec:
  replicas: 3                  # at most 6, the partition count of task-events
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: worker
          image: <registry-hostname>/taskboard/worker:<git-sha>
          envFrom:
            - secretRef: { name: taskboard-kafka }   # KAFKA_BOOTSTRAP
          volumeMounts:
            - { name: kafka-tls, mountPath: /etc/kafka, readOnly: true }
      volumes:
        - name: kafka-tls
          secret: { secretName: taskboard-kafka-tls }  # ca-cert.pem, client-cert.pem, client-key.pem

When a pod starts or stops, the group rebalances, and partitions move between pods. Handle SIGTERM by leaving the poll loop and calling consumer.close(), so the group rebalances at once instead of waiting for a session timeout. Keep each message's processing well below max.poll.interval.ms, or the broker removes the consumer from the group. To raise the ceiling, add partitions in multiples of 3; note that this changes which partition a key maps to.

API Reference Quick Card

Key Kafka API endpoints for this unit:

Method Endpoint Description
GET /clusters/{clusterId} Read the cluster state and brokerAddresses
GET /clusters/{clusterId}/users List users and their IDs
GET /clusters/{clusterId}/users/{userId}/access Retrieve the CA, client certificate and private key
POST /clusters/{clusterId}/topics Create a topic
DELETE /clusters/{clusterId}/topics/{topicId} Delete a topic and its data

Base URL: regional, for example https://kafka.de-txl.ionos.com Authentication: Authorization: Bearer <token> for the API; mTLS for producers and consumers

Code Lab

Objective: Produce and consume TaskBoard task-change events on Event Streams for Apache Kafka with a consumer group, manual offset commits and a dead-letter topic.

Prerequisites:

  • The Kafka cluster and task-events topic from Unit 2.5, plus the retry and dead-letter topics from Section 2.1
  • The outputs from Section 1.1, applied
  • A VM or pod in the cluster's LAN with Python 3.12, confluent-kafka and jsonschema

Step 1: Write the certificates

mkdir -p /etc/kafka && cd /etc/kafka
terraform -chdir=$TF_DIR output -raw kafka_ca   > ca-cert.pem
terraform -chdir=$TF_DIR output -raw kafka_cert > client-cert.pem
terraform -chdir=$TF_DIR output -raw kafka_key  > client-key.pem && chmod 600 client-key.pem
export KAFKA_BOOTSTRAP=$(terraform -chdir=$TF_DIR output -raw kafka_bootstrap)
openssl x509 -in client-cert.pem -noout -enddate

Expected output:

notAfter=<date about one year from issue>

Step 2: List the topics

from confluent_kafka.admin import AdminClient
print(sorted(AdminClient(KAFKA_TLS).list_topics(timeout=10).topics))

Expected output:

['task-events', 'task-events-dlq', 'task-events-retry', ...]

Step 3: Produce three events

for tid in (1, 2, 3):
    publish_task_event(tid, "task.completed", {"title": f"Task {tid}"})
print("unflushed:", producer.flush(10))

Expected output:

unflushed: 0

Step 4: Produce a poison message

producer.produce("task-events", key="99", value=b"not json"); producer.flush(10)

Expected output:

(no output)

Step 5: Run the consumer

Run the loop from Section 3.1 with send_notifications = lambda e: print("notify", e["task_id"]) and stop it after a few seconds. Expected output:

notify 1
notify 2
notify 3

Step 6: Inspect the dead-letter topic

dlq = Consumer({**KAFKA_TLS, "group.id": "lab-dlq-reader", "auto.offset.reset": "earliest"})
dlq.subscribe(["task-events-dlq"]); m = dlq.poll(10)
print(m.value(), dict(m.headers()))

Expected output:

b'not json' {'attempt': b'1', 'error': b'JSONDecodeError', 'source': b'task-events/...'}

Step 7: Verify the committed offsets

from confluent_kafka import TopicPartition
parts = [TopicPartition("task-events", p) for p in range(6)]
print([tp.offset for tp in consumer.committed(parts, timeout=10)])

Expected output:

[<offset or -1001 per partition>, ...]

Run this with a new consumer in group taskboard-notifier; partitions that received events show a committed offset.

Validation Checklist:

  • [ ] The client connects with mTLS using the cluster CA
  • [ ] Each valid event is notified once, even after a restart of the consumer
  • [ ] The poison message ends in task-events-dlq with its attempt count and source
  • [ ] Offsets are committed only after handling

Cleanup:

rm -f /etc/kafka/client-key.pem
terraform destroy -auto-approve -target=ionoscloud_kafka_cluster_topic.task_events_dlq

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. Handshake failures after a year

    • Problem: Producers and consumers suddenly fail with SSL handshake errors, though nothing changed.
    • Why it happens: Client certificates are valid for 365 days, and the pods still mount the old one.
    • Fix: Fetch the renewed certificate, which is available 30 days before expiry, update the Kubernetes Secret and roll the Deployments.
  2. Host name verification against IP addresses

    • Problem: The client cannot connect and logs a certificate verification failure for the broker.
    • Why it happens: Recent librdkafka versions check the broker's host name by default, while the documented client configuration for the cluster turns that check off.
    • Fix: Set ssl.endpoint.identification.algorithm to none and keep ssl.ca.location pointing to the cluster CA.
  3. More consumers, no more throughput

    • Problem: You scale the worker to 10 replicas, and lag does not drop.
    • Why it happens: Only as many consumers as partitions receive work; the rest stay idle.
    • Fix: Keep replicas at or below the partition count, or add partitions in multiples of 3.

Summary

You can now connect TaskBoard's services to Event Streams for Apache Kafka in production quality. Clients authenticate with mTLS using the CA, certificate and key that Terraform or the API provides, and the brokers are reached in the private LAN. The API produces keyed, schema-validated events with an idempotent acks=all producer. The worker commits offsets after handling, deduplicates by event ID, and moves poison messages through a retry topic into a dead-letter topic.

Key Points:

  • Authenticate with mTLS; validate the server with the cluster CA and disable host name checks
  • Client certificates last 365 days; automate their renewal
  • Use acks=all and idempotence for producers; commit offsets after processing for at-least-once
  • Route failures to retry and dead-letter topics so that one bad message never blocks a partition
  • The partition count caps consumer parallelism; use multiples of 3 for the three brokers

Important Terminology:

  • mTLS: Mutual TLS, where client and broker both present certificates that the other side verifies
  • Consumer group: Consumers sharing a group.id that split a topic's partitions among them
  • Offset commit: The stored position in a partition from which a group resumes after a restart
  • Dead-letter topic: A topic that holds messages that failed all retries, for later inspection
  • Idempotent producer: A producer whose retries never write a duplicate and keep partition order

Next Steps

Continue Learning: Unit 4.4: AI Model Hub Integration

Related Topics: