Unit 4.3: Event Streaming Integration
Introduction
When a user completes a task, TaskBoard should notify the task's watchers. Sending those notifications inside the API request makes the request slow and fragile. Instead, the API publishes a task.completed event, and the worker service consumes it and sends the notifications at its own pace.
In Unit 2.5 you provisioned an Event Streams for Apache Kafka cluster and the task-events topic with Terraform. Now you write the client side. You configure mutual TLS (mTLS) with the cluster's certificates, write a reliable producer, run a consumer group with explicit offset commits, and route failing messages to retry and dead-letter topics. You also define the event contract and scale the worker on Kubernetes. In the Code Lab you produce and consume TaskBoard events end to end.
1. Connecting with mTLS
1.1 Credentials and Brokers from Terraform
Communication with the cluster is TLS secured, and both sides authenticate each other through mTLS. The cluster does not have publicly signed certificates, so your client validates the server with the cluster's certificate authority (CA). The provider reads the CA, the client certificate and the key for a user:
data "ionoscloud_kafka_users" "events" {
cluster_id = ionoscloud_kafka_cluster.events.id
location = ionoscloud_kafka_cluster.events.location
}
data "ionoscloud_kafka_user_credentials" "admin" {
cluster_id = ionoscloud_kafka_cluster.events.id
location = ionoscloud_kafka_cluster.events.location
username = "admin"
}
output "kafka_bootstrap" {
value = join(",", ionoscloud_kafka_cluster.events.broker_addresses)
}
output "kafka_ca" {
value = data.ionoscloud_kafka_user_credentials.admin.certificate_authority
sensitive = true
}
output "kafka_cert" {
value = data.ionoscloud_kafka_user_credentials.admin.certificate
sensitive = true
}
output "kafka_key" {
value = data.ionoscloud_kafka_user_credentials.admin.private_key
sensitive = true
}
The computed broker_addresses attribute holds the IP address and port of each broker, for example 10.10.2.2:9093. The credentials belong to the Kafka administrator user, so treat them as admin secrets. The data source stores them in state; the ionoscloud_kafka_user_credentials ephemeral resource reads the same values without storing them.
Without Terraform, the same values come from the API:
curl -s https://kafka.de-txl.ionos.com/clusters/$CLUSTER_ID/users/$USER_ID/access \
-H "Authorization: Bearer $IONOS_TOKEN" \
| jq -r '.metadata.certificateAuthority' > ca-cert.pem
The .metadata.certificate and .metadata.privateKey fields hold the client certificate and key. Get the user ID from GET /clusters/{clusterId}/users.
1.2 Client Configuration
import os
from confluent_kafka import Producer
KAFKA_TLS = {
"bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
"security.protocol": "SSL",
"ssl.ca.location": "/etc/kafka/ca-cert.pem",
"ssl.certificate.location": "/etc/kafka/client-cert.pem",
"ssl.key.location": "/etc/kafka/client-key.pem",
"ssl.endpoint.identification.algorithm": "none",
}
The client configuration for the cluster uses security.protocol=SSL and an empty ssl.endpoint.identification.algorithm, which turns off the host name check while the CA still verifies the certificate chain. In librdkafka, the value for "off" is none. The brokers live in your private DB LAN, so the API and worker must run on servers or node pools attached to that LAN (Unit 3.2).
Each client certificate is valid for 365 days. Thirty days before it expires, a renewed certificate is available from the user credentials API. Re-read the credentials on a schedule and roll your pods, or the clients stop connecting on the expiry date.
2. Producing Task Events Reliably
2.1 Topic Design
Use one topic per event family and key each message by the entity whose order matters. TaskBoard keys task-events by task ID, so all events of one task land in one partition, in order. The topic from Unit 2.5 has 6 partitions and replication factor 3.
Create the retry and dead-letter topics the same way:
resource "ionoscloud_kafka_cluster_topic" "task_events_retry" {
cluster_id = ionoscloud_kafka_cluster.events.id
location = ionoscloud_kafka_cluster.events.location
name = "task-events-retry"
replication_factor = 3
number_of_partitions = 6
}
resource "ionoscloud_kafka_cluster_topic" "task_events_dlq" {
cluster_id = ionoscloud_kafka_cluster.events.id
location = ionoscloud_kafka_cluster.events.location
name = "task-events-dlq"
replication_factor = 3
number_of_partitions = 3
retention_time = 2592000000 # 30 days to investigate failures
}
Each cluster has three brokers, so use partition counts that are multiples of 3 for an even distribution. Without retention_time, a topic keeps messages for 604800000 ms (7 days); -1 removes the time limit.
2.2 The Producer
import json, uuid, datetime
producer = Producer({
**KAFKA_TLS,
"acks": "all",
"enable.idempotence": True,
"linger.ms": 20,
"batch.size": 65536,
"compression.type": "lz4",
})
def publish_task_event(task_id: int, event_type: str, data: dict) -> None:
event = {
"event_id": str(uuid.uuid4()),
"type": event_type, # "task.completed"
"schema_version": 1,
"occurred_at": datetime.datetime.now(datetime.UTC).isoformat(),
"task_id": task_id,
"data": data,
}
producer.produce("task-events", key=str(task_id), value=json.dumps(event),
on_delivery=_report)
producer.poll(0)
def _report(err, msg):
if err is not None:
log.error("delivery failed: %s", err) # alert; the event is not stored
acks=allwaits until all in-sync replicas have the write, so a single broker failure loses nothing.enable.idempotencestops retries from creating duplicates and keeps per-partition order.linger.msandbatch.sizetrade a few milliseconds of latency for fewer, larger requests.
Call producer.flush(10) on shutdown, for example in your FastAPI lifespan handler, or the last batch is lost when Kubernetes stops the pod.
3. Consuming with Offset Control
3.1 Consumer Group and Commits
from confluent_kafka import Consumer, KafkaException
consumer = Consumer({
**KAFKA_TLS,
"group.id": "taskboard-notifier",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"max.poll.interval.ms": 300000,
})
consumer.subscribe(["task-events", "task-events-retry"])
while running:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
raise KafkaException(msg.error())
handle_with_retry(msg) # Section 4
consumer.commit(message=msg, asynchronous=False)
consumer.close()
All worker pods share the group.id, so Kafka divides the partitions among them. With enable.auto.commit off, the worker commits an offset only after it has handled the message.
3.2 Delivery Semantics
| Commit point | Delivery | Failure effect |
|---|---|---|
| After processing (this code) | At-least-once | A crash between processing and commit replays the message |
| Before processing | At-most-once | A crash after commit loses the message |
As the table shows, at-least-once can deliver a message twice, so make the handler idempotent. The notifier records each event_id in PostgreSQL with a unique constraint and skips events it has already sent.
def already_sent(event_id: str) -> bool:
with engine.begin() as c:
r = c.execute(text("insert into sent_events (event_id) values (:e) on conflict do nothing"),
{"e": event_id})
return r.rowcount == 0
4. Retries, Dead Letters and Event Contracts
4.1 Retry and Dead-Letter Routing
A message that always fails, a poison message, must not block its partition. The worker retries a few times through the retry topic and then parks the message in the dead-letter topic:
from confluent_kafka import Message
from jsonschema import ValidationError
MAX_ATTEMPTS = 3
def handle_with_retry(msg: Message) -> None:
headers = dict(msg.headers() or [])
attempt = int(headers.get("attempt", b"0"))
try:
event = json.loads(msg.value())
validate_event(event) # Section 4.2
if not already_sent(event["event_id"]):
send_notifications(event)
except (json.JSONDecodeError, ValidationError) as exc:
forward(msg, "task-events-dlq", attempt, exc) # a retry cannot fix bad data
except Exception as exc:
target = "task-events-retry" if attempt + 1 < MAX_ATTEMPTS else "task-events-dlq"
forward(msg, target, attempt, exc)
def forward(msg: Message, target: str, attempt: int, exc: Exception) -> None:
producer.produce(target, key=msg.key(), value=msg.value(), headers={
"attempt": str(attempt + 1),
"error": type(exc).__name__,
"source": f"{msg.topic()}/{msg.partition()}/{msg.offset()}",
})
producer.flush(5)
The original offset is committed after the message is safely in the next topic, so the partition moves on. Invalid JSON or a failed schema check goes to the dead-letter topic at once, because a retry cannot fix it. A small tool can read the dead-letter topic, fix the cause and produce the messages back to task-events.
4.2 Schema Management
Kafka stores bytes; the contract between API and worker is your responsibility. Define it as a JSON Schema in a shared package and validate on both sides:
from jsonschema import validate
TASK_EVENT_V1 = {
"type": "object",
"required": ["event_id", "type", "schema_version", "task_id", "occurred_at"],
"properties": {
"event_id": {"type": "string"},
"type": {"enum": ["task.created", "task.updated", "task.completed"]},
"schema_version": {"const": 1},
"task_id": {"type": "integer"},
},
}
def validate_event(event: dict) -> None:
validate(instance=event, schema=TASK_EVENT_V1)
Evolve the schema additively: add optional fields, never rename or remove one. For a breaking change, raise schema_version and keep the consumer reading both versions until no old producer remains. Avro with a schema registry gives the same discipline with compiler support.
5. Scaling Consumers on Kubernetes
The partition count is the parallelism ceiling of a consumer group: each partition is read by at most one consumer in the group. With 6 partitions, a seventh worker pod stays idle.
apiVersion: apps/v1
kind: Deployment
metadata: { name: taskboard-worker }
spec:
replicas: 3 # at most 6, the partition count of task-events
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: worker
image: <registry-hostname>/taskboard/worker:<git-sha>
envFrom:
- secretRef: { name: taskboard-kafka } # KAFKA_BOOTSTRAP
volumeMounts:
- { name: kafka-tls, mountPath: /etc/kafka, readOnly: true }
volumes:
- name: kafka-tls
secret: { secretName: taskboard-kafka-tls } # ca-cert.pem, client-cert.pem, client-key.pem
When a pod starts or stops, the group rebalances, and partitions move between pods. Handle SIGTERM by leaving the poll loop and calling consumer.close(), so the group rebalances at once instead of waiting for a session timeout. Keep each message's processing well below max.poll.interval.ms, or the broker removes the consumer from the group. To raise the ceiling, add partitions in multiples of 3; note that this changes which partition a key maps to.
API Reference Quick Card
Key Kafka API endpoints for this unit:
| Method | Endpoint | Description |
|---|---|---|
GET |
/clusters/{clusterId} |
Read the cluster state and brokerAddresses |
GET |
/clusters/{clusterId}/users |
List users and their IDs |
GET |
/clusters/{clusterId}/users/{userId}/access |
Retrieve the CA, client certificate and private key |
POST |
/clusters/{clusterId}/topics |
Create a topic |
DELETE |
/clusters/{clusterId}/topics/{topicId} |
Delete a topic and its data |
Base URL: regional, for example https://kafka.de-txl.ionos.com
Authentication: Authorization: Bearer <token> for the API; mTLS for producers and consumers
Code Lab
Objective: Produce and consume TaskBoard task-change events on Event Streams for Apache Kafka with a consumer group, manual offset commits and a dead-letter topic.
Prerequisites:
- The Kafka cluster and
task-eventstopic from Unit 2.5, plus the retry and dead-letter topics from Section 2.1 - The outputs from Section 1.1, applied
- A VM or pod in the cluster's LAN with Python 3.12,
confluent-kafkaandjsonschema
Step 1: Write the certificates
mkdir -p /etc/kafka && cd /etc/kafka
terraform -chdir=$TF_DIR output -raw kafka_ca > ca-cert.pem
terraform -chdir=$TF_DIR output -raw kafka_cert > client-cert.pem
terraform -chdir=$TF_DIR output -raw kafka_key > client-key.pem && chmod 600 client-key.pem
export KAFKA_BOOTSTRAP=$(terraform -chdir=$TF_DIR output -raw kafka_bootstrap)
openssl x509 -in client-cert.pem -noout -enddate
Expected output:
notAfter=<date about one year from issue>
Step 2: List the topics
from confluent_kafka.admin import AdminClient
print(sorted(AdminClient(KAFKA_TLS).list_topics(timeout=10).topics))
Expected output:
['task-events', 'task-events-dlq', 'task-events-retry', ...]
Step 3: Produce three events
for tid in (1, 2, 3):
publish_task_event(tid, "task.completed", {"title": f"Task {tid}"})
print("unflushed:", producer.flush(10))
Expected output:
unflushed: 0
Step 4: Produce a poison message
producer.produce("task-events", key="99", value=b"not json"); producer.flush(10)
Expected output:
(no output)
Step 5: Run the consumer
Run the loop from Section 3.1 with send_notifications = lambda e: print("notify", e["task_id"]) and stop it after a few seconds.
Expected output:
notify 1
notify 2
notify 3
Step 6: Inspect the dead-letter topic
dlq = Consumer({**KAFKA_TLS, "group.id": "lab-dlq-reader", "auto.offset.reset": "earliest"})
dlq.subscribe(["task-events-dlq"]); m = dlq.poll(10)
print(m.value(), dict(m.headers()))
Expected output:
b'not json' {'attempt': b'1', 'error': b'JSONDecodeError', 'source': b'task-events/...'}
Step 7: Verify the committed offsets
from confluent_kafka import TopicPartition
parts = [TopicPartition("task-events", p) for p in range(6)]
print([tp.offset for tp in consumer.committed(parts, timeout=10)])
Expected output:
[<offset or -1001 per partition>, ...]
Run this with a new consumer in group taskboard-notifier; partitions that received events show a committed offset.
Validation Checklist:
- [ ] The client connects with mTLS using the cluster CA
- [ ] Each valid event is notified once, even after a restart of the consumer
- [ ] The poison message ends in
task-events-dlqwith its attempt count and source - [ ] Offsets are committed only after handling
Cleanup:
rm -f /etc/kafka/client-key.pem
terraform destroy -auto-approve -target=ionoscloud_kafka_cluster_topic.task_events_dlq
Common Pitfalls
Developer mistakes to avoid with this unit's topic:
-
Handshake failures after a year
- Problem: Producers and consumers suddenly fail with SSL handshake errors, though nothing changed.
- Why it happens: Client certificates are valid for 365 days, and the pods still mount the old one.
- Fix: Fetch the renewed certificate, which is available 30 days before expiry, update the Kubernetes Secret and roll the Deployments.
-
Host name verification against IP addresses
- Problem: The client cannot connect and logs a certificate verification failure for the broker.
- Why it happens: Recent librdkafka versions check the broker's host name by default, while the documented client configuration for the cluster turns that check off.
- Fix: Set
ssl.endpoint.identification.algorithmtononeand keepssl.ca.locationpointing to the cluster CA.
-
More consumers, no more throughput
- Problem: You scale the worker to 10 replicas, and lag does not drop.
- Why it happens: Only as many consumers as partitions receive work; the rest stay idle.
- Fix: Keep replicas at or below the partition count, or add partitions in multiples of 3.
Summary
You can now connect TaskBoard's services to Event Streams for Apache Kafka in production quality. Clients authenticate with mTLS using the CA, certificate and key that Terraform or the API provides, and the brokers are reached in the private LAN. The API produces keyed, schema-validated events with an idempotent acks=all producer. The worker commits offsets after handling, deduplicates by event ID, and moves poison messages through a retry topic into a dead-letter topic.
Key Points:
- Authenticate with mTLS; validate the server with the cluster CA and disable host name checks
- Client certificates last 365 days; automate their renewal
- Use
acks=alland idempotence for producers; commit offsets after processing for at-least-once - Route failures to retry and dead-letter topics so that one bad message never blocks a partition
- The partition count caps consumer parallelism; use multiples of 3 for the three brokers
Important Terminology:
- mTLS: Mutual TLS, where client and broker both present certificates that the other side verifies
- Consumer group: Consumers sharing a
group.idthat split a topic's partitions among them - Offset commit: The stored position in a partition from which a group resumes after a restart
- Dead-letter topic: A topic that holds messages that failed all retries, for later inspection
- Idempotent producer: A producer whose retries never write a duplicate and keep partition order
Next Steps
Continue Learning: Unit 4.4: AI Model Hub Integration
Related Topics: