13 min read

Learning Objectives

By the end of this module, you will be able to:

  • Call the AI Model Hub OpenAI-compatible API from Python with a bearer token and a model chosen from `GET /v1/models`
  • Implement streaming and non-streaming chat completions with timeouts and response validation
  • Handle `429`, `503` and `529` responses with `Retry-After`-based back-off and per-model rate limits
  • Integrate inference synchronously and asynchronously through Kafka, with results cached in In-Memory DB
  • Configure the IONOS CLOUD MCP Server for agent access to your account, read-only by default

Unit 4.4: AI Model Hub Integration

Introduction

TaskBoard users write long task descriptions, and the team wants a one-line summary on every task card. You add an endpoint that sends the description to a large language model (LLM) on AI Model Hub and stores the summary in PostgreSQL. Inference runs in Germany, and your prompts are not used for training, so task content stays in the same sovereign setup as the rest of TaskBoard.

In this unit you integrate AI Model Hub into application code. You authenticate, pick a model, call chat completions with and without streaming, and harden the client against rate limits and overload. Then you move bulk summarization onto the Kafka pipeline from Unit 4.3 and cache results in the In-Memory DB cluster from Unit 4.1. Finally, you connect AI agents to your IONOS CLOUD account with the IONOS CLOUD MCP Server. The Code Lab builds the summary feature end to end.

1. The AI Model Hub API

AI Model Hub offers an OpenAI-compatible API. You call it with the official openai Python library by changing the base URL:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["IONOS_API_TOKEN"],
    base_url="https://openai.inference.de-txl.ionos.com/v1",
    timeout=30.0,
    max_retries=3,
)
for m in client.models.list():
    print(m.id)

The base URL is https://openai.inference.de-txl.ionos.com/v1. Inference runs in Germany (Berlin), and the API is reachable globally.

1.1 Authentication

The API key is an IONOS CLOUD authentication token, sent as Authorization: Bearer <token>. Tokens are JSON Web Tokens (JWTs) bound to a user of an IONOS Public Cloud contract; usage is billed to the contract owner. You generate a token in the DCD under Management > Token Manager or with the Authentication API from Unit 1.1:

curl -s -H "Authorization: Bearer $IONOS_API_TOKEN" \
  https://openai.inference.de-txl.ionos.com/v1/models | jq -r '.data[].id'

A token's time to live (TTL) is between 1 hour and 365 days in the Token Manager, and between 60 seconds and 365 days with the Authentication API. Its value is shown only once. When calls that used to work return 401, check the token's exp claim first. Give TaskBoard its own token and store it in the taskboard-ai Kubernetes Secret.

1.2 Models and Endpoints

The OpenAI-compatible paths are POST /v1/chat/completions, POST /v1/responses, POST /v1/embeddings, POST /v1/images/generations, POST /v1/rerank and GET /v1/models. The following table lists the text models and their capabilities:

Model Name Type Input Modality Output Modality Streaming Tool Calling Context Window Model Size
Qwen3.5-9B Language Model Text, Image, and Video Text ✅ ✅ 262k Small
Mistral Small 24B Language Model Text, Image Text ✅ ✅ 128k Medium
Qwen3.8-27B Language Model Text, Image, and Video (beta) Text ✅ ✅ 262k Medium
Llama 3.3 70B Language Model Text Text ✅ ✅ 128k Medium
GPT-OSS 120B Language Model Text Text ✅ ✅ 128k Medium
Qwen3.5 397B A17B Language Model Text and Image Text ✅ ✅ 262k Large
Qwen3-Coder-Next 80B Coding Model Text Text ✅ ✅ 256k Medium

As shown above, every text model supports streaming and tool calling. In code you use the model identifier, such as mistralai/Mistral-Small-24B-Instruct for Mistral Small 24B. Read the identifiers from /v1/models at startup instead of hard-coding a list, because the catalog changes.

2. Inference Patterns

2.1 A Synchronous Summary

MODEL = os.environ.get("SUMMARY_MODEL", "mistralai/Mistral-Small-24B-Instruct")

def summarize(description: str, model: str = MODEL) -> str:
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Summarize the task in one sentence of at most 20 words."},
            {"role": "user", "content": description},
        ],
        max_tokens=60,
        temperature=0.2,
    )
    choice = resp.choices[0]
    summary = (choice.message.content or "").strip()
    if choice.finish_reason != "stop" or not summary:
        raise ValueError(f"unusable summary, finish_reason={choice.finish_reason}")
    return summary[:200]

Validate every response before you store it: a cut-off answer (finish_reason of length), an empty string or an overlong text must not reach the task card. The usage field reports prompt_tokens and completion_tokens, which you log for cost tracking (Section 4).

2.2 Streaming

For an interactive "explain this task" panel, stream tokens to the browser as they arrive:

stream = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": f"Explain the next steps for: {description}"}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        yield chunk.choices[0].delta.content
    if chunk.usage:
        log.info("tokens in=%s out=%s", chunk.usage.prompt_tokens, chunk.usage.completion_tokens)

With streaming, usage statistics are not included by default. Set "stream_options": {"include_usage": true} to get them in the final chunk. Use streaming where a user watches the output, and a plain request where code consumes the result.

3. Production Error Handling

3.1 Rate Limits and Overload

The default rate limit is 5 requests per second per model, with bursts of up to 10 requests in a 2-second window. Limits apply per contract and separately for each model, and your contract can have custom limits. Every counted response carries headers you can throttle on:

raw = client.chat.completions.with_raw_response.create(
    model=MODEL, messages=[{"role": "user", "content": "ping"}], max_tokens=1)
print(raw.headers.get("x-ratelimit-remaining-requests"),
      raw.headers.get("x-ratelimit-reset-requests"))
completion = raw.parse()
Status Meaning Reaction
429 RATE_LIMIT_EXCEEDED Your contract's limit for this model is exhausted Wait the Retry-After seconds, then retry
503 SERVICE_OVERLOADED Too many large requests in flight: requests above 4 MiB are limited to six concurrent Wait the Retry-After seconds
529 user_overloaded Your traffic contributes to a platform overload Wait Retry-After, then send at a lower rate
529 server_overloaded The model is out of capacity, independent of your traffic Wait Retry-After; consider another model

The openai library honors Retry-After automatically within max_retries. Add your own fallback for 529 server_overloaded:

import openai

FALLBACK = "meta-llama/Llama-3.3-70B-Instruct"

def summarize_resilient(description: str) -> str:
    try:
        return summarize(description)
    except openai.APIStatusError as e:
        body = e.body if isinstance(e.body, dict) else {}
        code = body.get("code") or (body.get("error") or {}).get("code")
        if e.status_code == 529 and code == "server_overloaded":
            return summarize(description, model=FALLBACK)   # same prompt, other model
        raise

3.2 Client Errors

Other error responses carry a JSON body with httpStatus and a messages list of errorCode and message entries. Branch on errorCode, not on the message text. A 400 VALIDATION_ERROR names the bad field and always fails again unchanged; 404 MODEL_NOT_SUPPORTED means the model does not exist or cannot serve this endpoint; 502 UPSTREAM_UNAVAILABLE is worth a retry after a brief wait. Do not retry 4xx errors other than 429.

4. Integration Patterns and Cost

4.1 Cache Results in In-Memory DB

LLMs are charged per million input tokens and per million output tokens, so the same description must never be summarized twice:

import hashlib

def cached_summary(description: str) -> str:
    key = "ai:summary:" + hashlib.sha256(f"{MODEL}\n{description}".encode()).hexdigest()
    hit = cache.get(key)                      # the redis client from Unit 4.1
    if hit:
        return hit
    summary = summarize_resilient(description)
    cache.set(key, summary, ex=7 * 24 * 3600)
    return summary

The key includes the model, so a model change produces new summaries instead of mixing outputs. You see AI Model Hub costs per model and unit in the Cost & Usage section of the DCD.

4.2 Asynchronous Inference via Kafka

A bulk import of 5,000 tasks would exceed the per-model rate limit at once. Queue the work instead: the API produces task.summarize events to a task-summaries topic, created like the topics in Unit 4.3, and a summarizer worker consumes them at a steady rate.

import time

def run_summarizer():
    consumer.subscribe(["task-summaries"])
    while running:
        msg = consumer.poll(1.0)
        if msg is None or msg.error():
            continue
        event = json.loads(msg.value())
        summary = cached_summary(event["data"]["description"])
        with engine.begin() as c:
            c.execute(text("update tasks set summary = :s where id = :i"),
                      {"s": summary, "i": event["task_id"]})
        consumer.commit(message=msg, asynchronous=False)
        time.sleep(0.25)          # about 4 requests/s, below the base limit

The synchronous path serves single tasks a user is waiting for; the Kafka path serves batches. Both share the cache, and the retry and dead-letter handling from Unit 4.3 catches descriptions that the model keeps rejecting.

5. The IONOS CLOUD MCP Server

The IONOS CLOUD MCP Server is a local binary that implements the Model Context Protocol (MCP). It gives AI assistants and agents 100+ tools across seven products: Compute Engine, IONOS Cloud Object Storage, Cloud DNS, Certificate Manager, Billing, Activity Log and Managed Kubernetes. It is an alternative to writing API or SDK calls when an agent should inspect or change your infrastructure.

5.1 Configuration

The AI client starts the binary and talks JSON-RPC over stdio. A project-scoped .mcp.json for your team:

{
  "mcpServers": {
    "ionoscloud": {
      "command": "/usr/local/bin/ionoscloud-mcp",
      "env": {
        "IONOS_TOKEN": "${env:IONOS_TOKEN}"
      }
    }
  }
}

IONOS_TOKEN is required; Object Storage tools also need IONOS_S3_ACCESS_KEY and IONOS_S3_SECRET_KEY. Create a dedicated token with the minimum required access, and never commit a token to the file.

5.2 Read-Only by Default, Writes by Opt-In

The server is read-only by default. IONOS_MCP_TOOL_SCOPE=write registers create and update tools, and destructive adds delete tools and disruptive actions such as stopping a server. Write tools cover Compute Engine and Managed Kubernetes. Every create, delete and disruptive action takes a two-step confirmation: a preview call returns a confirmation_token, valid once for five minutes, and a second call performs the operation. Do not commit IONOS_MCP_TOOL_SCOPE to a shared .mcp.json, because it activates write tools for everyone who opens the repository.

For one shared instance, run the Streamable HTTP transport:

export IONOS_TOKEN="$MCP_READONLY_TOKEN"
ionoscloud-mcp --transport http --http-addr 127.0.0.1:8080

The HTTP endpoint has no TLS and no authentication of its own. Bind it to 127.0.0.1, or put it behind a reverse proxy that terminates TLS and authenticates callers.

5.3 A Sovereign Agent Path

The MCP server calls the IONOS CLOUD API directly. The AI client, however, sends prompts and tool output to its own model backend. To keep the whole round trip in IONOS CLOUD, use a client that supports both MCP and a custom OpenAI-compatible base URL, and point that base URL at AI Model Hub. Tool calling is a prerequisite for MCP, and the models in Section 1.2 support it.

API Reference Quick Card

Key AI Model Hub endpoints for this unit:

Method Endpoint Description
GET /v1/models List the available model identifiers
POST /v1/chat/completions Text generation, with stream: true for streaming
POST /v1/responses Responses API with typed output items
POST /v1/embeddings Embedding vectors for semantic search
POST /v1/rerank Rerank documents for a query

Base URL: https://openai.inference.de-txl.ionos.com/v1 Authentication: Authorization: Bearer <token>

Code Lab

Objective: Call AI Model Hub from TaskBoard to generate task summaries, handle errors, cache results in In-Memory DB and store them in PostgreSQL.

Prerequisites:

  • An IONOS CLOUD token in IONOS_API_TOKEN
  • The PostgreSQL and cache connections from Unit 4.1 (engine, cache)
  • Python 3.12 with openai, sqlalchemy and redis

Step 1: List the models

curl -s -H "Authorization: Bearer $IONOS_API_TOKEN" \
  https://openai.inference.de-txl.ionos.com/v1/models | jq -r '.data[].id' | head -5

Expected output:

<model identifiers, for example mistralai/Mistral-Small-24B-Instruct>

Step 2: Add the summary column

with engine.begin() as c:
    c.execute(text("alter table tasks add column if not exists summary text"))
print("ok")

Expected output:

ok

Step 3: Summarize one description

d = "Migrate the attachment upload from API proxying to presigned PUT URLs, add CORS, and remove the old endpoint after one release."
print(summarize(d))

Expected output:

<one sentence of at most 20 words>

Step 4: Read the rate limit headers

Run the with_raw_response snippet from Section 3.1. Expected output:

4 200ms

Step 5: Cache and store

import time
for i in range(2):
    t = time.perf_counter(); s = cached_summary(d)
    print(i, f"{time.perf_counter() - t:.2f}s")
with engine.begin() as c:
    c.execute(text("update tasks set summary = :s where id = 1"), {"s": s})

Expected output:

0 <about a second>s
1 0.00s

Step 6: Force a validation error

try:
    client.chat.completions.create(model=MODEL, messages=[])
except openai.BadRequestError as e:
    print(e.status_code, e.body)

Expected output:

400 {... 'errorCode': 'VALIDATION_ERROR', ...}

Validation Checklist:

  • [ ] /v1/models returns identifiers with your bearer token
  • [ ] The summary passes the finish_reason and length checks
  • [ ] The second call is served from the cache without a model request
  • [ ] The summary is stored in the tasks.summary column

Cleanup:

for k in cache.scan_iter("ai:summary:*"):
    cache.delete(k)

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. Sudden 401 after weeks of success

    • Problem: Every call returns 401 although the code has not changed.
    • Why it happens: The token's TTL has expired.
    • Fix: Check the JWT exp claim, generate a new token, update the Secret, and alert before the expiry date.
  2. Bulk jobs that trip the rate limit

    • Problem: An import job fires hundreds of parallel requests and most fail with 429.
    • Why it happens: Each model allows 5 requests per second per contract, with short bursts.
    • Fix: Queue the work through Kafka and pace the consumer (Section 4.2); read X-RateLimit-Remaining-Requests.
  3. Storing unvalidated output

    • Problem: Task cards show half sentences or empty summaries.
    • Why it happens: The response hit max_tokens (finish_reason is length), or the model returned no content.
    • Fix: Check finish_reason and length before storing, and keep the old summary on failure (Section 2.1).

Summary

You can now add AI features to TaskBoard with AI Model Hub. The openai library works against the OpenAI-compatible base URL with an IONOS CLOUD bearer token. You validate responses, stream where users watch, and handle 429, 503 and 529 with Retry-After. Single requests run synchronously, batches flow through Kafka at a pace below the rate limit, and the cache avoids paying twice for the same tokens. For agent workflows, the IONOS CLOUD MCP Server exposes your account as read-only tools, with writes only on explicit opt-in.

Key Points:

  • Base URL https://openai.inference.de-txl.ionos.com/v1, bearer token, model IDs from /v1/models
  • Default limits: 5 requests per second per model, bursts of up to 10, per contract
  • Wait the Retry-After time on 429, 503 and 529; branch on error codes, not messages
  • Cache results by model and input; queue bulk work through Kafka
  • The MCP Server is read-only by default; writes need IONOS_MCP_TOOL_SCOPE, and creates, deletes and disruptive actions need a two-step confirmation

Important Terminology:

  • Model identifier: The string you pass as model, such as mistralai/Mistral-Small-24B-Instruct
  • Burst limit: The extra requests allowed within a 2-second window above the base rate
  • finish_reason: The field that tells you why generation stopped, for example stop or length
  • MCP: Model Context Protocol, the JSON-RPC protocol through which AI clients call tools
  • Tool scope: The IONOS_MCP_TOOL_SCOPE level, read, write or destructive, that decides which tools the server registers

Next Steps

Continue Learning: Unit 4.5: Knowledge Check - Service Integration

Related Topics: