Unit 6.5: AI Inference: Managed Model Hub
Introduction
FinCorp wants an assistant that answers advisors' questions from internal policy documents. The first architectural decision is where inference runs: on a managed endpoint billed per token, or on GPU servers that FinCorp operates. The second is how the model gets FinCorp's knowledge without being trained on it.
This unit makes both decisions on IONOS CLOUD terms and ends with a light build: an inference user, a test in the Playground and the same request sent from the private cluster fc-k8s-ai of Unit 6.3.
1. Managed Inference as the Default
1.1 What the Service Gives You
AI Model Hub is an inference service: it serves pre-trained models through REST APIs, and you do not manage hardware. It offers its own API and an OpenAI-compatible API, whose base URL is https://openai.inference.de-txl.ionos.com/v1. Its endpoints, such as models, chat completions, image generations and embeddings, mirror OpenAI's API structure, so tools that support OpenAI endpoints integrate by pointing them at this base URL and token. Requests authenticate with a Bearer token, an IONOS CLOUD authentication token that is bound to a user of the contract.
All inference is performed in IONOS CLOUD data centers in Germany. The inference API is stateless per request: no session or conversation is kept, so an application that holds a conversation sends the earlier turns with every request. Unit 6.6 treats the data-handling consequences.
1.2 Billing, Limits and Service Level
Usage is billed by tokens. Large language models are charged per million input tokens and per million output tokens; embedding and reranking models per million input tokens at a single rate; text-to-image models per image or per megapixel. Three details matter for cost design. A system prompt and tool definitions count as input. Every retry is measured on its own, including output that a failed attempt had already produced. And repeated input is not discounted, even when prompt caching makes it faster.
The default rate limit is 5 requests per second per model, with a burst of 10 requests in a 2-second window, enforced per contract rather than per user. Exceeding it returns HTTP 429 Too Many Requests with a Retry-After header, so clients need backoff and a queue in front of bulk jobs such as embedding a whole corpus. Your contract can have custom limits. The per-service uptime SLA is 99.9 percent, scoped to the API endpoints.
The model catalogue changes. Models are introduced and retired, and each retirement names a date and an alternative. Treat the model ID as configuration, track the retirement schedule, and re-run your quality tests before you switch.
1.3 Reaching the Endpoint From a Private Network
The endpoint is reached over the internet. Private workloads need an egress path: VMs use their NAT Gateway (Unit 3.6), and the nodes of fc-k8s-ai use the NAT Gateway that the cluster service deployed (Unit 6.3). No inbound path is needed, because the call is outbound.
2. When Self-Hosted GPU Serving Is Justified
2.1 The Cloud GPU VM
Cloud GPU VMs carry dedicated NVIDIA H200 Tensor Core GPUs, each with 141 GB of HBM3e memory and 4.8 TB/s of memory bandwidth, attached through PCIe passthrough. They come in four fixed templates:
| Template | GPU Model | GPU Type | Number of GPUs | Dedicated CPUs | RAM (GiB) | Storage (GB) |
|---|---|---|---|---|---|---|
| S | NVIDIA H200 | H200 PCIe | 1 | 15 | 267 | 1024 |
| M | NVIDIA H200 | H200 PCIe | 2 | 30 | 534 | 1536 |
| L | NVIDIA H200 | H200 PCIe | 4 | 60 | 1068 | 2048 |
| XL | NVIDIA H200 | H200 PCIe | 8 | 127 | 2136 | 4096 |
As shown above, CPU and RAM grow in proportion to the GPU count. The hourly template rates are 3.0, 6.0, 12.0 and 24.0 EUR for S, M, L and XL. The XL template organizes its eight GPUs as two clusters of four, connected through NVLink within each cluster, so address each cluster separately rather than as one 8-GPU pool.
2.2 The Operating Burden
The constraints are what make self-hosting a deliberate choice:
- No SLA. The Service Level Agreement does not apply to the Cloud GPU VM product, irrespective of variant or configuration.
- Restarts. There are no seamless background transfers during maintenance, so some updates require a shutdown or reboot, and emergency maintenance can come with little or no notice. Services must restart automatically on boot.
- Fixed size. The template is set at provisioning; CPU and RAM cannot be scaled up or down afterwards.
- Placement. Cloud GPU VMs are available only in Frankfurt (
de/fra/2) and Worcester (gb/bhx), with the availability zone always set to Auto. Only IONOS CLOUD Linux images are supported at start. - Quota. By default, a contract can deploy one Cloud GPU VM with the S template. Larger templates or more instances need a resource limit increase from support.
A redundant self-hosted endpoint therefore needs at least two instances, behind a load balancer of your own design, plus the quota request and your own model-serving stack. Self-hosting is justified for a model the catalogue does not offer, for fine-tuning, or for steady around-the-clock inference where a fixed hourly rate is predictable. Otherwise, managed inference is simpler to operate and comes with an SLA.
2.3 Inference Versus Fine-Tuning Sizing
Size inference by the memory that the model weights and the concurrent requests need on the GPU, and pick the smallest template that fits, because the template cannot grow later. Fine-tuning needs much more GPU memory than inference for the same model, because training also holds gradients and optimizer state; it is a batch job, so it can run on a larger template for its duration and be deleted. AI Model Hub is an inference service for its catalogue of pre-trained models, while fine-tuning is a documented use case of Cloud GPU VMs. In this design, a fine-tuned model is a self-hosted model.
3. Retrieval-Augmented Generation as the Pattern
RAG retrieves relevant passages from your data and places them in the prompt, so the model answers from your knowledge without being trained on it. On IONOS CLOUD you compose it from three services, all hosted in German data centers:
- Corpus in IONOS Cloud Object Storage. The source documents stay in a versioned bucket, such as
fc-ai-corpus-fra, which is the system of record for re-indexing (Unit 5.2). The embedding API accepts plain text, so text extraction from PDF or Word files is your pipeline step. - Embeddings from AI Model Hub. Each text chunk goes to the OpenAI-compatible embeddings endpoint. The documented guide uses
BAAI/bge-m3, with 1024-dimensional output. - Vectors in DBaaS for PostgreSQL. Managed PostgreSQL includes the
vectorextension (pgvector), activated withCREATE EXTENSION vector CASCADE;. Its vector data type andivfflatandhnswaccess methods turn a standard table into the vector store, managed alongside relational data.
At query time, the worker embeds the question, retrieves the nearest chunks with SQL, optionally reorders them with a reranking model at POST /v1/rerank, and sends them with the question to POST /v1/chat/completions. Because the embedding model fixes the vector dimension, changing it means re-embedding the corpus; record the model ID next to each vector.
FinCorp's design runs the workers in fc-k8s-ai, a PostgreSQL cluster fc-pg-rag on the fc-ai-app LAN for the vectors, and the corpus bucket in Frankfurt. Managed inference is the default; no GPU VM is planned until a fine-tuning need appears.
DCD Implementation Walkthrough
You will create a dedicated inference identity, test a model in the Playground and send the same request as an OpenAI-compatible call. This is the managed default of Section 1.
Prerequisites: a contract with the full version unlocked; Administrator or Contract Owner access for the group setup.
Build goal: Test a model and lift a working OpenAI-compatible call.
Steps (in the Data Center Designer):
- Open Management > Users & Groups. Under Groups, select + Create, enter the Group Name
fc-ai-inference, and select Create. Select the group and check Access and manage AI Model Hub. - Under Users, select + Create and create
svc-fc-ai-inference. In its Groups area, select + Add to Group and choosefc-ai-inference. - Sign in as
svc-fc-ai-inference, open Management > Token Manager, select a Time to Live (TTL) and select Generate Token. Copy the token now; it is not shown again. - Go to AI > AI Model Hub. The Models tab lists the available models with their pricing; note the ID of the model you will test.
- Open the Playground tab and test the model with a representative prompt. Playground requests are billed at the model's standard rates.
- From a pod in
fc-k8s-ai, send the same request to the chat completions endpoint, with the token in theIONOS_API_TOKENenvironment variable.
curl -X POST https://openai.inference.de-txl.ionos.com/v1/chat/completions \
-H "Authorization: Bearer ${IONOS_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Summarise the attached policy."}]}'
The call carries the architectural point of Section 1.1: the request is complete in itself, with no server-side session, and leaves the cluster through its NAT Gateway.
Common mistakes:
- Generating the inference token as the Contract Owner. Use a dedicated user in a group with only the AI Model Hub privilege.
- Losing the token. It is shown once; generate a new one and update the secret.
- Expecting the service to remember a conversation. It is stateless; send the earlier turns.
- Unbounded client retries. Each retry is billed, so cap them and honour
Retry-After. - Hard-coding a model ID with no owner for its retirement date.
- Placing a Cloud GPU VM in a VDC outside
de/fra/2orgb/bhx, or planning one instance as a production endpoint without an SLA.
Summary
Managed inference on AI Model Hub is FinCorp's default: an OpenAI-compatible, stateless API processed in Germany, billed per token, with a 99.9 percent SLA on its endpoints. Self-hosting on Cloud GPU VMs is a deliberate exception for fine-tuning, missing models or steady load, and it brings fixed templates, maintenance restarts and no SLA. Knowledge reaches the model through RAG: corpus in Object Storage, embeddings from the hub, vectors in PostgreSQL with pgvector.
Key Points:
- The OpenAI-compatible base URL is
https://openai.inference.de-txl.ionos.com/v1, with Bearer token authentication. - Default limits are 5 requests per second per model with a burst of 10, per contract; HTTP 429 means back off.
- The SLA does not apply to Cloud GPU VMs; their size is fixed and maintenance can restart them.
- AI Model Hub serves its catalogue of pre-trained models; fine-tuning is a self-hosted job on Cloud GPU VMs.
- RAG composes Object Storage, AI Model Hub embeddings and DBaaS for PostgreSQL with the
vectorextension.