13 min read

Learning Objectives

By the end of this module, you will be able to:

  • Configure boto3 against a regional IONOS Cloud Object Storage endpoint with an Access Key and Secret Key
  • Implement presigned upload and download URLs so that browsers transfer TaskBoard attachments directly
  • Upload large files with multipart upload and stream downloads with correct content types
  • Automate lifecycle rules through the S3 API to expire temporary objects and abandoned uploads
  • Check S3 feature support before you use an AWS-specific API in your code

Unit 4.2: Object Storage Integration

Introduction

TaskBoard users want to attach screenshots, logs and documents to tasks. Routing those bytes through the API pods would cost memory, bandwidth and request time. Instead, the browser uploads straight to the bucket from Unit 2.4 with a short-lived presigned URL, and the API stores only the object key in PostgreSQL.

IONOS Cloud Object Storage speaks the S3 protocol, so you use standard S3 SDKs. But it is not AWS: you set the endpoint explicitly, authenticate with an Access Key and Secret Key, and check which S3 features exist before you call them. In this unit you build the attachment service with boto3: client setup, presigned URLs, multipart uploads, streaming downloads and lifecycle rules. The Code Lab runs the full upload and download flow.

1. The boto3 Client

import os
import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.eu-central-3.ionoscloud.com",
    aws_access_key_id=os.environ["S3_ACCESS_KEY"],       # terraform output uploads_access_key
    aws_secret_access_key=os.environ["S3_SECRET_KEY"],   # terraform output uploads_secret_key
    region_name="eu-central-3",
    config=Config(signature_version="s3v4"),
)
print([b["Name"] for b in s3.list_buckets()["Buckets"]])

Without endpoint_url, boto3 talks to AWS and your keys fail. The credentials are an Access Key and Secret Key pair: Object Storage does not accept the bearer token of the Cloud API, and IAM roles and the Security Token Service (STS) are not supported. Your credentials are not tied to a specific region or bucket, so one key pair works against every endpoint.

1.1 Endpoints per Region

The endpoint depends on the bucket type and region. The following table lists the endpoints for contract-owned buckets:

Data Center Region Endpoint Static Website Endpoint
Frankfurt, Germany eu-central-4 s3.eu-central-4.ionoscloud.com s3-website.eu-central-4.ionoscloud.com
Berlin, Germany eu-central-3 s3.eu-central-3.ionoscloud.com s3-website.eu-central-3.ionoscloud.com
Lenexa, USA us-central-1 s3.us-central-1.ionoscloud.com s3-website.us-central-1.ionoscloud.com

User-owned buckets use other regions, for example s3.eu-central-2.ionoscloud.com for Berlin. TaskBoard's bucket is contract-owned in eu-central-3. The endpoints answer on HTTP and HTTPS; always use https://. Encryption in transit uses TLS 1.2 or 1.3.

1.2 The boto3 1.36 Checksum Change

From boto3 1.36.0, PutObject sends new checksum trailers that Object Storage does not yet support. Calls then fail with InvalidTrailer: "Invalid trailing header names in x-amz-trailer". Either pin boto3<=1.35.99 or set these variables for your process:

export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required

Put them in the API's ConfigMap from Unit 3.2 so that every pod gets them.

2. Buckets, Access and Presigned URLs

2.1 Bucket Operations

Terraform owns TaskBoard's long-lived bucket (Unit 2.4). Tests and tools can still create and list buckets in code:

from botocore.exceptions import ClientError

try:
    s3.create_bucket(Bucket="taskboard-ci-4711",
                     CreateBucketConfiguration={"LocationConstraint": "eu-central-3"})
except ClientError as e:
    if e.response["Error"]["Code"] != "BucketAlreadyOwnedByYou":
        raise
print(s3.get_bucket_acl(Bucket="taskboard-ci-4711")["Grants"])

Objects are private by default; only the bucket owner can access them. Keep it that way. For a contract-owned bucket, an ACL shares access only with grantees from other contracts; to give users of your own contract access, use a Bucket Policy. TaskBoard needs neither, because presigned URLs grant time-bound access to single objects.

2.2 Presigned Upload URLs

import uuid

BUCKET = "taskboard-uploads-dev-7f3a"

def create_upload(task_id: int, filename: str, content_type: str) -> dict:
    key = f"tasks/{task_id}/{uuid.uuid4()}/{filename}"
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=900,
    )
    return {"key": key, "url": url, "headers": {"Content-Type": content_type}}

Presigned URLs are generated locally by a signing algorithm with your credentials; there is no API call. Signature types v2 and v4 are supported. The browser then sends the file itself:

const { key, url, headers } = await (await fetch(`/api/tasks/${id}/attachments`, { method: "POST",
  body: JSON.stringify({ filename: file.name, contentType: file.type }) })).json();
await fetch(url, { method: "PUT", headers, body: file });
await fetch(`/api/tasks/${id}/attachments/${encodeURIComponent(key)}/confirm`, { method: "POST" });

Because ContentType is part of the signature, the browser must send exactly that header. The confirm call lets the API check the object with head_object before it writes the key to PostgreSQL.

Only the bucket owner can generate a valid presigned URL; for a contract-owned bucket, an administrator can as well. The key that signs must belong to one of them.

2.3 CORS for Browser Uploads

A browser on https://taskboard.example.com may only PUT to the bucket if the bucket's CORS configuration allows it:

resource "ionoscloud_s3_bucket_cors_configuration" "uploads" {
  bucket = ionoscloud_s3_bucket.uploads.name
  cors_rule {
    allowed_methods = ["PUT", "GET"]
    allowed_origins = ["https://taskboard.example.com"]
    allowed_headers = ["Content-Type"]
    expose_headers  = ["ETag"]
    max_age_seconds = 3000
  }
}

allowed_methods accepts GET, PUT, HEAD, POST and DELETE.

3. Upload and Download Patterns

3.1 Multipart Upload for Large Files

A single upload request is limited to 5 GiB for contract-owned buckets and 4.65 GiB for user-owned buckets. Above that, use multipart upload, up to the maximum object size of 5 TiB. boto3's managed transfer splits files for you:

from boto3.s3.transfer import TransferConfig

cfg = TransferConfig(multipart_threshold=64 * 1024**2,
                     multipart_chunksize=64 * 1024**2,
                     max_concurrency=4)
s3.upload_file("export.tar.gz", BUCKET, "exports/2026-11/export.tar.gz",
               ExtraArgs={"ContentType": "application/gzip"}, Config=cfg)

Server-side jobs such as the worker's exports use this pattern. For multi-gigabyte browser uploads, the API calls create_multipart_upload, presigns one upload_part URL per part, and finishes with complete_multipart_upload using the returned ETags. An abandoned multipart upload keeps its parts, which the lifecycle rule in Section 4 removes.

3.2 Streaming Downloads

For downloads, the API usually returns a redirect to a presigned GET. When it must process the file itself, stream it instead of loading it into memory:

from fastapi.responses import StreamingResponse

def download(key: str):
    obj = s3.get_object(Bucket=BUCKET, Key=key)
    return StreamingResponse(obj["Body"].iter_chunks(chunk_size=1024 * 1024),
                             media_type=obj["ContentType"])

Store the content type at upload time; get_object returns it. User-defined metadata per object is limited to 2 kB, and an object name, including its folder path, can have up to 1024 characters.

3.3 Rate Limits on Listing and Deleting

On contract-owned buckets, resource-heavy use of List Objects and Delete Objects triggers rate limiting, enforced per contract. The response is 429 Too Many Requests with a Retry-After header:

import time

def list_all(prefix: str):
    kwargs = {"Bucket": BUCKET, "Prefix": prefix}
    while True:
        try:
            page = s3.list_objects_v2(**kwargs)
        except ClientError as e:
            if e.response["ResponseMetadata"]["HTTPStatusCode"] != 429:
                raise
            time.sleep(int(e.response["ResponseMetadata"]["HTTPHeaders"].get("retry-after", "1")))
            continue
        yield from page.get("Contents", [])
        if not page.get("IsTruncated"):
            return
        kwargs["ContinuationToken"] = page["NextContinuationToken"]

Do not poll List Objects on a fixed schedule; keep the list of attachments in PostgreSQL.

4. Lifecycle Rules via the API

Lifecycle rules automate the deletion of objects and their versions. Apply them with the S3 API:

s3.put_bucket_lifecycle_configuration(
    Bucket=BUCKET,
    LifecycleConfiguration={"Rules": [
        {"ID": "expire-tmp", "Status": "Enabled",
         "Filter": {"Prefix": "tmp/"}, "Expiration": {"Days": 7}},
        {"ID": "abort-multipart", "Status": "Enabled", "Filter": {},
         "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 5}},
    ]},
)
print(s3.get_bucket_lifecycle_configuration(Bucket=BUCKET)["Rules"])

A rule supports four actions: expire current versions, permanently delete noncurrent versions, delete expired object delete markers, and delete incomplete multipart uploads. A configuration can hold up to 1000 rules. Terraform manages the same configuration with ionoscloud_s3_bucket_lifecycle_configuration.

Transition rules do not exist here. Only the STANDARD storage class is available, so lifecycle rules cannot move objects to another storage class. Cost management on IONOS Cloud Object Storage means deleting what you no longer need.

5. S3 Compatibility Checks

IONOS Cloud Object Storage supports most S3 APIs, but not all. The following table shows the support per feature:

Feature Supported Notes
Bucket CRUD Yes
Object CRUD Yes
Object Copy Yes Cross-regional copying is not supported
Multipart Uploads Yes
Pre-Signed URLs Yes Signature types v2 and v4 are supported
Bucket ACLs Yes
Object ACLs Yes
Block Public Access Yes Only via the API
Bucket Policies Yes
Object Policies Yes
CORS Configuration Yes
Bucket Versioning Yes
Bucket Replication Yes Intraregional and cross-regional replication are supported
Bucket Tagging Yes Only via the API
Object Tagging Yes Only via the API
Bucket Lifecycle Yes
Bucket Access Logging Yes
Bucket Encryption Configuration Yes Server-side encryption is used by default in the web interface. The encryption with customer-managed encryption keys is available via the API.
Object Encryption Yes Only via the API
Bucket Websites Yes
Bucket Inventory Yes Only via the API
Object Lock Yes
Legal Hold Yes
Identity and Access Management (IAM) No
Security Token Service (STS) No
Multi-factor Authentication No
Bucket Notifications No
Request Payment No
Bucket Metrics No
Bucket Analytics No
Bucket Accelerate No
S3 Select No

As the table shows, the data path is broadly compatible, while AWS account features are missing. Two consequences for TaskBoard: without Bucket Notifications, the API's confirm call (Section 2.2) replaces an "object created" event, and without STS, every service gets its own long-lived key pair. Bucket types also differ: Bucket Access Logging is not supported for contract-owned buckets, for example. Check the bucket type before you rely on a feature.

API Reference Quick Card

Key S3 operations behind this unit's code:

Method Endpoint Description
PUT /{Bucket}/{Key} Upload an object (the target of a presigned upload URL)
GET /{Bucket}/{Key} Download an object (the target of a presigned download URL)
GET /{Bucket}?list-type=2 List objects (ListObjectsV2, rate limited on contract-owned buckets)
POST /{Bucket}/{Key}?uploads Start a multipart upload
PUT /{Bucket}?lifecycle Set the lifecycle configuration

Base URL: https://s3.eu-central-3.ionoscloud.com (one endpoint per region) Authentication: AWS Signature (v4 or v2) with Access Key and Secret Key, not a bearer token

Code Lab

Objective: Upload and retrieve TaskBoard attachments with boto3: a presigned upload URL, a multipart upload and a presigned download, all authenticated with an Access Key and Secret Key.

Prerequisites:

  • The bucket and access key from Unit 2.4, applied, plus the CORS resource from Section 2.3
  • Python 3.12 with boto3 and requests
  • curl and a 100 MB test file: head -c 100M /dev/urandom > big.bin

Step 1: Configure credentials

export S3_ACCESS_KEY=$(terraform output -raw uploads_access_key)
export S3_SECRET_KEY=$(terraform output -raw uploads_secret_key)
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
python -c "import boto3; print(boto3.__version__)"

Expected output:

1.x.y

Step 2: Prove the auth model

curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $IONOS_TOKEN" \
  https://s3.eu-central-3.ionoscloud.com/taskboard-uploads-dev-7f3a

Expected output:

4xx status code (not 200)

The Cloud API token is not an S3 credential, so the request is rejected. Now list the bucket with the client from Section 1:

print(s3.list_objects_v2(Bucket=BUCKET).get("KeyCount"))

Expected output:

0

Step 3: Create a presigned upload URL

up = create_upload(42, "screenshot.png", "image/png")
print(up["key"]); print(up["url"][:60], "...")

Expected output:

tasks/42/<uuid>/screenshot.png
https://s3.eu-central-3.ionoscloud.com/taskboard-uploads-dev ...

Step 4: Upload like the browser

curl -s -o /dev/null -w "%{http_code}\n" -X PUT -H "Content-Type: image/png" \
  --upload-file screenshot.png "<url from Step 3>"

Expected output:

200

Step 5: Upload a large file in parts

s3.upload_file("big.bin", BUCKET, "tmp/big.bin", Config=cfg)
print(s3.head_object(Bucket=BUCKET, Key="tmp/big.bin")["ETag"])

Expected output:

"<hash>-2"

The -2 suffix shows a multipart object of two parts: 64 MB and the remaining 36 MB.

Step 6: Download through a presigned URL

url = s3.generate_presigned_url("get_object", Params={"Bucket": BUCKET, "Key": up["key"]}, ExpiresIn=300)
import requests; r = requests.get(url); print(r.status_code, r.headers["Content-Type"])

Expected output:

200 image/png

Step 7: Apply the lifecycle rules

Run the code from Section 4. Expected output:

[{'ID': 'expire-tmp', ...}, {'ID': 'abort-multipart', ...}]

Validation Checklist:

  • [ ] The bearer token is rejected, the key pair gets access
  • [ ] The presigned PUT uploads without credentials in the request
  • [ ] The multipart ETag ends with a part count
  • [ ] The downloaded object has the content type set at upload

Cleanup:

for k in [up["key"], "tmp/big.bin"]:
    s3.delete_object(Bucket=BUCKET, Key=k)

Common Pitfalls

Developer mistakes to avoid with this unit's topic:

  1. InvalidTrailer on every upload

    • Problem: put_object and upload_file fail after a dependency update with "Invalid trailing header names in x-amz-trailer".
    • Why it happens: boto3 1.36.0 and later send checksum trailers that Object Storage does not yet support.
    • Fix: Set AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required, or pin boto3<=1.35.99 (Section 1.2).
  2. SignatureDoesNotMatch on presigned uploads

    • Problem: The browser's PUT returns 403 with SignatureDoesNotMatch.
    • Why it happens: The browser sends a different Content-Type than the one signed, or the URL was signed against another endpoint or region.
    • Fix: Return the signed headers with the URL and send them unchanged; sign with the bucket's regional endpoint.
  3. Waiting for an AWS feature that is not there

    • Problem: Code that calls put_bucket_notification_configuration, or a lifecycle rule with Transitions, fails.
    • Why it happens: Bucket Notifications are not supported, and only the STANDARD storage class exists.
    • Fix: Check the compatibility table (Section 5) first; use an explicit confirm call instead of events and expiration instead of transitions.

Summary

You can now integrate IONOS Cloud Object Storage into TaskBoard the S3 way, with IONOS CLOUD specifics. The boto3 client gets an explicit regional endpoint and an Access Key and Secret Key. Browsers upload and download through presigned URLs, so the API never proxies the bytes, and large files use multipart upload. Lifecycle rules delete temporary files and abandoned uploads, and you check the compatibility table before you use an AWS-only feature.

Key Points:

  • Always set endpoint_url; authenticate with an Access Key and Secret Key, never a bearer token
  • Set the two checksum variables, or pin boto3 below 1.36.0
  • Presigned URLs are signed locally; the signing key must belong to the bucket owner or an administrator
  • Use multipart upload above the single-request limit; add a rule that aborts incomplete uploads
  • Lifecycle rules expire and delete objects; there are no transition rules

Important Terminology:

  • Access Key and Secret Key: The credential pair that signs every S3 request; not tied to a region or bucket
  • Presigned URL: A URL that carries a time-limited signature for one operation on one object
  • Contract-owned bucket: A bucket owned by the contract owner, in eu-central-3, eu-central-4 or us-central-1
  • Multipart upload: An upload split into parts that are sent separately and then combined into one object
  • Lifecycle rule: A bucket rule that expires objects, noncurrent versions or incomplete uploads after a set time

Next Steps

Continue Learning: Unit 4.3: Event Streaming Integration

Related Topics: