Unit 4.2: Object Storage Integration
Introduction
TaskBoard users want to attach screenshots, logs and documents to tasks. Routing those bytes through the API pods would cost memory, bandwidth and request time. Instead, the browser uploads straight to the bucket from Unit 2.4 with a short-lived presigned URL, and the API stores only the object key in PostgreSQL.
IONOS Cloud Object Storage speaks the S3 protocol, so you use standard S3 SDKs. But it is not AWS: you set the endpoint explicitly, authenticate with an Access Key and Secret Key, and check which S3 features exist before you call them. In this unit you build the attachment service with boto3: client setup, presigned URLs, multipart uploads, streaming downloads and lifecycle rules. The Code Lab runs the full upload and download flow.
1. The boto3 Client
import os
import boto3
from botocore.config import Config
s3 = boto3.client(
"s3",
endpoint_url="https://s3.eu-central-3.ionoscloud.com",
aws_access_key_id=os.environ["S3_ACCESS_KEY"], # terraform output uploads_access_key
aws_secret_access_key=os.environ["S3_SECRET_KEY"], # terraform output uploads_secret_key
region_name="eu-central-3",
config=Config(signature_version="s3v4"),
)
print([b["Name"] for b in s3.list_buckets()["Buckets"]])
Without endpoint_url, boto3 talks to AWS and your keys fail. The credentials are an Access Key and Secret Key pair: Object Storage does not accept the bearer token of the Cloud API, and IAM roles and the Security Token Service (STS) are not supported. Your credentials are not tied to a specific region or bucket, so one key pair works against every endpoint.
1.1 Endpoints per Region
The endpoint depends on the bucket type and region. The following table lists the endpoints for contract-owned buckets:
| Data Center | Region | Endpoint | Static Website Endpoint |
|---|---|---|---|
| Frankfurt, Germany | eu-central-4 |
s3.eu-central-4.ionoscloud.com |
s3-website.eu-central-4.ionoscloud.com |
| Berlin, Germany | eu-central-3 |
s3.eu-central-3.ionoscloud.com |
s3-website.eu-central-3.ionoscloud.com |
| Lenexa, USA | us-central-1 |
s3.us-central-1.ionoscloud.com |
s3-website.us-central-1.ionoscloud.com |
User-owned buckets use other regions, for example s3.eu-central-2.ionoscloud.com for Berlin. TaskBoard's bucket is contract-owned in eu-central-3. The endpoints answer on HTTP and HTTPS; always use https://. Encryption in transit uses TLS 1.2 or 1.3.
1.2 The boto3 1.36 Checksum Change
From boto3 1.36.0, PutObject sends new checksum trailers that Object Storage does not yet support. Calls then fail with InvalidTrailer: "Invalid trailing header names in x-amz-trailer". Either pin boto3<=1.35.99 or set these variables for your process:
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
Put them in the API's ConfigMap from Unit 3.2 so that every pod gets them.
2. Buckets, Access and Presigned URLs
2.1 Bucket Operations
Terraform owns TaskBoard's long-lived bucket (Unit 2.4). Tests and tools can still create and list buckets in code:
from botocore.exceptions import ClientError
try:
s3.create_bucket(Bucket="taskboard-ci-4711",
CreateBucketConfiguration={"LocationConstraint": "eu-central-3"})
except ClientError as e:
if e.response["Error"]["Code"] != "BucketAlreadyOwnedByYou":
raise
print(s3.get_bucket_acl(Bucket="taskboard-ci-4711")["Grants"])
Objects are private by default; only the bucket owner can access them. Keep it that way. For a contract-owned bucket, an ACL shares access only with grantees from other contracts; to give users of your own contract access, use a Bucket Policy. TaskBoard needs neither, because presigned URLs grant time-bound access to single objects.
2.2 Presigned Upload URLs
import uuid
BUCKET = "taskboard-uploads-dev-7f3a"
def create_upload(task_id: int, filename: str, content_type: str) -> dict:
key = f"tasks/{task_id}/{uuid.uuid4()}/{filename}"
url = s3.generate_presigned_url(
"put_object",
Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
ExpiresIn=900,
)
return {"key": key, "url": url, "headers": {"Content-Type": content_type}}
Presigned URLs are generated locally by a signing algorithm with your credentials; there is no API call. Signature types v2 and v4 are supported. The browser then sends the file itself:
const { key, url, headers } = await (await fetch(`/api/tasks/${id}/attachments`, { method: "POST",
body: JSON.stringify({ filename: file.name, contentType: file.type }) })).json();
await fetch(url, { method: "PUT", headers, body: file });
await fetch(`/api/tasks/${id}/attachments/${encodeURIComponent(key)}/confirm`, { method: "POST" });
Because ContentType is part of the signature, the browser must send exactly that header. The confirm call lets the API check the object with head_object before it writes the key to PostgreSQL.
Only the bucket owner can generate a valid presigned URL; for a contract-owned bucket, an administrator can as well. The key that signs must belong to one of them.
2.3 CORS for Browser Uploads
A browser on https://taskboard.example.com may only PUT to the bucket if the bucket's CORS configuration allows it:
resource "ionoscloud_s3_bucket_cors_configuration" "uploads" {
bucket = ionoscloud_s3_bucket.uploads.name
cors_rule {
allowed_methods = ["PUT", "GET"]
allowed_origins = ["https://taskboard.example.com"]
allowed_headers = ["Content-Type"]
expose_headers = ["ETag"]
max_age_seconds = 3000
}
}
allowed_methods accepts GET, PUT, HEAD, POST and DELETE.
3. Upload and Download Patterns
3.1 Multipart Upload for Large Files
A single upload request is limited to 5 GiB for contract-owned buckets and 4.65 GiB for user-owned buckets. Above that, use multipart upload, up to the maximum object size of 5 TiB. boto3's managed transfer splits files for you:
from boto3.s3.transfer import TransferConfig
cfg = TransferConfig(multipart_threshold=64 * 1024**2,
multipart_chunksize=64 * 1024**2,
max_concurrency=4)
s3.upload_file("export.tar.gz", BUCKET, "exports/2026-11/export.tar.gz",
ExtraArgs={"ContentType": "application/gzip"}, Config=cfg)
Server-side jobs such as the worker's exports use this pattern. For multi-gigabyte browser uploads, the API calls create_multipart_upload, presigns one upload_part URL per part, and finishes with complete_multipart_upload using the returned ETags. An abandoned multipart upload keeps its parts, which the lifecycle rule in Section 4 removes.
3.2 Streaming Downloads
For downloads, the API usually returns a redirect to a presigned GET. When it must process the file itself, stream it instead of loading it into memory:
from fastapi.responses import StreamingResponse
def download(key: str):
obj = s3.get_object(Bucket=BUCKET, Key=key)
return StreamingResponse(obj["Body"].iter_chunks(chunk_size=1024 * 1024),
media_type=obj["ContentType"])
Store the content type at upload time; get_object returns it. User-defined metadata per object is limited to 2 kB, and an object name, including its folder path, can have up to 1024 characters.
3.3 Rate Limits on Listing and Deleting
On contract-owned buckets, resource-heavy use of List Objects and Delete Objects triggers rate limiting, enforced per contract. The response is 429 Too Many Requests with a Retry-After header:
import time
def list_all(prefix: str):
kwargs = {"Bucket": BUCKET, "Prefix": prefix}
while True:
try:
page = s3.list_objects_v2(**kwargs)
except ClientError as e:
if e.response["ResponseMetadata"]["HTTPStatusCode"] != 429:
raise
time.sleep(int(e.response["ResponseMetadata"]["HTTPHeaders"].get("retry-after", "1")))
continue
yield from page.get("Contents", [])
if not page.get("IsTruncated"):
return
kwargs["ContinuationToken"] = page["NextContinuationToken"]
Do not poll List Objects on a fixed schedule; keep the list of attachments in PostgreSQL.
4. Lifecycle Rules via the API
Lifecycle rules automate the deletion of objects and their versions. Apply them with the S3 API:
s3.put_bucket_lifecycle_configuration(
Bucket=BUCKET,
LifecycleConfiguration={"Rules": [
{"ID": "expire-tmp", "Status": "Enabled",
"Filter": {"Prefix": "tmp/"}, "Expiration": {"Days": 7}},
{"ID": "abort-multipart", "Status": "Enabled", "Filter": {},
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 5}},
]},
)
print(s3.get_bucket_lifecycle_configuration(Bucket=BUCKET)["Rules"])
A rule supports four actions: expire current versions, permanently delete noncurrent versions, delete expired object delete markers, and delete incomplete multipart uploads. A configuration can hold up to 1000 rules. Terraform manages the same configuration with ionoscloud_s3_bucket_lifecycle_configuration.
Transition rules do not exist here. Only the STANDARD storage class is available, so lifecycle rules cannot move objects to another storage class. Cost management on IONOS Cloud Object Storage means deleting what you no longer need.
5. S3 Compatibility Checks
IONOS Cloud Object Storage supports most S3 APIs, but not all. The following table shows the support per feature:
| Feature | Supported | Notes |
|---|---|---|
| Bucket CRUD | Yes | |
| Object CRUD | Yes | |
| Object Copy | Yes | Cross-regional copying is not supported |
| Multipart Uploads | Yes | |
| Pre-Signed URLs | Yes | Signature types v2 and v4 are supported |
| Bucket ACLs | Yes | |
| Object ACLs | Yes | |
| Block Public Access | Yes | Only via the API |
| Bucket Policies | Yes | |
| Object Policies | Yes | |
| CORS Configuration | Yes | |
| Bucket Versioning | Yes | |
| Bucket Replication | Yes | Intraregional and cross-regional replication are supported |
| Bucket Tagging | Yes | Only via the API |
| Object Tagging | Yes | Only via the API |
| Bucket Lifecycle | Yes | |
| Bucket Access Logging | Yes | |
| Bucket Encryption Configuration | Yes | Server-side encryption is used by default in the web interface. The encryption with customer-managed encryption keys is available via the API. |
| Object Encryption | Yes | Only via the API |
| Bucket Websites | Yes | |
| Bucket Inventory | Yes | Only via the API |
| Object Lock | Yes | |
| Legal Hold | Yes | |
| Identity and Access Management (IAM) | No | |
| Security Token Service (STS) | No | |
| Multi-factor Authentication | No | |
| Bucket Notifications | No | |
| Request Payment | No | |
| Bucket Metrics | No | |
| Bucket Analytics | No | |
| Bucket Accelerate | No | |
| S3 Select | No |
As the table shows, the data path is broadly compatible, while AWS account features are missing. Two consequences for TaskBoard: without Bucket Notifications, the API's confirm call (Section 2.2) replaces an "object created" event, and without STS, every service gets its own long-lived key pair. Bucket types also differ: Bucket Access Logging is not supported for contract-owned buckets, for example. Check the bucket type before you rely on a feature.
API Reference Quick Card
Key S3 operations behind this unit's code:
| Method | Endpoint | Description |
|---|---|---|
PUT |
/{Bucket}/{Key} |
Upload an object (the target of a presigned upload URL) |
GET |
/{Bucket}/{Key} |
Download an object (the target of a presigned download URL) |
GET |
/{Bucket}?list-type=2 |
List objects (ListObjectsV2, rate limited on contract-owned buckets) |
POST |
/{Bucket}/{Key}?uploads |
Start a multipart upload |
PUT |
/{Bucket}?lifecycle |
Set the lifecycle configuration |
Base URL: https://s3.eu-central-3.ionoscloud.com (one endpoint per region)
Authentication: AWS Signature (v4 or v2) with Access Key and Secret Key, not a bearer token
Code Lab
Objective: Upload and retrieve TaskBoard attachments with boto3: a presigned upload URL, a multipart upload and a presigned download, all authenticated with an Access Key and Secret Key.
Prerequisites:
- The bucket and access key from Unit 2.4, applied, plus the CORS resource from Section 2.3
- Python 3.12 with
boto3andrequests curland a 100 MB test file:head -c 100M /dev/urandom > big.bin
Step 1: Configure credentials
export S3_ACCESS_KEY=$(terraform output -raw uploads_access_key)
export S3_SECRET_KEY=$(terraform output -raw uploads_secret_key)
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
python -c "import boto3; print(boto3.__version__)"
Expected output:
1.x.y
Step 2: Prove the auth model
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $IONOS_TOKEN" \
https://s3.eu-central-3.ionoscloud.com/taskboard-uploads-dev-7f3a
Expected output:
4xx status code (not 200)
The Cloud API token is not an S3 credential, so the request is rejected. Now list the bucket with the client from Section 1:
print(s3.list_objects_v2(Bucket=BUCKET).get("KeyCount"))
Expected output:
0
Step 3: Create a presigned upload URL
up = create_upload(42, "screenshot.png", "image/png")
print(up["key"]); print(up["url"][:60], "...")
Expected output:
tasks/42/<uuid>/screenshot.png
https://s3.eu-central-3.ionoscloud.com/taskboard-uploads-dev ...
Step 4: Upload like the browser
curl -s -o /dev/null -w "%{http_code}\n" -X PUT -H "Content-Type: image/png" \
--upload-file screenshot.png "<url from Step 3>"
Expected output:
200
Step 5: Upload a large file in parts
s3.upload_file("big.bin", BUCKET, "tmp/big.bin", Config=cfg)
print(s3.head_object(Bucket=BUCKET, Key="tmp/big.bin")["ETag"])
Expected output:
"<hash>-2"
The -2 suffix shows a multipart object of two parts: 64 MB and the remaining 36 MB.
Step 6: Download through a presigned URL
url = s3.generate_presigned_url("get_object", Params={"Bucket": BUCKET, "Key": up["key"]}, ExpiresIn=300)
import requests; r = requests.get(url); print(r.status_code, r.headers["Content-Type"])
Expected output:
200 image/png
Step 7: Apply the lifecycle rules
Run the code from Section 4. Expected output:
[{'ID': 'expire-tmp', ...}, {'ID': 'abort-multipart', ...}]
Validation Checklist:
- [ ] The bearer token is rejected, the key pair gets access
- [ ] The presigned
PUTuploads without credentials in the request - [ ] The multipart ETag ends with a part count
- [ ] The downloaded object has the content type set at upload
Cleanup:
for k in [up["key"], "tmp/big.bin"]:
s3.delete_object(Bucket=BUCKET, Key=k)
Common Pitfalls
Developer mistakes to avoid with this unit's topic:
-
InvalidTrailer on every upload
- Problem:
put_objectandupload_filefail after a dependency update with "Invalid trailing header names in x-amz-trailer". - Why it happens: boto3 1.36.0 and later send checksum trailers that Object Storage does not yet support.
- Fix: Set
AWS_REQUEST_CHECKSUM_CALCULATION=when_requiredandAWS_RESPONSE_CHECKSUM_VALIDATION=when_required, or pinboto3<=1.35.99(Section 1.2).
- Problem:
-
SignatureDoesNotMatch on presigned uploads
- Problem: The browser's
PUTreturns403withSignatureDoesNotMatch. - Why it happens: The browser sends a different
Content-Typethan the one signed, or the URL was signed against another endpoint or region. - Fix: Return the signed headers with the URL and send them unchanged; sign with the bucket's regional endpoint.
- Problem: The browser's
-
Waiting for an AWS feature that is not there
- Problem: Code that calls
put_bucket_notification_configuration, or a lifecycle rule withTransitions, fails. - Why it happens: Bucket Notifications are not supported, and only the
STANDARDstorage class exists. - Fix: Check the compatibility table (Section 5) first; use an explicit confirm call instead of events and expiration instead of transitions.
- Problem: Code that calls
Summary
You can now integrate IONOS Cloud Object Storage into TaskBoard the S3 way, with IONOS CLOUD specifics. The boto3 client gets an explicit regional endpoint and an Access Key and Secret Key. Browsers upload and download through presigned URLs, so the API never proxies the bytes, and large files use multipart upload. Lifecycle rules delete temporary files and abandoned uploads, and you check the compatibility table before you use an AWS-only feature.
Key Points:
- Always set
endpoint_url; authenticate with an Access Key and Secret Key, never a bearer token - Set the two checksum variables, or pin boto3 below 1.36.0
- Presigned URLs are signed locally; the signing key must belong to the bucket owner or an administrator
- Use multipart upload above the single-request limit; add a rule that aborts incomplete uploads
- Lifecycle rules expire and delete objects; there are no transition rules
Important Terminology:
- Access Key and Secret Key: The credential pair that signs every S3 request; not tied to a region or bucket
- Presigned URL: A URL that carries a time-limited signature for one operation on one object
- Contract-owned bucket: A bucket owned by the contract owner, in
eu-central-3,eu-central-4orus-central-1 - Multipart upload: An upload split into parts that are sent separately and then combined into one object
- Lifecycle rule: A bucket rule that expires objects, noncurrent versions or incomplete uploads after a set time
Next Steps
Continue Learning: Unit 4.3: Event Streaming Integration
Related Topics: