S3 Backend¶

Connects to an AWS S3 or S3-compatible object-storage service over HTTP. Objects are stored in a bucket; each blob key maps to an object key. An optional prefix places all objects beneath a shared key prefix.

Installation¶

The S3 backend uses gufo-http, which is an optional dependency. Install Gufo Blob with the http extra:

pip install "gufo_blob[http]"

For an editable checkout, install the same extra with:

pip install -e ".[http]"

Properties¶

Property	Level
Data durability	Depends on the S3-compatible service configuration
Data sharing	Network-distributed

Parameters¶

Parameter	Default	Description
`bucket`	(required)	Bucket name.
`prefix`	`""`	Optional key prefix within the bucket.
`endpoint`	`https://s3.amazonaws.com`	S3 endpoint URL; set this for compatible services.
`region`	`"us-east-1"`	AWS signing region.
`access_key`	`None`	Access key ID.
`secret_key`	`None`	Secret access key; provide it together with `access_key`.

When credentials are provided, requests are signed with AWS Signature Version 4. If both credentials are omitted, requests are sent unsigned. The backend does not load credentials from the AWS environment or shared credentials files.

Behavior¶

Put: Sends a single PutObject request. Existing objects are overwritten.

Get: Reads the object body as bytes. A missing object raises KeyError.

Exists: Uses HEAD; a missing object returns False.

Delete: Checks that the object exists with HEAD, then sends DELETE. A missing object raises KeyError.

Scan: Uses the S3 ListObjectsV2 API, follows continuation tokens, and returns keys relative to the configured bucket prefix. The scan argument is an additional string prefix; it does not support globs or directory collapsing.

Object keys are URL-encoded for requests. A configured prefix is stripped from keys returned by scan().

Limitations¶

The backend uses path-style requests (/{bucket}/{key}). The selected service must support that addressing style.
Uploads use a single PutObject request; multipart and streaming uploads are not implemented.
Bucket creation and deletion, object versioning, ACLs, and other S3 management APIs are not implemented.
When delete() checks existence and then deletes, another client can change the object between those requests.
Provider-specific durability and availability depend on the service configuration.

URL Routing¶

Schema: s3://. The path after the bucket is the optional object prefix. Supported query parameters are endpoint, region, access_key, and secret_key.

s3://my-bucket/archive?endpoint=https://s3.example.com&region=eu-central-1&access_key=ACCESS_KEY&secret_key=SECRET_KEY

Percent-encode query parameter values when they contain reserved URL characters. The bucket name must be 3–63 characters, contain lowercase letters, digits, dots, or hyphens, and start and end with a lowercase letter or digit.

Examples¶

URLSyncAsync

s3://my-bucket/archive?endpoint=https://s3.example.com&region=eu-central-1&access_key=ACCESS_KEY&secret_key=SECRET_KEY

from gufo.blob.sync.s3 import S3Blob

with S3Blob(
    bucket="my-bucket",
    prefix="archive",
    endpoint="https://s3.example.com",
    region="eu-central-1",
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
) as blob:
    blob.put("report.bin", b"data")
    data = blob.get("report.bin")

from gufo.blob.aio.s3 import S3Blob

async with S3Blob(
    bucket="my-bucket",
    prefix="archive",
    endpoint="https://s3.example.com",
    region="eu-central-1",
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
) as blob:
    await blob.put("report.bin", b"data")
    data = await blob.get("report.bin")

Compatibility¶

The backend tests run against Moto's S3-compatible HTTP server.