S3 filesystem

PyAthena ships its own fsspec-compatible filesystem implementation for Amazon S3 (S3FileSystem), built on boto3, with an API surface compatible with s3fs for users migrating from it.

The filesystem is used internally by the pandas, Polars, and S3FS result sets to read query results from S3, and can also be used independently for S3 file operations.

fsspec registration

Importing pyathena.pandas or pyathena.polars registers S3FileSystem as the fsspec s3 / s3a protocols via pyathena.filesystem.register_s3_filesystem. This replaces fsspec’s default lazy mapping of the s3 protocol to s3fs, which means fsspec.filesystem("s3") returns PyAthena’s implementation and s3fs-specific settings (such as the S3FS_LOGGING_LEVEL environment variable) have no effect. The registered class is the synchronous S3FileSystem, whose async_impl is False. Like fsspec’s other synchronous filesystems, it accepts and ignores asynchronous=True, so fsspec.filesystem("s3", asynchronous=True) returns a synchronous filesystem; libraries such as zarr check async_impl and wrap it. For a native asynchronous filesystem, use AioS3FileSystem.

A filesystem class that has already been registered explicitly is also overwritten, with a warning log, so that the replacement is diagnosable. To restore another implementation, re-register it afterwards:

import fsspec
import s3fs

import pyathena.pandas  # Registers PyAthena's S3FileSystem.

fsspec.register_implementation("s3", s3fs.S3FileSystem, clobber=True)

Basic usage

The filesystem can be created from a PyAthena connection with s3_filesystem(), which gives it the S3 client and the retry policy of the connection (see “S3 client” in Usage), with a boto3 S3 client, or with s3fs-compatible credential arguments:

import boto3

from pyathena import connect
from pyathena.filesystem import s3_filesystem
from pyathena.filesystem.s3 import S3FileSystem

fs = s3_filesystem(connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
                           region_name="us-west-2"))

# Or with a boto3 S3 client, and optionally a pyathena.util.RetryConfig.
fs = S3FileSystem(client=boto3.client("s3", region_name="us-west-2"))

# Or with direct credentials (s3fs-compatible arguments).
fs = S3FileSystem(key="YOUR_ACCESS_KEY", secret="YOUR_SECRET_KEY")

# Or with a named profile.
fs = S3FileSystem(profile="YOUR_PROFILE")

# Or anonymously for public buckets.
fs = S3FileSystem(anon=True)

The filesystem uses a given client as it is and does not close it. Combining client with an argument from which the filesystem builds a client, such as key, secret, token, profile, anon, endpoint_url, client_kwargs, config_kwargs, or region_name, raises TypeError; such settings from fsspec’s configuration, such as the FSSPEC_S3_ENDPOINT_URL environment variable, are ignored. requester_pays, max_workers, the cache options, and the arguments of fsspec can be combined with it. s3_filesystem() takes the other arguments of the filesystem, and AioS3FileSystem as its second argument.

The connection argument of S3FileSystem and AioS3FileSystem is deprecated and emits a DeprecationWarning; it will be removed in the next major version. It is subject to the same restrictions as client.

The S3 client computes checksums of uploads and validates those of downloads by botocore’s request_checksum_calculation and response_checksum_validation options, set with s3_config of the connection, with config_kwargs, or in the configuration of a given client; see “Checksums” under “S3 client” in Usage for their settings and their effect on data integrity.

Standard fsspec operations work as expected:

fs.ls("s3://YOUR_S3_BUCKET/path/to/")
fs.find("s3://YOUR_S3_BUCKET/path/to/")
fs.exists("s3://YOUR_S3_BUCKET/path/to/object")
fs.info("s3://YOUR_S3_BUCKET/path/to/object")

with fs.open("s3://YOUR_S3_BUCKET/path/to/object", "rb") as f:
    data = f.read()

fs.pipe("s3://YOUR_S3_BUCKET/path/to/object", b"data")
fs.cat("s3://YOUR_S3_BUCKET/path/to/object")
fs.cp("s3://YOUR_S3_BUCKET/src", "s3://YOUR_S3_BUCKET/dst")
fs.rm("s3://YOUR_S3_BUCKET/path/to/", recursive=True)

cat, cat_ranges, get, put, pipe with a dict of values, cp, and mv transfer their objects in parallel, batch_size objects at a time:

fs.get("s3://YOUR_S3_BUCKET/path/to/", "local/dir/", recursive=True, batch_size=16)

batch_size is an argument of these methods, of rm, and of S3FileSystem. By default, and with -1, it is the filesystem’s max_workers. The transfers of a batch run in threads of their own. With several transfers at once, each object sends at most max(1, max_workers // n) range reads, parts, or part copies at once, n being the smaller of batch_size and the number of objects, so that a batch keeps about max_workers requests in the request pool and as many parts in memory; a smaller max_workers given to the method is kept. A single object keeps the filesystem’s max_workers. cat, cat_ranges, and a small pipe send their single request from the transfer’s thread, so a batch_size larger than max_workers sends that many requests at once.

When a transfer fails, no further transfer starts, the retries of the running ones stop, and the error is raised once they have ended. A running multipart upload or copy sends no further part and is aborted instead of being completed, and a running download reads and writes no further block. A request already sent, such as the PutObject of a small object, runs to its end; a multipart copy that has been completed still copies its annotations. The transfers that ended before the failure stay done, and their number is not bounded by batch_size. mv deletes the sources only once every copy has succeeded. on_error of cat, cat_ranges, and cp keeps its fsspec meaning; with on_error="omit" or "return" of cat, "return" of cat_ranges, or "ignore" of cp for a missing source, those errors do not stop the batch.

Pairs whose result would depend on the order of the transfers raise ValueError before anything is transferred: two sources of get, put, cp, or pipe with the same destination, including two paths of one existing local file or one file-like object, and a cp destination that is another source. A local directory, which put skips, does not conflict, and neither do two directories that get creates at one local path; the sources of get are looked up for that only when their local paths are the same. Local paths that do not exist yet are compared as strings, after os.path.normcase, so on a case-insensitive filesystem other than Windows, such as the default one of macOS, give get destinations that differ by more than case. A cp of an object onto itself, such as with MetadataDirective="REPLACE", is no conflict. mv raises for the same pairs as cp. A source of cp or mv without an object at its key, such as a key prefix given in a list, writes no destination and does not conflict; the sources without a version whose destinations conflict are looked up with HeadObject before anything is copied. get with outfile writes the objects into it one after another, in order.

rm deletes up to 1,000 objects per DeleteObjects request, batch_size requests at a time. Its max_workers argument is a deprecated alias of batch_size.

get and get_file download each object in waves of parallel range reads of the block size: at most max_workers ranges at a time, and no more than the request pool runs at once. The ranges of a wave are written in order before the next wave is read, so a download holds at most one wave of data, max_workers × the block size, apart from a range that a thread of the request pool has just read until its call returns. cache_type and cache_options do not apply: no data cache reads the object. The range reads are conditional on the ETag, and the version, if any, of the object looked up when the download starts: an object replaced during the download fails it with OSError, unless the download reads a version other than null, which a replacement does not change. A failed download leaves the part written so far in the local file, as fsspec’s get_file does. With compression, the object is decompressed into the destination, read in order through a block cache, which reads ahead of the decompressor; a block size that is not positive takes the filesystem’s.

Writes with pipe/pipe_file issue a single PutObject request for data up to the block size (5 MiB by default); larger data is uploaded as a parallel multipart upload through the buffered file path. Inside an fsspec transaction, writes are deferred until the transaction commits and are discarded on rollback. With the compression argument (a codec of fsspec, or "infer" from the extension of the path), pipe/pipe_file compress the data before uploading it, and the sizes in this section apply to the compressed data.

A multipart write of a file returned by open keeps at most max_workers parts in flight, counting the parts of the written data and the parts copied from the existing object in an append, and waits for one of them to finish before it submits another. The parts take about max_workers × the block size of memory, in addition to the buffered data of the file. If a part has failed before the write submits another part, the write raises its error instead of uploading the rest of the data, and aborts the multipart upload after the parts being uploaded or copied finish. A part that fails after the last write raises its error at the next write, when the file is closed, or when the transaction commits, and the multipart upload is aborted. max_workers is an argument of S3FileSystem, AioS3FileSystem, open, put, and pipe.

The requests that the files and operations of a filesystem run in parallel (range reads, parts, part copies, the ACLs of a recursive chmod, the DeleteObjects batches of rm, and the aborts of clear_multipart_uploads) share one request pool of the filesystem, with max_workers threads. Each file or operation runs at most its own max_workers requests at once in that pool. The max_workers of the filesystem defaults to the max_pool_connections of its S3 client: 10 unless configured, or that of the given client, such as the client of the connection. A client that the filesystem builds takes its max_pool_connections from max_workers unless config_kwargs sets it; max_workers does not change the connection pool of a given client. For more parallel requests, raise max_workers, and with a given client, also its max_pool_connections, such as with the s3_config of the connection. close(), or the end of a with block, releases the threads of the pool once they are idle, and the filesystem stays usable.

Pickling and processes

A filesystem created with s3fs-compatible arguments can be pickled, as for Dask, Ray, or multiprocessing workers: the unpickled filesystem builds its own client from the same arguments. A filesystem created with client, connection, or s3_filesystem(), and the files that it opens for reading, raise TypeError when they are pickled, as a boto3 client cannot be pickled; as in fsspec, a file opened for writing cannot be pickled. To use a filesystem in such workers, create it with s3fs-compatible arguments, such as profile or region_name, or create a filesystem in each worker.

In a forked child process, create a new filesystem. A file that the parent opened raises RuntimeError there from its reads, writes, and flushes, commit(), and discard(), and closing it only marks it closed, without uploading or aborting anything.

Close a file returned by open explicitly or with a with block. A file that is not closed is closed when it is finalized, which commits its written data unless autocommit is False:

  • When it is finalized during a pass of the cyclic garbage collector, it is closed in a new thread that starts once the collection has ended. When it is finalized in a thread of an S3ThreadPool, such as the request pool of a filesystem, it is closed in a new thread. The interpreter waits for these closes when it exits. If no thread can be started, the file is left unclosed and its multipart upload incomplete.

  • When its last reference is dropped elsewhere, it is closed before that returns.

  • During interpreter finalization, or after the exit has waited for the closes in new threads, it is closed in the finalizing thread instead of a new thread. The request pools take no more calls then, so a file with data left to upload as a part is not committed, and its multipart upload is aborted.

If the garbage collector finalizes the write buffer of a file before the file, the data of the file is not committed, and its multipart upload is aborted.

close() called during a pass of the cyclic garbage collector, such as by another finalizer, closes the file in the same way, in a new thread that starts once the collection has ended, and returns before the file is closed. A later close(), commit(), or discard() of the file waits for that close to end. During a collection or in a thread of an S3ThreadPool, close() returns without waiting instead, and commit() and discard() raise RuntimeError. If no thread could be started for that close, a later close(), commit(), or discard() of the file elsewhere closes it first.

A file opened in text mode is closed by its text wrapper when the wrapper is finalized. The wrapper writes its remaining text to the file in the thread that finalizes it, and then closes the file there, or as above during a collection. That write waits as any write does, so in a thread of an S3ThreadPool it can wait forever for a part of the file that is queued behind that thread.

An interrupt, such as Ctrl+C, or a failure while a file reads ranges in parallel cancels the range reads that have not started, and the running ones make no further attempt. The interrupt or the error is raised once they have ended, and their data is released. A second interrupt stops that wait, and the reads still finish. rm and clear_multipart_uploads send every request also after one fails; an interrupt cancels those that have not started and is raised once the running ones have ended. An interrupt of a batch operation, such as get or cp, does the same as a failed transfer. Closing the file waits for its running requests; an interrupt of that wait stops it, and the requests still finish. An interrupt while a request is being submitted can let one more request than max_workers run at once until the request pool is idle. These guarantees hold for one interrupt at a time. If another interrupt arrives while the cleanup of an interrupt or an error runs, the call still does not wait forever, but the cleanup may stop early: the data of finished requests can stay referenced until the garbage collector frees it, and a multipart upload can be left incomplete. clear_multipart_uploads removes such uploads.

A file returned by open whose write or commit has failed, or which has been discarded, cannot be committed: its commit() raises S3MultipartStateError. Inside a transaction, a write error that the caller catches therefore makes the transaction fail when it commits, and fsspec discards the files of the transaction that are not committed yet.

The block size for writing, given by the block_size argument of open or by the filesystem’s default_block_size, must be between 5 MiB and 5 GiB, inclusive, the part size limits of a multipart upload. Otherwise, open raises ValueError.

A multipart upload consists of at most 10,000 parts. put and pipe upload one part per block, so with the default block size they can upload up to about 48.8 GiB (10,000 × 5 MiB). To upload a larger object, use a block size of at least its size divided by 10,000, either with the block_size argument of put, pipe, and open or with the default_block_size argument of S3FileSystem. put and pipe check the size before uploading anything and raise ValueError with the minimum block size if the data needs more parts. A file written with open can take more parts, because each write that fills the buffer uploads the data beyond its last full block as a separate part when that data is at least 5 MiB. In an append, the parts copied from the existing object also count toward the limit. A write with open that reaches the limit raises ValueError and aborts its multipart upload. Multipart copies with cp use parts large enough to stay within the limit.

cp copies an object larger than 5 GiB with a multipart upload instead of a single CopyObject request, with the same result as CopyObject. If HeadObject reports a size of at most 5 GiB when the copy starts, for example because the size that cp found came from a cached listing, the object is copied with CopyObject instead. CopyObject parameters given as keyword arguments are sent to the multipart requests that accept them, such as CopySourceIfMatch to each part copy. With the default COPY value of MetadataDirective, TaggingDirective, and AnnotationDirective, the content headers (such as ContentType) and user-defined metadata, the tags, and the annotations of the source are copied, and the values given for them are ignored, as CopyObject does. A REPLACE directive uses the given values instead, and AnnotationDirective="EXCLUDE" skips the annotations. Copying the tags needs s3:GetObjectTagging on the source, and copying the annotations needs s3:ListObjectAnnotations and s3:GetObjectAnnotation on the source and s3:PutObjectAnnotation on the destination. A source without a ?versionId= suffix whose HeadObject reports a version ID other than null is copied from that version, which needs s3:GetObjectVersion and, to copy the tags, s3:GetObjectVersionTagging on the source. A null version is not pinned. The annotations are listed before anything is written and copied after the upload completes, so the destination exists without them until the last one is written. If an annotation fails to copy, the error is raised and the destination is kept. A failed part copy aborts the multipart upload. So does an interrupt, or the cancellation of an AioS3FileSystem copy, including one by the shutdown of the event loop at the end of asyncio.run(), unless the upload has already completed. A CreateMultipartUpload request and the part copies in flight finish first, without a further attempt. The CompleteMultipartUpload request of an AioS3FileSystem copy also finishes first, with its retries; the requests of an AioS3FileSystem copy that have not started are cancelled. A further interrupt while the CreateMultipartUpload request is waited for is ignored. One while the part copies are waited for or the upload is aborted stops the cleanup, and the upload ID is logged.

Every method of the filesystems also accepts paths as os.PathLike objects, such as pathlib.PurePosixPath("YOUR_S3_BUCKET/dir/key"), and uses them as their strings. pathlib removes a trailing slash and collapses //, so a path with the s3:// protocol or a trailing slash needs to be a string. On Windows, pathlib.Path joins its parts with backslashes, so S3 paths need PurePosixPath.

Paths are normalized as in fsspec, which drops a trailing slash, so info, isfile, and open treat s3://YOUR_S3_BUCKET/dir/ as s3://YOUR_S3_BUCKET/dir: the object dir if it exists, and otherwise the directory dir. An object whose key ends in a slash, such as a folder marker, is therefore not a file for these methods. Opening dir/ for reading reads the object dir or raises FileNotFoundError. Opening it for writing, pipe, pipe_file, and put_file write the object dir. A path with a ?versionId= suffix keeps the slash and refers to the object. find, and ls of the directory, list the object as a file entry. cat_file uses the key as written. Without a ?versionId= suffix, it reads such an object without a range, with a non-empty range of non-negative offsets, or with a negative start and no end, and raises FileNotFoundError for other ranges.

A directory is named after its key prefix without the last slash. A key with an empty segment, such as dir//x, therefore lies in the directory dir/, which ls of dir lists and find, walk, and tree descend into. Since ls drops a trailing slash, ls of dir/ lists dir. walk yields the directory under its name dir/, and names it "" among the directories of dir. find, walk, and tree also descend into a directory whose name ends in a version ID query, such as dir?versionId=v1 for the key dir?versionId=v1/x, while ls of that name looks up version v1 of dir.

S3 request parameters, such as ContentType, ServerSideEncryption, or RequestPayer, can be given to open, pipe, and put as keyword arguments or in s3_additional_kwargs, and to all of them through the s3_additional_kwargs argument of S3FileSystem. The parameters of a call take precedence over those of the filesystem. A file sends each of its requests only the parameters that the S3 operation accepts, so, for example, ServerSideEncryption for writes is not sent with reads. A pipe of data up to the block size sends its parameters with a single PutObject request as given. put sets ContentType from the file extension unless the call or the filesystem gives one.

The parameters in the s3_additional_kwargs argument of S3FileSystem that authorize the access to the objects, RequestPayer, ExpectedBucketOwner, and the SSECustomer* parameters, are sent with every request of the filesystem whose operation accepts them, listings and copies included. A copy also sends them for its source, as ExpectedSourceBucketOwner and the CopySourceSSECustomer* parameters. The parameters of a call take precedence. With an SSECustomerKey there, every object that the filesystem reads or copies must be encrypted with that key, and every object that it writes is: a call that asks for another encryption with ServerSideEncryption sends both, which S3 rejects. sign signs only the parameters given to it.

fs = S3FileSystem(s3_additional_kwargs={"RequestPayer": "requester", **sse_c})
data = fs.cat_file("s3://YOUR_S3_BUCKET/path/to/encrypted.csv")
fs = S3FileSystem(s3_additional_kwargs={"ServerSideEncryption": "AES256"})
with fs.open("s3://YOUR_S3_BUCKET/path/to/data.csv", "wb", ContentType="text/csv") as f:
    f.write(b"col1\n1\n")

info and exists accept ExpectedBucketOwner, RequestPayer, and the SSECustomer* parameters of an object encrypted with a customer-provided key (SSE-C) as keyword arguments, and ignore other request parameters. The lookups of the object by open, and the existence check of pipe with mode="create", send these parameters of the file or the write. A lookup with them uses only the cached results of lookups with the same values, not cached listings.

sse_c = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": YOUR_32_BYTE_KEY}
with fs.open("s3://YOUR_S3_BUCKET/path/to/encrypted.csv", "rb", **sse_c) as f:
    data = f.read()

The lookups of the sources of cp_file, copy, and mv, and of the object of setxattr, send the source’s parameters of the copy: RequestPayer, ExpectedSourceBucketOwner as ExpectedBucketOwner, and the CopySourceSSECustomer* parameters as SSECustomer*. mv deletes the copied sources with RequestPayer and that expected bucket owner. The requests that copy and mv send to expand and pair their paths, such as the listings and the directory checks of the sources and of the destination, do not send these parameters of the call, but those of the filesystem described above. copy and mv with lists of both sources and destinations pair the paths as given, without these requests.

fs.cp_file(
    "s3://YOUR_S3_BUCKET/path/to/encrypted.csv",
    "s3://YOUR_S3_BUCKET/path/to/copy.csv",
    CopySourceSSECustomerAlgorithm="AES256",
    CopySourceSSECustomerKey=YOUR_32_BYTE_KEY,
    **sse_c,
)

A file opened for reading sends its reads on the condition of the ETag looked up when it was opened (IfMatch), so reading an object replaced since raises OSError; an IfMatch given to open is sent instead. A file that fsspec reopens, such as an unpickled file, keeps the request parameters of the file, including the SSECustomerKey of an SSE-C object, and this condition. metadata(), getxattr(), url(), and setxattr() of a file send its lookup parameters; setxattr() sends them also under the names of the copy source. In an append, the existing object is read when the file is opened, or copied as parts when its upload starts, on the condition of the ETag looked up when the file was opened. If the object was replaced before it is read or copied, the append raises OSError, and its multipart upload, if any, is aborted. The write of the appended object is not conditional, so it replaces an object written after the existing object was read or copied. The appended object keeps the attributes of the existing object that neither the call nor the s3_additional_kwargs of the filesystem give: its system-defined metadata, such as ContentType, its user-defined metadata, its storage class, and its encryption, unless they give any encryption parameter, such as ServerSideEncryption. It does not keep the object lock of the existing object. A key that is only a prefix of other keys has no object: reading it raises FileNotFoundError, and appending, exclusive creation ("xb"), and pipe/put with mode="create" write a new object.

Read caching

A file opened for reading fetches its data through an fsspec cache, chosen by the cache_type argument of open or by the default_cache_type argument of S3FileSystem, and "bytes" by default. The value is the name of a cache registered in fsspec.caching.caches, so the available names depend on the installed fsspec version. The common ones are:

  • "none": keeps nothing; each read requests the bytes it reads.

  • "bytes": reads one block beyond each read and keeps one contiguous range, which later reads extend.

  • "readahead": reads one block beyond each read and keeps only the last range read, which suits many small sequential reads.

  • "blockcache": requests whole blocks and keeps the most recently used ones, up to maxblocks (32 by default).

  • "background": like "blockcache", and also reads the next block in a thread while the current one is consumed.

The block size is the block_size argument of open or the filesystem’s default_block_size (5 MiB by default). Options of the cache, such as maxblocks, are given in the cache_options argument of open. See Read Buffering in the fsspec documentation for the other cache types.

fs = S3FileSystem(default_cache_type="readahead")
with fs.open(
    "s3://YOUR_S3_BUCKET/path/to/data.csv",
    "rb",
    block_size=32 * 2**20,
    cache_type="background",
    cache_options={"maxblocks": 2},
) as f:
    data = f.read()

The cache_type option of PandasCursor and PolarsCursor sets the default_cache_type of the filesystem that reads their results; see S3 read options and S3 read options. For the cache that S3FSCursor uses, see S3 read options.

Error translation

S3 error responses are translated into standard Python exceptions, so filesystem operations raise natural errors instead of botocore’s ClientError:

S3 error

Python exception

404 / NoSuchKey / NoSuchBucket

FileNotFoundError

405 of a version that is a delete marker (x-amz-delete-marker: true)

FileNotFoundError

403 / AccessDenied

PermissionError

BucketAlreadyExists / BucketAlreadyOwnedByYou

FileExistsError

PreconditionFailed of an If-None-Match condition

FileExistsError

RequestTimeout

TimeoutError

Others

OSError with the matching errno

S3ClientError.code_of() returns the S3 error code of a translated error, or None for an error that is not translated from an S3 error response:

from pyathena.filesystem.s3_errors import S3ClientError

try:
    fs.pipe_file("s3://YOUR_S3_BUCKET/path/to/object", b"data")
except OSError as e:
    if S3ClientError.code_of(e) == "SlowDown":
        ...

Request parameters that botocore rejects before sending the request (ParamValidationError), such as a misspelled parameter name, raise ValueError.

exists() returns True for a bucket that HeadBucket answers with 403. For an object, a 403 raises PermissionError: without the s3:ListBucket permission on the bucket, S3 also answers 403 for a key that does not exist.

Retries

Each request is sent with a pyathena.util.RetryConfig: the retry_config of the filesystem, the retry policy of the connection for s3_filesystem(), or RetryConfig(). If the exceptions of the policy are the default codes (THROTTLING_ERROR_CODES), the requests also retry the S3 error codes SlowDown, ServiceUnavailable, InternalError, and RequestTimeout (S3Core.RETRY_ERROR_CODES), with the attempts and waits of the policy. That includes a policy that gives the default codes explicitly. A policy with other exceptions is used as given:

from pyathena.util import RetryConfig

# Retries only SlowDown, in up to 3 attempts.
fs = S3FileSystem(retry_config=RetryConfig(exceptions=["SlowDown"], attempt=3))

botocore also retries these errors within each attempt, as the retry mode of the S3 client sets (retries in its botocore.config.Config, or the AWS_RETRY_MODE and AWS_MAX_ATTEMPTS settings). Unless the maximum attempts are set, botocore sends a request up to 3 times in standard mode, its default since botocore 1.44.0, and up to 5 times in legacy mode, the default of earlier versions. So with the default policy (8 attempts) and botocore’s default settings, a request whose error persists is sent up to 24 times before the operation raises the error, or up to 40 times with botocore earlier than 1.44.0. The waits of the policy, which add up to 127 seconds plus jitter, come on top of botocore’s own.

A request body given as a stream, such as S3Core.put_object(path, Body=file), is sent from the position where the call found it on each attempt of the policy. A stream whose position cannot be read with tell() and restored with seek() is sent without the retries of the policy.

Object metadata, tags, and ACLs

# User-defined metadata (x-amz-meta-*).
fs.setxattr("s3://YOUR_S3_BUCKET/path/to/object", attr1="value1")
metadata = fs.metadata("s3://YOUR_S3_BUCKET/path/to/object")
metadata["attr1"]        # User-defined metadata via the mapping interface.
metadata.content_type    # System-defined metadata as typed properties.
fs.getxattr("s3://YOUR_S3_BUCKET/path/to/object", "attr1")

# Object tagging.
fs.put_tags("s3://YOUR_S3_BUCKET/path/to/object", {"tag1": "value1"})
fs.put_tags("s3://YOUR_S3_BUCKET/path/to/object", {"tag2": "value2"}, mode="m")  # Merge.
fs.get_tags("s3://YOUR_S3_BUCKET/path/to/object")

# Canned ACLs.
fs.chmod("s3://YOUR_S3_BUCKET/path/to/object", "bucket-owner-full-control")
fs.chmod("s3://YOUR_S3_BUCKET/path/to/", "private", recursive=True)

A recursive chmod updates the object at the path, if any, and the objects below it in parallel. A path with a version ID updates that version only. Once an update fails, the updates that have not started are cancelled, the running ones make no further attempt, and the error is raised once they have ended.

Note that setxattr rewrites the object by copying it onto itself (S3 does not allow updating the metadata of an existing object in place), which updates its last-modified time. The copy keeps the system-defined metadata (such as ContentType and CacheControl), the storage class, and the server-side encryption algorithm and KMS key of the object; parameters given in copy_kwargs take precedence over them, and any encryption parameter replaces all of the kept encryption settings. HeadObject does not return the KMS encryption context, so pass it in copy_kwargs together with the other encryption parameters. An Expires value that botocore cannot parse as a date is not kept. A path with a ?versionId= suffix raises ValueError, since the metadata of an existing version cannot be changed.

Multipart upload management

Incomplete multipart uploads continue to accrue storage costs until they are completed or aborted. The filesystem can discover and abort them:

uploads = fs.list_multipart_uploads("s3://YOUR_S3_BUCKET")
for upload in uploads:
    print(upload.key, upload.upload_id, upload.initiated)

# Abort all incomplete uploads to a key and the keys under it.
fs.clear_multipart_uploads("s3://YOUR_S3_BUCKET/path/to/")

Versioning

With version_aware=True, reads pin the object version observed at open time, so a file handle keeps returning consistent data even if the object is overwritten while reading. The file path carries the pinned version as a ?versionId= suffix, which metadata(), getxattr() and url() of the file also use. Explicit versions can always be read with the ?versionId= suffix or the version_id argument. Only a ?versionId= (or ?versionID=, ?versionid=, ?version_id=) query at the end of a path is a version; any other ? is part of the key. A path with a version is not a glob pattern: copy(), mv() and get() copy that version to a destination named after its key. ls() and find() of a key path with a version return only that version, named as ls(versions=True) names it, without listing the keys under the path. They look the version up as info() does: if HeadObject reports it missing, ls() raises FileNotFoundError and find() returns nothing, and other errors, such as the 400 Bad Request of a version ID that the bucket rejects, are raised.

fs = S3FileSystem(
    connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2"),
    version_aware=True,
)

with fs.open("s3://YOUR_S3_BUCKET/path/to/object", "rb") as f:
    data = f.read()  # Pinned to the version observed at open time.

# List all versions of the objects under a prefix. Each version is named
# "bucket/key?versionId=<id>", except the "null" version, which is named by its key.
fs.ls("s3://YOUR_S3_BUCKET/path/to/", versions=True)

# Typed version information, including delete markers if requested.
versions = fs.object_version_info("s3://YOUR_S3_BUCKET/path/to/object")
for version in versions:
    print(version.version_id, version.is_latest, version.last_modified)

Version-aware operations require the s3:GetObjectVersion and s3:ListBucketVersions permissions.

Moving a ?versionId=null source onto its own key copies that version and deletes the source version when bucket versioning is enabled. If versioning has never been enabled or is suspended, the move leaves the source in place: copying onto the key would replace the null version that the subsequent deletion removes. The same distinction applies to move conflict checks.

A move compares a null version with its unversioned key using GetBucketVersioning, which requires s3:GetBucketVersioning. Each relevant bucket is looked up once during planning; other moves do not make this request. The result is not cached between moves. A failed lookup stops the move before any copy or deletion. Directory buckets do not support versioning, so their null versions are treated as the key itself without a versioning lookup.

The opt-in S3 versioning integration tests create dedicated temporary buckets and check unversioned, enabled, and suspended states. See Testing for the command and required test permissions.

Bucket lifecycle

Bucket creation and deletion are infrastructure-level changes and are disabled by default: mkdir/makedirs and rmdir raise PermissionError when they would create or delete a bucket. Pass the opt-in flags to enable them:

fs = S3FileSystem(
    connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2"),
    allow_bucket_creation=True,
    allow_bucket_deletion=True,
)
fs.mkdir("s3://YOUR_NEW_BUCKET")
fs.rmdir("s3://YOUR_NEW_BUCKET")  # The bucket must be empty.

Creating a key prefix under an existing bucket requires no operation (S3 has no real directories below the bucket level) and is always a no-op.

Typed S3 operations

Parse an S3 path

Use S3Path.parse() to obtain the bucket, key and version ID of a path:

from pyathena.filesystem.s3_path import S3Path

path = S3Path.parse("s3://bucket/key?versionId=v1")
print(path.bucket, path.key, path.version_id)

In 4.0, S3FileSystem.PATTERN_PATH and the parse_path() methods of S3FileSystem and AioS3FileSystem are removed. S3Path.parse() returns an S3Path object; access its attributes instead of unpacking the tuple returned by parse_path(). For direct access to the regular expression, use S3Path.PATTERN instead of S3FileSystem.PATTERN_PATH.

Send S3 requests

S3FileSystem.core is an S3Core, the typed operations that the filesystem sends its listing, lookup, read, write, delete, multipart upload, copy, tagging, ACL and metadata replacement requests with. It can also be built on a boto3 S3 client. Each operation sends one request (one per page for the iterators and list_object_annotations()); plan_multipart_copy() and copy_object_annotation(), described below, send several, and generate_presigned_url() signs a URL locally without a request. The requests are sent with the retry policy (see “Retries” above). An operation raises FileNotFoundError for a missing bucket or multipart upload, or for a missing object or version that it reads, and caches nothing. copy_object() and upload_part_copy() with a source version that is a delete marker raise OSError with EINVAL, while the filesystem’s cp_file() raises FileNotFoundError for it, as for a missing file. Requests sent through fs.core do not invalidate the filesystem’s cache: call fs.invalidate_cache() after a change, or make it through the filesystem.

import boto3

from pyathena.filesystem.s3_core import S3Core
from pyathena.filesystem.s3_path import S3Path

core = fs.core  # or S3Core(boto3.client("s3"))

metadata = core.head_object(S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object"))
print(metadata.content_length, metadata.version_id)

for page in core.list_objects("YOUR_S3_BUCKET", prefix="path/to/", delimiter="/"):
    print([o.key for o in page.objects], [p.prefix for p in page.common_prefixes])

list_object_versions(), list_multipart_uploads() and list_buckets() return pages in the same way. The core sends the prefix of list_multipart_uploads() as given, and S3 matches it as a plain string prefix, so it also lists the uploads to sibling keys that start with it, which fs.list_multipart_uploads() excludes.

get_object() reads an object, a version of it, or a byte range and returns the bytes. A range has an exclusive end; None as the end reads to the end of the object, and a negative start without an end reads the last bytes. The response body is closed after the read; a failure to read it raises the botocore exception and is not retried. put_object() writes bytes and returns an S3PutObject.

data = core.get_object(S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object"), (0, 1024))
result = core.put_object(S3Path("YOUR_S3_BUCKET", "path/to/copy"), data)
print(result.etag, result.version_id)

delete_objects() deletes an S3DeleteBatch, the objects of one bucket that one DeleteObjects request accepts. S3DeleteBatch.from_paths() groups paths into batches of up to 1,000 objects per bucket. The objects that S3 could not delete are in the errors of the returned S3DeleteResult, not raised; as in S3, deleting a key that does not exist is not an error.

from pyathena.filesystem.s3_core import S3DeleteBatch

paths = [
    S3Path.parse("s3://YOUR_S3_BUCKET/path/to/a"),
    S3Path.parse("s3://YOUR_S3_BUCKET/path/to/b"),
]
for batch in S3DeleteBatch.from_paths(paths):
    result = core.delete_objects(batch)
    for error in result.errors:
        print(error)  # path (code: message)

get_object_tagging() returns the tags of an object as a dictionary, and put_object_tagging() replaces all of its tags with the given ones. Both act on the version ID of the path, if any, including null. put_object_acl() and put_bucket_acl() apply a canned ACL, which must be in S3Core.OBJECT_ACLS or S3Core.BUCKET_ACLS, respectively; another value raises ValueError, and an object ACL also applies to the version ID of the path.

replace_object_metadata() replaces the user-defined metadata of an object, given the head_object() result of the same path, by copying the object onto itself. The copy rewrites the object, or creates a new version in a versioned bucket, so the path must not have a version ID. The copy retains the content headers (CacheControl, ContentDisposition, ContentEncoding, ContentLanguage, ContentType), Expires, WebsiteRedirectLocation, StorageClass, and the ServerSideEncryption, SSEKMSKeyId and BucketKeyEnabled of the object. Additional CopyObject parameters take precedence over the retained fields, and any encryption parameter replaces all of the retained encryption fields.

path = S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object")
core.put_object_tagging(path, {**core.get_object_tagging(path), "tag2": "value2"})

head = core.head_object(path)
core.replace_object_metadata(path, head, {**head, "attr1": "value1"})

generate_presigned_url() signs a URL locally, for get_object unless another client method is given, and sends no request. Its parameters take precedence over the Bucket, Key and VersionId of the path. The request_kwargs of the core, such as RequestPayer, are not included in the signed parameters; pass them as parameters of generate_presigned_url() to sign them.

url = core.generate_presigned_url(path, expires_in=600)
upload_url = core.generate_presigned_url(path, "put_object", ContentType="text/csv")

create_bucket() and delete_bucket() create and delete a bucket when called. The allow_bucket_creation and allow_bucket_deletion options apply only to the filesystem’s mkdir/makedirs and rmdir, not to calls through the core. get_bucket_versioning() returns the versioning state of a bucket, Enabled or Suspended, or None if versioning has never been enabled on it. S3Path.is_directory_bucket tells whether the bucket of a path is a directory bucket.

create_bucket() sends a LocationConstraint for the region_name argument, or for the client’s region by default, except in us-east-1. The fields of a CreateBucketConfiguration passed as a parameter are kept, and the LocationConstraint is added to it only if it has none of LocationConstraint, Location and Bucket; an empty or None configuration is treated as not passed. To create a directory bucket, pass its Location and Bucket in CreateBucketConfiguration; no LocationConstraint is added.

Multipart writer

S3MultipartWriter provides synchronous multipart requests and part planning without fsspec. It is the writer used by S3File; applications can also construct it on an S3Core. The constructor validates the destination and block size without sending a request. The caller owns the buffer and executor, collects part results in number order, and waits for running requests before completing or aborting. Serialize initiation and finalization; part requests can run in parallel after initiation.

from io import BytesIO

from pyathena.filesystem.s3_writer import S3MultipartWriter

writer = S3MultipartWriter(
    core,
    S3Path("YOUR_S3_BUCKET", "path/to/object"),
    block_size=5 * 2**20,
    request_kwargs={"ChecksumAlgorithm": "SHA256"},
)
writer.initiate()
try:
    parts = [
        writer.upload_part(number, body)
        for number, body in writer.iter_parts(BytesIO(b"your data"))
    ]
    result = writer.complete(parts)
except BaseException:
    writer.abort()
    raise

iter_parts() consumes a blocking binary stream from its current position and produces numbered part bodies without sending requests. It merges a short tail into the preceding block and splits the result if it reaches the maximum part size. A stream shorter than the minimum part size must be the final part. iter_copy_parts(size) produces numbered source ranges for copying an existing object before appending data. A source smaller than the minimum part size must be the final part; S3File reads a small existing object into its buffer when appending. Both planners accept first_part_number to include previously copied or uploaded parts in the 10,000-part limit. upload_part_copy() sends a planned range with an exclusive end; None sends no range of its own, as with the core primitive.

The writer filters inherited and per-request parameters by operation; per-request parameters take precedence. It retains the upload identity after completion and after an abort failure, and clears it only after a successful abort. The caller can therefore retry a failed abort. The writer does not invalidate the filesystem cache; call fs.invalidate_cache() after a direct write.

When an S3File wait for multipart creation is interrupted, it waits for any creation that has already started and recovers the successful response before aborting. It drops the buffer and closes the file before waiting, so a later close cannot upload the interrupted write, and a deferred commit raises S3MultipartStateError. It suppresses Ctrl-C during the blocking recovery wait and re-raises the original error when cleanup finishes. An additional interruption outside that wait can stop cleanup; once the upload identity is known, discard() can retry the abort. An abort failure retains the upload identity for discard() to retry. A failed multipart commit() also closes the file without its buffered data and aborts the upload; a later commit() raises S3MultipartStateError. Cancelling an async buffered write through asyncio.to_thread still allows its writer thread to finish.

Multipart requests

create_multipart_upload(), upload_part(), upload_part_copy(), complete_multipart_upload() and abort_multipart_upload() send the requests of a multipart upload. part_ranges() sends no request: it splits an object into the byte ranges of the parts that copy it, by the part limits MULTIPART_UPLOAD_MIN_PART_SIZE (5 MiB), MULTIPART_UPLOAD_MAX_PART_SIZE (5 GiB) and MULTIPART_UPLOAD_MAX_PARTS (10,000) of S3Core. check_multipart_upload_size() sends no request either: it raises ValueError with the minimum block size if data of the given size takes more than MULTIPART_UPLOAD_MAX_PARTS parts of the block size, the check that put and pipe make before uploading.

copy_object() copies an object with one CopyObject request, which accepts objects up to MULTIPART_UPLOAD_MAX_PART_SIZE. For a larger object, plan_multipart_copy() reads the source and returns an S3MultipartCopyPlan: the version to copy, the byte ranges of the parts, the parameters of each multipart upload request, and the annotations to copy, so that the multipart upload writes the metadata, tags and annotations that CopyObject would. It sends HeadObject, then GetObjectTagging and ListObjectAnnotations unless the directives or the source exclude them, and writes nothing. If HeadObject reports a size that fits in one CopyObject request, nothing else is read, and the plan’s fits_single_request says to copy with copy_object() instead. copy_object_annotation() copies one annotation onto the destination after the upload completes, with GetObjectAnnotation and PutObjectAnnotation. As with get_object(), the payload is closed after the read, and a failure to read it raises the botocore exception and is not retried. The filesystems’ cp_file(), copy() and mv() run these plans.

The multipart primitives take the S3MultipartUpload returned by creation, keeping its bucket, key, upload ID and checksum configuration together. The core retains no upload state. upload_part() passes the upload’s checksum algorithm to the SDK; completion selects the matching part checksum and sends the upload’s ChecksumType when present, including FULL_OBJECT. Without a creation algorithm, completion sends the ETag and part number without checksums that the SDK may add to part uploads. abort_multipart_upload() also accepts uploads returned by list_multipart_uploads().

upload = core.create_multipart_upload(
    S3Path("YOUR_S3_BUCKET", "path/to/object"), ChecksumAlgorithm="SHA256"
)
try:
    part = core.upload_part(upload, 1, b"data")
    result = core.complete_multipart_upload(upload, [part])
except Exception:
    core.abort_multipart_upload(upload)
    raise

Path pairing

S3PathPairing pairs the paths of one copy(), get() or mv(). Its delete_paths() splits the paths of an rm() into those deleted as given and those expanded. The pairing is fsspec’s, except that a path with a version ID names that version, and its destination is named after its key.

A source may also be an S3Path, as the filesystems pass the keys that their listings return. It names its bucket and key as they are, also a key that ends in what looks like a version ID query, such as a/x?versionId=null, and its destination is named after that key. s3_copy_pairs() pairs the sources as copy_pairs() does, and returns the S3 destination of an S3Path source as an S3Path. conflict_candidates() returns the candidates of S3Path sources as S3Path objects, and missing accepts them.

A pairing is a frozen dataclass of path1, path2, recursive and maxdepth, and holds no filesystem. The filesystems read from it what to look up (expands, skips_directories, looks_up_destination and conflict_candidates()), expand the sources, leave out the directories when skips_directories is true, look up the paths, and pass the results to copy_pairs() and move_pairs(). A sources, destination_is_dir or missing that a rule needs and that is not passed raises ValueError.

For moves that compare a null version with its unversioned key, the caller also looks up bucket versioning and passes the names of the versioning-enabled buckets as versioning_enabled_buckets to both conflict_candidates() and move_pairs(). Pass a collection of bucket names, such as a set, rather than a single string. The default empty collection treats null versions as their keys, as in unversioned or suspended buckets. The model makes no AWS requests.

from pyathena.filesystem.s3_path_pairing import S3PathPairing

pairing = S3PathPairing("s3://YOUR_S3_BUCKET/src/", "s3://YOUR_S3_BUCKET/dst/", recursive=True)
sources = fs.expand_path(pairing.path1, recursive=pairing.recursive)
for source, destination in pairing.copy_pairs(sources):
    print(source, "->", destination)

Async filesystem

AioS3FileSystem provides the same functionality on top of fsspec’s AsyncFileSystem, running its blocking calls in a thread pool sized by max_workers, and its parallel requests in the request pool of its internal S3FileSystem. Its open_async() returns a file whose reads and writes are coroutines. See AioS3FileSystem for details.