File System Integration

This section covers S3 filesystem integration and object management.

S3 FileSystem

class pyathena.filesystem.s3.S3FileSystem(*args, **kwargs)

A filesystem interface for Amazon S3 that implements the fsspec protocol.

This class provides a file-system like interface to Amazon S3, allowing you to use familiar file operations (ls, open, cp, rm, etc.) with S3 objects. It’s designed to be compatible with s3fs while offering PyAthena-specific optimizations.

The filesystem supports standard S3 operations including:

• Listing objects and directories

• Reading and writing files

• Copying and moving objects

• Reading and writing object metadata, tags, and canned ACLs

• Multipart uploads for large files, including management of incomplete uploads

• Version-aware reads and object version listing (see version_aware)

• Creating and removing buckets (disabled by default; see allow_bucket_creation / allow_bucket_deletion)

• Various S3 storage classes and encryption options

• Translating S3 error responses into standard Python exceptions (e.g., 404 -> FileNotFoundError, 403 -> PermissionError)

allow_bucket_creation

Whether mkdir/makedirs may create buckets. Defaults to False.

allow_bucket_deletion

Whether rmdir may delete buckets. Defaults to False.

version_aware

Whether reads pin the object version observed at open time and ls may list all versions. Requires the s3:GetObjectVersion / s3:ListBucketVersions permissions. Defaults to False.

Example

>>> from pyathena.filesystem.s3 import S3FileSystem
>>> fs = S3FileSystem()
>>>
>>> # List objects in a bucket
>>> files = fs.ls('s3://my-bucket/data/')
>>>
>>> # Read a file
>>> with fs.open('s3://my-bucket/data/file.csv', 'r') as f:
...     content = f.read()
>>>
>>> # Write a file
>>> with fs.open('s3://my-bucket/output/result.txt', 'w') as f:
...     f.write('Hello, S3!')
>>>
>>> # Copy files
>>> fs.cp('s3://source-bucket/file.txt', 's3://dest-bucket/file.txt')

Note

This filesystem is used internally by PyAthena for handling query results stored in S3, but can also be used independently for S3 file operations.

DEFAULT_BLOCK_SIZE: int = 5242880

protocol: str | tuple[str, ...] = ('s3', 's3a')

__init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int = 20, s3_additional_kwargs=None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, *args, **kwargs) → None

Create a filesystem for Amazon S3.

Parameters:

• connection – A PyAthena connection whose S3 client (Connection.s3_client) and retry policy the filesystem uses. Without one, the client is built from s3fs-compatible arguments in kwargs.

• default_block_size – The block size for reads and writes; defaults to DEFAULT_BLOCK_SIZE.

• default_cache_type – The fsspec cache type for reads, the name of a cache in fsspec.caching.caches; defaults to "bytes". The available names depend on the installed fsspec version. Common ones are "none", which keeps nothing and requests the bytes of each read; "bytes", which reads one block beyond each read and keeps one contiguous range; "readahead", which reads one block beyond each read and keeps only the last range read; "blockcache", which requests whole blocks and keeps the most recently used ones; and "background", which is like "blockcache" and also reads the next block in a thread.

• max_workers – The number of threads for parallel transfers.

• s3_additional_kwargs – Extra arguments for the object requests of open() and pipe_file(); listings and other requests do not use them. Each request receives those that its operation accepts, and the parameters of a call take precedence.

• allow_bucket_creation – Whether mkdir/makedirs may create a bucket.

• allow_bucket_deletion – Whether rmdir may delete a bucket.

• version_aware – Whether reads pin the object version observed at open time.

• *args – Passed to fsspec.AbstractFileSystem.

• **kwargs – Passed to fsspec.AbstractFileSystem; without a connection, also s3fs-compatible client arguments. requester_pays=True sends requester-pays requests with the operations that accept RequestPayer.

property core: S3Core

The typed S3 operations that the filesystem sends its requests with.

property request_kwargs: dict[str, Any]

The parameters sent with every operation that accepts them.

{"RequestPayer": "requester"} with requester_pays=True; see core.request_kwargs.

ls(path: str, detail: bool = False, refresh: bool = False, **kwargs) → list[S3Object] | list[str]

List contents of an S3 path.

Lists buckets (when path is root) or objects within a bucket/prefix. Compatible with fsspec interface for filesystem operations.

Parameters:

• path – S3 path to list (e.g., “s3://bucket” or “s3://bucket/prefix”).

• detail – If True, return S3Object instances; if False, return paths as strings.

• refresh – If True, bypass cache and fetch fresh results from S3.

• **kwargs –

  Additional arguments including: versions: If True, list all versions of the objects, named

  bucket/key?versionId=<id> except for the null version. Requires the filesystem to be constructed with version_aware=True.

Returns:

List of S3Object instances (if detail=True) or paths as strings (if detail=False).

Example

>>> fs = S3FileSystem()
>>> fs.ls("s3://my-bucket")  # List objects in bucket
>>> fs.ls("s3://my-bucket/", detail=True)  # Get detailed object info

info(path: str, **kwargs) → S3Object

Return information about an S3 path.

Uses the directory cache first: a cached entry for the path is returned, the entry of the path in a cached listing of its parent is returned, preferring an object to a key prefix of the same name as HeadObject does, and a cached listing of its parent without it means it does not exist. The cached bucket listing holds only the buckets that the caller owns, so a bucket missing from it is looked up with HeadBucket. Otherwise, a key path is looked up with HeadObject and, if no object exists, with a ListObjectsV2 request (Delimiter="/", MaxKeys=1) that checks whether it is a key prefix; a bucket path is looked up with HeadBucket. If these requests find a listed object missing, or find a key prefix, the cached listing of the parent is removed. With version_aware, a cached file entry without a version ID is looked up again. With an explicit version, the cached entries of the path are skipped, the HeadObject result is cached under the version-qualified path apart from other versions, except for the null version, which an overwrite replaces, and a missing version is not looked up as a key prefix, since a version names an object. With request parameters on which the authorization of the requests depends (ExpectedBucketOwner, RequestPayer, and the SSECustomer* parameters of an object encrypted with a customer-provided key), each request receives those that its operation accepts, and only the cached HeadObject or HeadBucket results of lookups with the same values of these parameters are used.

Parameters:

• path – S3 path (e.g., “s3://bucket” or “s3://bucket/key”).

• **kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. version_id: The version ID to look up when the path has none. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters described above. Other request parameters are ignored.

Returns:

S3Object describing the bucket, directory, or file. The root path ("", "/" or "s3://") is a directory.

Raises:

FileNotFoundError – If the path does not exist.

find(path: str, maxdepth: int | None = None, withdirs: bool | None = None, detail: bool = False, **kwargs) → dict[str, S3Object] | list[str]

Find all files below a given S3 path.

Recursively searches for files under the specified path, with optional depth limiting and directory inclusion. Uses efficient S3 list operations with delimiter handling for performance. As in fsspec, the result includes the path itself if withdirs is True and objects exist below it, unless it is a bucket, or if it is an object and nothing is listed below it.

Parameters:

• path – S3 path to search under (e.g., “s3://bucket/prefix”).

• maxdepth – Maximum number of levels to descend, at least 1 (None for unlimited). With 1, only the entries directly under the path are listed.

• withdirs – Whether to include directories in results (None = default behavior).

• detail – If True, return dict of {path: S3Object}; if False, return list of paths.

• **kwargs –

  Additional arguments including: prefix: Key prefix, relative to the path, to filter the listed keys

  by. Each slash in the prefix counts as one level of maxdepth. With withdirs, the directories above the prefix, such as sub for sub/deep/, are not included.

  refresh: If True, bypass the cache and list from S3.

Returns:

Dictionary mapping paths to S3Objects (if detail=True) or list of paths (if detail=False).

Raises:

ValueError – If maxdepth is less than 1 or the path is the root.

Example

>>> fs = S3FileSystem()
>>> fs.find("s3://bucket/data/", maxdepth=2)  # Limit depth
>>> fs.find("s3://bucket/", withdirs=True)    # Include directories

expand_path(path, recursive=False, maxdepth=None, **kwargs) → list[str]

Expand glob patterns and directories into the paths they match.

As in fsspec, except that a path with a version ID names that version of an object: it is not a glob pattern, although its ? is one in fsspec, nor is anything expanded below it. With recursive, it is included only if the version exists, as fsspec includes a path that exists.

Parameters:

• path – S3 path, glob pattern, or list of them.

• recursive – Whether to include the paths below the directories.

• maxdepth – Maximum depth of the expansion, at least 1.

• **kwargs – Additional arguments passed to glob and find.

Returns:

The sorted matching paths.

Raises:

• ValueError – If maxdepth is less than 1.

• FileNotFoundError – If nothing matches.

exists(path: str, **kwargs) → bool

Check if an S3 path exists.

Determines whether a bucket, object, or prefix exists in S3. Uses caching and efficient head operations to minimize API calls.

Parameters:

• path – S3 path to check (e.g., “s3://bucket” or “s3://bucket/key”).

• **kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters on which the authorization of the requests depends, as described in info(). With them, cached listings are not used. Other request parameters are ignored.

Returns:

True if the path exists, False otherwise. A bucket that HeadBucket denies access to (403) exists.

Example

>>> fs = S3FileSystem()
>>> fs.exists("s3://my-bucket/file.txt")
>>> fs.exists("s3://my-bucket/")

rm_file(path: str, **kwargs) → None

Delete an S3 object with DeleteObject.

Does nothing for a bucket path. If the path has a version ID, that version is deleted.

Parameters:

• path – S3 path (s3://bucket/key) of the object to delete.

• **kwargs – Accepted for fsspec compatibility; not used in the request.

Raises:

TypeError – If kwargs has bucket, key or version_id, which the path gives.

rm(path, recursive=False, maxdepth=None, **kwargs) → None

Delete objects with DeleteObjects requests.

Expands the paths with expand_path and deletes the matched objects in parallel requests of up to S3DeleteBatch.MAX_KEYS keys each, one set of requests per bucket. A path with a version ID deletes that version without expansion.

Parameters:

• path – S3 path (s3://bucket/key) or list of paths to delete.

• recursive – Whether to delete all objects below the paths.

• maxdepth – Maximum depth to expand when recursive is True.

• **kwargs – Additional parameters passed to the DeleteObjects API. Quiet (default True) sets the quiet mode of the requests.

Raises:

• ValueError – If a path is a bucket.

• OSError – If S3 could not delete some of the objects.

mkdir(path: str, create_parents: bool = True, **kwargs) → None

Create an S3 bucket.

S3 has no real directories below the bucket level; creating a key prefix requires no operation. This method creates the bucket when the path points at a bucket (or when create_parents is True and the bucket does not exist yet), and does nothing for key prefixes under an existing bucket.

Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass allow_bucket_creation=True to the filesystem constructor to enable bucket creation.

Parameters:

• path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).

• create_parents – If True, create the bucket when it does not exist, even if the path contains a key prefix.

• **kwargs –

  Additional arguments including: acl: Canned ACL to apply to the bucket. region_name: Region to create the bucket in. Defaults to the

  client’s region.

Raises:

• FileExistsError – If the path is a bucket that already exists.

• FileNotFoundError – If the bucket does not exist and create_parents is False.

• PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.

• ValueError – If the ACL is invalid or the path is empty.

makedirs(path: str, exist_ok: bool = False) → None

Recursively create a directory, creating the bucket if necessary.

Creating the bucket requires allow_bucket_creation=True on the filesystem constructor; see mkdir().

Parameters:

• path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).

• exist_ok – If False, raise FileExistsError when the path is a bucket that already exists.

Raises:

• FileExistsError – If the path is a bucket that already exists and exist_ok is False.

• PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.

rmdir(path: str) → None

Remove an S3 bucket, which must be empty.

S3 has no real directories below the bucket level, so only bucket paths can be removed.

Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass allow_bucket_deletion=True to the filesystem constructor to enable bucket deletion.

Parameters:

path – S3 bucket path (e.g., “s3://bucket”).

Raises:

• FileExistsError – If the path contains a key that exists. The user may have meant rm(path, recursive=True).

• FileNotFoundError – If the path contains a key that does not exist, or the bucket does not exist.

• PermissionError – If bucket deletion is not enabled on this filesystem instance.

• OSError – If the bucket is not empty.

touch(path: str, truncate: bool = True, **kwargs) → dict[str, Any]

Create an empty object with PutObject.

Parameters:

• path – S3 path (s3://bucket/key) of the object.

• truncate – If True, replace an existing object with an empty one; if False, raise if the object exists.

• **kwargs – Additional parameters passed to the PutObject API.

Returns:

The PutObject response as a dictionary (see S3PutObject.to_dict()).

Raises:

ValueError – If the path has a version ID, is a bucket, or exists while truncate is False.

mv(path1, path2, recursive=False, maxdepth=None, **kwargs) → None

Move files from one S3 location to another.

Copies the files to the destinations that copy() gives them, and then deletes only the source objects that were copied. fsspec’s AbstractFileSystem.mv() instead removes path1 by expanding it again, which also deletes copies placed where path1 matches them and files that maxdepth kept from being copied. A file whose destination is the file itself is left in place, and directories, which S3 does not store as objects, are not copied. A null version moved onto its key is copied and deleted only when bucket versioning is enabled. Without versioning, or with versioning suspended, that move is left in place because the copy would replace the version that the deletion removes.

Comparing a null version with its unversioned key calls GetBucketVersioning once per bucket during the move’s planning and requires s3:GetBucketVersioning. The result is not cached across moves. Directory buckets do not support versioning and need no lookup.

Parameters:

• path1 – Source S3 path, glob pattern, or list of paths.

• path2 – Destination S3 path, or list of paths when path1 is a list.

• recursive – Whether to move the directories with their contents.

• maxdepth – Maximum depth of a recursive move.

• **kwargs – Additional S3 copy parameters passed to cp_file().

Raises:

• ValueError – If two sources have the same destination, or a destination is another source, including one left in place, which is checked before anything is copied. A directory with no object at its key, which is not copied, does not conflict.

• OSError – If bucket versioning cannot be read.

copy(path1, path2, recursive=False, maxdepth=None, on_error=None, **kwargs) → None

Copy files within S3.

As fsspec’s copy(), except that a source with a version ID copies that version of the object to a destination named after its key, as copy_pairs() pairs them.

Parameters:

• path1 – Source S3 path, glob pattern, or list of them.

• path2 – Destination S3 path, or list of paths when path1 is a list.

• recursive – Whether to copy the directories with their contents.

• maxdepth – Maximum depth of a recursive copy.

• on_error – "raise" or "ignore" for a missing source; by default "ignore" with recursive and "raise" otherwise.

• **kwargs – Additional S3 copy parameters passed to cp_file().

get(rpath, lpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, **kwargs) → None

Copy files from S3 to the local filesystem.

As fsspec’s get(), except that a source with a version ID downloads that version of the object to a local path named after its key, as copy_pairs() pairs them. Those destinations are checked to lie under lpath.

Parameters:

• rpath – Source S3 path, glob pattern, or list of them.

• lpath – Local destination path, or list of paths when rpath is a list.

• recursive – Whether to copy the directories with their contents.

• callback – Progress callback.

• maxdepth – Maximum depth of a recursive copy.

• **kwargs – Additional parameters passed to get_file().

Raises:

ValueError – If a source with a version ID is paired, and a destination lies outside lpath.

cp_file(path1: str, path2: str, recursive=False, maxdepth=None, on_error=None, **kwargs)

Copy an S3 object to another S3 location.

Performs server-side copy of S3 objects, which is more efficient than downloading and re-uploading. Automatically chooses between simple copy and multipart copy based on object size.

Parameters:

• path1 – Source S3 path (s3://bucket/key).

• path2 – Destination S3 path (s3://bucket/key).

• recursive – Unused parameter for fsspec compatibility.

• maxdepth – Unused parameter for fsspec compatibility.

• on_error – Unused parameter for fsspec compatibility.

• **kwargs – Additional S3 copy parameters (e.g., metadata, storage class). The block_size and max_workers parameters control a multipart copy and are not sent to S3.

Raises:

ValueError – If trying to copy to a versioned file or copy buckets, or if a directive of a multipart copy has an invalid value.

Note

Uses multipart copy for objects larger than the maximum part size to optimize performance for large files. The copy operation is performed entirely on the S3 service without data transfer. The multipart copy applies the metadata, tagging and annotation directives as CopyObject does (default COPY). A directory path1, which recursive copy() passes along with the files under it, is skipped.

pipe_file(path: str, value: bytes | bytearray | memoryview, mode: str = 'overwrite', **kwargs) → None

Write bytes into the path.

Writes data up to the block size with a single PutObject request instead of the inherited open() + write() path. Larger data and writes inside an fsspec transaction go through the buffered path, which uploads the data as a parallel multipart upload and keeps the deferred-commit semantics of transactions. A write that fails on that path leaves the existing object unchanged. Both paths write to the path without a trailing slash, as open() does.

Parameters:

• path – S3 path (s3://bucket/key) to write to.

• value – The bytes to write.

• mode – “overwrite” (default) or “create”. With “create”, raise FileExistsError when the object already exists, including one created during the write, which is not replaced.

• **kwargs – Additional parameters passed to the PutObject API (e.g., ContentType, StorageClass) on the single-request path. The block_size, max_workers, and s3_additional_kwargs parameters of the open() path are also accepted, and so is compression: the codec of open() to compress the value with before it is uploaded, or "infer" to take it from the extension of the path.

Raises:

• FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the write is committed.

• ValueError – If the path does not contain a key or specifies a version, if the compression is not supported, or if the data takes more than S3Core.MULTIPART_UPLOAD_MAX_PARTS blocks.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs) → bytes

Read the contents of an S3 object with GetObject.

start and end select bytes like a slice of the object: an empty range, or one that starts at or past the end of the object, returns b"", and an end past the object reads up to its end. Non-negative offsets are sent to S3 as they are, and so is a negative start without an end, as a suffix range of the last bytes. Other negative offsets are resolved against the size from info(), which also checks that the object exists for an empty range and receives the lookup parameters among kwargs.

Parameters:

• path – S3 path (s3://bucket/key) of the object.

• start – Byte offset to start reading at. A negative value counts from the end of the object.

• end – Byte offset to stop reading at (exclusive). A negative value counts from the end of the object.

• **kwargs – Additional parameters passed to the GetObject API, except version_id: the version ID to read when the path has none.

Returns:

The bytes read from the object.

Raises:

FileNotFoundError – If the path has no key or the key does not exist.

put_file(lpath: str, rpath: str, callback=<fsspec.callbacks.NoOpCallback object>, mode: str = 'overwrite', **kwargs)

Upload a local file to S3.

Uploads a file from the local filesystem to an S3 location. Supports automatic content type detection based on file extension and provides progress callback functionality. An upload that fails before it is completed, including one of a local file that cannot be read, leaves the existing object unchanged.

Parameters:

• lpath – Local file path to upload.

• rpath – S3 destination path (s3://bucket/key).

• callback – Progress callback for tracking upload progress.

• mode – “overwrite” (default) or “create”. With “create”, the file is written as with open() in xb mode: raise FileExistsError when the object already exists, including one created during the upload, which is not replaced.

• **kwargs – Additional S3 parameters (e.g., ContentType, StorageClass). The block_size, max_workers, and s3_additional_kwargs parameters of open() are also accepted.

Raises:

• FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the upload is committed.

• ValueError – If the file takes more than S3Core.MULTIPART_UPLOAD_MAX_PARTS blocks.

Note

Directories are not supported for upload. If lpath is a directory, the method returns without performing any operation. Bucket-only destinations (without key) are also not supported.

get_file(rpath: str, lpath=None, callback=<fsspec.callbacks.NoOpCallback object>, outfile=None, **kwargs)

Download an S3 file to local filesystem.

Downloads a file from S3 to the local filesystem with progress tracking. Reads the file in chunks to handle large files efficiently.

As with fsspec’s AbstractFileSystem.get_file(), a directory rpath creates the local directory lpath, and the parent directories of a local file lpath are created as needed.

Parameters:

• rpath – S3 source path (s3://bucket/key).

• lpath – Local destination path, or a file-like object to write to. Not needed when outfile is given.

• callback – Progress callback for tracking download progress.

• outfile – A file-like object to write to instead of lpath.

• **kwargs – Additional S3 parameters passed to open().

checksum(path: str, **kwargs)

Get checksum for S3 object or directory.

Computes a checksum for the specified S3 path. For individual objects, returns the ETag converted to an integer. For directories, returns a checksum based on the directory’s tokenized representation.

Parameters:

• path – S3 path (s3://bucket/key) to get checksum for.

• **kwargs – Additional arguments including: refresh: If True, refresh cached info before computing checksum.

Returns:

Integer checksum value derived from S3 ETag or directory token.

Note

For multipart uploads, ETag format is different and only the first part before the dash is used for checksum calculation.

sign(path: str, expiration: int = 3600, **kwargs)

Generate a presigned URL for S3 object access.

Creates a presigned URL that allows temporary access to an S3 object without requiring AWS credentials. Useful for sharing files or providing time-limited access to resources.

Parameters:

• path – S3 path (s3://bucket/key) to generate URL for.

• expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).

• **kwargs –

  Additional parameters including: client_method: S3 operation (‘get_object’, ‘put_object’, etc.).

  Defaults to ‘get_object’.

  Additional parameters passed to the S3 operation.

Returns:

Presigned URL string that provides temporary access to the S3 object.

Example

>>> fs = S3FileSystem()
>>> url = fs.sign("s3://my-bucket/file.txt", expiration=7200)
>>> # URL valid for 2 hours
>>>
>>> # Generate upload URL
>>> upload_url = fs.sign(
...     "s3://my-bucket/upload.txt",
...     client_method="put_object"
... )

metadata(path: str, **kwargs) → S3Metadata

Return the metadata of the path.

Parameters:

• path – S3 path (s3://bucket/key) to get metadata for.

• **kwargs – Additional parameters passed to the HeadObject API.

Returns:

S3Metadata, which behaves as a read-only mapping of the user-defined metadata (x-amz-meta-*) and exposes the system-defined metadata (content type, encryption settings, etc.) as typed properties.

getxattr(path: str, attr_name: str, **kwargs) → str | None

Get an attribute from the user-defined metadata of the path.

Parameters:

• path – S3 path (s3://bucket/key) to get the attribute for.

• attr_name – The name of the attribute.

• **kwargs – Additional parameters passed to metadata().

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(path: str, copy_kwargs: dict[str, Any] | None = None, **kw_args) → None

Set the user-defined metadata of the path.

S3 does not allow updating the metadata of an existing object in place, so the object is copied onto itself with the REPLACE metadata directive. Note that this rewrites the object and updates its last-modified time. The system-defined metadata (e.g., ContentType and CacheControl), the storage class, and the server-side encryption algorithm and KMS key of the object are kept. HeadObject does not return the KMS encryption context, and an Expires value that botocore cannot parse as a date is not kept.

Parameters:

• path – S3 path (s3://bucket/key) to set metadata for. A path with a version ID is rejected, since copying a version onto the key would replace the current object with it.

• copy_kwargs – Additional parameters to use for the underlying CopyObject API call. They take precedence over the kept system-defined metadata and storage class. Any encryption parameter replaces all kept encryption settings.

• **kw_args – Key-value pairs to set, where the values must be strings. The keys are used as-is; names that are not valid Python identifiers (e.g., containing hyphens) can be passed by unpacking a dictionary. Does not alter existing fields, unless the field appears here - if the value is None, delete the field.

Example

>>> fs = S3FileSystem()
>>> fs.setxattr("s3://bucket/key", attribute1="value1")
>>> fs.setxattr("s3://bucket/key", **{"attribute-2": "value2"})

Raises:

ValueError – If the path is a bucket or has a version ID.

get_tags(path: str) → dict[str, str]

Retrieve the tag key/values for the given path.

Parameters:

path – S3 path (s3://bucket/key) to get tags for.

Returns:

Dictionary mapping tag keys to tag values.

put_tags(path: str, tags: dict[str, str], mode: str = 'o') → None

Set the tags for the given existing key.

Tags are a str:str mapping that can be attached to any key, distinct from the user-defined metadata, which is usually set at key creation time. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html

Parameters:

• path – S3 path (s3://bucket/key) of the existing key to attach tags to.

• tags – Tags to apply.

• mode – One of ‘o’ or ‘m’. ‘o’ will over-write any existing tags. ‘m’ will merge in new tags with existing tags, which incurs two remote calls.

chmod(path: str, acl: str, recursive: bool = False, **kwargs) → None

Set the Access Control on a bucket/key.

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl

Parameters:

• path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.

• acl – The value of the canned ACL to apply.

• recursive – Whether to apply the ACL to all keys below the given path too.

• **kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.

list_multipart_uploads(path: str) → list[S3MultipartUpload]

List in-progress (incomplete) multipart uploads in a bucket.

Incomplete multipart uploads continue to accrue storage costs until they are completed or aborted. Use clear_multipart_uploads() to abort all of them.

Parameters:

path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under key/ are listed, not those to sibling keys that merely start with the same characters (e.g., prefix2/a).

Returns:

List of S3MultipartUpload instances describing the in-progress multipart uploads.

object_version_info(path: str, delete_markers: bool = False, **kwargs) → list[S3ObjectVersion]

List the versions of the object or of the objects under the path.

A key path without a trailing slash selects that key if it has any versions or delete markers, and otherwise the keys under key/. The choice does not depend on delete_markers, so a key that has only delete markers yields no versions without them. A key path with a trailing slash selects the keys under it, and a bucket path selects all the keys in the bucket. Sibling keys that merely start with the same characters (e.g., key.bak) are never included.

Parameters:

• path – S3 path (s3://bucket/key or a key prefix) to list the versions for.

• delete_markers – Whether to include delete markers in the result.

• **kwargs – Additional parameters passed to the ListObjectVersions API.

Returns:

List of S3ObjectVersion instances describing the versions.

clear_multipart_uploads(path: str) → None

Abort any incomplete multipart uploads in the bucket.

Uploads completed or aborted after listing are already cleared. All abort results are checked before another abort error is raised.

Parameters:

path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under key/ are aborted, as listed by list_multipart_uploads().

created(path: str) → datetime

Return the creation time of the path.

Returns the same value as modified().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

modified(path: str) → datetime

Return the last-modified time of the path.

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last_modified field from info(), which is None for buckets and directories.

invalidate_cache(path: str | None = None) → None

Remove the cached entries of the path and its parent paths.

The cached bucket listing is removed only by the root path ("", "/" or "s3://"), not by the paths of buckets or keys. A version-qualified path invalidates the version under every query spelling that S3Path.parse accepts, and also the object path without the version, because deleting or copying a version can change the current version of the object.

Parameters:

path – The path to invalidate. If None, clear the whole cache.

class pyathena.filesystem.s3.S3File(fs: S3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int = 20, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)

A buffered file object for reading and writing an S3 object.

Instances are returned by S3FileSystem.open().

fs: S3FileSystem

buffer: BytesIO | None

__init__(fs: S3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int = 20, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → None

Initialize the file for the path and mode.

In read mode, the object is looked up with info() and the reads are made conditional on its ETag (IfMatch). In append mode, an existing object smaller than S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE is read into the write buffer; a larger one is copied with UploadPartCopy as the first parts of a multipart upload, whatever the block size. In exclusive-create mode, the object must not exist when the file is opened, and the upload is committed with IfNoneMatch="*" so that it does not replace an object created in the meantime.

Parameters:

• fs – The filesystem that the file belongs to.

• path – S3 path (s3://bucket/key) of the file.

• mode – The file mode: rb, wb, ab, or xb.

• version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.

• max_workers – The number of parallel workers for range reads, part copies, and part uploads. A write keeps at most max_workers uploaded parts in flight, waiting for one of them to finish before it submits another.

• executor – The executor for parallel operations. If None, a new S3ThreadPoolExecutor is created.

• block_size – The block size for reads and writes. Must be between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.

• cache_type – The fsspec cache type for reads; see S3FileSystem.__init__().

• autocommit – Whether to commit the written data when the file is closed. If False, commit() must be called.

• cache_options – Options for the fsspec cache.

• size – The size of the object, if known. Passed to fsspec.spec.AbstractBufferedFile.

• s3_additional_kwargs – Additional parameters for the S3 requests of the file, such as ContentType or RequestPayer. Each request receives those that its operation accepts.

• **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Raises:

• FileExistsError – If an object exists at the path in exclusive-create mode.

• FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.

• ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE for writing.

close() → None

Close the file, flushing any written data, and shut down its executor.

property multipart_upload: S3MultipartUpload | None

The upload identity retained by the core writer, if initialized.

commit() → None

Complete the upload of the written data.

Creates an empty object if nothing was written, uploads the buffered data with PutObject if no multipart upload part was submitted, and otherwise completes the multipart upload, which is aborted if a part or the completion fails or is interrupted. A failed part is handled as soon as it fails, without waiting for the parts before it. If the abort also fails, the upload is kept so that discard() can abort it. Invalidates the cache of the path afterwards. Does nothing for a file whose failed write dropped the written data.

Raises:

• FileExistsError – If an object was created at the path after the file was opened in exclusive-create mode.

• RuntimeError – If parts were submitted but no multipart upload is initialized.

discard() → None

Abort the multipart upload, if any.

The part uploads that have not started are cancelled, and the running ones are waited for before the abort.

url(expiration: int = 3600, **kwargs) → str

Generate a presigned HTTP URL to read this file (if it already exists).

Parameters:

• expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).

• **kwargs – Additional parameters passed to S3FileSystem.sign().

Returns:

Presigned URL string that provides temporary access to the S3 object.

metadata(**kwargs) → S3Metadata

Return the metadata of the file.

See S3FileSystem.metadata().

Parameters:

**kwargs – Additional parameters passed to the HeadObject API.

Returns:

S3Metadata, which behaves as a read-only mapping of the user-defined metadata and exposes the system-defined metadata as typed properties.

getxattr(xattr_name: str, **kwargs) → str | None

Get an attribute from the user-defined metadata of the file.

See S3FileSystem.getxattr().

Parameters:

• xattr_name – The name of the attribute.

• **kwargs – Additional parameters passed to the HeadObject API.

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(copy_kwargs: dict[str, Any] | None = None, **kwargs) → None

Set the user-defined metadata of the file.

See S3FileSystem.setxattr().

Parameters:

• copy_kwargs – Additional parameters to use for the underlying CopyObject API call.

• **kwargs – Key-value pairs of metadata to set.

class pyathena.filesystem.s3.S3FileTarget(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None)

The object that a file reads or writes, resolved when it is opened.

In read mode, the object is looked up, and the reads are made conditional on its ETag (IfMatch). In append mode, an existing object is looked up; one smaller than S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE is also read, so that the file rewrites it from its buffer, and a larger one is copied by the file as the first parts of a multipart upload. In exclusive-create mode, the object must not exist, and the upload is committed with IfNoneMatch="*".

path

The path of the file. A version that is read is carried in the path as a ?versionId= suffix, so that a reopened file reads the same version.

Type:

str

bucket

The bucket of the object.

Type:

str

key

The key of the object.

Type:

str

version_id

The version that is read, if any.

Type:

str | None

request_kwargs

The parameters of the S3 requests of the file, with the conditions above and, in append mode, the metadata of the existing object, which the rewritten object keeps.

Type:

dict[str, Any]

details

The object that is read or appended, or an empty dictionary.

Type:

pyathena.filesystem.s3_object.S3Object | dict[str, Any]

size

The size of the object that is read, if known.

Type:

int | None

appended

The existing object in append mode, if any.

Type:

pyathena.filesystem.s3_object.S3Object | None

append_data

The data of an appended object that the file rewrites from its buffer, if any.

Type:

bytes | None

path: str

bucket: str

key: str

version_id: str | None

request_kwargs: dict[str, Any]

details: S3Object | dict[str, Any]

size: int | None = None

appended: S3Object | None = None

append_data: bytes | None = None

classmethod resolve(fs: S3FileSystem, path: str, mode: str, *, block_size: int, version_id: str | None = None, size: int | None = None, request_kwargs: Mapping[str, Any] | None = None) → S3FileTarget

Validate the arguments of a file and look up its object.

Sends the lookup requests of the mode, which block.

Parameters:

• fs – The filesystem that the file belongs to.

• path – S3 path (s3://bucket/key) of the file.

• mode – The file mode: rb, wb, ab, or xb.

• block_size – The block size of the file.

• version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.

• size – The size of the object to read, if known.

• request_kwargs – The parameters of the S3 requests of the file. A copy is kept.

Returns:

The resolved target.

Raises:

• FileExistsError – If an object exists at the path in exclusive-create mode.

• FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.

• ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE for writing.

__init__(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None) → None

Async S3 FileSystem

class pyathena.filesystem.s3_async.AioS3FileSystem(*args, **kwargs)

An async filesystem interface for Amazon S3 using fsspec’s AsyncFileSystem.

This class wraps S3FileSystem to provide native asyncio support. Its coroutines run the blocking boto3 calls in an S3AioThreadPool with max_workers threads that the filesystem owns, instead of the default executor of the event loop, and run parallel requests with asyncio.gather.

The implementation uses composition: an internal S3FileSystem instance handles all boto3 calls, while this class delegates to it in the threads of its pool. This avoids diamond inheritance issues and keeps all boto3 logic in one place.

File handles created by _open use S3AioExecutor so that parallel operations (range reads, multipart uploads) are dispatched through the event loop to a pool of max_workers threads of each file. An instance created with asynchronous=True has no event loop of its own, so its file handles use a ThreadPoolExecutor. open_async() returns an AioS3StreamedFile, whose reads and writes are coroutines.

_sync_fs

The internal synchronous S3FileSystem instance.

Example

>>> from pyathena.filesystem.s3_async import AioS3FileSystem
>>> fs = AioS3FileSystem(asynchronous=True)
>>>
>>> # Use in async context
>>> files = await fs._ls('s3://my-bucket/data/')
>>>
>>> # Sync wrappers (auto-generated by fsspec) need an instance created
>>> # without asynchronous=True; they block the caller until done
>>> files = AioS3FileSystem().ls('s3://my-bucket/data/')

protocol: str | tuple[str, ...] = ('s3', 's3a')

mirror_sync_methods = True

async_impl = True

__init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int = 20, s3_additional_kwargs: dict[str, Any] | None = None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, asynchronous: bool = False, loop: Any | None = None, batch_size: int | None = None, **kwargs) → None

Initialize the filesystem and its internal S3FileSystem.

Parameters:

• connection – Passed to the internal S3FileSystem.

• default_block_size – Passed to the internal S3FileSystem.

• default_cache_type – Passed to the internal S3FileSystem.

• max_workers – The number of threads that run the blocking calls of the filesystem, and the default number of parallel requests of its files and copies. If None, the default of ThreadPoolExecutor: min(32, CPUs + 4). Passed to the internal S3FileSystem.

• s3_additional_kwargs – Passed to the internal S3FileSystem.

• allow_bucket_creation – Passed to the internal S3FileSystem.

• allow_bucket_deletion – Passed to the internal S3FileSystem.

• version_aware – Passed to the internal S3FileSystem.

• asynchronous – Passed to fsspec.asyn.AsyncFileSystem.

• loop – Passed to fsspec.asyn.AsyncFileSystem.

• batch_size – Passed to fsspec.asyn.AsyncFileSystem.

• **kwargs – Passed to both fsspec.asyn.AsyncFileSystem and the internal S3FileSystem.

property core: S3Core

The typed S3 operations of the wrapped S3FileSystem.

mv(path1, path2, recursive=False, maxdepth=None, **kwargs) → None

Move files from one S3 location to another.

Runs _mv(); fsspec does not provide the synchronous wrapper of _mv.

Parameters:

• path1 – Source S3 path, glob pattern, or list of paths.

• path2 – Destination S3 path, or list of paths when path1 is a list.

• recursive – Whether to move the directories with their contents.

• maxdepth – Maximum depth of a recursive move.

• **kwargs – Additional S3 copy parameters.

Raises:

• ValueError – If the move has conflicting paths.

• OSError – If bucket versioning cannot be read.

async open_async(path: str, mode: str = 'rb', block_size: int | None = None, max_workers: int | None = None, version_id: str | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → AioS3StreamedFile

Open a file whose reads and writes are coroutines.

The lookups of the mode, as in open(), run in the threads of the filesystem. See AioS3StreamedFile for how the file reads, writes, and commits.

Parameters:

• path – S3 path (s3://bucket/key) of the file.

• mode – The file mode: rb, wb, ab, or xb.

• block_size – The block size for reads and writes. Defaults to the default block size of the filesystem.

• max_workers – The number of threads of the file, which bounds its parallel requests. Defaults to the max_workers of the filesystem.

• version_id – The version ID to read.

• s3_additional_kwargs – Additional parameters for the S3 requests of the file, which take precedence over those of the filesystem.

• **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Returns:

The opened file.

Raises:

• ValueError – If the mode is not binary, compression is requested, or the arguments are invalid, as in open().

• NotImplementedError – If the mode is not supported, or the file is opened for writing in a transaction or with autocommit=False, as the file commits when it is closed.

• FileExistsError – If an object exists at the path in exclusive-create mode.

• FileNotFoundError – If no object exists at the path when reading.

rmdir(path: str) → None

Remove an S3 bucket, which must be empty.

See S3FileSystem.rmdir().

Parameters:

path – S3 bucket path (e.g., “s3://bucket”).

sign(path: str, expiration: int = 3600, **kwargs) → str

Generate a presigned URL for S3 object access.

See S3FileSystem.sign().

Parameters:

• path – S3 path (s3://bucket/key) to generate the URL for.

• expiration – URL expiration time in seconds.

• **kwargs – Additional parameters passed to S3FileSystem.sign().

Returns:

The presigned URL.

metadata(path: str, **kwargs) → S3Metadata

Return the metadata of the path.

See S3FileSystem.metadata().

Parameters:

• path – S3 path (s3://bucket/key) to get metadata for.

• **kwargs – Additional parameters passed to the HeadObject API.

Returns:

S3Metadata of the object.

getxattr(path: str, attr_name: str, **kwargs) → str | None

Get an attribute from the user-defined metadata of the path.

See S3FileSystem.getxattr().

Parameters:

• path – S3 path (s3://bucket/key) to get the attribute for.

• attr_name – The name of the attribute.

• **kwargs – Additional parameters passed to the HeadObject API.

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(path: str, copy_kwargs: dict[str, Any] | None = None, **kwargs) → None

Set the user-defined metadata of the path.

See S3FileSystem.setxattr().

Parameters:

• path – S3 path (s3://bucket/key) to set metadata for.

• copy_kwargs – Additional parameters to use for the underlying CopyObject API call.

• **kwargs – Key-value pairs of metadata to set; a None value deletes the key.

get_tags(path: str) → dict[str, str]

Retrieve the tag key/values for the given path.

See S3FileSystem.get_tags().

Parameters:

path – S3 path (s3://bucket/key) to get tags for.

Returns:

Dictionary mapping tag keys to tag values.

put_tags(path: str, tags: dict[str, str], mode: str = 'o') → None

Set the tags for the given existing key.

See S3FileSystem.put_tags().

Parameters:

• path – S3 path (s3://bucket/key) of the existing key.

• tags – Tags to apply.

• mode – o to overwrite the existing tags or m to merge with them.

chmod(path: str, acl: str, recursive: bool = False, **kwargs) → None

Set the canned ACL of a bucket or key.

See S3FileSystem.chmod().

Parameters:

• path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.

• acl – The canned ACL to apply.

• recursive – Whether to apply the ACL to all keys below the path too.

• **kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.

object_version_info(path: str, delete_markers: bool = False, **kwargs) → list[S3ObjectVersion]

List the versions of the object or of the objects under the path.

See S3FileSystem.object_version_info().

Parameters:

• path – S3 path (s3://bucket/key or a key prefix) to list the versions for.

• delete_markers – Whether to include delete markers in the result.

• **kwargs – Additional parameters passed to the ListObjectVersions API.

Returns:

List of S3ObjectVersion instances describing the versions.

list_multipart_uploads(path: str) → list[S3MultipartUpload]

List in-progress (incomplete) multipart uploads in a bucket.

See S3FileSystem.list_multipart_uploads().

Parameters:

path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).

Returns:

List of S3MultipartUpload instances describing the uploads.

clear_multipart_uploads(path: str) → None

Abort any incomplete multipart uploads in the bucket.

See S3FileSystem.clear_multipart_uploads().

Parameters:

path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).

checksum(path: str, **kwargs) → int

Get the checksum of an S3 object or directory.

See S3FileSystem.checksum().

Parameters:

• path – S3 path (s3://bucket/key) to get the checksum for.

• **kwargs – Additional arguments passed to S3FileSystem.checksum().

Returns:

Integer checksum derived from the ETag or the directory token.

created(path: str) → datetime

Return the creation time of the path.

See S3FileSystem.created().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

modified(path: str) → datetime

Return the last-modified time of the path.

See S3FileSystem.modified().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

invalidate_cache(path: str | None = None) → None

Remove the cached entries of the path and its parent paths.

See S3FileSystem.invalidate_cache().

Parameters:

path – The path to invalidate. If None, clear the whole cache.

touch(path: str, truncate: bool = True, **kwargs) → dict[str, Any]

Create an empty object with PutObject.

See S3FileSystem.touch().

Parameters:

• path – S3 path (s3://bucket/key) of the object.

• truncate – If True, replace an existing object with an empty one; if False, raise if the object exists.

• **kwargs – Additional parameters passed to the PutObject API.

Returns:

The PutObject response as a dictionary.

class pyathena.filesystem.s3_async.AioS3File(fs: S3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int = 20, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)

Async-aware S3 file handle using S3AioExecutor.

Functionally identical to S3File; exists as a distinct type for isinstance checks and to document the async execution model. All parallel operations (range reads, multipart uploads) are dispatched through the S3Executor interface — the S3AioExecutor provided by AioS3FileSystem dispatches them through the event loop to a pool of max_workers threads of the file. For an AioS3FileSystem created with asynchronous=True, it is an S3ThreadPoolExecutor. For reads and writes that are coroutines, see AioS3StreamedFile.

class pyathena.filesystem.s3_async.AioS3StreamedFile(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int = 20, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)

An S3 file whose reads and writes are coroutines.

Returned by AioS3FileSystem.open_async(). The requests of the file run in an S3AioThreadPool of max_workers threads of its own, and the coroutines wait for them without blocking the event loop:

• A read splits the requested range into ranges of the block size, read in parallel. Nothing is cached. As with S3File, the reads are conditional on the ETag of the object, and on its version when the filesystem is version-aware.

• A write buffers the data, and uploads each full block as a part of a multipart upload, with at most max_workers parts in flight. A write that would submit another part waits for one of them to finish first, and a part found failed by that wait, or a part copy of an append found failed before a part is submitted, stops the write with its error. An append copies an existing object with UploadPartCopy, or rewrites one smaller than a part from the buffer, as S3File does.

• Closing the file commits the written data, as S3File.close() does: with PutObject if no part was uploaded, otherwise by completing the multipart upload, which is aborted if the completion fails.

• Leaving an async with block with an exception, a failed write, and a cancelled write close the file without committing it: the requests that have started are waited for and the multipart upload is aborted, so that the existing object is left unchanged.

The commit and the abort run to their end in the threads of the file, also if the task that waits for them is cancelled, including a commit that still waits for a thread; once the commit has been submitted, nothing else aborts the upload. A file opened for writing must be closed with await close() or an async with block; garbage collection cannot await it, so the data of a file that is collected before it is closed or its commit has been submitted is never committed; its multipart upload, if any, is aborted, with a ResourceWarning.

fs: AioS3FileSystem

buffer: BytesIO | None

forced: bool

offset: int | None

__init__(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int = 20, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → None

Initialize the file for the path and mode.

Sends the lookup requests of the mode (see S3FileTarget), which block; AioS3FileSystem.open_async() runs this in a thread.

Parameters:

• fs – The filesystem that the file belongs to.

• path – S3 path (s3://bucket/key) of the file.

• mode – The file mode: rb, wb, ab, or xb.

• version_id – The version ID to read.

• max_workers – The number of threads of the file, which bounds its parallel requests.

• block_size – The block size for reads and writes. Must be between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.

• size – The size of the object to read, if known.

• s3_additional_kwargs – Additional parameters for the S3 requests of the file. Each request receives those that its operation accepts.

• **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Raises:

• FileExistsError – If an object exists at the path in exclusive-create mode.

• FileNotFoundError – If no object exists at the path when reading.

• NotImplementedError – If the mode is not supported.

• ValueError – If max_workers is not positive, or the arguments are invalid as for S3FileTarget.resolve().

__enter__() → AioS3StreamedFile

Reject a synchronous with block, whose exit cannot close the file.

Raises:

TypeError – Always; use async with.

async write(data: bytes) → int

Write data to the buffer, uploading each full block as a part.

Parameters:

data – The bytes to write.

Returns:

The number of bytes written.

Raises:

• ValueError – If the file is not open for writing, is closed, or has been force-flushed.

• BaseException – The error of a part that the write found failed while it waited for a part in flight, or of a part copy of an append found failed before a part is submitted.

async flush(force: bool = False) → None

Upload the buffered data if it fills a block, or all of it if forced.

Parameters:

force – Whether this is the final flush of the file, after which the written data is committed.

Raises:

ValueError – If the file is closed, or force-flushed twice.

async close() → None

Close the file, committing the written data.

Read-ahead Stream

class pyathena.filesystem.reader.ReadAheadStream(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None)

Sequential stream over a file that reads ranges ahead in parallel.

The file is read in parts of part_size bytes with a range-read function, and up to max_parts parts are read in a thread pool ahead of the consumer, so the transfer is not limited to one request at a time. Parts are returned in order, and at most max_parts parts plus the one being consumed are held in memory. A read that returns fewer bytes than requested raises OSError, and an error from the range-read function is raised by the read that reaches its part; every later read raises the same error.

Closing the stream cancels the queued reads, waits for the running ones, and then calls on_close.

Example

>>> file = fs.open_input_file(path)
>>> stream = ReadAheadStream(
...     lambda offset, size: file.read_at(size, offset), file.size(), on_close=file.close
... )
>>> with pyarrow.PythonFile(stream, mode="r") as f:
...     table = pyarrow.csv.read_csv(f)

DEFAULT_PART_SIZE: int = 8388608

Bytes requested by one range read unless part_size is given.

DEFAULT_MAX_PARTS: int = 16

The most parts read ahead at once unless max_parts is given.

__init__(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None) → None

Initialize the stream without reading the file.

Parameters:

• read_range – A function that reads size bytes of the file from offset, called as read_range(offset, size) from several threads at once.

• size – The size of the file in bytes.

• part_size – Bytes requested by one read_range() call; if None, DEFAULT_PART_SIZE.

• max_parts – The most parts read ahead of the consumer at once; if None, DEFAULT_MAX_PARTS.

• on_close – A function called once when the stream is closed, after the running reads have finished, such as the close() of the file.

Raises:

ValueError – If part_size or max_parts is not positive.

readable() → bool

Return whether object was opened for reading.

If False, read() will raise OSError.

tell() → int

Return current stream position.

readinto(buffer: Any) → int

Fill buffer from the parts read ahead.

Parameters:

buffer – A writable buffer.

Returns:

The number of bytes read, less than the size of buffer only at the end of the file.

Raises:

ValueError – If the stream is closed, or its closing was interrupted.

close() → None

Cancel the queued reads, wait for the running ones, and call on_close.

If the wait is interrupted, the stream stays open and on_close is not called, so that the file is not closed under the running reads; closing again waits for them. Reads raise ValueError from then on.

S3 Executor

class pyathena.filesystem.s3_executor.S3Executor

Abstract executor for parallel S3 operations.

Defines the interface used by S3File and S3FileSystem for submitting work to run in parallel and for shutting down the executor when done. Both submit and shutdown mirror the concurrent.futures.Executor interface so that as_completed() and Future.cancel() work unchanged.

abstractmethod submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T]

Submit a callable for execution and return a Future.

abstractmethod shutdown(wait: bool = True) → None

Shut down the executor, freeing any resources.

submit_to(future: Future[T], fn: Callable[[...], T], *args: Any, **kwargs: Any) → None

Submit a callable whose outcome resolves a future of the caller.

The caller creates the future before calling this method, so that it keeps a reference to the future even if an interrupt stops the scheduling midway, before submit() returns. Cancelling the future keeps a callable that has not started from running; a started callable sets the future’s result or exception. If the executor drops the callable before it starts, as S3AioExecutor does when its event loop shuts down, the future is cancelled or gets the error that the executor reported.

Parameters:

• future – A pending future that no executor has started.

• fn – The callable to run.

• *args – Positional arguments passed to the callable.

• **kwargs – Keyword arguments passed to the callable.

class pyathena.filesystem.s3_executor.S3ThreadPoolExecutor(max_workers: int)

Executor that delegates to a ThreadPoolExecutor.

This is the default executor used by S3File and S3FileSystem for synchronous parallel operations.

__init__(max_workers: int) → None

Initialize the executor with a new ThreadPoolExecutor.

Parameters:

max_workers – The maximum number of threads of the thread pool.

submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T]

Submit a callable for execution and return a Future.

shutdown(wait: bool = True) → None

Shut down the executor, freeing any resources.

class pyathena.filesystem.s3_executor.S3AioExecutor(loop: AbstractEventLoop | None = None, max_workers: int = 20)

Executor that schedules work on an asyncio event loop.

Uses asyncio.run_coroutine_threadsafe() to dispatch blocking functions through the event loop to an S3AioThreadPool of its own, returning concurrent.futures.Future objects that are compatible with as_completed(), wait() and Future.cancel(). As with ThreadPoolExecutor, a future cannot be cancelled once its function has started. At most max_workers of the submitted functions run at once, whatever the size of the default executor of the event loop.

Parameters:

• loop – A running asyncio event loop.

• max_workers – The maximum number of submitted functions that run at once.

Raises:

RuntimeError – If the event loop is not running when submit is called.

__init__(loop: AbstractEventLoop | None = None, max_workers: int = 20) → None

Initialize the executor with the event loop to schedule work on.

Parameters:

• loop – The asyncio event loop. submit raises RuntimeError if it is None or not running.

• max_workers – The maximum number of submitted functions that run at once.

Raises:

ValueError – If max_workers is not positive.

submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T]

Submit a callable for execution and return a Future.

shutdown(wait: bool = True) → None

Shut down the executor, freeing any resources.

class pyathena.filesystem.s3_executor.S3AioThreadPool(max_workers: int | None = None)

A thread pool that runs the blocking calls of asyncio code.

Unlike asyncio.to_thread(), which runs a call in the default executor of the event loop, this pool runs at most max_workers calls at once in threads of its own, so that neither the size nor the load of the default executor limits or blocks them. The threads are created as the calls need them, and each call runs with a copy of the caller’s context variables, as with asyncio.to_thread(). In a forked child process, whose copy has no threads, the pool creates new ones. The pool can be shared by several event loops and threads.

Parameters:

max_workers – The maximum number of calls that run at once.

Raises:

ValueError – If max_workers is not positive.

__init__(max_workers: int | None = None) → None

Initialize the pool without starting any thread.

Parameters:

max_workers – The maximum number of calls that run at once. If None, the default of ThreadPoolExecutor: min(32, CPUs + 4).

Raises:

ValueError – If max_workers is not positive.

submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T]

Schedule a blocking call in a thread of the pool.

As with ThreadPoolExecutor, the returned future can be cancelled only before the call starts. The calls start in the order they were submitted, so that a call that waits for the calls submitted before it does not wait for a thread that it holds.

Parameters:

• fn – The blocking callable.

• *args – Positional arguments passed to the callable.

• **kwargs – Keyword arguments passed to the callable.

Returns:

The future of the call.

Raises:

RuntimeError – If the pool has been shut down.

async run(fn: Callable[[...], T], *args: Any, **kwargs: Any) → T

Run a blocking call in a thread of the pool and wait for it.

Cancelling the waiting task cancels a call that has not started; a running call keeps running, as with asyncio.to_thread().

Parameters:

• fn – The blocking callable.

• *args – Positional arguments passed to the callable.

• **kwargs – Keyword arguments passed to the callable.

Returns:

The return value of the call.

Raises:

RuntimeError – If the pool has been shut down.

async run_shielded(fn: Callable[[...], T], *args: Any, **kwargs: Any) → T

Run a blocking call that a cancellation of the waiting task does not stop.

The call runs to its end, even if it has not started when the waiting task is cancelled; its outcome is then dropped.

Parameters:

• fn – The blocking callable.

• *args – Positional arguments passed to the callable.

• **kwargs – Keyword arguments passed to the callable.

Returns:

The return value of the call.

Raises:

RuntimeError – If the pool has been shut down.

shutdown(wait: bool = True) → None

Stop accepting calls and release the threads once they are idle.

The calls that were submitted before still run.

Parameters:

wait – Whether to wait for the submitted calls to finish.

S3 Paths

class pyathena.filesystem.s3_path.S3Path(bucket: str, key: str | None = None, version_id: str | None = None)

An S3 path: a bucket, optionally a key, and optionally a version ID.

Paths are parsed from strings such as s3://bucket/key, s3a://bucket/key or bucket/key, optionally followed by a version ID query (?versionId=, ?versionID=, ?versionid= or ?version_id=). Only a query at the end of the path is a version ID; any other ? is part of the key. The root path, which names no bucket, is not an S3Path.

bucket

The name of the bucket.

Type:

str

key

The key, or None for a bucket path. A trailing slash is kept, and a key of only slashes (bucket//) names the bucket (see is_bucket).

Type:

str | None

version_id

The version ID, or None for a path without a version.

Type:

str | None

Example

>>> path = S3Path.parse("s3://bucket/dir/key?versionId=v1")
>>> path.bucket, path.key, path.version_id
('bucket', 'dir/key', 'v1')
>>> path.name
'bucket/dir/key'
>>> str(path)  # The name, with a version ID query if there is one.
'bucket/dir/key?versionId=v1'
>>> path.uri
's3://bucket/dir/key?versionId=v1'

VERSION_QUERY: ClassVar[Pattern[str]] = re.compile('\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z')

PATTERN: ClassVar[Pattern[str]] = re.compile('(^s3://|^s3a://|^)(?P<bucket>[a-zA-Z0-9.\\-_]+)(/|/(?P<key>.+?))?(\\Z|\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z)', re.DOTALL)

bucket: str

key: str | None = None

version_id: str | None = None

classmethod parse(path: str) → S3Path

Parse a string into an S3 path.

Parameters:

path – The S3 path (e.g., “s3://bucket/key?versionId=…”).

Returns:

The path. The key is None for a bucket path (bucket or bucket/), and keeps a trailing slash otherwise.

Raises:

ValueError – If the string is not a valid S3 path.

classmethod split_version_id(path: str) → tuple[str, str | None]

Split the version ID query from the end of a path string.

Unlike parse(), any string is accepted, such as the names that fsspec builds.

Parameters:

path – The path string.

Returns:

Tuple of the string without the version ID query and the version ID, or of the string itself and None if it ends with no version ID query.

classmethod has_version_id(path: str | PathLike[str]) → bool

Return whether a path string ends with a version ID query.

Unlike parse(), any string is accepted, as with split_version_id().

Parameters:

path – The path.

Returns:

Whether the path ends with a version ID query.

property is_bucket: bool

it has no key, or a key of only slashes.

Type:

Whether the path names the bucket

property is_directory_bucket: bool

Whether the bucket is a directory bucket (S3 Express One Zone).

The names of directory buckets end with --x-s3.

property name: str

The path without a scheme or version, in bucket/key form, or the bucket.

property uri: str

The path as an s3:// URI, with its version ID query, if any.

property target: S3Path

The default move target, assuming bucket versioning is not enabled.

A null version is taken to name the key itself: in a bucket without versioning, or with versioning suspended, a write to the key replaces its null version. With versioning enabled, a write adds a new version instead, so callers with that state must keep the null version distinct from its key. Any other path is its own target.

with_version_id(version_id: str | None) → S3Path

Return the path with another version ID.

Parameters:

version_id – The version ID, or None for the path without a version.

Returns:

The path with the version ID.

__init__(bucket: str, key: str | None = None, version_id: str | None = None) → None

class pyathena.filesystem.s3_path_pairing.S3PathPairing(path1: str | list[str], path2: str | list[str], recursive: bool = False, maxdepth: int | None = None)

The pairing of the paths of one copy(), get() or mv().

The sources are paired with their destinations as fsspec’s copy() pairs them, except that a path with a version ID names that version of an object: it is not a glob pattern, and its destination is named after its key without the version. A move compares the paths by what they name (see target), keeping null versions distinct from their keys in the caller’s versioning-enabled buckets. delete_paths() splits the paths of an rm().

The pairing is a frozen dataclass of the paths as given, and holds no filesystem. The caller makes the lookups that it asks for (expands, skips_directories, looks_up_destination and conflict_candidates()) with its own requests and cache: it expands the sources, leaves out the directories when skips_directories says so, and passes the results in. A sources, destination_is_dir or missing that a rule needs and that is not passed raises ValueError.

path1

The source path, glob pattern, or list of them, as given to the copy.

Type:

str | list[str]

path2

The destination path, or list of paths.

Type:

str | list[str]

recursive

Whether the copy includes the contents of directories.

Type:

bool

maxdepth

The maximum depth of the expansion.

Type:

int | None

Example

>>> pairing = S3PathPairing("s3://bucket/dir/", "s3://bucket/copy/", recursive=True)
>>> sources = fs.expand_path(pairing.path1, recursive=True)
>>> pairing.copy_pairs(sources)

path1: str | list[str]

path2: str | list[str]

recursive: bool = False

maxdepth: int | None = None

property expands: bool

Whether the sources are expanded.

False if both paths are lists, which are paired as given, without expansion or lookups.

property skips_directories: bool

Whether the directories among the expanded sources are left out.

True for a string source copied without recursive, or with a maxdepth. A directory is a source that ends with a slash or that the filesystem reports as a directory.

property looks_up_destination: bool

Whether the pairing depends on the destination being a directory.

True for a string source that is neither a glob pattern nor ends with a slash, copied to a string destination that does not end with a slash. copy_pairs() then needs destination_is_dir.

copy_pairs(sources: Sequence[str] | None = None, destination_is_dir: bool | None = None) → list[tuple[str, str]]

Pair the sources with their destinations as fsspec’s copy() does.

Parameters:

• sources – The expansion of path1, without the directories that skips_directories leaves out; needed when expands.

• destination_is_dir – Whether path2 is a directory; needed when looks_up_destination is true and there are sources.

Returns:

The sources and their destinations, which keep the form of path2. When both paths are lists, they are paired as given, up to the end of the shorter list, as in fsspec’s copy(). Empty if there are no sources.

Raises:

ValueError – If sources or destination_is_dir is needed and None.

conflict_candidates(pairs: Sequence[tuple[str, str]], *, versioning_enabled_buckets: Collection[str] = ()) → list[str]

Return the sources of a move whose conflicts depend on an object at their key.

A source with another source below it may be a directory. If no object exists at its key, it is not copied, so it writes no destination and is left out of the conflict checks of move_pairs(). A path with a version always names an object.

Parameters:

• pairs – The sources and destinations of the move, as copy_pairs() of this pairing returns them.

• versioning_enabled_buckets – Buckets whose versioning is enabled. Their null versions are distinct from their unversioned keys. Use the same collection for move_pairs().

Returns:

The sources, in bucket/key form (their target), that have another source below them and a destination that conflicts, one per pair in the order of the pairs; empty if nothing needs to be looked up.

Raises:

TypeError – If versioning_enabled_buckets is a string.

move_pairs(pairs: Sequence[tuple[str, str]], missing: Collection[str] | None = None, *, versioning_enabled_buckets: Collection[str] = ()) → list[tuple[str, str]]

Check the pairs of a move and leave out the sources that stay in place.

Parameters:

• pairs – The sources and destinations of the move, as copy_pairs() of this pairing returns them.

• missing – The conflict_candidates() without an object at their key, in any form that names them; needed when there are candidates.

• versioning_enabled_buckets – Buckets whose versioning is enabled. Their null versions are distinct from their unversioned keys. Use the same collection for conflict_candidates().

Returns:

The pairs, except those whose destination is the source itself or, for a null version in a bucket without enabled versioning, the key of the source.

Raises:

• ValueError – If two sources have the same destination, or a destination is another source, including one left in place, except for a directory with no object at its key, which is not copied. Also if missing is needed and None.

• TypeError – If missing is a string instead of a collection. Also if versioning_enabled_buckets is a string.

classmethod delete_paths(path: str | list[str]) → tuple[list[str], list[str]]

Split the paths that rm() deletes into those deleted as given and those expanded.

Parameters:

path – S3 path or list of paths.

Returns:

The paths with a version ID, which are deleted as given, and the other paths, which the filesystem expands.

Raises:

ValueError – If a path is a bucket.

__init__(path1: str | list[str], path2: str | list[str], recursive: bool = False, maxdepth: int | None = None) → None

S3 Core

class pyathena.filesystem.s3_core.S3Core(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None)

Typed S3 operations on a boto3 S3 client.

Each operation sends one request, or one per page for the iterators, except those that say otherwise, such as plan_multipart_copy(), copy_object_annotation() and generate_presigned_url(). The requests are sent with the retry policy, and S3 errors are translated into OSError subclasses (see S3ClientError): a missing bucket or multipart upload, or a missing object or version that an operation reads, raises FileNotFoundError, and a denied request PermissionError. As in S3, deleting a missing key is not an error. Nothing is cached.

Example

>>> core = S3Core(boto3.client("s3"))
>>> core.head_object(S3Path.parse("s3://bucket/key")).content_length
>>> for page in core.list_objects("bucket", prefix="dir/", delimiter="/"):
...     print([o.key for o in page.objects])

MULTIPART_UPLOAD_MIN_PART_SIZE: int = 5242880

MULTIPART_UPLOAD_MAX_PART_SIZE: int = 5368709120

MULTIPART_UPLOAD_MAX_PARTS: int = 10000

OBJECT_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'aws-exec-read', 'bucket-owner-full-control', 'bucket-owner-read', 'private', 'public-read', 'public-read-write'})

BUCKET_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'private', 'public-read', 'public-read-write'})

__init__(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None) → None

Create the operations on a client.

Parameters:

• client – The boto3 S3 client to send the requests with.

• retry_config – The retry policy of the requests; defaults to RetryConfig().

• request_kwargs – Parameters sent with every operation that accepts them, such as {"RequestPayer": "requester"}.

property client: BaseClient

The boto3 S3 client.

property retry_config: RetryConfig

The retry policy of the requests.

property request_kwargs: dict[str, Any]

A copy of the parameters sent with every operation that accepts them.

operation_params(method: str, params: Mapping[str, Any]) → dict[str, Any]

Select the parameters that an S3 operation accepts.

Parameters inherited by several requests are filtered by the input shape of each operation, so that, e.g., ServerSideEncryption for writes is not sent with GetObject.

Parameters:

• method – The name of the client method, such as get_object.

• params – The parameters to select from.

Returns:

The parameters that the operation accepts. Empty for a method that is not an S3 API operation, such as generate_presigned_url.

call(method: str | Callable[[...], Any], **request) → dict[str, Any]

Send a request with the retry policy and translate its errors.

request_kwargs that the operation accepts are added; a parameter of the request takes precedence.

Parameters:

• method – The name of the client method, or the method itself.

• **request – The request parameters, sent as given.

Returns:

The response.

Raises:

OSError – The translation of an S3 error response (see S3ClientError).

head_object(path: S3Path, **params) → S3Metadata

Look up an object, or a version of it, with HeadObject.

Parameters:

• path – The path of the object, with the version ID to look up, if any.

• **params – Additional request parameters, sent as given.

Returns:

The metadata of the object.

Raises:

• ValueError – If the path has no key.

• FileNotFoundError – If the object or version does not exist.

head_bucket(bucket: str, **params) → S3Bucket

Look up a bucket with HeadBucket.

Parameters:

• bucket – The name of the bucket.

• **params – Additional request parameters, sent as given.

Returns:

The bucket, with its region if returned.

Raises:

FileNotFoundError – If the bucket does not exist.

get_object(path: S3Path, range_: tuple[int, int | None] | None = None, **params) → bytes

Read an object, a version of it, or a byte range of it with GetObject.

The body of the response is read whole and then closed, also when the read fails. A failure to read the body is neither translated nor retried: the botocore exception, such as ReadTimeoutError or IncompleteReadError, propagates.

Parameters:

• path – The path of the object, with the version ID to read, if any.

• range – The (start, end) byte range to read, with an exclusive end, or None as the end to read to the end of the object. A negative start without an end reads the last -start bytes, or the whole object if it is shorter. None sends no range of its own, so the whole object is read unless params or request_kwargs have Range.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The bytes read.

Raises:

• ValueError – If the path has no key, or the range is empty or has both a negative start and an end, which S3 would ignore and return the whole object for.

• FileNotFoundError – If the object or version does not exist.

• OSError – If the range starts at or past the end of the object. Its __cause__ is the ClientError with the InvalidRange code.

put_object(path: S3Path, body: bytes | None = None, **params) → S3PutObject

Write an object with PutObject.

Parameters:

• path – The path of the object to write, without a version ID.

• body – The data to write. None or empty bytes send no body of their own, so an empty object is written unless params or request_kwargs have Body.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The result of the write.

Raises:

ValueError – If the path has no key, or has a version ID, which a write cannot replace.

create_bucket(bucket: str, acl: str | None = None, region_name: str | None = None, **params) → None

Create a bucket with CreateBucket.

A call creates the bucket; the allow_bucket_creation option of the filesystem applies only to the filesystem’s methods.

Parameters:

• bucket – The name of the bucket.

• acl – The canned ACL of the bucket, one of BUCKET_ACLS. None or an empty string sends no ACL.

• region_name – The region to create the bucket in. None or an empty string uses the region of the client. A location constraint is sent for every region except us-east-1, which does not accept one.

• **params – Additional request parameters, sent as given, except CreateBucketConfiguration. Its fields are kept, and the LocationConstraint of region_name is added unless the region is us-east-1 or the configuration has a LocationConstraint, Location or Bucket. A directory bucket is created with Location and Bucket, so no LocationConstraint is added to its configuration. An empty or None configuration is treated as not given. The given configuration is not modified.

Raises:

ValueError – If the ACL is not a canned ACL of buckets.

delete_bucket(bucket: str, **params) → None

Delete a bucket, which must be empty, with DeleteBucket.

A call deletes the bucket; the allow_bucket_deletion option of the filesystem applies only to the filesystem’s methods.

Parameters:

• bucket – The name of the bucket.

• **params – Additional request parameters, sent as given.

Raises:

FileNotFoundError – If the bucket does not exist.

get_bucket_versioning(bucket: str, **params) → str | None

Get the versioning state of a bucket with GetBucketVersioning.

Parameters:

• bucket – The name of the bucket.

• **params – Additional request parameters, sent as given.

Returns:

Enabled or Suspended, or None if versioning has never been enabled on the bucket.

Raises:

FileNotFoundError – If the bucket does not exist.

delete_object(path: S3Path, **params) → None

Delete an object, or a version of it, with DeleteObject.

Parameters:

• path – The path of the object, with the version ID to delete, if any.

• **params – Additional request parameters, sent as given.

Raises:

• ValueError – If the path has no key.

• FileNotFoundError – If the bucket does not exist.

delete_objects(batch: S3DeleteBatch, **params) → S3DeleteResult

Delete the objects of a batch with one DeleteObjects request.

Parameters:

• batch – The objects to delete.

• **params – Additional request parameters, sent as given.

Returns:

The result, with the objects that S3 could not delete in its errors.

Raises:

FileNotFoundError – If the bucket does not exist.

create_multipart_upload(path: S3Path, **params) → S3MultipartUpload

Start a multipart upload to an object with CreateMultipartUpload.

Parameters:

• path – The path of the object to write, without a version ID.

• **params – Additional request parameters. The bucket and key of the path take precedence over parameters of the same name.

Returns:

The upload retaining the path’s bucket and key, including an access point alias or ARN, and the response’s upload ID and checksum configuration.

Raises:

ValueError – If the path has no key, or has a version ID, which a write cannot replace.

upload_part(upload: S3MultipartUpload, part_number: int, body: bytes, **params) → S3MultipartUploadPart

Upload a part of a multipart upload with UploadPart.

Parameters:

• upload – The multipart upload returned by creation. Its checksum algorithm is passed to the SDK to calculate the part checksum.

• part_number – The number of the part, from 1.

• body – The data of the part.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The uploaded part.

Raises:

ValueError – If the upload has no bucket, key, or upload ID.

upload_part_copy(upload: S3MultipartUpload, part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params) → S3MultipartUploadPart

Copy a part of a multipart upload from an object with UploadPartCopy.

Parameters:

• upload – The multipart upload returned by creation.

• part_number – The number of the part, from 1.

• source – The path of the object to copy, with the version ID to copy, if any.

• range – The (start, end) byte range of the source to copy, with an exclusive end. None sends no range of its own, so the whole source is copied unless params or request_kwargs have CopySourceRange.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The copied part.

Raises:

ValueError – If the upload has no bucket, key, or upload ID, or the source has no key.

complete_multipart_upload(upload: S3MultipartUpload, parts: Sequence[S3MultipartUploadPart], **params) → S3CompleteMultipartUpload

Complete a multipart upload with CompleteMultipartUpload.

Parameters:

• upload – The multipart upload returned by creation. Its checksum algorithm selects the matching part checksum, and its checksum type is sent when present. Without an algorithm, only the ETag and part number are sent, even if the SDK added a part checksum.

• parts – The uploaded parts, in part-number order.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The completed upload.

Raises:

ValueError – If the upload has no bucket, key, or upload ID.

abort_multipart_upload(upload: S3MultipartUpload, **params) → None

Abort a multipart upload with AbortMultipartUpload.

Parameters:

• upload – The multipart upload returned by creation or listing.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Raises:

• ValueError – If the upload has no bucket, key, or upload ID.

• FileNotFoundError – If the upload does not exist, for example because it was completed or aborted.

part_ranges(size: int, block_size: int) → list[tuple[int, int]]

Split an object into the source ranges of the parts that copy it.

The object is split into ranges of block_size bytes, or of a larger size that splits it into at most MULTIPART_UPLOAD_MAX_PARTS ranges. A last range shorter than MULTIPART_UPLOAD_MIN_PART_SIZE is merged into the previous one, which is split in half if the result exceeds MULTIPART_UPLOAD_MAX_PART_SIZE. Every range is then within the S3 part size limits, including the last one unless the whole object is smaller than the minimum part size, so that more parts can follow the copied ones, as in an append.

Parameters:

• size – The size of the source object in bytes.

• block_size – The size in bytes to split the object by, between MULTIPART_UPLOAD_MIN_PART_SIZE and MULTIPART_UPLOAD_MAX_PART_SIZE. It is raised to size divided by MULTIPART_UPLOAD_MAX_PARTS, rounded up, if smaller. The range that a short last range is merged into can be longer, up to MULTIPART_UPLOAD_MAX_PART_SIZE.

Returns:

The (start, end) byte ranges, with an exclusive end, that cover the whole object in order.

check_multipart_upload_size(size: int, block_size: int) → None

Check that data fits in a multipart upload before it is uploaded.

Sends no request.

Parameters:

• size – The size of the data in bytes.

• block_size – The size in bytes of the parts that upload the data.

Raises:

ValueError – If the data takes more than MULTIPART_UPLOAD_MAX_PARTS blocks. The message gives the minimum block size, which is at least MULTIPART_UPLOAD_MIN_PART_SIZE.

copy_object(source: S3Path, destination: S3Path, **params) → None

Copy an object, or a version of it, with CopyObject.

Parameters:

• source – The path of the object to copy, with the version ID to copy, if any.

• destination – The path of the object to write, without a version ID.

• **params – Additional request parameters, sent as given.

Raises:

ValueError – If the source or the destination has no key, or the destination has a version ID, which a write cannot replace.

plan_multipart_copy(source: S3Path, destination: S3Path, block_size: int | None = None, **params) → S3MultipartCopyPlan

Plan a copy with a multipart upload that copies as CopyObject does.

The source is read with HeadObject. Without a version in the path, a version ID other than null that it reports is the version to copy, so that the parts, the tags and the annotations come from the same object even if the source is replaced during the copy. A null version, which a write can replace, is not pinned. If the reported size fits in a single CopyObject request, nothing else is read and the plan says so.

No multipart request accepts the directives of CopyObject, so the plan applies them as CopyObject does. With the COPY metadata directive (the default), the content headers and the user-defined metadata of the source are used, and the values of params are ignored. With the COPY tagging directive (the default), the tags are read with GetObjectTagging, and the Tagging of params is ignored. REPLACE uses the values of params instead. With the COPY annotation directive (the default), the annotations are listed with ListObjectAnnotations, unless the source is encrypted with SSE-C or is in a directory bucket, which cannot have annotations; an object in a directory bucket has no tags either. The requests that read the source receive RequestPayer, and the source’s expected bucket owner and SSE-C parameters (ExpectedSourceBucketOwner and CopySourceSSECustomer*) under their names in those requests.

Parameters:

• source – The path of the object to copy, with the version ID to copy, if any.

• destination – The path of the object to write, without a version ID.

• block_size – The size in bytes of the copied ranges, between MULTIPART_UPLOAD_MIN_PART_SIZE and MULTIPART_UPLOAD_MAX_PART_SIZE (the default); see part_ranges().

• **params – The CopyObject parameters of the copy. Each request receives those that it accepts; CreateMultipartUpload also receives those that CopyObject does not accept either, so that botocore rejects them as it would for CopyObject.

Returns:

The plan.

Raises:

ValueError – If the source or the destination has no key, the destination has a version ID, block_size is out of the part size limits, a directive has a value that CopyObject does not accept, or HeadObject reports no size.

list_object_annotations(path: S3Path, **params) → list[str]

List the names of the annotations of an object with ListObjectAnnotations.

Sends one request per page.

Parameters:

• path – The path of the object, with the version ID to list, if any.

• **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The annotation names, across all pages.

Raises:

ValueError – If the path has no key.

copy_object_annotation(name: str, source: S3Path, destination: S3Path, version_id: str | None, etag: str | None, **params) → None

Copy an annotation of an object onto the object that a copy wrote.

Reads the annotation with GetObjectAnnotation and writes it with PutObjectAnnotation. The annotation is written to the version that the copy created, if the bucket is versioned, and only if the destination still has the ETag of the copy, so that it is not attached to an object written over the copy.

Parameters:

• name – The annotation name.

• source – The path of the copied object, with the version ID that was copied, if any.

• destination – The path of the object that the copy wrote, without a version ID.

• version_id – The version ID that the copy created, if any.

• etag – The ETag of the object that the copy wrote, if any.

• **params – The CopyObject parameters of the copy. GetObjectAnnotation receives those of the source, mapped as for plan_multipart_copy(), and PutObjectAnnotation those that it accepts; the fields that the other arguments set take precedence.

Raises:

ValueError – If the source or the destination has no key.

get_object_tagging(path: S3Path, **params) → dict[str, str]

Get the tags of an object, or of a version of it, with GetObjectTagging.

Parameters:

• path – The path of the object, with the version ID to read, if any, including null.

• **params – Additional request parameters, sent as given.

Returns:

The tags, mapping each key to its value, in the order of the response.

Raises:

• ValueError – If the path has no key.

• FileNotFoundError – If the object or version does not exist.

put_object_tagging(path: S3Path, tags: Mapping[str, str], **params) → None

Replace the tags of an object, or of a version of it, with PutObjectTagging.

Parameters:

• path – The path of the object, with the version ID to tag, if any, including null.

• tags – The tags, mapping each key to its value. They replace all the existing tags.

• **params – Additional request parameters, sent as given.

Raises:

• ValueError – If the path has no key.

• FileNotFoundError – If the object or version does not exist.

put_object_acl(path: S3Path, acl: str, **params) → None

Apply a canned ACL to an object, or to a version of it, with PutObjectAcl.

Parameters:

• path – The path of the object, with the version ID to apply the ACL to, if any, including null.

• acl – The canned ACL, one of OBJECT_ACLS.

• **params – Additional request parameters, sent as given.

Raises:

• ValueError – If the path has no key, or the ACL is not in OBJECT_ACLS.

• FileNotFoundError – If the object or version does not exist.

put_bucket_acl(bucket: str, acl: str, **params) → None

Apply a canned ACL to a bucket with PutBucketAcl.

Parameters:

• bucket – The name of the bucket.

• acl – The canned ACL, one of BUCKET_ACLS.

• **params – Additional request parameters, sent as given.

Raises:

• ValueError – If the ACL is not in BUCKET_ACLS.

• FileNotFoundError – If the bucket does not exist.

replace_object_metadata(path: S3Path, head: S3Metadata, metadata: Mapping[str, str], **params) → None

Replace the user-defined metadata of an object by copying it onto itself.

S3 does not update the metadata of an object in place, so the object is copied onto itself with CopyObject and the REPLACE metadata directive, which writes a new object, or a new version in a versioned bucket. With that directive, S3 does not copy what the request omits, so the request also sends the content headers, Expires, WebsiteRedirectLocation and StorageClass of head, and, unless params set an encryption parameter, its ServerSideEncryption, SSEKMSKeyId and BucketKeyEnabled. Fields that are None in head are not sent; note that S3Metadata reports STANDARD when HeadObject omits the storage class. HeadObject does not return the KMS encryption context, so it is not retained.

Parameters:

• path – The path of the object, without a version ID.

• head – The HeadObject result of path (see head_object()), whose fields are retained.

• metadata – The user-defined metadata of the copy, which replaces all the existing user-defined metadata.

• **params – Additional CopyObject parameters. They take precedence over the retained fields; one that the copy itself sets, such as Metadata, MetadataDirective or Key, raises TypeError.

Raises:

ValueError – If the path has no key or has a version ID, which a write cannot replace.

generate_presigned_url(path: S3Path, client_method: str = 'get_object', expires_in: int = 3600, **params) → str

Generate a presigned URL for a request on an object.

The URL is signed locally with the client’s credentials; no request is sent. request_kwargs are not added to the signed parameters.

Parameters:

• path – The path of the object, with the version ID to sign for, if any. A path without a key is not rejected; botocore validates the parameters of the method.

• client_method – The name of the client method to sign, such as get_object or put_object.

• expires_in – The number of seconds for which the URL is valid.

• **params – Parameters of the method to sign. They take precedence over the bucket, key and version ID of the path.

Returns:

The presigned URL.

list_objects_page(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) → S3ListObjectsPage

List one page of objects with ListObjectsV2.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list.

• delimiter – The delimiter to group keys by; None or "" lists recursively.

• max_keys – The maximum number of keys of the page.

• continuation_token – The token of the page to list.

• **params – Additional request parameters, sent as given.

Returns:

The page.

list_objects(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) → Iterator[S3ListObjectsPage]

List the pages of objects with ListObjectsV2, one request each.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list.

• delimiter – The delimiter to group keys by; None or "" lists recursively.

• max_keys – The maximum number of keys per page.

• continuation_token – The token of the first page to list.

• **params – Additional request parameters, sent as given.

Yields:

The pages, until one has no continuation token.

Raises:

TypeError – If params has ContinuationToken, which the iterator advances; pass continuation_token instead.

list_object_versions_page(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) → S3ListObjectVersionsPage

List one page of versions with ListObjectVersions.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list.

• delimiter – The delimiter to group keys by; None or "" lists recursively.

• key_marker – The key marker of the page to list.

• version_id_marker – The version ID marker of the page to list. S3 accepts it only with a key marker.

• **params – Additional request parameters, sent as given.

Returns:

The page.

list_object_versions(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) → Iterator[S3ListObjectVersionsPage]

List the pages of versions with ListObjectVersions, one request each.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list.

• delimiter – The delimiter to group keys by; None or "" lists recursively.

• key_marker – The key marker of the first page to list.

• version_id_marker – The version ID marker of the first page to list, sent with key_marker.

• **params – Additional request parameters, sent as given.

Yields:

The pages, until one is not truncated or has no key marker.

Raises:

TypeError – If params has KeyMarker or VersionIdMarker, which the iterator advances; pass key_marker and version_id_marker instead.

list_multipart_uploads_page(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) → S3ListMultipartUploadsPage

List one page of in-progress multipart uploads with ListMultipartUploads.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.

• key_marker – The key marker of the page to list.

• upload_id_marker – The upload ID marker of the page to list. S3 accepts it only with a key marker.

• **params – Additional request parameters, sent as given.

Returns:

The page.

Raises:

FileNotFoundError – If the bucket does not exist.

list_multipart_uploads(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) → Iterator[S3ListMultipartUploadsPage]

List the pages of in-progress multipart uploads, one request each.

Parameters:

• bucket – The bucket to list.

• prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.

• key_marker – The key marker of the first page to list.

• upload_id_marker – The upload ID marker of the first page to list, sent with key_marker.

• **params – Additional request parameters, sent as given.

Yields:

The pages, until one is not truncated or lacks either next marker.

Raises:

• TypeError – If params has KeyMarker or UploadIdMarker, which the iterator advances; pass key_marker and upload_id_marker instead.

• FileNotFoundError – If the bucket does not exist.

list_buckets_page(continuation_token: str | None = None, **params) → S3ListBucketsPage

List one page of the buckets of the caller with ListBuckets.

Parameters:

• continuation_token – The token of the page to list.

• **params – Additional request parameters, sent as given.

Returns:

The page.

list_buckets(continuation_token: str | None = None, **params) → Iterator[S3ListBucketsPage]

List the pages of the buckets of the caller, one request each.

Parameters:

• continuation_token – The token of the first page to list.

• **params – Additional request parameters, sent as given.

Yields:

The pages, until one has no continuation token.

Raises:

TypeError – If params has ContinuationToken, which the iterator advances; pass continuation_token instead.

class pyathena.filesystem.s3_core.S3ObjectSummary(bucket: str, key: str, size: int | None = None, etag: str | None = None, last_modified: datetime | None = None, storage_class: str | None = None)

An object listed by ListObjectsV2 (a Contents entry).

bucket

The bucket of the object.

Type:

str

key

The key of the object.

Type:

str

size

The size of the object in bytes.

Type:

int | None

etag

The entity tag of the object.

Type:

str | None

last_modified

When the object was last modified.

Type:

datetime.datetime | None

storage_class

The storage class of the object.

Type:

str | None

bucket: str

key: str

size: int | None = None

etag: str | None = None

last_modified: datetime | None = None

storage_class: str | None = None

classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3ObjectSummary

Build the object from a Contents entry of ListObjectsV2.

Parameters:

• bucket – The bucket that was listed.

• entry – The Contents entry.

Returns:

The listed object.

property path: S3Path

The path of the object.

__init__(bucket: str, key: str, size: int | None = None, etag: str | None = None, last_modified: datetime | None = None, storage_class: str | None = None) → None

class pyathena.filesystem.s3_core.S3CommonPrefix(bucket: str, prefix: str)

A key prefix listed with a delimiter (a CommonPrefixes entry).

bucket

The bucket that was listed.

Type:

str

prefix

The key prefix, ending with the delimiter.

Type:

str

bucket: str

prefix: str

classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3CommonPrefix

Build the prefix from a CommonPrefixes entry.

Parameters:

• bucket – The bucket that was listed.

• entry – The CommonPrefixes entry.

Returns:

The listed prefix.

__init__(bucket: str, prefix: str) → None

class pyathena.filesystem.s3_core.S3Bucket(name: str, creation_date: datetime | None = None, bucket_region: str | None = None)

A bucket, as listed by ListBuckets or found by HeadBucket.

name

The name of the bucket.

Type:

str

creation_date

When the bucket was created, if returned.

Type:

datetime.datetime | None

bucket_region

The region of the bucket, if returned.

Type:

str | None

name: str

creation_date: datetime | None = None

bucket_region: str | None = None

classmethod from_response(entry: Mapping[str, Any]) → S3Bucket

Build the bucket from a Buckets entry of ListBuckets.

Parameters:

entry – The Buckets entry.

Returns:

The listed bucket.

__init__(name: str, creation_date: datetime | None = None, bucket_region: str | None = None) → None

class pyathena.filesystem.s3_core.S3ListObjectsPage(bucket: str, objects: tuple[S3ObjectSummary, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), key_count: int | None = None, is_truncated: bool = False, next_continuation_token: str | None = None)

One page of a ListObjectsV2 listing.

bucket

The bucket that was listed.

Type:

str

objects

The listed objects (Contents).

Type:

tuple[pyathena.filesystem.s3_core.S3ObjectSummary, …]

common_prefixes

The listed key prefixes (CommonPrefixes).

Type:

tuple[pyathena.filesystem.s3_core.S3CommonPrefix, …]

key_count

The number of objects and prefixes of the page (KeyCount), if returned.

Type:

int | None

is_truncated

Whether more pages follow.

Type:

bool

next_continuation_token

The token of the next page, if any.

Type:

str | None

bucket: str

objects: tuple[S3ObjectSummary, ...] = ()

common_prefixes: tuple[S3CommonPrefix, ...] = ()

key_count: int | None = None

is_truncated: bool = False

next_continuation_token: str | None = None

classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListObjectsPage

Build the page from a ListObjectsV2 response.

Parameters:

• bucket – The bucket that was listed.

• response – The ListObjectsV2 response.

Returns:

The page.

__init__(bucket: str, objects: tuple[S3ObjectSummary, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), key_count: int | None = None, is_truncated: bool = False, next_continuation_token: str | None = None) → None

class pyathena.filesystem.s3_core.S3ListObjectVersionsPage(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None)

One page of a ListObjectVersions listing.

bucket

The bucket that was listed.

Type:

str

versions

The listed versions (Versions).

Type:

tuple[pyathena.filesystem.s3_object.S3ObjectVersion, …]

delete_markers

The listed delete markers (DeleteMarkers).

Type:

tuple[pyathena.filesystem.s3_object.S3ObjectVersion, …]

common_prefixes

The listed key prefixes (CommonPrefixes).

Type:

tuple[pyathena.filesystem.s3_core.S3CommonPrefix, …]

is_truncated

Whether more pages follow.

Type:

bool

next_key_marker

The key marker of the next page, if any.

Type:

str | None

next_version_id_marker

The version ID marker of the next page, if any.

Type:

str | None

bucket: str

versions: tuple[S3ObjectVersion, ...] = ()

delete_markers: tuple[S3ObjectVersion, ...] = ()

common_prefixes: tuple[S3CommonPrefix, ...] = ()

is_truncated: bool = False

next_key_marker: str | None = None

next_version_id_marker: str | None = None

classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListObjectVersionsPage

Build the page from a ListObjectVersions response.

Parameters:

• bucket – The bucket that was listed.

• response – The ListObjectVersions response.

Returns:

The page.

__init__(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None) → None

class pyathena.filesystem.s3_core.S3ListBucketsPage(buckets: tuple[S3Bucket, ...] = (), continuation_token: str | None = None)

One page of a ListBuckets listing.

buckets

The listed buckets.

Type:

tuple[pyathena.filesystem.s3_core.S3Bucket, …]

continuation_token

The token of the next page, if any.

Type:

str | None

buckets: tuple[S3Bucket, ...] = ()

continuation_token: str | None = None

classmethod from_response(response: Mapping[str, Any]) → S3ListBucketsPage

Build the page from a ListBuckets response.

Parameters:

response – The ListBuckets response.

Returns:

The page.

__init__(buckets: tuple[S3Bucket, ...] = (), continuation_token: str | None = None) → None

class pyathena.filesystem.s3_core.S3ListMultipartUploadsPage(bucket: str, uploads: tuple[S3MultipartUpload, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_upload_id_marker: str | None = None)

One page of a ListMultipartUploads listing.

bucket

The bucket that was listed.

Type:

str

uploads

The listed in-progress multipart uploads (Uploads), with the listed bucket as their bucket.

Type:

tuple[pyathena.filesystem.s3_object.S3MultipartUpload, …]

is_truncated

Whether more pages follow.

Type:

bool

next_key_marker

The key marker of the next page, if any.

Type:

str | None

next_upload_id_marker

The upload ID marker of the next page, if any.

Type:

str | None

bucket: str

uploads: tuple[S3MultipartUpload, ...] = ()

is_truncated: bool = False

next_key_marker: str | None = None

next_upload_id_marker: str | None = None

classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListMultipartUploadsPage

Build the page from a ListMultipartUploads response.

Parameters:

• bucket – The bucket that was listed.

• response – The ListMultipartUploads response.

Returns:

The page.

__init__(bucket: str, uploads: tuple[S3MultipartUpload, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_upload_id_marker: str | None = None) → None

class pyathena.filesystem.s3_core.S3DeleteBatch(bucket: str, objects: tuple[S3Path, ...], quiet: bool = True)

The objects of one bucket that one DeleteObjects request deletes.

A batch is valid by construction: it has 1 to MAX_KEYS objects, each with a key in the bucket of the batch. A path with a version ID deletes that version.

bucket

The bucket of the objects.

Type:

str

objects

The paths of the objects to delete.

Type:

tuple[pyathena.filesystem.s3_path.S3Path, …]

quiet

Whether S3 omits the deleted objects from the response.

Type:

bool

MAX_KEYS: ClassVar[int] = 1000

bucket: str

objects: tuple[S3Path, ...]

quiet: bool = True

__post_init__() → None

Validate the objects of the batch.

Raises:

ValueError – If the batch has no objects or more than MAX_KEYS, or an object has no key or another bucket.

classmethod from_paths(paths: Iterable[S3Path], quiet: bool = True) → list[S3DeleteBatch]

Group the paths into batches by bucket, in the order of the paths.

Parameters:

• paths – The paths of the objects to delete.

• quiet – Whether S3 omits the deleted objects from the responses.

Returns:

The batches of up to MAX_KEYS objects of one bucket each.

Raises:

ValueError – If a path has no key.

__init__(bucket: str, objects: tuple[S3Path, ...], quiet: bool = True) → None

class pyathena.filesystem.s3_core.S3DeleteResult(bucket: str, deleted: tuple[S3Path, ...] = (), errors: tuple[S3DeleteError, ...] = ())

The result of a DeleteObjects request.

S3 answers a request with 200 also when it could not delete some of the objects, and lists them in errors.

bucket

The bucket of the request.

Type:

str

deleted

The deleted objects, with the version ID of the request, if any. Empty for a quiet batch.

Type:

tuple[pyathena.filesystem.s3_path.S3Path, …]

errors

The objects that S3 could not delete.

Type:

tuple[pyathena.filesystem.s3_core.S3DeleteError, …]

bucket: str

deleted: tuple[S3Path, ...] = ()

errors: tuple[S3DeleteError, ...] = ()

classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3DeleteResult

Build the result from a DeleteObjects response.

Parameters:

• bucket – The bucket of the request.

• response – The DeleteObjects response.

Returns:

The result.

__init__(bucket: str, deleted: tuple[S3Path, ...] = (), errors: tuple[S3DeleteError, ...] = ()) → None

class pyathena.filesystem.s3_core.S3DeleteError(path: S3Path, code: str | None = None, message: str | None = None)

An object that DeleteObjects could not delete (an Errors entry).

str() gives "path (code: message)".

path

The path of the object, with the version ID of the request, if any.

Type:

pyathena.filesystem.s3_path.S3Path

code

The error code.

Type:

str | None

message

The error message.

Type:

str | None

path: S3Path

code: str | None = None

message: str | None = None

classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3DeleteError

Build the error from an Errors entry of DeleteObjects.

Parameters:

• bucket – The bucket of the request.

• entry – The Errors entry.

Returns:

The error.

__init__(path: S3Path, code: str | None = None, message: str | None = None) → None

class pyathena.filesystem.s3_core.S3MultipartCopyPlan(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False)

The requests of a copy with a multipart upload, as CopyObject copies.

S3Core.plan_multipart_copy() reads the source and builds the plan; the caller schedules the requests: CreateMultipartUpload with create_params, one UploadPartCopy per range with part_params, then CompleteMultipartUpload with complete_params, or AbortMultipartUpload with abort_params after a failure, and finally the copy of each annotation with S3Core.copy_object_annotation(). If fits_single_request is true, the source is copied with S3Core.copy_object() instead, and ranges, the parameters and annotations are empty.

source

The object to copy, with the version that HeadObject reported unless the path has one or the version is null.

Type:

pyathena.filesystem.s3_path.S3Path

destination

The object that the copy writes.

Type:

pyathena.filesystem.s3_path.S3Path

size

The size in bytes of the source, from HeadObject.

Type:

int

ranges

The (start, end) byte ranges of the source that the parts copy, with an exclusive end, in part-number order.

Type:

tuple[tuple[int, int], …]

create_params

The parameters of CreateMultipartUpload, with the metadata and the tags that CopyObject would write.

Type:

collections.abc.Mapping[str, Any]

part_params

The parameters of each UploadPartCopy.

Type:

collections.abc.Mapping[str, Any]

complete_params

The parameters of CompleteMultipartUpload.

Type:

collections.abc.Mapping[str, Any]

abort_params

The parameters of AbortMultipartUpload.

Type:

collections.abc.Mapping[str, Any]

annotations

The names of the annotations to copy; empty if the directive excludes them or the source cannot have any.

Type:

tuple[str, …]

fits_single_request

Whether the source fits in a single CopyObject request.

Type:

bool

source: S3Path

destination: S3Path

size: int

ranges: tuple[tuple[int, int], ...] = ()

create_params: Mapping[str, Any]

part_params: Mapping[str, Any]

complete_params: Mapping[str, Any]

abort_params: Mapping[str, Any]

annotations: tuple[str, ...] = ()

fits_single_request: bool = False

__init__(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False) → None

S3 Objects

class pyathena.filesystem.s3_object.S3Object(init: dict[str, Any], **kwargs)

Represents an S3 object with metadata and filesystem-like properties.

This class provides a dictionary-like interface to S3 object metadata, making it easier to work with S3 objects in filesystem operations. It handles the mapping between S3 API field names and more pythonic property names.

The object supports both dictionary-style access and property-style access to metadata fields like content type, storage class, encryption settings, and object lock configurations. Dictionary-style access behaves like a dictionary, so a missing key raises KeyError. Property-style access returns None for a known field that the object does not have, and raises AttributeError for any other missing name.

Example

>>> s3_obj = S3Object({"ContentType": "text/csv", "ContentLength": 1024})
>>> print(s3_obj.content_type)  # "text/csv"
>>> print(s3_obj["content_length"])  # 1024
>>> s3_obj.storage_class = "STANDARD_IA"

Note

This class is primarily used internally by S3FileSystem for representing S3 objects in filesystem operations.

__init__(init: dict[str, Any], **kwargs) → None

Initialize the object from an S3 API response.

Only the fields of init that have a property mapping are kept, under their property names (e.g., ContentType -> content_type). storage_class defaults to STANDARD when init has no StorageClass, and size is taken from Size or ContentLength, or is 0, also for an empty init. name is set to bucket/key, or to the bucket when there is no key.

Parameters:

• init – An S3 API response or listing entry, such as a HeadObject response or a ListObjectsV2 Contents entry.

• **kwargs – Additional fields stored as-is, such as type, bucket, key and version_id. S3 API field names are stored under their property names.

get(k[, d]) → D[k] if k in D, else d.  d defaults to None.

__getattr__(item: str) → Any

Return None for a known field that the object does not have.

Called only when normal attribute lookup fails, so fields that the object has are returned without reaching this method.

Parameters:

item – The attribute name.

Returns:

None, if item is a known S3 object field.

Raises:

AttributeError – If item is not a known S3 object field.

copy() → S3Object

Return a shallow copy of the object.

Returns:

A new S3Object with the same fields.

to_dict() → dict[str, Any]

Convert S3Object to dictionary representation.

Returns:

Deep copy of the object’s attributes as a dictionary.

to_api_repr() → dict[str, Any]

Convert the object metadata to S3 API request parameters.

Returns:

Dictionary keyed by S3 API field names (e.g., ContentType) of the fields that are set, excluding ETag, ContentLength and LastModified.

classmethod from_summary(summary: S3ObjectSummary) → S3Object

Build the file entry of an object listed by ListObjectsV2.

Parameters:

summary – The listed object.

Returns:

The file entry, named bucket/key.

classmethod from_version(version: S3ObjectVersion) → S3Object

Build the file entry of a version listed by ListObjectVersions.

The entry is named bucket/key?versionId=<id> so that the version can be addressed, except for the null version, which a write to the key replaces and which is named bucket/key.

Parameters:

version – The listed version.

Returns:

The file entry.

classmethod from_metadata(metadata: S3Metadata, version_id: str | None = None) → S3Object

Build the file entry of an object looked up with HeadObject.

The fields that the response does not have are left out, except Metadata, which botocore always returns.

Parameters:

• metadata – The metadata of the object, with the looked up path.

• version_id – The version ID of the entry, which may be pinned or omitted apart from the looked up path.

Returns:

The file entry, named bucket/key.

Raises:

ValueError – If the metadata has no object path.

classmethod from_bucket(bucket: S3Bucket) → S3Object

Build the directory entry of a bucket.

Parameters:

bucket – The bucket.

Returns:

The directory entry, named after the bucket.

class pyathena.filesystem.s3_object.S3ObjectType

Constants for S3 object types in filesystem operations.

These constants are used to distinguish between directories and files when working with S3 paths through the S3FileSystem interface.

S3_OBJECT_TYPE_DIRECTORY: str = 'directory'

S3_OBJECT_TYPE_FILE: str = 'file'

class pyathena.filesystem.s3_object.S3StorageClass

Constants for Amazon S3 storage classes.

S3 storage classes determine the availability, durability, and cost characteristics of stored objects. Each class is optimized for different access patterns and use cases.

Storage classes:

• STANDARD: Default storage for frequently accessed data

• REDUCED_REDUNDANCY: Lower cost, reduced durability (deprecated)

• STANDARD_IA: Infrequently accessed data with rapid retrieval

• ONEZONE_IA: Lower cost IA storage in single availability zone

• INTELLIGENT_TIERING: Automatic tiering between frequent/infrequent

• GLACIER: Archive storage for long-term backup

• DEEP_ARCHIVE: Lowest cost archive storage

• GLACIER_IR: Archive with faster retrieval than standard Glacier

• OUTPOSTS: Storage on AWS Outposts

• BUCKET: Pseudo storage class PyAthena assigns to bucket entries

• DIRECTORY: Pseudo storage class PyAthena assigns to directory entries

See also

AWS S3 storage classes documentation: https://docs.aws.amazon.com/s3/latest/userguide/storage-class-intro.html

S3_STORAGE_CLASS_STANDARD: str = 'STANDARD'

S3_STORAGE_CLASS_REDUCED_REDUNDANCY: str = 'REDUCED_REDUNDANCY'

S3_STORAGE_CLASS_STANDARD_IA: str = 'STANDARD_IA'

S3_STORAGE_CLASS_ONEZONE_IA: str = 'ONEZONE_IA'

S3_STORAGE_CLASS_INTELLIGENT_TIERING: str = 'INTELLIGENT_TIERING'

S3_STORAGE_CLASS_GLACIER: str = 'GLACIER'

S3_STORAGE_CLASS_DEEP_ARCHIVE: str = 'DEEP_ARCHIVE'

S3_STORAGE_CLASS_OUTPOSTS: str = 'OUTPOSTS'

S3_STORAGE_CLASS_GLACIER_IR: str = 'GLACIER_IR'

S3_STORAGE_CLASS_BUCKET: str = 'BUCKET'

S3_STORAGE_CLASS_DIRECTORY: str = 'DIRECTORY'

class pyathena.filesystem.s3_object.S3Metadata(response: dict[str, Any], path: S3Path | None = None)

Represents the metadata of an S3 object as returned by HeadObject.

Behaves as a read-only mapping of the user-defined metadata (x-amz-meta-*, whose keys are arbitrary user-chosen strings), so it is a drop-in for implementations that return the user-defined metadata as a plain dictionary, and compares equal to such dictionaries. The system-defined metadata (content type, encryption settings, storage class, etc.) is exposed as typed properties.

Example

>>> metadata = fs.metadata("s3://bucket/key")
>>> metadata["attr1"]  # user-defined metadata
'value1'
>>> metadata.content_type  # system-defined metadata
'text/plain'

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html

__init__(response: dict[str, Any], path: S3Path | None = None) → None

Initialize the metadata from a HeadObject response.

Parameters:

• response – The HeadObject response.

• path – The path that was looked up, if known.

property cache_control: str | None

The CacheControl header of the object.

property content_disposition: str | None

The ContentDisposition header of the object.

property content_encoding: str | None

The ContentEncoding header of the object.

property content_language: str | None

The ContentLanguage header of the object.

property content_length: int | None

The ContentLength of the object in bytes.

property content_type: str | None

The ContentType of the object.

property etag: str | None

The ETag (entity tag) of the object.

property expiration: str | None

The Expiration header of the object.

property expires: datetime | None

The Expires date of the object.

property last_modified: datetime | None

The LastModified time of the object.

property storage_class: str

The StorageClass of the object; STANDARD when S3 omits it.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the object.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the object.

property bucket_key_enabled: bool | None

Whether the object uses an S3 Bucket Key (BucketKeyEnabled).

property website_redirect_location: str | None

The WebsiteRedirectLocation of the object.

property version_id: str | None

The VersionId of the object.

property path: S3Path | None

The path that was looked up, or None if it is not known.

property object_lock_mode: str | None

The ObjectLockMode of the object.

property object_lock_retain_until_date: datetime | None

The ObjectLockRetainUntilDate of the object.

property object_lock_legal_hold_status: str | None

The ObjectLockLegalHoldStatus of the object.

property user_metadata: dict[str, str]

A copy of the user-defined metadata (x-amz-meta-*).

The keys are arbitrary user-chosen strings, returned as stored in S3 (S3 normalizes them to lowercase). The same key/value pairs are also accessible directly through the mapping interface of this class.

class pyathena.filesystem.s3_object.S3ObjectVersion(bucket: str, is_delete_marker: bool, response: dict[str, Any])

Represents a version of an S3 object as returned by ListObjectVersions.

bucket

S3 bucket name.

key

Object key.

version_id

The version ID of the object.

is_latest

Whether the version is the latest version of the object.

is_delete_marker

Whether the version is a delete marker.

last_modified

Date and time when the version was last modified.

etag

Entity tag of the version. None for delete markers.

size

Size in bytes of the version. None for delete markers.

storage_class

Storage class of the version. None for delete markers.

owner

Owner of the version.

__init__(bucket: str, is_delete_marker: bool, response: dict[str, Any]) → None

Initialize the version from a ListObjectVersions entry.

Parameters:

• bucket – The name of the bucket that contains the version.

• is_delete_marker – Whether the entry comes from DeleteMarkers rather than Versions.

• response – A Versions or DeleteMarkers entry of the ListObjectVersions response.

property bucket: str

The name of the bucket that contains the version.

property key: str

The Key of the object.

property name: str

The path of the version in bucket/key form.

property version_id: str | None

The VersionId of the version.

property is_latest: bool

Whether the version is the latest version of the object (IsLatest).

property is_delete_marker: bool

Whether the version is a delete marker.

property last_modified: datetime | None

The LastModified time of the version.

property etag: str | None

The ETag of the version; None for delete markers.

property size: int | None

The Size of the version in bytes; None for delete markers.

property storage_class: str | None

The StorageClass of the version; None for delete markers.

property owner: S3Owner | None

The Owner of the version, or None if the response has none.

S3 Upload Operations

class pyathena.filesystem.s3_object.S3PutObject(response: dict[str, Any])

Represents the response from an S3 PUT object operation.

This class encapsulates the metadata returned when uploading an object to S3, including encryption details, versioning information, and integrity checksums.

expiration

Object expiration time if lifecycle policy applies.

version_id

Version ID if bucket versioning is enabled.

etag

Entity tag for the uploaded object.

server_side_encryption

Server-side encryption method used.

Various checksum properties

For data integrity verification.

Note

This class is used internally by S3FileSystem operations and typically not instantiated directly by users.

__init__(response: dict[str, Any]) → None

Initialize the result from a PutObject response.

Parameters:

response – The PutObject response.

property expiration: str | None

The Expiration header of the uploaded object.

property version_id: str | None

The VersionId of the uploaded object.

property etag: str | None

The ETag of the uploaded object.

property checksum_crc32: str | None

The ChecksumCRC32 of the uploaded object.

property checksum_crc32c: str | None

The ChecksumCRC32C of the uploaded object.

property checksum_sha1: str | None

The ChecksumSHA1 of the uploaded object.

property checksum_sha256: str | None

The ChecksumSHA256 of the uploaded object.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the uploaded object.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the uploaded object.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the uploaded object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the uploaded object.

property sse_kms_encryption_context: str | None

The SSEKMSEncryptionContext of the uploaded object.

property bucket_key_enabled: bool | None

Whether the uploaded object uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_dict() → dict[str, Any]

Convert the response to a dictionary.

Returns:

Deep copy of the instance attributes, keyed by their attribute names (e.g., _etag).

class pyathena.filesystem.s3_object.S3MultipartUpload(response: dict[str, Any])

Represents an S3 multipart upload operation.

This class manages the metadata for multipart uploads, which allow uploading large files in chunks for better reliability and performance. It tracks upload identifiers, encryption settings, and lifecycle rules.

bucket

S3 bucket name for the upload.

key

Object key being uploaded.

upload_id

Unique identifier for the multipart upload.

server_side_encryption

Encryption method applied to the upload.

abort_date/abort_rule_id

Lifecycle rule information for upload cleanup.

initiated/storage_class/owner/initiator

Fields returned by the ListMultipartUploads API for in-progress uploads.

Note

Used internally by S3FileSystem for large file upload operations, and returned by S3FileSystem.list_multipart_uploads.

__init__(response: dict[str, Any]) → None

Initialize the upload from an S3 API response.

Parameters:

response – A CreateMultipartUpload response or an Uploads entry of a ListMultipartUploads response.

property abort_date: datetime | None

The AbortDate of the upload set by a lifecycle rule.

property abort_rule_id: str | None

The AbortRuleId of the lifecycle rule that applies to the upload.

property bucket: str | None

The Bucket of the upload.

property key: str | None

The Key of the object being uploaded.

property upload_id: str | None

The UploadId of the multipart upload.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the upload.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the upload.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the upload.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the upload.

property sse_kms_encryption_context: str | None

The SSEKMSEncryptionContext of the upload.

property bucket_key_enabled: bool | None

Whether the upload uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

property checksum_algorithm: str | None

The ChecksumAlgorithm of the upload.

property checksum_type: str | None

COMPOSITE or FULL_OBJECT.

Type:

The ChecksumType of the upload

property initiated: datetime | None

The Initiated time of the upload, returned by ListMultipartUploads.

property storage_class: str | None

The StorageClass of the upload, returned by ListMultipartUploads.

property owner: S3Owner | None

The Owner of the upload, returned by ListMultipartUploads.

property initiator: S3Owner | None

The Initiator of the upload, returned by ListMultipartUploads.

class pyathena.filesystem.s3_object.S3MultipartUploadPart(part_number: int, response: dict[str, Any])

Represents a single part in an S3 multipart upload operation.

Each part in a multipart upload has its own metadata including checksums, encryption details, and part identification. This class manages that metadata and provides methods to convert it to API-compatible formats.

part_number

The sequential part number (1-based).

etag

Entity tag for this specific part.

checksum_*

Various integrity checksums for the part data.

server_side_encryption

Encryption settings for this part.

Note

Parts must be at least 5MB except for the last part. Used internally by S3FileSystem for chunked upload operations.

__init__(part_number: int, response: dict[str, Any]) → None

Initialize the part from an UploadPart or UploadPartCopy response.

For an UploadPartCopy response, the ETag, LastModified and checksums are read from its CopyPartResult.

Parameters:

• part_number – The part number of the part.

• response – The UploadPart or UploadPartCopy response.

property part_number: int

The part number of the part.

property copy_source_version_id: str | None

The CopySourceVersionId of the source object of a copied part.

property last_modified: datetime | None

The LastModified time from CopyPartResult; None for uploaded parts.

property etag: str | None

The ETag of the part.

property checksum_crc32: str | None

The ChecksumCRC32 of the part.

property checksum_crc32c: str | None

The ChecksumCRC32C of the part.

property checksum_sha1: str | None

The ChecksumSHA1 of the part.

property checksum_sha256: str | None

The ChecksumSHA256 of the part.

property checksum_crc64nvme: str | None

The ChecksumCRC64NVME of the part.

property checksum_sha512: str | None

The ChecksumSHA512 of the part.

property checksum_md5: str | None

The ChecksumMD5 of the part.

property checksum_xxhash64: str | None

The ChecksumXXHASH64 of the part.

property checksum_xxhash3: str | None

The ChecksumXXHASH3 of the part.

property checksum_xxhash128: str | None

The ChecksumXXHASH128 of the part.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the part.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the part.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the part.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the part.

property bucket_key_enabled: bool | None

Whether the part uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_api_repr() → dict[str, Any]

Convert the part to a part entry of a CompleteMultipartUpload request.

Returns:

Dictionary with the ETag, checksum and PartNumber fields of the part, omitting fields whose value is None.

class pyathena.filesystem.s3_object.S3CompleteMultipartUpload(response: dict[str, Any])

Represents the completion of an S3 multipart upload operation.

This class encapsulates the final response when a multipart upload is completed, including the final object location, versioning information, and consolidated metadata from all parts.

location

Final S3 URL of the completed object.

bucket

S3 bucket containing the object.

key

Final object key.

version_id

Version ID if bucket versioning is enabled.

etag

Final entity tag of the complete object.

server_side_encryption

Encryption applied to the final object.

Note

This represents the successful completion of a multipart upload. Used internally by S3FileSystem operations.

__init__(response: dict[str, Any]) → None

Initialize the result from a CompleteMultipartUpload response.

Parameters:

response – The CompleteMultipartUpload response.

property location: str | None

The Location URI of the completed object.

property bucket: str | None

The Bucket of the completed object.

property key: str | None

The Key of the completed object.

property expiration: str | None

The Expiration header of the completed object.

property version_id: str | None

The VersionId of the completed object.

property etag: str | None

The ETag of the completed object.

property checksum_crc32: str | None

The ChecksumCRC32 of the completed object.

property checksum_crc32c: str | None

The ChecksumCRC32C of the completed object.

property checksum_sha1: str | None

The ChecksumSHA1 of the completed object.

property checksum_sha256: str | None

The ChecksumSHA256 of the completed object.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the completed object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the completed object.

property bucket_key_enabled: bool | None

Whether the completed object uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_dict()

Convert the response to a dictionary.

Returns:

Deep copy of the instance attributes, keyed by their attribute names (e.g., _etag).

Multipart Writer

class pyathena.filesystem.s3_writer.S3MultipartWriter(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None)

Plan parts and send the requests of a multipart write.

The caller owns the buffer, schedules requests, collects part results in number order, and waits for running requests before completing or aborting. Initiation and finalization must be serialized; part requests may run in parallel after initiation. No request is sent by the constructor or planners.

upload

The identity returned by creation, retained after completion and after an abort failure. A successful abort clears it.

__init__(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None) → None

Initialize a writer without starting its upload.

Parameters:

• core – The synchronous S3 operations to use.

• path – The destination, with a key and without a version ID.

• block_size – The nominal part size, between the core’s minimum and maximum multipart part sizes, inclusive.

• request_kwargs – Parameters inherited by each request that accepts them. A copy is kept; parameters given to a request override it.

Raises:

ValueError – If the destination or block size is invalid.

iter_parts(stream: BinaryIO, first_part_number: int = 1) → Iterator[tuple[int, bytes]]

Plan numbered parts from the stream’s current position to EOF.

A short trailing block is merged into the preceding block, splitting the result in half if it reaches the maximum part size. This keeps every part large enough for later writes, unless the entire stream is shorter than the minimum part size and must be the final part. Reads are bounded by the block size; the stream is not rewound.

Parameters:

• stream – A blocking binary stream of buffered data.

• first_part_number – The first number, including any earlier copied or uploaded parts in the caller’s count.

Yields:

The part number and bytes, in order, without sending requests.

Raises:

ValueError – If a generated part number is outside the S3 limit.

iter_copy_parts(size: int, first_part_number: int = 1) → Iterator[tuple[int, tuple[int, int]]]

Plan numbered copy ranges, with an exclusive end, without requests.

Parameters:

• size – The nonnegative size of the existing object in bytes.

• first_part_number – The first part number to use.

Yields:

The part number and source range, in order. The ranges use the core’s maximum part size and keep a short tail within S3 limits so that appended parts may follow, unless the whole source is smaller than the minimum part size and must be the final part.

Raises:

ValueError – If the size is negative or a part number exceeds the limit.

initiate(**params: Any) → S3MultipartUpload

Create the upload, or return its already known identity.

Parameters:

**params – Parameters accepted by CreateMultipartUpload.

Returns:

The upload, stored before this method returns so that a caller can recover it after an interrupted wait for a scheduled request.

upload_part(part_number: int, body: bytes, **params: Any) → S3MultipartUploadPart

Send an UploadPart request for the initialized upload.

Parameters:

• part_number – The part number, between 1 and the core’s maximum.

• body – The bytes to upload, with the size selected by the caller.

• **params – Parameters accepted by UploadPart.

Returns:

The uploaded part, retaining its number and checksums.

Raises:

• RuntimeError – If no upload is initialized.

• ValueError – If the part number is outside the S3 limit.

upload_part_copy(part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params: Any) → S3MultipartUploadPart

Send an UploadPartCopy request for the initialized upload.

Parameters:

• part_number – The part number, between 1 and the core’s maximum.

• source – The source object, with an optional version ID.

• range – The source range with an exclusive end, or None to send no range of its own, as in S3Core.upload_part_copy().

• **params – Parameters accepted by UploadPartCopy.

Returns:

The copied part, retaining its number and checksums.

Raises:

• RuntimeError – If no upload is initialized.

• ValueError – If the part number is outside the S3 limit.

complete(parts: Sequence[S3MultipartUploadPart], **params: Any) → S3CompleteMultipartUpload

Complete the upload after the caller has collected all its parts.

Parameters:

• parts – Successfully uploaded parts in part-number order.

• **params – Parameters accepted by CompleteMultipartUpload.

Returns:

The completion result. The upload identity is retained.

Raises:

RuntimeError – If no upload is initialized. Request failures propagate; the caller must settle running work before aborting.

abort(**params: Any) → None

Abort a known upload after the caller has settled running requests.

Does nothing if no upload is known. A successful abort clears the identity; a failed or interrupted abort retains it for a later retry.

Parameters:

**params – Parameters accepted by AbortMultipartUpload.