File System Integration

This section covers S3 filesystem integration and object management.

S3 FileSystem

pyathena.filesystem.s3_filesystem(connection: Connection[Any], **kwargs: Any) → S3FileSystem[source]
pyathena.filesystem.s3_filesystem(connection: Connection[Any], filesystem_class: type[_FileSystemT], **kwargs: Any) → _FileSystemT

Create an S3 filesystem with the S3 client and retry policy of a connection.

The filesystem borrows connection.s3_client, which the connection shares with its result sets, and uses connection.retry_config. A policy with the default exceptions also retries the transient S3 error codes (RETRY_ERROR_CODES). Closing the filesystem does not close the client. A filesystem created this way cannot be pickled; see S3FileSystem.

Example

>>> from pyathena import connect
>>> from pyathena.filesystem import s3_filesystem
>>> fs = s3_filesystem(connect(s3_staging_dir="s3://my-bucket/path/"))
Parameters:
  • connection – The PyAthena connection.

  • filesystem_class – S3FileSystem, AioS3FileSystem, or a subclass of either.

  • **kwargs – The other arguments of the filesystem, except the arguments from which it builds a client.

Returns:

The filesystem.

class pyathena.filesystem.s3.S3FileSystem(*args, **kwargs)[source]

A filesystem interface for Amazon S3 that implements the fsspec protocol.

This class provides a file-system like interface to Amazon S3, allowing you to use familiar file operations (ls, open, cp, rm, etc.) with S3 objects. It’s designed to be compatible with s3fs while offering PyAthena-specific optimizations.

The filesystem supports standard S3 operations including:

  • Listing objects and directories

  • Reading and writing files

  • Copying and moving objects

  • Reading and writing object metadata, tags, and canned ACLs

  • Multipart uploads for large files, including management of incomplete uploads

  • Version-aware reads and object version listing (see version_aware)

  • Creating and removing buckets (disabled by default; see allow_bucket_creation / allow_bucket_deletion)

  • Various S3 storage classes and encryption options

  • Translating S3 error responses into standard Python exceptions (e.g., 404 -> FileNotFoundError, 403 -> PermissionError)

allow_bucket_creation

Whether mkdir/makedirs may create buckets. Defaults to False.

allow_bucket_deletion

Whether rmdir may delete buckets. Defaults to False.

version_aware

Whether reads pin the object version observed at open time and ls may list all versions. Requires the s3:GetObjectVersion / s3:ListBucketVersions permissions. Defaults to False.

Example

>>> from pyathena.filesystem.s3 import S3FileSystem
>>> fs = S3FileSystem()
>>>
>>> # List objects in a bucket
>>> files = fs.ls('s3://my-bucket/data/')
>>>
>>> # Read a file
>>> with fs.open('s3://my-bucket/data/file.csv', 'r') as f:
...     content = f.read()
>>>
>>> # Write a file
>>> with fs.open('s3://my-bucket/output/result.txt', 'w') as f:
...     f.write('Hello, S3!')
>>>
>>> # Copy files
>>> fs.cp('s3://source-bucket/file.txt', 's3://dest-bucket/file.txt')

Note

This filesystem is used internally by PyAthena for handling query results stored in S3, but can also be used independently for S3 file operations.

DEFAULT_BLOCK_SIZE: int = 5242880
protocol: str | tuple[str, ...] = ('s3', 's3a')
__init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int | None = None, s3_additional_kwargs=None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, *args, batch_size: int | None = None, client: BaseClient | None = None, retry_config: RetryConfig | None = None, **kwargs) → None[source]

Create a filesystem for Amazon S3.

The filesystem sends its requests with the boto3 S3 client given as client, or with a client that it builds from s3fs-compatible arguments in kwargs. pyathena.filesystem.s3_filesystem() creates one with the S3 client and retry policy of a PyAthena connection.

Parameters:
  • connection – Deprecated; use client and retry_config, or pyathena.filesystem.s3_filesystem(). A PyAthena connection whose S3 client (Connection.s3_client) and retry policy the filesystem uses, with a DeprecationWarning.

  • default_block_size – The block size for reads and writes; defaults to DEFAULT_BLOCK_SIZE.

  • default_cache_type – The fsspec cache type for reads, the name of a cache in fsspec.caching.caches; defaults to "bytes". The available names depend on the installed fsspec version. Common ones are "none", which keeps nothing and requests the bytes of each read; "bytes", which reads one block beyond each read and keeps one contiguous range; "readahead", which reads one block beyond each read and keeps only the last range read; "blockcache", which requests whole blocks and keeps the most recently used ones; and "background", which is like "blockcache" and also reads the next block in a thread.

  • max_workers – The number of threads of the request pool of the filesystem, which its files and parallel operations share, and the default number of parallel requests of a file. If None, the max_pool_connections of the S3 client (10 unless configured). A client that the filesystem builds gets max_pool_connections from max_workers unless config_kwargs sets it; the connection pool of a given client keeps its size.

  • s3_additional_kwargs – Extra arguments for the object requests of open(), pipe_file() and put_file(). Their authorization parameters (RequestPayer, ExpectedBucketOwner and SSECustomer*) are sent with every request, listings and copies included; a copy also sends them for its source (ExpectedSourceBucketOwner and CopySourceSSECustomer*). Each request receives those that its operation accepts, and the parameters of a call take precedence.

  • allow_bucket_creation – Whether mkdir/makedirs may create a bucket.

  • allow_bucket_deletion – Whether rmdir may delete a bucket.

  • version_aware – Whether reads pin the object version observed at open time.

  • *args – Passed to fsspec.AbstractFileSystem.

  • batch_size – The number of per-object calls of a batch operation (cat, get, put, copy, and the like) that run at once, and of DeleteObjects requests of rm, unless the call gives its own. None or -1 is max_workers.

  • client – The boto3 S3 client that the filesystem sends its requests with. The filesystem does not close it. A filesystem with a given client cannot be pickled.

  • retry_config – The retry policy of the requests; defaults to RetryConfig(). A policy with the default exceptions also retries the transient S3 error codes (RETRY_ERROR_CODES).

  • **kwargs – Passed to fsspec.AbstractFileSystem; without client or connection, also s3fs-compatible client arguments. requester_pays=True sends requester-pays requests with the operations that accept RequestPayer.

Raises:
  • ValueError – If batch_size is neither positive, -1, nor None.

  • TypeError – If connection is given with client or retry_config, or client or connection with an argument from which the filesystem builds a client, such as key, profile, anon, endpoint_url, client_kwargs, or region_name, other than one that fsspec’s configuration supplies.

__reduce__() → tuple[Any, ...][source]

Reduce the filesystem for pickling, as fsspec does.

Returns:

The reduction of fsspec.AbstractFileSystem, which creates the filesystem again from its arguments.

Raises:

TypeError – If the filesystem was built with client or connection, whose boto3 client cannot be pickled.

property core: S3Core

The typed S3 operations that the filesystem sends its requests with.

property request_kwargs: dict[str, Any]

The parameters sent with every operation that accepts them.

{"RequestPayer": "requester"} with requester_pays=True, and the authorization parameters of s3_additional_kwargs, also under the names of a copy source; see core.request_kwargs.

close() → None[source]

Release the threads of the request pool, which stays usable, once they are idle.

See S3ThreadPool.release(). The calls in progress, including the requests of open files, run to their end, and a call after the release creates new threads.

__enter__() → S3FileSystem[source]

Return the filesystem, whose threads the end of the block releases.

ls(path: str | PathLike[str], detail: bool = False, refresh: bool = False, **kwargs) → list[S3Object] | list[str][source]

List contents of an S3 path.

Lists buckets (when path is root) or objects within a bucket/prefix. Compatible with fsspec interface for filesystem operations.

Parameters:
  • path – S3 path to list (e.g., “s3://bucket” or “s3://bucket/prefix”).

  • detail – If True, return S3Object instances; if False, return paths as strings.

  • refresh – If True, bypass cache and fetch fresh results from S3.

  • **kwargs –

    Additional arguments including: versions: If True, list all versions of the objects, named

    bucket/key?versionId=<id> except for the null version. Requires the filesystem to be constructed with version_aware=True.

Returns:

List of S3Object instances (if detail=True) or paths as strings (if detail=False).

Raises:

FileNotFoundError – If the bucket does not exist, or if a key path is neither an object nor a directory, a path that some key starts with followed by /, or if HeadObject reports the version that a key path with a version ID names missing.

Example

>>> fs = S3FileSystem()
>>> fs.ls("s3://my-bucket")  # List objects in bucket
>>> fs.ls("s3://my-bucket/", detail=True)  # Get detailed object info
info(path: str | PathLike[str] | S3Path, **kwargs) → S3Object[source]

Return information about an S3 path.

Uses the directory cache first: a cached entry for the path is returned, the entry of the path in a cached listing of its parent is returned, preferring an object to a key prefix of the same name as HeadObject does, and a cached listing of its parent without it means it does not exist. The cached bucket listing holds only the buckets that the caller owns, so a bucket missing from it is looked up with HeadBucket. Otherwise, a key path is looked up with HeadObject and, if no object exists, with a ListObjectsV2 request (Delimiter="/", MaxKeys=1) that checks whether it is a key prefix; a bucket path is looked up with HeadBucket. If these requests find a listed object missing, or find a key prefix, the cached listing of the parent is removed. With version_aware, a cached file entry without a version ID is looked up again. With an explicit version, the cached entries of the path are skipped, the HeadObject result is cached under the name and the version ID, apart from the key and other versions, except for the null version, which an overwrite replaces, and a missing version is not looked up as a key prefix, since a version names an object. With request parameters on which the authorization of the requests depends (ExpectedBucketOwner, RequestPayer, and the SSECustomer* parameters of an object encrypted with a customer-provided key), each request receives those that its operation accepts, and only the cached HeadObject or HeadBucket results of lookups with the same values of these parameters are used.

Parameters:
  • path – S3 path (e.g., “s3://bucket” or “s3://bucket/key”), or an S3Path, whose bucket, key and version are looked up as they are, such as a key that a listing returns.

  • **kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. version_id: The version ID to look up when the path has none. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters described above. Other request parameters are ignored.

Returns:

S3Object describing the bucket, directory, or file. The root path ("", "/" or "s3://") is a directory.

Raises:

FileNotFoundError – If the path does not exist.

find(path: str | PathLike[str], maxdepth: int | None = None, withdirs: bool | None = None, detail: bool = False, **kwargs) → dict[str, S3Object] | list[str][source]

Find all files below a given S3 path.

Recursively searches for files under the specified path, with optional depth limiting and directory inclusion. Uses efficient S3 list operations with delimiter handling for performance. As in fsspec, the result includes the path itself if withdirs is True and objects exist below it, unless it is a bucket, or if it is an object and nothing is listed below it. A key path with a version ID names that version of the object: the result is the version, named as ls(versions=True) names it, or empty if HeadObject reports it missing.

Parameters:
  • path – S3 path to search under (e.g., “s3://bucket/prefix”).

  • maxdepth – Maximum number of levels to descend, at least 1 (None for unlimited). With 1, only the entries directly under the path are listed.

  • withdirs – Whether to include directories in results (None = default behavior).

  • detail – If True, return dict of {path: S3Object}; if False, return list of paths.

  • **kwargs –

    Additional arguments including: prefix: Key prefix, relative to the path, to filter the listed keys

    by. Each slash in the prefix counts as one level of maxdepth. With withdirs, the directories above the prefix, such as sub for sub/deep/, are not included.

    refresh: If True, bypass the cache and list from S3.

Returns:

Dictionary mapping paths to S3Objects (if detail=True) or list of paths (if detail=False).

Raises:

ValueError – If maxdepth is less than 1 or the path is the root.

Example

>>> fs = S3FileSystem()
>>> fs.find("s3://bucket/data/", maxdepth=2)  # Limit depth
>>> fs.find("s3://bucket/", withdirs=True)    # Include directories
walk(path, maxdepth=None, topdown=True, on_error='omit', **kwargs)[source]

Walk the directories below a path, as fsspec’s walk() does.

Unlike fsspec, a listed directory is listed by its key prefix rather than by its name (see _directory_prefix()), so a directory whose name ends in a version ID query or in an empty segment is walked as it was listed. Its name in the listing of its parent is the part of its name after the parent’s name, such as "" for the directory bucket/dir/ of the keys dir//x in bucket/dir, and it is walked under its name as listed.

Parameters:
  • path – The path to walk.

  • maxdepth – Maximum number of levels to walk, at least 1 (None for unlimited).

  • topdown – Whether to yield a directory before the directories below it. When True, removing names from the yielded directory names skips their directories.

  • on_error – For the listing of the path, "omit" to yield nothing if it fails, "raise" to raise its error, or a callable that receives the error. As in fsspec, a directory below the path whose listing fails is skipped.

  • **kwargs – Additional arguments passed to ls(), such as refresh and versions; detail=True yields the entries by name instead of the names.

Yields:

Tuples of a directory path, its directories, and its files.

Raises:

ValueError – If maxdepth is less than 1.

du(path, total=True, maxdepth=None, withdirs=False, **kwargs)[source]

Return the sizes of the objects below a path, as fsspec’s du() does.

Unlike fsspec, each entry that find() returns is looked up by its bucket and key (see _entry_path()), never by parsing its name.

Parameters:
  • path – The path to search under.

  • total – Whether to return the sum of the sizes instead of the sizes by name.

  • maxdepth – Maximum number of levels to descend, at least 1 (None for unlimited).

  • withdirs – Whether to include the directories, and the path itself if it is a directory.

  • **kwargs – Additional arguments passed to find().

Returns:

The total size, or the sizes by the names that info() gives.

glob(path, maxdepth=None, **kwargs)[source]

Find the paths that match a glob pattern.

As fsspec’s glob(), except that only the keys that start with the part of the last path component before the first wildcard are listed, as with fsspec’s asynchronous _glob().

Parameters:
  • path – S3 glob pattern, as a string or an os.PathLike.

  • maxdepth – Maximum depth of the paths below the literal part of the pattern, at least 1.

  • **kwargs – Additional arguments passed to find, such as detail.

Returns:

The matching paths, or a dictionary of them and their objects with detail=True.

tree(path: str | PathLike[str] = '/', recursion_limit: int = 2, max_display: int = 25, display_size: bool = False, prefix: str = '', is_last: bool = True, first: bool = True, indent_size: int = 4) → str[source]

Return a tree of the paths below a path, as fsspec’s tree() does.

Unlike fsspec, a listed directory is listed by its key prefix rather than by its name, as in walk().

Parameters:
  • path – The root path, as a string or an os.PathLike.

  • recursion_limit – Maximum depth of the tree.

  • max_display – Maximum number of entries shown per directory.

  • display_size – Whether to show the sizes of the files.

  • prefix – The prefix of each line.

  • is_last – Whether the path is the last entry of its directory.

  • first – Whether the path is the root of the tree.

  • indent_size – Number of spaces per level.

Returns:

The tree as a string.

get_mapper(root: str | PathLike[str] = '', check: bool = False, create: bool = False, missing_exceptions=None)[source]

Return a key/value store at a path, as fsspec’s get_mapper() does.

Parameters:
  • root – The root path of the store, as a string or an os.PathLike.

  • check – Whether to check that the root is writable.

  • create – Whether to create the root if it does not exist.

  • missing_exceptions – The exceptions that mean a missing key.

Returns:

The fsspec.mapping.FSMap of the root.

unstrip_protocol(name: str | PathLike[str]) → str[source]

Add the protocol to a path, as fsspec’s unstrip_protocol() does.

Parameters:

name – The path, as a string or an os.PathLike.

Returns:

The path with the s3:// protocol.

expand_path(path, recursive=False, maxdepth=None, **kwargs) → list[str][source]

Expand glob patterns and directories into the paths they match.

As in fsspec, except that a path with a version ID names that version of an object: it is not a glob pattern, although its ? is one in fsspec, nor is anything expanded below it. With recursive, it is included only if the version exists, as fsspec includes a path that exists.

Parameters:
  • path – S3 path, glob pattern, or list of them.

  • recursive – Whether to include the paths below the directories.

  • maxdepth – Maximum depth of the expansion, at least 1.

  • **kwargs – Additional arguments passed to glob and find.

Returns:

The sorted matching paths.

Raises:
exists(path: str | PathLike[str], **kwargs) → bool[source]

Check if an S3 path exists.

Determines whether a bucket, object, or prefix exists in S3. Uses caching and efficient head operations to minimize API calls.

Parameters:
  • path – S3 path to check (e.g., “s3://bucket” or “s3://bucket/key”).

  • **kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters on which the authorization of the requests depends, as described in info(). With them, cached listings are not used. Other request parameters are ignored.

Returns:

True if the path exists, False otherwise. A bucket that HeadBucket denies access to (403) exists.

Raises:

PermissionError – If the lookup of an object is denied (403). Without s3:ListBucket on the bucket, S3 also answers 403 for a key that does not exist.

Example

>>> fs = S3FileSystem()
>>> fs.exists("s3://my-bucket/file.txt")
>>> fs.exists("s3://my-bucket/")
rm_file(path: str | PathLike[str], **kwargs) → None[source]

Delete an S3 object with DeleteObject.

Does nothing for a bucket path. If the path has a version ID, that version is deleted.

Parameters:
  • path – S3 path (s3://bucket/key) of the object to delete.

  • **kwargs – Additional parameters passed to the DeleteObject API, such as RequestPayer. Those that it does not accept are ignored, for fsspec compatibility.

Raises:

TypeError – If kwargs has bucket, key or version_id, or Bucket, Key or VersionId, which the path gives.

rm(path, recursive=False, maxdepth=None, batch_size=None, **kwargs) → None[source]

Delete objects with DeleteObjects requests.

Expands the paths with expand_path and deletes the matched objects in parallel requests of up to S3DeleteBatch.MAX_KEYS keys each, one set of requests per bucket. A path with a version ID deletes that version without expansion.

Parameters:
  • path – S3 path (s3://bucket/key) or list of paths to delete.

  • recursive – Whether to delete all objects below the paths.

  • maxdepth – Maximum depth to expand when recursive is True.

  • batch_size – Number of DeleteObjects requests to send at the same time. By default the batch_size of the filesystem; None or -1 there is max_workers. max_workers is a deprecated alias.

  • **kwargs – Additional parameters passed to the DeleteObjects API. Quiet (default True) sets the quiet mode of the requests.

Raises:
  • ValueError – If a path is a bucket, or batch_size is neither positive, -1, nor None.

  • TypeError – If both batch_size and max_workers are given.

  • OSError – If S3 could not delete some of the objects.

mkdir(path: str | PathLike[str], create_parents: bool = True, **kwargs) → None[source]

Create an S3 bucket.

S3 has no real directories below the bucket level; creating a key prefix requires no operation. This method creates the bucket when the path points at a bucket (or when create_parents is True and the bucket does not exist yet), and does nothing for key prefixes under an existing bucket.

Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass allow_bucket_creation=True to the filesystem constructor to enable bucket creation.

Parameters:
  • path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).

  • create_parents – If True, create the bucket when it does not exist, even if the path contains a key prefix.

  • **kwargs –

    Additional arguments including: acl: Canned ACL to apply to the bucket. region_name: Region to create the bucket in. Defaults to the

    client’s region.

    Other parameters that the CreateBucket API accepts, such as ObjectLockEnabledForBucket, are passed to it; the others are ignored, for fsspec compatibility.

Raises:
  • FileExistsError – If the path is a bucket that already exists.

  • FileNotFoundError – If the bucket does not exist and create_parents is False.

  • PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.

  • ValueError – If the ACL is invalid, the path is empty, or botocore rejects the parameters of the CreateBucket request.

makedirs(path: str | PathLike[str], exist_ok: bool = False) → None[source]

Recursively create a directory, creating the bucket if necessary.

Creating the bucket requires allow_bucket_creation=True on the filesystem constructor; see mkdir().

Parameters:
  • path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).

  • exist_ok – If False, raise FileExistsError when the path is a bucket that already exists.

Raises:
  • FileExistsError – If the path is a bucket that already exists and exist_ok is False.

  • PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.

rmdir(path: str | PathLike[str]) → None[source]

Remove an S3 bucket, which must be empty.

S3 has no real directories below the bucket level, so only bucket paths can be removed.

Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass allow_bucket_deletion=True to the filesystem constructor to enable bucket deletion.

Parameters:

path – S3 bucket path (e.g., “s3://bucket”).

Raises:
  • FileExistsError – If the path contains a key that exists. The user may have meant rm(path, recursive=True).

  • FileNotFoundError – If the path contains a key that does not exist, or the bucket does not exist.

  • PermissionError – If bucket deletion is not enabled on this filesystem instance.

  • OSError – If the bucket is not empty.

touch(path: str | PathLike[str], truncate: bool = True, **kwargs) → dict[str, Any][source]

Create an empty object with PutObject.

Parameters:
  • path – S3 path (s3://bucket/key) of the object.

  • truncate – If True, replace an existing object with an empty one; if False, raise if the object exists. A key that is only a prefix of other keys has no object.

  • **kwargs – Additional parameters passed to the PutObject API, except Body: the object is empty.

Returns:

The PutObject response as a dictionary (see S3PutObject.to_dict()).

Raises:
  • TypeError – If kwargs has Body.

  • ValueError – If the path has a version ID, is a bucket, or exists while truncate is False.

mv(path1, path2, recursive=False, maxdepth=None, batch_size=None, **kwargs) → None[source]

Move files from one S3 location to another.

Copies the files to the destinations that copy() gives them, and then deletes only the source objects that were copied. fsspec’s AbstractFileSystem.mv() instead removes path1 by expanding it again, which also deletes copies placed where path1 matches them and files that maxdepth kept from being copied. A file whose destination is the file itself is left in place, and directories, which S3 does not store as objects, are not copied. A null version moved onto its key is copied and deleted only when bucket versioning is enabled. Without versioning, or with versioning suspended, that move is left in place because the copy would replace the version that the deletion removes.

Comparing a null version with its unversioned key calls GetBucketVersioning once per bucket during the move’s planning and requires s3:GetBucketVersioning. The result is not cached across moves. Directory buckets do not support versioning and need no lookup. A source without a version whose destination conflicts with another pair is looked up with HeadObject during the planning too.

The files are copied in parallel as copy() copies them, and are deleted once every copy has succeeded; nothing is deleted after a failed copy.

Parameters:
  • path1 – Source S3 path, glob pattern, or list of paths.

  • path2 – Destination S3 path, or list of paths when path1 is a list.

  • recursive – Whether to move the directories with their contents.

  • maxdepth – Maximum depth of a recursive move.

  • batch_size – Number of copies to run at the same time, as in copy().

  • **kwargs – Additional S3 copy parameters passed to cp_file(). The lookups and the deletion of the sources receive RequestPayer and the source’s expected bucket owner (see S3Core.copy_source_params()); the lookups also receive the source’s SSE-C parameters.

Raises:
  • ValueError – If two sources have the same destination, or a destination is another source, including one left in place, which is checked before anything is copied. A source without an object at its key, such as a directory, writes no destination and does not conflict. Also if batch_size is neither positive, -1, nor None.

  • FileNotFoundError – If a source does not exist, once the running copies have ended; nothing is deleted.

  • OSError – If bucket versioning cannot be read.

copy(path1, path2, recursive=False, maxdepth=None, on_error=None, batch_size=None, **kwargs) → None[source]

Copy files within S3.

As fsspec’s copy(), except that a source with a version ID copies that version of the object to a destination named after its key, as copy_pairs() pairs them, and that the files are copied in parallel, batch_size at a time. Pairs that would depend on the order of the copies raise before anything is copied (see check_copy_pairs()).

An error that on_error does not ignore starts no further copy, stops the running ones before their next request where it can (a multipart copy sends no further part and is aborted), and is raised once they have ended. Copies that ended before it stay in place.

Parameters:
  • path1 – Source S3 path, glob pattern, or list of them.

  • path2 – Destination S3 path, or list of paths when path1 is a list.

  • recursive – Whether to copy the directories with their contents.

  • maxdepth – Maximum depth of a recursive copy.

  • on_error – "raise" or "ignore" for a missing source; by default "ignore" with recursive and "raise" otherwise.

  • batch_size – Number of copies to run at the same time. By default the batch_size of the filesystem; None or -1 there is max_workers. With n copies at once, the smaller of batch_size and the number of pairs, each multipart copy sends at most max(1, max_workers // n) parts at once.

  • **kwargs – Additional S3 copy parameters passed to cp_file().

Raises:
  • ValueError – If two sources have the same destination, or a destination is another source, except for a source without an object at its key, such as a directory, which is not copied. Also if batch_size is neither positive, -1, nor None.

  • OSError – If bucket versioning cannot be read for the checks.

get(rpath, lpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, batch_size=None, **kwargs) → None[source]

Copy files from S3 to the local filesystem.

As fsspec’s get(), except that a source with a version ID downloads that version of the object to a local path named after its key, as copy_pairs() pairs them, and that the files are downloaded in parallel, as with copy(). The destinations named after the sources are checked to lie under lpath. With outfile, the files are written to it one after another, in order.

A failed download starts no further one, the running ones read and write no further block, and the error is raised once they have ended.

Parameters:
  • rpath – Source S3 path, glob pattern, or list of them.

  • lpath – Local destination path, or list of paths or file-like objects when rpath is a list.

  • recursive – Whether to copy the directories with their contents.

  • callback – Progress callback. It and its branches for the files are used from the threads that download them, one at a time.

  • maxdepth – Maximum depth of a recursive copy.

  • batch_size – Number of downloads to run at the same time, as in copy().

  • **kwargs – Additional parameters passed to get_file().

Raises:

ValueError – If a destination named after a source lies outside lpath, two downloads write the same local file, or batch_size is neither positive, -1, nor None.

cp_file(path1: str | PathLike[str] | S3Path, path2: str | PathLike[str] | S3Path, recursive=False, maxdepth=None, on_error=None, **kwargs)[source]

Copy an S3 object to another S3 location.

Performs server-side copy of S3 objects, which is more efficient than downloading and re-uploading. Automatically chooses between simple copy and multipart copy based on object size.

Parameters:
  • path1 – Source S3 path (s3://bucket/key), or an S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns.

  • path2 – Destination S3 path (s3://bucket/key), or an S3Path.

  • recursive – Unused parameter for fsspec compatibility.

  • maxdepth – Unused parameter for fsspec compatibility.

  • on_error – Unused parameter for fsspec compatibility.

  • **kwargs – Additional S3 copy parameters (e.g., metadata, storage class). The block_size and max_workers parameters control a multipart copy and are not sent to S3. The lookup of the source receives RequestPayer, and the source’s expected bucket owner and SSE-C parameters under the names of the requests that read it (see S3Core.copy_source_params()).

Raises:

ValueError – If trying to copy to a versioned file or copy buckets, or if a directive of a multipart copy has an invalid value.

Note

Uses multipart copy for objects larger than the maximum part size to optimize performance for large files. The copy operation is performed entirely on the S3 service without data transfer. The multipart copy applies the metadata, tagging and annotation directives as CopyObject does (default COPY). A directory path1, which recursive copy() passes along with the files under it, is skipped.

pipe(path, value=None, batch_size=None, **kwargs) → None[source]

Write bytes into one or more paths.

As fsspec’s pipe(), except that the values of a dict are written in parallel with pipe_file(), as copy() copies its files.

Parameters:
  • path – S3 path to write value to, or a dict of the values to write by path.

  • value – The bytes to write to a single path; ignored with a dict.

  • batch_size – Number of writes to run at the same time, as in copy().

  • **kwargs – Additional parameters passed to pipe_file().

Raises:

ValueError – If path is neither a string nor a dict, two paths of a dict name the same object, or batch_size is neither positive, -1, nor None.

pipe_file(path: str | PathLike[str], value: bytes | bytearray | memoryview, mode: str = 'overwrite', **kwargs) → None[source]

Write bytes into the path.

Writes data up to the block size with a single PutObject request instead of the inherited open() + write() path. Larger data and writes inside an fsspec transaction go through the buffered path, which uploads the data as a parallel multipart upload and keeps the deferred-commit semantics of transactions. A write that fails on that path leaves the existing object unchanged. Both paths write to the path without a trailing slash, as open() does.

Parameters:
  • path – S3 path (s3://bucket/key) to write to.

  • value – The bytes to write.

  • mode – “overwrite” (default) or “create”. With “create”, raise FileExistsError when the object already exists, including one created during the write, which is not replaced.

  • **kwargs – Additional parameters passed to the PutObject API (e.g., ContentType, StorageClass) on the single-request path. The block_size, max_workers, and s3_additional_kwargs parameters of the open() path are also accepted, and so is compression: the codec of open() to compress the value with before it is uploaded, or "infer" to take it from the extension of the path.

Raises:
  • FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the write is committed.

  • ValueError – If the path does not contain a key or specifies a version, if the compression is not supported, or if the data takes more than S3Core.MULTIPART_UPLOAD_MAX_PARTS blocks.

cat(path, recursive=False, on_error='raise', batch_size=None, **kwargs)[source]

Read the contents of one or more paths.

As fsspec’s cat(), except that the objects are read in parallel with cat_file(), as copy() copies its files. With on_error="raise", an error starts no further read and is raised once the running reads have ended.

Parameters:
  • path – S3 path, glob pattern, or list of them.

  • recursive – Whether to read the files below the directories.

  • on_error – "raise" to raise an error of a read, "omit" to leave out the paths whose read failed, or "return" to return the error in place of their contents.

  • batch_size – Number of reads to run at the same time, as in copy().

  • **kwargs – Additional parameters passed to expand_path() and cat_file().

Returns:

The contents of the path, or a dict of the contents by path if there are several paths or the path was expanded.

Raises:

ValueError – If batch_size is neither positive, -1, nor None.

cat_ranges(paths, starts, ends, max_gap=None, on_error='return', batch_size=None, **kwargs) → list[bytes | Exception][source]

Read byte ranges of one or more paths.

As fsspec’s cat_ranges(), except that the ranges are read in parallel with cat_file(), as copy() copies its files. Unless on_error is "return", an error starts no further read and is raised once the running reads have ended.

Parameters:
  • paths – The S3 paths, one per range.

  • starts – The start offset of each range, or one for all of them.

  • ends – The end offset of each range, or one for all of them.

  • max_gap – Not supported; must be None.

  • on_error – "return" to return the error of a read in place of its bytes; otherwise the error is raised.

  • batch_size – Number of reads to run at the same time, as in copy().

  • **kwargs – Additional parameters passed to cat_file().

Returns:

The bytes of each range, or the error of its read.

Raises:
  • NotImplementedError – If max_gap is given.

  • TypeError – If paths is not a list.

  • ValueError – If the numbers of paths, starts, and ends differ, or if batch_size is neither positive, -1, nor None.

cat_file(path: str | PathLike[str] | S3Path, start: int | None = None, end: int | None = None, **kwargs) → bytes[source]

Read the contents of an S3 object with GetObject.

start and end select bytes like a slice of the object: an empty range, or one that starts at or past the end of the object, returns b"", and an end past the object reads up to its end. Non-negative offsets are sent to S3 as they are, and so is a negative start without an end, as a suffix range of the last bytes. Other negative offsets are resolved against the size from info(), which also checks that the object exists for an empty range and receives the lookup parameters among kwargs.

Parameters:
  • path – S3 path (s3://bucket/key) of the object, or an S3Path, whose bucket, key and version are read as they are, such as a key that a listing returns.

  • start – Byte offset to start reading at. A negative value counts from the end of the object.

  • end – Byte offset to stop reading at (exclusive). A negative value counts from the end of the object.

  • **kwargs – Additional parameters passed to the GetObject API, except version_id: the version ID to read when the path has none.

Returns:

The bytes read from the object.

Raises:

FileNotFoundError – If the path has no key or the key does not exist.

put(lpath, rpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, batch_size=None, **kwargs) → None[source]

Copy files from the local filesystem to S3.

As fsspec’s put(), except that the files are uploaded in parallel with put_file(), as copy() copies its files. A failed upload starts no further one, the running ones send no further part and do not commit, and the error is raised once they have ended.

Parameters:
  • lpath – Local source path, glob pattern, or list of them.

  • rpath – S3 destination path, or list of paths when lpath is a list.

  • recursive – Whether to copy the directories with their contents.

  • callback – Progress callback. It and its branches for the files are used from the threads that upload them, one at a time.

  • maxdepth – Maximum depth of a recursive copy.

  • batch_size – Number of uploads to run at the same time, as in copy().

  • **kwargs – Additional parameters passed to put_file().

Raises:

ValueError – If two files have the same destination, or batch_size is neither positive, -1, nor None.

put_file(lpath: str | ~os.PathLike[str], rpath: str | ~os.PathLike[str], callback=<fsspec.callbacks.NoOpCallback object>, mode: str = 'overwrite', **kwargs)[source]

Upload a local file to S3.

Uploads a file from the local filesystem to an S3 location. Supports automatic content type detection based on file extension and provides progress callback functionality. An upload that fails before it is completed, including one of a local file that cannot be read, leaves the existing object unchanged.

Parameters:
  • lpath – Local file path to upload.

  • rpath – S3 destination path (s3://bucket/key).

  • callback – Progress callback for tracking upload progress.

  • mode – “overwrite” (default) or “create”. With “create”, the file is written as with open() in xb mode: raise FileExistsError when the object already exists, including one created during the upload, which is not replaced.

  • **kwargs – Additional S3 parameters (e.g., ContentType, StorageClass). The block_size, max_workers, and s3_additional_kwargs parameters of open() are also accepted.

Raises:
  • FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the upload is committed.

  • ValueError – If the file takes more than S3Core.MULTIPART_UPLOAD_MAX_PARTS blocks.

Note

Directories are not supported for upload. If lpath is a directory, the method returns without performing any operation. Bucket-only destinations (without key) are also not supported.

get_file(rpath: str | ~os.PathLike[str] | ~pyathena.filesystem.s3_path.S3Path, lpath=None, callback=<fsspec.callbacks.NoOpCallback object>, outfile=None, **kwargs)[source]

Download an S3 file to local filesystem.

The object is read in waves of parallel range requests of the block size, at most max_workers of them at once, but no more than the request pool of the filesystem runs, and the bodies of each wave are written in order before the next wave is read, so that the download holds at most one wave of bodies; a thread of the pool that has just read a range drops its body when its call returns (see S3File._download()). The reads are conditional on the version and the ETag of the object looked up when the download starts, as with open(), so that an object replaced during the download fails it instead of mixing two objects, unless the download reads a version other than null, which a replacement does not change. A failed download leaves the part written so far in a local file lpath, as fsspec’s AbstractFileSystem.get_file() does.

As with fsspec’s AbstractFileSystem.get_file(), a directory rpath creates the local directory lpath, and the parent directories of a local file lpath are created as needed.

Parameters:
  • rpath – S3 source path (s3://bucket/key), or an S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns.

  • lpath – Local destination path, or a file-like object to write to. Not needed when outfile is given.

  • callback – Progress callback for tracking download progress, with the size of the object.

  • outfile – A file-like object to write to instead of lpath.

  • **kwargs – The arguments of open(), such as block_size, max_workers, version_id, and s3_additional_kwargs, and additional S3 parameters. cache_type and cache_options are not used. With compression, the object is decompressed into the destination, read in order through open() with a block cache, whose block size, if not positive, is that of the filesystem, or DEFAULT_BLOCK_SIZE if that is not positive either, and the size of the callback is unknown.

checksum(path: str | PathLike[str], **kwargs)[source]

Get checksum for S3 object or directory.

Computes a checksum for the specified S3 path. For individual objects, returns the ETag converted to an integer. For directories, returns a checksum based on the directory’s tokenized representation.

Parameters:
  • path – S3 path (s3://bucket/key) to get checksum for.

  • **kwargs – Additional arguments including: refresh: If True, refresh cached info before computing checksum.

Returns:

Integer checksum value derived from S3 ETag or directory token.

Note

For multipart uploads, ETag format is different and only the first part before the dash is used for checksum calculation.

sign(path: str | PathLike[str], expiration: int = 3600, **kwargs)[source]

Generate a presigned URL for S3 object access.

Creates a presigned URL that allows temporary access to an S3 object without requiring AWS credentials. Useful for sharing files or providing time-limited access to resources.

Parameters:
  • path – S3 path (s3://bucket/key) to generate URL for.

  • expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).

  • **kwargs –

    Additional parameters including: client_method: S3 operation (‘get_object’, ‘put_object’, etc.).

    Defaults to ‘get_object’.

    Additional parameters passed to the S3 operation.

Returns:

Presigned URL string that provides temporary access to the S3 object.

Example

>>> fs = S3FileSystem()
>>> url = fs.sign("s3://my-bucket/file.txt", expiration=7200)
>>> # URL valid for 2 hours
>>>
>>> # Generate upload URL
>>> upload_url = fs.sign(
...     "s3://my-bucket/upload.txt",
...     client_method="put_object"
... )
metadata(path: str | PathLike[str], **kwargs) → S3Metadata[source]

Return the metadata of the path.

Parameters:
  • path – S3 path (s3://bucket/key) to get metadata for.

  • **kwargs – Additional parameters passed to the HeadObject API.

Returns:

S3Metadata, which behaves as a read-only mapping of the user-defined metadata (x-amz-meta-*) and exposes the system-defined metadata (content type, encryption settings, etc.) as typed properties.

getxattr(path: str | PathLike[str], attr_name: str, **kwargs) → str | None[source]

Get an attribute from the user-defined metadata of the path.

Parameters:
  • path – S3 path (s3://bucket/key) to get the attribute for.

  • attr_name – The name of the attribute.

  • **kwargs – Additional parameters passed to metadata().

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(path: str | PathLike[str], copy_kwargs: dict[str, Any] | None = None, **kw_args) → None[source]

Set the user-defined metadata of the path.

S3 does not allow updating the metadata of an existing object in place, so the object is copied onto itself with the REPLACE metadata directive. Note that this rewrites the object and updates its last-modified time. The system-defined metadata (e.g., ContentType and CacheControl), the storage class, and the server-side encryption algorithm and KMS key of the object are kept. HeadObject does not return the KMS encryption context, and an Expires value that botocore cannot parse as a date is not kept.

Parameters:
  • path – S3 path (s3://bucket/key) to set metadata for. A path with a version ID is rejected, since copying a version onto the key would replace the current object with it.

  • copy_kwargs – Additional parameters to use for the underlying CopyObject API call. They take precedence over the kept system-defined metadata and storage class. Any encryption parameter replaces all kept encryption settings. The lookup of the object receives RequestPayer, and the source’s expected bucket owner and SSE-C parameters (e.g., CopySourceSSECustomerKey) under the names of HeadObject (see S3Core.copy_source_params()).

  • **kw_args – Key-value pairs to set, where the values must be strings. The keys are used as-is; names that are not valid Python identifiers (e.g., containing hyphens) can be passed by unpacking a dictionary. Does not alter existing fields, unless the field appears here - if the value is None, delete the field.

Example

>>> fs = S3FileSystem()
>>> fs.setxattr("s3://bucket/key", attribute1="value1")
>>> fs.setxattr("s3://bucket/key", **{"attribute-2": "value2"})
Raises:

ValueError – If the path is a bucket or has a version ID.

get_tags(path: str | PathLike[str]) → dict[str, str][source]

Retrieve the tag key/values for the given path.

Parameters:

path – S3 path (s3://bucket/key) to get tags for.

Returns:

Dictionary mapping tag keys to tag values.

put_tags(path: str | PathLike[str], tags: dict[str, str], mode: str = 'o') → None[source]

Set the tags for the given existing key.

Tags are a str:str mapping that can be attached to any key, distinct from the user-defined metadata, which is usually set at key creation time. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html

Parameters:
  • path – S3 path (s3://bucket/key) of the existing key to attach tags to.

  • tags – Tags to apply.

  • mode – One of ‘o’ or ‘m’. ‘o’ will over-write any existing tags. ‘m’ will merge in new tags with existing tags, which incurs two remote calls.

chmod(path: str | PathLike[str], acl: str, recursive: bool = False, **kwargs) → None[source]

Set the Access Control on a bucket/key.

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl

Parameters:
  • path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.

  • acl – The value of the canned ACL to apply.

  • recursive – Whether to apply the ACL to all keys below the given path too, and to the object at the key, if any. The keys are updated in parallel; once one fails, the updates that have not started are cancelled, and the running ones make no further attempt. A path with a version ID updates that version only.

  • **kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.

list_multipart_uploads(path: str | PathLike[str]) → list[S3MultipartUpload][source]

List in-progress (incomplete) multipart uploads in a bucket.

Incomplete multipart uploads continue to accrue storage costs until they are completed or aborted. Use clear_multipart_uploads() to abort all of them.

Parameters:

path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under key/ are listed, not those to sibling keys that merely start with the same characters (e.g., prefix2/a).

Returns:

List of S3MultipartUpload instances describing the in-progress multipart uploads.

object_version_info(path: str | PathLike[str], delete_markers: bool = False, **kwargs) → list[S3ObjectVersion][source]

List the versions of the object or of the objects under the path.

A key path without a trailing slash selects that key if it has any versions or delete markers, and otherwise the keys under key/. The choice does not depend on delete_markers, so a key that has only delete markers yields no versions without them. A key path with a trailing slash selects the keys under it, and a bucket path selects all the keys in the bucket. Sibling keys that merely start with the same characters (e.g., key.bak) are never included.

Parameters:
  • path – S3 path (s3://bucket/key or a key prefix) to list the versions for.

  • delete_markers – Whether to include delete markers in the result.

  • **kwargs – Additional parameters passed to the ListObjectVersions API.

Returns:

List of S3ObjectVersion instances describing the versions.

clear_multipart_uploads(path: str | PathLike[str]) → None[source]

Abort any incomplete multipart uploads in the bucket.

Uploads completed or aborted after listing are already cleared. All abort results are checked before another abort error is raised.

Parameters:

path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under key/ are aborted, as listed by list_multipart_uploads().

created(path: str | PathLike[str]) → datetime[source]

Return the creation time of the path.

Returns the same value as modified().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

modified(path: str | PathLike[str]) → datetime[source]

Return the last-modified time of the path.

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last_modified field from info(), which is None for buckets and directories.

invalidate_cache(path: str | PathLike[str] | None = None) → None[source]

Remove the cached entries of the path and its parent paths.

The cached bucket listing is removed only by the root path ("", "/" or "s3://"), not by the paths of buckets or keys. A version-qualified path invalidates the lookup of the version, and also the object path without the version, because deleting or copying a version can change the current version of the object. It also invalidates the keys that end in its query, in each spelling that names the version, such as the keys a/x?versionId=v1 and a/x?version_id=v1 of bucket/a/x?versionId=v1, which a listing names as the paths. Only the path itself is parsed for a version: its parent paths are directories, named as they are listed, such as bucket/a/dir?versionId=v1 for the key a/dir?versionId=v1/x. A path that ends in slashes is invalidated both with and without them, since lookups strip them while a key such as dir// is listed in the directory bucket/dir/.

Parameters:

path – The path to invalidate. If None, clear the whole cache.

class pyathena.filesystem.s3.S3File(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]

A buffered file object for reading and writing an S3 object.

Instances are returned by S3FileSystem.open().

fs: S3FileSystem
buffer: BytesIO | None
__init__(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → None[source]

Initialize the file for the path and mode.

In read mode, the object is looked up with info() and the reads are made conditional on its ETag (IfMatch). In append mode, an existing object smaller than S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE is read into the write buffer; a larger one is copied with UploadPartCopy as the first parts of a multipart upload, whatever the block size. In exclusive-create mode, the object must not exist when the file is opened, and the upload is committed with IfNoneMatch="*" so that it does not replace an object created in the meantime.

Parameters:
  • fs – The filesystem that the file belongs to.

  • path – S3 path (s3://bucket/key) of the file, or an S3Path (see S3FileTarget.resolve()).

  • mode – The file mode: rb, wb, ab, or xb.

  • version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.

  • max_workers – The number of parallel requests of the file: range reads, part copies, and part uploads. A write keeps at most max_workers parts, including the part copies of an append, in flight, waiting for one of them to finish before it submits another. If None, the max_workers of the filesystem.

  • executor – The executor for parallel operations. If None, one on the request pool of the filesystem (fs._create_executor()).

  • block_size – The block size for reads and writes. Must be between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.

  • cache_type – The fsspec cache type for reads; see S3FileSystem.__init__().

  • autocommit – Whether to commit the written data when the file is closed. If False, commit() must be called.

  • cache_options – Options for the fsspec cache.

  • size – The size of the object, if known. Passed to fsspec.spec.AbstractBufferedFile.

  • s3_additional_kwargs – Additional parameters for the S3 requests of the file, such as ContentType or RequestPayer. Each request receives those that its operation accepts.

  • **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Raises:
  • FileExistsError – If an object exists at the path in exclusive-create mode.

  • FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.

  • ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE for writing, or max_workers is not positive.

close() → None[source]

Close the file, flushing any written data, and shut down its executor.

In a forked child process, a file that the parent opened is only marked closed: its data is neither flushed nor committed, and its upload is not aborted, since they belong to the parent. Called by another finalizer while the cyclic garbage collector runs in this thread, such as that of a TextIOWrapper, the file is closed in a new thread once the collection has ended (see __del__()), and this returns before that. A close handed over to another thread is waited for, except during a collection or in a thread of an S3ThreadPool, where this returns without waiting.

__del__() → None[source]

Close the file when it is finalized without having been closed.

A file whose last reference is dropped in a user thread is closed here. A file that the cyclic garbage collector finalizes, or that is finalized in a thread of an S3ThreadPool, is closed in a new thread instead, since a wait here could last forever (T1): the requests of the file may be queued behind this thread or include the one that runs the collection, and this thread may hold a lock that the close takes. During a collection, the thread starts once the collection has ended, after the other finalizers of the collection, such as that of a TextIOWrapper, have written to the file.

read(length: int = -1) → bytes[source]

Read bytes from the file.

Parameters:

length – The number of bytes to read; -1 reads to the end.

Returns:

The bytes read.

Raises:
  • RuntimeError – In a forked child process of the process that opened the file.

  • ValueError – If the file is closed or not open for reading.

flush(force: bool = False) → None[source]

Upload the buffered data as a part once a block is full, or all of it with force.

Parameters:

force – Whether to upload the remaining data and commit the file, after which it cannot be written.

Raises:
  • RuntimeError – In a forked child process of the process that opened the file.

  • ValueError – If force is given again after a forced flush.

write(data: bytes | bytearray | memoryview) → int[source]

Write data to the buffer, uploading each full block as a part.

Parameters:

data – The bytes to write.

Returns:

The number of bytes written.

Raises:
  • BaseException – The error of a request of the write that failed since the last write, after which the file is closed without committing it.

  • RuntimeError – In a forked child process of the process that opened the file.

  • S3MultipartStateError – If the write was discarded.

  • ValueError – If the file is not open for writing.

property multipart_upload: S3MultipartUpload | None

The multipart upload of the write, from its creation until it is aborted.

property completed_parts: tuple[S3MultipartUploadPart, ...]

The uploaded or copied parts in part-number order; empty after an abort.

commit() → None[source]

Commit the written data.

Creates an empty object if nothing was written, uploads the buffered data with PutObject if no multipart upload was started, and otherwise completes the multipart upload once its parts have been uploaded. A failed part is raised as soon as it fails, without waiting for the parts before it. If a part or the completion fails or is interrupted, the file is closed without committing it, and its upload is aborted; if the abort also fails, the upload is kept so that discard() can abort it. Committing again after a successful commit sends no request. Invalidates the cache of the path afterwards. Does nothing for a file opened for reading. Waits first for a close of the file that was handed over to another thread (see close()).

Raises:
  • FileExistsError – If an object was created at the path after the file was opened in exclusive-create mode.

  • RuntimeError – In a forked child process of the process that opened the file, or if a close of the file that was handed over to another thread runs and this is called during a garbage collection or in a thread of an S3ThreadPool.

  • S3MultipartStateError – If the write was discarded, or the write or an earlier commit failed.

discard() → None[source]

Discard the written data, and abort the multipart upload, if any.

The part requests that have not started are cancelled, and the running ones make no further attempt and are waited for before the abort, whose own retries are not stopped. If the abort fails or is interrupted, the upload is kept, and calling this again retries the abort. Afterwards, commit() raises S3MultipartStateError. Does nothing for a committed file or a file opened for reading. Waits first for a close of the file that was handed over to another thread (see close()).

Raises:

RuntimeError – In a forked child process of the process that opened the file, or if a close of the file that was handed over to another thread runs and this is called during a garbage collection or in a thread of an S3ThreadPool.

url(expiration: int = 3600, **kwargs) → str[source]

Generate a presigned HTTP URL to read this file (if it already exists).

The URL is signed with the lookup parameters of the file that the signed operation accepts, such as RequestPayer or the SSECustomer* parameters, whose headers the request with the URL must then send.

Parameters:
  • expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).

  • **kwargs – Additional parameters passed to S3FileSystem.sign(), which take precedence over those of the file.

Returns:

Presigned URL string that provides temporary access to the S3 object.

metadata(**kwargs) → S3Metadata[source]

Return the metadata of the file.

See S3FileSystem.metadata(). The HeadObject request receives the lookup parameters of the file, such as RequestPayer or the SSECustomer* parameters.

Parameters:

**kwargs – Additional parameters passed to the HeadObject API, which take precedence over those of the file.

Returns:

S3Metadata, which behaves as a read-only mapping of the user-defined metadata and exposes the system-defined metadata as typed properties.

getxattr(xattr_name: str, **kwargs) → str | None[source]

Get an attribute from the user-defined metadata of the file.

See S3FileSystem.getxattr(), and metadata() for the parameters of the request.

Parameters:
  • xattr_name – The name of the attribute.

  • **kwargs – Additional parameters passed to the HeadObject API, which take precedence over those of the file.

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(copy_kwargs: dict[str, Any] | None = None, **kwargs) → None[source]

Set the user-defined metadata of the file.

See S3FileSystem.setxattr(). The copy of the object onto itself receives the lookup parameters of the file both for its source and for its destination: RequestPayer, the expected bucket owner as ExpectedSourceBucketOwner and ExpectedBucketOwner, and the SSECustomer* parameters also as CopySourceSSECustomer*, so that an object encrypted with a customer-provided key keeps it. A ServerSideEncryption in copy_kwargs replaces the SSECustomer* parameters of the file for the destination; those in the s3_additional_kwargs of the filesystem are still sent.

Parameters:
  • copy_kwargs – Additional parameters to use for the underlying CopyObject API call, which take precedence over those of the file.

  • **kwargs – Key-value pairs of metadata to set.

class pyathena.filesystem.s3.S3FileTarget(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None)[source]

The object that a file reads or writes, resolved when it is opened.

In read mode, the object is looked up, and the reads are made conditional on its ETag (IfMatch). In append mode, an existing object is looked up; one smaller than S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE is also read, so that the file rewrites it from its buffer, and a larger one is copied by the file as the first parts of a multipart upload. In exclusive-create mode, the object must not exist, and the upload is committed with IfNoneMatch="*".

path

The path of the file. A version that is read is carried in the path as a ?versionId= suffix, so that a reopened file reads the same version.

Type:

str

bucket

The bucket of the object.

Type:

str

key

The key of the object.

Type:

str

version_id

The version that is read, if any.

Type:

str | None

request_kwargs

The parameters of the S3 requests of the file, with the conditions above and, in append mode, the attributes of the existing object that the request parameters (those of the call and of the filesystem) do not give, which the rewritten object keeps: its system-defined and user-defined metadata, storage class, and encryption, unless they give any encryption parameter. Its object lock is not kept.

Type:

dict[str, Any]

details

The object that is read or appended, or an empty dictionary.

Type:

pyathena.filesystem.s3_object.S3Object | dict[str, Any]

size

The size of the object that is read, if known.

Type:

int | None

appended

The existing object in append mode, if any.

Type:

pyathena.filesystem.s3_object.S3Object | None

append_data

The data of an appended object that the file rewrites from its buffer, if any.

Type:

bytes | None

path: str
bucket: str
key: str
version_id: str | None
request_kwargs: dict[str, Any]
details: S3Object | dict[str, Any]
size: int | None = None
appended: S3Object | None = None
append_data: bytes | None = None
classmethod resolve(fs: S3FileSystem, path: str | S3Path, mode: str, *, block_size: int, version_id: str | None = None, size: int | None = None, request_kwargs: Mapping[str, Any] | None = None) → S3FileTarget[source]

Validate the arguments of a file and look up its object.

Sends the lookup requests of the mode, which block.

Parameters:
  • fs – The filesystem that the file belongs to.

  • path – S3 path (s3://bucket/key) of the file, or an S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns, and which is looked up by them. The path of the file is then the string of the S3Path.

  • mode – The file mode: rb, wb, ab, or xb.

  • block_size – The block size of the file.

  • version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.

  • size – The size of the object to read, if known.

  • request_kwargs – The parameters of the S3 requests of the file. A copy is kept.

Returns:

The resolved target.

Raises:
  • FileExistsError – If an object exists at the path in exclusive-create mode.

  • FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.

  • ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE for writing.

__init__(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None) → None

Async S3 FileSystem

class pyathena.filesystem.s3_async.AioS3FileSystem(*args, **kwargs)[source]

An async filesystem interface for Amazon S3 using fsspec’s AsyncFileSystem.

This class wraps S3FileSystem to provide native asyncio support. Its coroutines run the blocking boto3 calls in an S3ThreadPool with max_workers threads that the filesystem owns, instead of the default executor of the event loop, and run parallel per-object calls, such as the copies of _mv(), with asyncio.gather. The single requests that these calls run in parallel, such as the parts of a file or a copy, and the DeleteObjects batches of _rm(), run in the request pool of the wrapped S3FileSystem, which its files share, through map_async() or a write session.

The implementation uses composition: an internal S3FileSystem instance handles all boto3 calls, while this class delegates to it in the threads of its pool. This avoids diamond inheritance issues and keeps all boto3 logic in one place.

File handles created by _open run their parallel operations (range reads, multipart uploads) in that request pool, as S3File does. open_async() returns an AioS3StreamedFile, whose reads and writes are coroutines. close(), or the end of a with or async with block, releases the threads of the filesystem, which stays usable.

_sync_fs

The internal synchronous S3FileSystem instance.

Example

>>> from pyathena.filesystem.s3_async import AioS3FileSystem
>>> fs = AioS3FileSystem(asynchronous=True)
>>>
>>> # Use in async context
>>> files = await fs._ls('s3://my-bucket/data/')
>>>
>>> # Sync wrappers (auto-generated by fsspec) need an instance created
>>> # without asynchronous=True; they block the caller until done
>>> files = AioS3FileSystem().ls('s3://my-bucket/data/')
protocol: str | tuple[str, ...] = ('s3', 's3a')
mirror_sync_methods = True
async_impl = True
__init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, asynchronous: bool = False, loop: Any | None = None, batch_size: int | None = None, *, client: BaseClient | None = None, retry_config: RetryConfig | None = None, **kwargs) → None[source]

Initialize the filesystem and its internal S3FileSystem.

Parameters:
  • connection – Deprecated; use client and retry_config, or pyathena.filesystem.s3_filesystem() with AioS3FileSystem. A PyAthena connection whose S3 client and retry policy the filesystem uses, with a DeprecationWarning.

  • default_block_size – Passed to the internal S3FileSystem.

  • default_cache_type – Passed to the internal S3FileSystem.

  • max_workers – The number of threads that run the blocking calls of the filesystem, and, passed to the internal S3FileSystem, the size of its request pool and the default number of parallel requests of its files and copies. If None, the max_pool_connections of the S3 client (10 unless configured), as for S3FileSystem.

  • s3_additional_kwargs – Passed to the internal S3FileSystem.

  • allow_bucket_creation – Passed to the internal S3FileSystem.

  • allow_bucket_deletion – Passed to the internal S3FileSystem.

  • version_aware – Passed to the internal S3FileSystem.

  • asynchronous – Passed to fsspec.asyn.AsyncFileSystem.

  • loop – Passed to fsspec.asyn.AsyncFileSystem.

  • batch_size – Passed to fsspec.asyn.AsyncFileSystem.

  • client – Passed to the internal S3FileSystem. A filesystem with a given client cannot be pickled.

  • retry_config – Passed to the internal S3FileSystem.

  • **kwargs – Passed to both fsspec.asyn.AsyncFileSystem and the internal S3FileSystem.

Raises:

TypeError – If connection is given with client or retry_config, or the internal S3FileSystem rejects the arguments.

__reduce__() → tuple[Any, ...][source]

Reduce the filesystem for pickling, as fsspec does.

Returns:

The reduction of fsspec.AbstractFileSystem, which creates the filesystem again from its arguments.

Raises:

TypeError – If the filesystem was built with client or connection, whose boto3 client cannot be pickled.

property core: S3Core

The typed S3 operations of the wrapped S3FileSystem.

close() → None[source]

Release the threads of the filesystem, which stays usable, once they are idle.

Releases the threads of the blocking calls and those of the request pool of the internal S3FileSystem (see S3FileSystem.close() and S3ThreadPool.release()). The calls in progress, including the requests of open files, run to their end.

__enter__() → AioS3FileSystem[source]

Return the filesystem, whose threads the end of the block releases.

__exit__(exc_type: Any, exc_val: Any, exc_tb: Any) → None[source]

Release the threads with close().

async __aenter__() → AioS3FileSystem[source]

Return the filesystem, whose threads the end of the block releases.

async __aexit__(exc_type: Any, exc_val: Any, exc_tb: Any) → None[source]

Release the threads with close().

mv(path1, path2, recursive=False, maxdepth=None, **kwargs) → None[source]

Move files from one S3 location to another.

Runs _mv(); fsspec does not provide the synchronous wrapper of _mv.

Parameters:
  • path1 – Source S3 path, glob pattern, or list of paths.

  • path2 – Destination S3 path, or list of paths when path1 is a list.

  • recursive – Whether to move the directories with their contents.

  • maxdepth – Maximum depth of a recursive move.

  • **kwargs – Additional S3 copy parameters.

Raises:
  • ValueError – If the move has conflicting paths.

  • OSError – If bucket versioning cannot be read.

walk(path, maxdepth=None, topdown=True, on_error='omit', **kwargs)[source]

Walk the directories below a path, as fsspec’s walk() does.

Runs the walk of S3FileSystem.walk(), which lists a listed directory by its key prefix rather than by its name. As fsspec’s walk() of an asynchronous filesystem, which runs _walk(), it yields a directory whose listing fails with no entries.

Parameters:
  • path – The path to walk.

  • maxdepth – Maximum number of levels to walk, at least 1 (None for unlimited).

  • topdown – Whether to yield a directory before the directories below it.

  • on_error – "omit", "raise", or a callable that receives the error of the listing of the path, as in S3FileSystem.walk().

  • **kwargs – Additional arguments passed to ls().

Yields:

Tuples of a directory path, its directories, and its files.

tree(path: str | PathLike[str] = '/', recursion_limit: int = 2, max_display: int = 25, display_size: bool = False, prefix: str = '', is_last: bool = True, first: bool = True, indent_size: int = 4) → str[source]

Return a tree of the paths below a path, as fsspec’s tree() does.

Runs S3FileSystem.tree(), which lists a listed directory by its key prefix rather than by its name.

Parameters:
  • path – The root path, as a string or an os.PathLike.

  • recursion_limit – Maximum depth of the tree.

  • max_display – Maximum number of entries shown per directory.

  • display_size – Whether to show the sizes of the files.

  • prefix – The prefix of each line.

  • is_last – Whether the path is the last entry of its directory.

  • first – Whether the path is the root of the tree.

  • indent_size – Number of spaces per level.

Returns:

The tree as a string.

get_mapper(root: str | PathLike[str] = '', check: bool = False, create: bool = False, missing_exceptions=None)[source]

Return a key/value store at a path, as fsspec’s get_mapper() does.

Parameters:
  • root – The root path of the store, as a string or an os.PathLike.

  • check – Whether to check that the root is writable.

  • create – Whether to create the root if it does not exist.

  • missing_exceptions – The exceptions that mean a missing key.

Returns:

The fsspec.mapping.FSMap of the root.

unstrip_protocol(name: str | PathLike[str]) → str[source]

Add the protocol to a path, as fsspec’s unstrip_protocol() does.

Parameters:

name – The path, as a string or an os.PathLike.

Returns:

The path with the s3:// protocol.

async open_async(path: str | PathLike[str], mode: str = 'rb', block_size: int | None = None, max_workers: int | None = None, version_id: str | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → AioS3StreamedFile[source]

Open a file whose reads and writes are coroutines.

The lookups of the mode, as in open(), run in the threads of the filesystem. If the caller is cancelled, a file opened after the cancellation is closed without committing anything (see create_in_thread()). See AioS3StreamedFile for how the file reads, writes, and commits.

Parameters:
  • path – S3 path (s3://bucket/key) of the file.

  • mode – The file mode: rb, wb, ab, or xb.

  • block_size – The block size for reads and writes. Defaults to the default block size of the filesystem.

  • max_workers – The number of parallel requests of the file, in the request pool of the filesystem. Defaults to the max_workers of the filesystem.

  • version_id – The version ID to read.

  • s3_additional_kwargs – Additional parameters for the S3 requests of the file, which take precedence over those of the filesystem.

  • **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Returns:

The opened file.

Raises:
  • ValueError – If the mode is not binary, compression is requested, or the arguments are invalid, as in open().

  • NotImplementedError – If the mode is not supported, or the file is opened for writing in a transaction or with autocommit=False, as the file commits when it is closed.

  • FileExistsError – If an object exists at the path in exclusive-create mode.

  • FileNotFoundError – If no object exists at the path when reading.

rmdir(path: str | PathLike[str]) → None[source]

Remove an S3 bucket, which must be empty.

See S3FileSystem.rmdir().

Parameters:

path – S3 bucket path (e.g., “s3://bucket”).

sign(path: str | PathLike[str], expiration: int = 3600, **kwargs) → str[source]

Generate a presigned URL for S3 object access.

See S3FileSystem.sign().

Parameters:
  • path – S3 path (s3://bucket/key) to generate the URL for.

  • expiration – URL expiration time in seconds.

  • **kwargs – Additional parameters passed to S3FileSystem.sign().

Returns:

The presigned URL.

metadata(path: str | PathLike[str], **kwargs) → S3Metadata[source]

Return the metadata of the path.

See S3FileSystem.metadata().

Parameters:
  • path – S3 path (s3://bucket/key) to get metadata for.

  • **kwargs – Additional parameters passed to the HeadObject API.

Returns:

S3Metadata of the object.

getxattr(path: str | PathLike[str], attr_name: str, **kwargs) → str | None[source]

Get an attribute from the user-defined metadata of the path.

See S3FileSystem.getxattr().

Parameters:
  • path – S3 path (s3://bucket/key) to get the attribute for.

  • attr_name – The name of the attribute.

  • **kwargs – Additional parameters passed to the HeadObject API.

Returns:

The value of the attribute, or None if the attribute is not set.

setxattr(path: str | PathLike[str], copy_kwargs: dict[str, Any] | None = None, **kwargs) → None[source]

Set the user-defined metadata of the path.

See S3FileSystem.setxattr().

Parameters:
  • path – S3 path (s3://bucket/key) to set metadata for.

  • copy_kwargs – Additional parameters to use for the underlying CopyObject API call.

  • **kwargs – Key-value pairs of metadata to set; a None value deletes the key.

get_tags(path: str | PathLike[str]) → dict[str, str][source]

Retrieve the tag key/values for the given path.

See S3FileSystem.get_tags().

Parameters:

path – S3 path (s3://bucket/key) to get tags for.

Returns:

Dictionary mapping tag keys to tag values.

put_tags(path: str | PathLike[str], tags: dict[str, str], mode: str = 'o') → None[source]

Set the tags for the given existing key.

See S3FileSystem.put_tags().

Parameters:
  • path – S3 path (s3://bucket/key) of the existing key.

  • tags – Tags to apply.

  • mode – o to overwrite the existing tags or m to merge with them.

chmod(path: str | PathLike[str], acl: str, recursive: bool = False, **kwargs) → None[source]

Set the canned ACL of a bucket or key.

See S3FileSystem.chmod().

Parameters:
  • path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.

  • acl – The canned ACL to apply.

  • recursive – Whether to apply the ACL to all keys below the path too.

  • **kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.

object_version_info(path: str | PathLike[str], delete_markers: bool = False, **kwargs) → list[S3ObjectVersion][source]

List the versions of the object or of the objects under the path.

See S3FileSystem.object_version_info().

Parameters:
  • path – S3 path (s3://bucket/key or a key prefix) to list the versions for.

  • delete_markers – Whether to include delete markers in the result.

  • **kwargs – Additional parameters passed to the ListObjectVersions API.

Returns:

List of S3ObjectVersion instances describing the versions.

list_multipart_uploads(path: str | PathLike[str]) → list[S3MultipartUpload][source]

List in-progress (incomplete) multipart uploads in a bucket.

See S3FileSystem.list_multipart_uploads().

Parameters:

path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).

Returns:

List of S3MultipartUpload instances describing the uploads.

clear_multipart_uploads(path: str | PathLike[str]) → None[source]

Abort any incomplete multipart uploads in the bucket.

See S3FileSystem.clear_multipart_uploads().

Parameters:

path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).

checksum(path: str | PathLike[str], **kwargs) → int[source]

Get the checksum of an S3 object or directory.

See S3FileSystem.checksum().

Parameters:
  • path – S3 path (s3://bucket/key) to get the checksum for.

  • **kwargs – Additional arguments passed to S3FileSystem.checksum().

Returns:

Integer checksum derived from the ETag or the directory token.

created(path: str | os.PathLike[str]) → datetime[source]

Return the creation time of the path.

See S3FileSystem.created().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

modified(path: str | os.PathLike[str]) → datetime[source]

Return the last-modified time of the path.

See S3FileSystem.modified().

Parameters:

path – S3 path (s3://bucket/key).

Returns:

The last-modified time of the object.

invalidate_cache(path: str | PathLike[str] | None = None) → None[source]

Remove the cached entries of the path and its parent paths.

See S3FileSystem.invalidate_cache().

Parameters:

path – The path to invalidate. If None, clear the whole cache.

touch(path: str | PathLike[str], truncate: bool = True, **kwargs) → dict[str, Any][source]

Create an empty object with PutObject.

See S3FileSystem.touch().

Parameters:
  • path – S3 path (s3://bucket/key) of the object.

  • truncate – If True, replace an existing object with an empty one; if False, raise if the object exists.

  • **kwargs – Additional parameters passed to the PutObject API.

Returns:

The PutObject response as a dictionary.

class pyathena.filesystem.s3_async.AioS3File(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]

The file handle that AioS3FileSystem.open() returns.

Functionally identical to S3File, whose parallel operations (range reads, multipart uploads) it runs in the request pool of the filesystem, at most max_workers of them at once; it exists as a distinct type for isinstance checks. For reads and writes that are coroutines, see AioS3StreamedFile.

class pyathena.filesystem.s3_async.AioS3StreamedFile(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]

An S3 file whose reads and writes are coroutines.

Returned by AioS3FileSystem.open_async(). The requests of the file run in the request pool of the filesystem, at most max_workers of them at once, and the coroutines wait for them without blocking the event loop:

  • A read splits the requested range into ranges of the block size, read in parallel. Nothing is cached. As with S3File, the reads are conditional on the ETag of the object, and on its version when the filesystem is version-aware.

  • A write buffers the data, and uploads each full block as a part of a multipart upload, with at most max_workers parts, including the part copies of an append, in flight. A write that would submit another part waits for one of them to finish first. A request found failed by the next write, or before a part is submitted, stops the write with its error. An append copies an existing object with UploadPartCopy, or rewrites one smaller than a part from the buffer, as S3File does.

  • Closing the file commits the written data, as S3File.close() does: with PutObject if no part was uploaded, otherwise by completing the multipart upload, which is aborted if the completion fails.

  • Leaving an async with block with an exception, a failed write, and a cancelled write close the file without committing it: the requests that have started are waited for and the multipart upload is aborted, so that the existing object is left unchanged.

The commit and the abort, which wait for the requests, run to their end in the threads of the filesystem, also if the task that waits for them is cancelled, including a commit that still waits for a thread; once the commit has been submitted, nothing else aborts the upload. In a forked child process, a file that the parent opened raises RuntimeError from its reads, writes, and flushes, and closing it only marks it closed. A file opened for writing must be closed with await close() or an async with block; garbage collection cannot await it, so the data of a file that is collected before it is closed or its commit has been submitted is never committed; its multipart upload, if any, is aborted, with a ResourceWarning.

fs: AioS3FileSystem
buffer: BytesIO | None
forced: bool
offset: int | None
__init__(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) → None[source]

Initialize the file for the path and mode.

Sends the lookup requests of the mode (see S3FileTarget), which block; AioS3FileSystem.open_async() runs this in a thread.

Parameters:
  • fs – The filesystem that the file belongs to.

  • path – S3 path (s3://bucket/key) of the file.

  • mode – The file mode: rb, wb, ab, or xb.

  • version_id – The version ID to read.

  • max_workers – The number of parallel requests of the file, in the request pool of the filesystem. If None, the max_workers of the filesystem.

  • block_size – The block size for reads and writes. Must be between S3Core.MULTIPART_UPLOAD_MIN_PART_SIZE and S3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.

  • size – The size of the object to read, if known.

  • s3_additional_kwargs – Additional parameters for the S3 requests of the file. Each request receives those that its operation accepts.

  • **kwargs – Additional parameters for the S3 requests of the file, which take precedence over s3_additional_kwargs.

Raises:
  • FileExistsError – If an object exists at the path in exclusive-create mode.

  • FileNotFoundError – If no object exists at the path when reading.

  • NotImplementedError – If the mode is not supported.

  • ValueError – If max_workers is not positive, or the arguments are invalid as for S3FileTarget.resolve().

__enter__() → AioS3StreamedFile[source]

Reject a synchronous with block, whose exit cannot close the file.

Raises:

TypeError – Always; use async with.

async read(length: int = -1) → bytes[source]

Read bytes from the file.

Parameters:

length – The number of bytes to read; -1 reads to the end.

Returns:

The bytes read.

Raises:
  • RuntimeError – In a forked child process of the process that opened the file.

  • ValueError – If the file is closed or not open for reading.

async write(data: bytes) → int[source]

Write data to the buffer, uploading each full block as a part.

Parameters:

data – The bytes to write.

Returns:

The number of bytes written.

Raises:
  • RuntimeError – In a forked child process of the process that opened the file.

  • ValueError – If the file is not open for writing, is closed, or has been force-flushed.

  • BaseException – The error of a request of the write that failed since the last write, or before a part is submitted, after which the file is closed without committing it.

async flush(force: bool = False) → None[source]

Upload the buffered data if it fills a block, or all of it if forced.

Parameters:

force – Whether this is the final flush of the file, after which the written data is committed.

Raises:
  • RuntimeError – In a forked child process of the process that opened the file.

  • ValueError – If the file is closed, or force-flushed twice.

async close() → None[source]

Close the file, committing the written data.

Returns once the requests of the file and its scheduled commit have ended, also when the close or an earlier read or flush was cancelled; the event loop runs meanwhile. In a forked child process, a file that the parent opened is only marked closed, without committing or aborting anything.

Read-ahead Stream

class pyathena.filesystem.reader.ReadAheadStream(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None)[source]

Sequential stream over a file that reads ranges ahead in parallel.

The file is read in parts of part_size bytes with a range-read function, and up to max_parts parts are read in a thread pool ahead of the consumer, so the transfer is not limited to one request at a time. Parts are returned in order, and at most max_parts parts plus the one being consumed are held in memory. A read that returns fewer bytes than requested raises OSError, and an error from the range-read function is raised by the read that reaches its part; every later read raises the same error.

Closing the stream cancels the queued reads, waits for the running ones, and then calls on_close.

Example

>>> file = fs.open_input_file(path)
>>> stream = ReadAheadStream(
...     lambda offset, size: file.read_at(size, offset), file.size(), on_close=file.close
... )
>>> with pyarrow.PythonFile(stream, mode="r") as f:
...     table = pyarrow.csv.read_csv(f)
DEFAULT_PART_SIZE: int = 8388608

Bytes requested by one range read unless part_size is given.

DEFAULT_MAX_PARTS: int = 16

The most parts read ahead at once unless max_parts is given.

__init__(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None) → None[source]

Initialize the stream without reading the file.

Parameters:
  • read_range – A function that reads size bytes of the file from offset, called as read_range(offset, size) from several threads at once.

  • size – The size of the file in bytes.

  • part_size – Bytes requested by one read_range() call; if None, DEFAULT_PART_SIZE.

  • max_parts – The most parts read ahead of the consumer at once; if None, DEFAULT_MAX_PARTS.

  • on_close – A function called once when the stream is closed, after the running reads have finished, such as the close() of the file.

Raises:

ValueError – If part_size or max_parts is not positive.

readable() → bool[source]

Return whether object was opened for reading.

If False, read() will raise OSError.

tell() → int[source]

Return current stream position.

readinto(buffer: Any) → int[source]

Fill buffer from the parts read ahead.

Parameters:

buffer – A writable buffer.

Returns:

The number of bytes read, less than the size of buffer only at the end of the file.

Raises:

ValueError – If the stream is closed, or its closing was interrupted.

close() → None[source]

Cancel the queued reads, wait for the running ones, and call on_close.

If the wait is interrupted, the stream stays open and on_close is not called, so that the file is not closed under the running reads; closing again waits for them. Reads raise ValueError from then on.

S3 Executor

class pyathena.filesystem.s3_executor.S3Executor[source]

Abstract executor for parallel S3 operations.

Defines the interface used by S3File and S3FileSystem for submitting work to run in parallel and for shutting down the executor when done. Both submit and shutdown mirror the concurrent.futures.Executor interface so that as_completed() and Future.cancel() work unchanged. Each submitted callable runs with a copy of the context variables of its submitter, so that the stop scopes of stop_retries_on() reach the requests that it makes.

abstractmethod submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T][source]

Submit a callable for execution and return a Future.

An implementation runs the callable in a copy of the submitter’s context, such as with contextvars.copy_context().run, so that the stop scopes of the submitter reach it.

abstractmethod shutdown(wait: bool = True) → None[source]

Shut down the executor, freeing any resources.

async run(fn: Callable[[...], T], *args: Any, **kwargs: Any) → T[source]

Run a blocking call in the executor and wait for it.

Cancelling the waiting task cancels a call that has not started. A running call makes no further attempt of its AWS API calls, as with run_in_thread(), and a write in it sends no further part and does not commit (see raise_if_stopped()); the task waits for the call to end before it re-raises the cancellation, also when the event loop shuts down with asyncio.run(). A repeated cancellation stops that wait, and the call then ends in the executor.

Parameters:
  • fn – The blocking callable.

  • *args – Positional arguments passed to the callable.

  • **kwargs – Keyword arguments passed to the callable.

Returns:

The return value of the call.

Raises:

RuntimeError – If the executor has been shut down.

async run_shielded(fn: Callable[[...], T], *args: Any, **kwargs: Any) → T[source]

Run a blocking call that a cancellation of the waiting task does not stop.

The call runs to its end, even if it has not started when the waiting task is cancelled, and its retries are not stopped; its outcome is then dropped. See wait_shielded().

Parameters:
  • fn – The blocking callable.

  • *args – Positional arguments passed to the callable.

  • **kwargs – Keyword arguments passed to the callable.

Returns:

The return value of the call.

Raises:

RuntimeError – If the executor has been shut down.

async wait_shielded(future: Future[T]) → T[source]

Wait for a submitted call that a cancellation of the waiting task does not cancel.

Unlike awaiting asyncio.wrap_future(future), a cancellation of the waiting task leaves the future to the caller, which can still wait for it or cancel it, and its outcome is retrieved, so that asyncio does not log a failure that ends after the cancellation as never retrieved.

Parameters:

future – The future of a call submitted to this executor.

Returns:

The return value of the call.

static waiter(future: Future[T]) → Future[T][source]

Wrap a future of an executor for asyncio, retrieving its outcome.

The outcome belongs to the caller that awaits the wrapper; it is retrieved, so that asyncio does not log a failure as never retrieved when no one awaits it, such as after a cancellation.

Parameters:

future – The future.

Returns:

The asyncio future.

submit_to(future: Future[T], fn: Callable[[...], T], *args: Any, **kwargs: Any) → None[source]

Submit a callable whose outcome resolves a future of the caller.

The caller creates the future before calling this method, so that it keeps a reference to the future even if an interrupt stops the scheduling midway, before submit() returns. Cancelling the future keeps a callable that has not started from running; a started callable sets the future’s result or exception. If the executor drops the callable before it starts, such as by cancelling it at its shutdown, the future is cancelled or gets the error that the executor reported.

The arguments of a failed callable, such as the body of a part, are freed when it fails: its error keeps the frames that it was raised from, without their local variables. The errors that it was raised from keep theirs; S3Core.call() clears those of the ClientError of a request.

Parameters:
  • future – A pending future that no executor has started.

  • fn – The callable to run.

  • *args – Positional arguments passed to the callable.

  • **kwargs – Keyword arguments passed to the callable.

map(fn: Callable[[ItemT], T], items: Iterable[ItemT], *, stop_on_failure: bool = True, counts: Callable[[BaseException], bool] | None = None) → S3MapResult[T][source]

Run a callable for each item in the executor and wait for the calls.

Every item is submitted at once, and the executor applies its own limit, such as the max_workers of a scope. Each call runs in a scope of stop_retries_on() that the stop of the batch ends.

With stop_on_failure, the first failure that counts stops the batch: the calls that have not started are cancelled, the running ones make no further attempt, and the failure is raised once they have ended. An interrupt of the waiting thread, such as KeyboardInterrupt, does the same whatever stop_on_failure is, and is raised once the running calls have ended. A further interrupt stops that wait; the running calls then end in the executor.

A thread of a scope’s pool that maps on that scope runs the calls itself. A thread of a bare S3ThreadPool that maps on that pool waits for threads that it may hold, so a pool must not map on itself.

Parameters:
  • fn – The callable to run for each item.

  • items – The items.

  • stop_on_failure – Whether a failure that counts stops the batch.

  • counts – Whether a failure counts, by default every one. A failure that does not count neither stops the batch nor becomes S3MapResult.failure.

Returns:

The outcomes of the calls.

Raises:
async map_async(fn: Callable[[ItemT], T], items: Iterable[ItemT], *, stop_on_failure: bool = True, counts: Callable[[BaseException], bool] | None = None) → S3MapResult[T][source]

Run a callable for each item in the executor and await the calls.

As map(), with a cancellation of the waiting task in place of an interrupt. The task waits for the running calls also when the event loop shuts down with asyncio.run(), which cancels it and runs the loop until it ends; a loop that is closed without cancelling it leaves the running calls to end in the executor.

Parameters:
  • fn – The callable to run for each item.

  • items – The items.

  • stop_on_failure – Whether a failure that counts stops the batch.

  • counts – Whether a failure counts, by default every one.

Returns:

The outcomes of the calls.

Raises:
class pyathena.filesystem.s3_executor.S3ThreadPool(max_workers: int | None = None)[source]

A thread pool for the requests of the S3 filesystems.

It runs at most max_workers calls at once in threads of its own, so that, unlike asyncio.to_thread(), which runs a call in the default executor of the event loop, neither the size nor the load of that executor limits or blocks the calls of asyncio code. The calls are queued in the order they were submitted. The threads are created as the calls need them, and each call runs with a copy of the caller’s context variables, as with asyncio.to_thread(). In a forked child process, whose copy has no threads, the pool creates new ones, as it does after release(). The pool can be shared by several threads and event loops.

S3ThreadPoolExecutor and S3AioThreadPool are aliases of this class.

Parameters:

max_workers – The maximum number of calls that run at once.

Raises:

ValueError – If max_workers is not positive.

__init__(max_workers: int | None = None) → None[source]

Initialize the pool without starting any thread.

Parameters:

max_workers – The maximum number of calls that run at once. If None, the default of ThreadPoolExecutor: min(32, CPUs + 4).

Raises:

ValueError – If max_workers is not positive.

submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) → Future[T][source]

Schedule a blocking call in a thread of the pool.

As with ThreadPoolExecutor, the returned future can be cancelled only before the call starts. The calls are queued in the order they were submitted, so that the calls submitted before a call that waits for them are taken by threads before it, and it does not wait for a thread that it holds.

Parameters:
  • fn – The blocking callable.

  • *args – Positional arguments passed to the callable.

  • **kwargs – Keyword arguments passed to the callable.

Returns:

The future of the call.

Raises:

RuntimeError – If the pool has been shut down.

shutdown(wait: bool = True) → None[source]

Stop accepting calls and release the threads once they are idle.

The calls that were submitted before still run.

Parameters:

wait – Whether to wait for the submitted calls to finish.

release() → None[source]

Release the threads once no call runs or waits, keeping the pool usable.

Until then, the calls, including those submitted after this method, run in the threads of the pool, at most max_workers at once. A call submitted after the threads have been released creates new ones. This method does not block.

owns_current_thread() → bool[source]

Whether the calling thread is a thread of this pool.

Returns:

True in a thread that this pool started.

static in_pool_thread() → bool[source]

Whether the calling thread is a thread of any pool of this type.

Returns:

True in a thread that a pool of this type started, also once the pool has been collected.

pyathena.filesystem.s3_executor.S3ThreadPoolExecutor = <class 'pyathena.filesystem.s3_executor.S3ThreadPool'>

An alias of S3ThreadPool, kept for compatibility.

pyathena.filesystem.s3_executor.S3AioThreadPool = <class 'pyathena.filesystem.s3_executor.S3ThreadPool'>

An alias of S3ThreadPool, kept for compatibility.

class pyathena.filesystem.s3_executor.S3AioExecutor(loop: AbstractEventLoop | None = None, max_workers: int = 20)[source]

A thread pool kept for compatibility, an S3ThreadPool.

It used to dispatch the calls through an asyncio event loop. The calls now run directly in the pool, so they need no running loop, and they run to their end when the loop stops.

Parameters:
  • loop – Ignored, with a DeprecationWarning if given.

  • max_workers – The maximum number of calls that run at once.

Raises:

ValueError – If max_workers is not positive.

__init__(loop: AbstractEventLoop | None = None, max_workers: int = 20) → None[source]

Initialize the pool without starting any thread.

Parameters:
  • loop – Ignored, with a DeprecationWarning if given.

  • max_workers – The maximum number of calls that run at once.

Raises:

ValueError – If max_workers is not positive.

S3 Paths

class pyathena.filesystem.s3_path.S3Path(bucket: str, key: str | None = None, version_id: str | None = None)[source]

An S3 path: a bucket, optionally a key, and optionally a version ID.

Paths are parsed from strings such as s3://bucket/key, s3a://bucket/key or bucket/key, optionally followed by a version ID query (?versionId=, ?versionID=, ?versionid= or ?version_id=). Only a query at the end of the path is a version ID; any other ? is part of the key. The root path, which names no bucket, is not an S3Path.

bucket

The name of the bucket.

Type:

str

key

The key, or None for a bucket path. An empty key is stored as None. A trailing slash is kept, and a key of only slashes (bucket//) names the bucket (see is_bucket).

Type:

str | None

version_id

The version ID, or None for a path without a version.

Type:

str | None

Example

>>> path = S3Path.parse("s3://bucket/dir/key?versionId=v1")
>>> path.bucket, path.key, path.version_id
('bucket', 'dir/key', 'v1')
>>> path.name
'bucket/dir/key'
>>> str(path)  # The name, with a version ID query if there is one.
'bucket/dir/key?versionId=v1'
>>> path.uri
's3://bucket/dir/key?versionId=v1'
VERSION_QUERY: ClassVar[Pattern[str]] = re.compile('\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z')
PATTERN: ClassVar[Pattern[str]] = re.compile('(^s3://|^s3a://|^)(?P<bucket>[a-zA-Z0-9.\\-_]+)(/|/(?P<key>.+?))?(\\Z|\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z)', re.DOTALL)
bucket: str
key: str | None = None
version_id: str | None = None
__post_init__() → None[source]

Store an empty key as None, so that the path is the bucket path it prints as.

classmethod parse(path: str) → S3Path[source]

Parse a string into an S3 path.

Parameters:

path – The S3 path (e.g., “s3://bucket/key?versionId=…”).

Returns:

The path. The key is None for a bucket path (bucket or bucket/), and keeps a trailing slash otherwise.

Raises:

ValueError – If the string is not a valid S3 path.

classmethod of(path: str | S3Path) → S3Path[source]

Return a path string parsed, or an S3 path as it is.

An S3Path, such as a key that a listing returns, is never parsed again, so that a key that ends in what looks like a version ID query keeps naming that key.

Parameters:

path – The S3 path string, as for parse(), or an S3Path.

Returns:

The path.

Raises:

ValueError – If the string is not a valid S3 path.

classmethod split_version_id(path: str) → tuple[str, str | None][source]

Split the version ID query from the end of a path string.

Unlike parse(), any string is accepted, such as the names that fsspec builds.

Parameters:

path – The path string.

Returns:

Tuple of the string without the version ID query and the version ID, or of the string itself and None if it ends with no version ID query.

classmethod has_version_id(path: str | PathLike[str]) → bool[source]

Return whether a path string ends with a version ID query.

Unlike parse(), any string is accepted, as with split_version_id().

Parameters:

path – The path.

Returns:

Whether the path ends with a version ID query.

property is_bucket: bool

it has no key, or a key of only slashes.

Type:

Whether the path names the bucket

property is_directory_bucket: bool

Whether the bucket is a directory bucket (S3 Express One Zone).

The names of directory buckets end with --x-s3.

property name: str

The path without a scheme or version, in bucket/key form, or the bucket.

property uri: str

The path as an s3:// URI, with its version ID query, if any.

property target: S3Path

The default move target, assuming bucket versioning is not enabled.

A null version is taken to name the key itself: in a bucket without versioning, or with versioning suspended, a write to the key replaces its null version. With versioning enabled, a write adds a new version instead, so callers with that state must keep the null version distinct from its key. Any other path is its own target.

with_version_id(version_id: str | None) → S3Path[source]

Return the path with another version ID.

Parameters:

version_id – The version ID, or None for the path without a version.

Returns:

The path with the version ID.

__init__(bucket: str, key: str | None = None, version_id: str | None = None) → None
class pyathena.filesystem.s3_path_pairing.S3PathPairing(path1: str | list[str], path2: str | list[str], recursive: bool = False, maxdepth: int | None = None)[source]

The pairing of the paths of one copy(), get() or mv().

The sources are paired with their destinations as fsspec’s copy() pairs them, except that a path with a version ID names that version of an object: it is not a glob pattern, and its destination is named after its key without the version. A source may also be an S3Path, such as a key that a listing returns, which names its bucket and key as they are, also when the key ends in what looks like a version ID query, and which is never parsed again (see s3_copy_pairs()). A move compares the paths by what they name (see target), keeping null versions distinct from their keys in the caller’s versioning-enabled buckets. delete_paths() splits the paths of an rm().

The pairing is a frozen dataclass of the paths as given, and holds no filesystem. The caller makes the lookups that it asks for (expands, skips_directories, looks_up_destination and conflict_candidates()) with its own requests and cache: it expands the sources, leaves out the directories when skips_directories says so, and passes the results in. A sources, destination_is_dir or missing that a rule needs and that is not passed raises ValueError.

path1

The source path, glob pattern, or list of them, as given to the copy.

Type:

str | list[str]

path2

The destination path, or list of paths.

Type:

str | list[str]

recursive

Whether the copy includes the contents of directories.

Type:

bool

maxdepth

The maximum depth of the expansion.

Type:

int | None

Example

>>> pairing = S3PathPairing("s3://bucket/dir/", "s3://bucket/copy/", recursive=True)
>>> sources = fs.expand_path(pairing.path1, recursive=True)
>>> pairing.copy_pairs(sources)
path1: str | list[str]
path2: str | list[str]
recursive: bool = False
maxdepth: int | None = None
property expands: bool

Whether the sources are expanded.

False if both paths are lists, which are paired as given, without expansion or lookups.

property skips_directories: bool

Whether the directories among the expanded sources are left out.

True for a string source copied without recursive, or with a maxdepth. A directory is a source that ends with a slash or that the filesystem reports as a directory.

property looks_up_destination: bool

Whether the pairing depends on the destination being a directory.

True for a string source that is neither a glob pattern nor ends with a slash, copied to a string destination that does not end with a slash. copy_pairs() then needs destination_is_dir.

copy_pairs(sources: Sequence[str | S3Path] | None = None, destination_is_dir: bool | None = None) → list[tuple[str | S3Path, str]][source]

Pair the sources with their destinations as fsspec’s copy() does.

A source string with a version ID is named after its key without the version, and an S3Path source after its bucket and key (its name), whose suffix the destination keeps as it is.

Parameters:
  • sources – The expansion of path1, without the directories that skips_directories leaves out; needed when expands.

  • destination_is_dir – Whether path2 is a directory; needed when looks_up_destination is true and there are sources.

Returns:

The sources and their destinations, which keep the form of path2. When both paths are lists, they are paired as given, up to the end of the shorter list, as in fsspec’s copy(). Empty if there are no sources.

Raises:

ValueError – If sources or destination_is_dir is needed and None.

s3_copy_pairs(sources: Sequence[str | S3Path] | None = None, destination_is_dir: bool | None = None) → list[tuple[str | S3Path, str | S3Path]][source]

Pair the sources with their S3 destinations, as copy_pairs() does.

The destination of an S3Path source, when path2 is a string, is an S3Path too: path2 parsed, with the suffix that copy_pairs() appends to it added to its key as it is, so that a key such as a/x?versionId=null is copied to the key b/x?versionId=null instead of being parsed as a version. The other destinations are strings, as copy_pairs() returns them.

Parameters:
Returns:

The sources and their destinations.

Raises:

ValueError – If sources or destination_is_dir is needed and None, or if path2 names a version, which cannot be written, and a destination is an S3Path.

conflict_candidates(pairs: Sequence[tuple[str | S3Path, str | S3Path]], *, versioning_enabled_buckets: Collection[str] = ()) → list[str | S3Path][source]

Return the sources of a move or a copy whose conflicts depend on an object at their key.

A source without a version may be a directory, such as a prefix given in a list or one with other sources below it. If no object exists at its key, it is not copied, so it writes no destination and is left out of the conflict checks of move_pairs() and check_copy_pairs(). A path with a version always names an object.

Parameters:
  • pairs – The sources and destinations of the move or the copy, as copy_pairs() or s3_copy_pairs() of this pairing returns them. A string is parsed; an S3Path is used as it is.

  • versioning_enabled_buckets – Buckets whose versioning is enabled. Their null versions are distinct from their unversioned keys. Use the same collection for move_pairs() or check_copy_pairs().

Returns:

The sources without a version whose destination conflicts, one per pair in the order of the pairs, as their target: in bucket/key form for a source string, and as an S3Path for an S3Path source. Empty if nothing needs to be looked up.

Raises:

TypeError – If versioning_enabled_buckets is a string.

move_pairs(pairs: Sequence[tuple[str | S3Path, str | S3Path]], missing: Collection[str | S3Path] | None = None, *, versioning_enabled_buckets: Collection[str] = ()) → list[tuple[str | S3Path, str | S3Path]][source]

Check the pairs of a move and leave out the sources that stay in place.

Parameters:
  • pairs – The sources and destinations of the move, as copy_pairs() or s3_copy_pairs() of this pairing returns them.

  • missing – The conflict_candidates() without an object at their key, as strings in any form that names them or as S3Path objects; needed when there are candidates.

  • versioning_enabled_buckets – Buckets whose versioning is enabled. Their null versions are distinct from their unversioned keys. Use the same collection for conflict_candidates().

Returns:

The pairs, except those whose destination is the source itself or, for a null version in a bucket without enabled versioning, the key of the source.

Raises:
  • ValueError – If two sources have the same destination, or a destination is another source, including one left in place, except for a source without a version in missing, which writes no destination. Also if missing is needed and None.

  • TypeError – If missing is a string instead of a collection. Also if versioning_enabled_buckets is a string.

check_copy_pairs(pairs: Sequence[tuple[str | S3Path, str | S3Path]], missing: Collection[str | S3Path] | None = None, *, versioning_enabled_buckets: Collection[str] = ()) → None[source]

Check the pairs of a copy whose objects are copied in parallel.

A copy has the conflicts of a move (see move_pairs()), since a parallel copy has no order in which one pair would overwrite another or read a source after another pair has written it. A pair whose destination is its own source is no conflict: it rewrites the object in place, such as with MetadataDirective="REPLACE".

Parameters:
  • pairs – The sources and destinations of the copy, as copy_pairs() or s3_copy_pairs() of this pairing returns them.

  • missing – The conflict_candidates() without an object at their key, as strings in any form that names them or as S3Path objects; needed when there are candidates.

  • versioning_enabled_buckets – Buckets whose versioning is enabled. Use the same collection for conflict_candidates().

Raises:
  • ValueError – If two sources have the same destination, or a destination is another source, except for a source without a version in missing, which writes no destination. Also if missing is needed and None.

  • TypeError – If missing is a string instead of a collection. Also if versioning_enabled_buckets is a string.

classmethod delete_paths(path: str | list[str]) → tuple[list[str], list[str]][source]

Split the paths that rm() deletes into those deleted as given and those expanded.

Parameters:

path – S3 path or list of paths.

Returns:

The paths with a version ID, which are deleted as given, and the other paths, which the filesystem expands.

Raises:

ValueError – If a path is a bucket.

__init__(path1: str | list[str], path2: str | list[str], recursive: bool = False, maxdepth: int | None = None) → None

S3 Core

class pyathena.filesystem.s3_core.S3Core(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None)[source]

Typed S3 operations on a boto3 S3 client.

Each operation sends one request, or one per page for the iterators, except those that say otherwise, such as plan_multipart_copy(), copy_object_annotation() and generate_presigned_url(). The requests are sent with the retry policy, and S3 errors are translated into OSError subclasses (see S3ClientError): a missing bucket or multipart upload, or a missing object or version that an operation reads, raises FileNotFoundError, and a denied request PermissionError. copy_object() and upload_part_copy() with a source version that is a delete marker raise OSError with EINVAL. As in S3, deleting a missing key is not an error. Nothing is cached.

Example

>>> core = S3Core(boto3.client("s3"))
>>> core.head_object(S3Path.parse("s3://bucket/key")).content_length
>>> for page in core.list_objects("bucket", prefix="dir/", delimiter="/"):
...     print([o.key for o in page.objects])
MULTIPART_UPLOAD_MIN_PART_SIZE: int = 5242880
MULTIPART_UPLOAD_MAX_PART_SIZE: int = 5368709120
MULTIPART_UPLOAD_MAX_PARTS: int = 10000
OBJECT_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'aws-exec-read', 'bucket-owner-full-control', 'bucket-owner-read', 'private', 'public-read', 'public-read-write'})
BUCKET_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'private', 'public-read', 'public-read-write'})
RETRY_ERROR_CODES: ClassVar[tuple[str, ...]] = ('SlowDown', 'ServiceUnavailable', 'InternalError', 'RequestTimeout')

The S3 error codes of transient failures (status 503, 500, and 400), which botocore’s retry modes also retry. A retry policy with the default codes retries them too.

__init__(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None) → None[source]

Create the operations on a client.

Parameters:
  • client – The boto3 S3 client to send the requests with.

  • retry_config – The retry policy of the requests; defaults to RetryConfig(). The requests are sent with the policy that resolve_retry_config() returns for it.

  • request_kwargs – Parameters sent with every operation that accepts them, such as {"RequestPayer": "requester"}.

classmethod resolve_retry_config(retry_config: RetryConfig) → RetryConfig[source]

Resolve the retry policy that S3 requests are sent with.

A policy whose exceptions are the default codes (THROTTLING_ERROR_CODES), in any order, also retries RETRY_ERROR_CODES. A policy with other exceptions is used as given.

Parameters:

retry_config – The retry policy.

Returns:

A copy of the policy that also retries RETRY_ERROR_CODES with the same attempts and waits, or the policy itself.

property client: BaseClient

The boto3 S3 client.

property retry_config: RetryConfig

The retry policy of the requests, with RETRY_ERROR_CODES if they were added.

property request_kwargs: dict[str, Any]

A copy of the parameters sent with every operation that accepts them.

operation_params(method: str, params: Mapping[str, Any]) → dict[str, Any][source]

Select the parameters that an S3 operation accepts.

Parameters inherited by several requests are filtered by the input shape of each operation, so that, e.g., ServerSideEncryption for writes is not sent with GetObject.

Parameters:
  • method – The name of the client method, such as get_object.

  • params – The parameters to select from.

Returns:

The parameters that the operation accepts. Empty for a method that is not an S3 API operation, such as generate_presigned_url.

call(method: str | Callable[[...], Any], **request) → dict[str, Any][source]

Send a request with the retry policy and translate its errors.

request_kwargs that the operation accepts are added; a parameter of the request takes precedence. The operation is identified by the name of the method; a functools.partial is identified by the method it wraps, and its bound keywords are parameters of the request. A callable whose __name__ is not that of a client method is sent without them.

A Body that is a stream, given to the call or bound by a functools.partial, is sent from the position where the call found it on each attempt of the policy. One whose position cannot be read with tell() and restored with seek() is sent once, without the retries of the policy, since a retry would send only the rest of the stream.

Parameters:
  • method – The name of the client method, or the method itself.

  • **request – The request parameters, sent as given.

Returns:

The response.

Raises:
  • ValueError – If botocore rejects the parameters of the request (ParamValidationError), which is not sent.

  • OSError – The translation of an S3 error response (see S3ClientError).

Note

The frames of a raised error below this method, those of the retries and of botocore, have no local variables, so that they keep neither the request, such as the body of a part, nor the retry state, which keeps the error in a reference cycle.

head_object(path: S3Path, **params) → S3Metadata[source]

Look up an object, or a version of it, with HeadObject.

Parameters:
  • path – The path of the object, with the version ID to look up, if any.

  • **params – Additional request parameters, sent as given.

Returns:

The metadata of the object.

Raises:
head_bucket(bucket: str, **params) → S3Bucket[source]

Look up a bucket with HeadBucket.

Parameters:
  • bucket – The name of the bucket.

  • **params – Additional request parameters, sent as given.

Returns:

The bucket, with its region if returned.

Raises:

FileNotFoundError – If the bucket does not exist.

get_object(path: S3Path, range_: tuple[int, int | None] | None = None, **params) → bytes[source]

Read an object, a version of it, or a byte range of it with GetObject.

The body of the response is read whole and then closed, also when the read fails. A failure to read the body is neither translated nor retried: the botocore exception, such as ReadTimeoutError or IncompleteReadError, propagates.

Parameters:
  • path – The path of the object, with the version ID to read, if any.

  • range – The (start, end) byte range to read, with an exclusive end, or None as the end to read to the end of the object. A negative start without an end reads the last -start bytes, or the whole object if it is shorter. None sends no range of its own, so the whole object is read unless params or request_kwargs have Range.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The bytes read.

Raises:
  • ValueError – If the path has no key, or the range is empty or has both a negative start and an end, which S3 would ignore and return the whole object for.

  • FileNotFoundError – If the object or version does not exist.

  • OSError – If the range starts at or past the end of the object. Its __cause__ is the ClientError with the InvalidRange code.

put_object(path: S3Path, body: bytes | None = None, **params) → S3PutObject[source]

Write an object with PutObject.

Parameters:
  • path – The path of the object to write, without a version ID.

  • body – The data to write. None or empty bytes send no body of their own, so an empty object is written unless params or request_kwargs have Body.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The result of the write.

Raises:

ValueError – If the path has no key, or has a version ID, which a write cannot replace.

create_bucket(bucket: str, acl: str | None = None, region_name: str | None = None, **params) → None[source]

Create a bucket with CreateBucket.

A call creates the bucket; the allow_bucket_creation option of the filesystem applies only to the filesystem’s methods.

Parameters:
  • bucket – The name of the bucket.

  • acl – The canned ACL of the bucket, one of BUCKET_ACLS. None or an empty string sends no ACL.

  • region_name – The region to create the bucket in. None or an empty string uses the region of the client. A location constraint is sent for every region except us-east-1, which does not accept one.

  • **params – Additional request parameters, sent as given, except CreateBucketConfiguration. Its fields are kept, and the LocationConstraint of region_name is added unless the region is us-east-1 or the configuration has a LocationConstraint, Location or Bucket. A directory bucket is created with Location and Bucket, so no LocationConstraint is added to its configuration. An empty or None configuration is treated as not given. The given configuration is not modified.

Raises:

ValueError – If the ACL is not a canned ACL of buckets.

delete_bucket(bucket: str, **params) → None[source]

Delete a bucket, which must be empty, with DeleteBucket.

A call deletes the bucket; the allow_bucket_deletion option of the filesystem applies only to the filesystem’s methods.

Parameters:
  • bucket – The name of the bucket.

  • **params – Additional request parameters, sent as given.

Raises:

FileNotFoundError – If the bucket does not exist.

get_bucket_versioning(bucket: str, **params) → str | None[source]

Get the versioning state of a bucket with GetBucketVersioning.

Parameters:
  • bucket – The name of the bucket.

  • **params – Additional request parameters, sent as given.

Returns:

Enabled or Suspended, or None if versioning has never been enabled on the bucket.

Raises:

FileNotFoundError – If the bucket does not exist.

delete_object(path: S3Path, **params) → None[source]

Delete an object, or a version of it, with DeleteObject.

Parameters:
  • path – The path of the object, with the version ID to delete, if any.

  • **params – Additional request parameters, sent as given.

Raises:
delete_objects(batch: S3DeleteBatch, **params) → S3DeleteResult[source]

Delete the objects of a batch with one DeleteObjects request.

Parameters:
  • batch – The objects to delete.

  • **params – Additional request parameters, sent as given.

Returns:

The result, with the objects that S3 could not delete in its errors.

Raises:

FileNotFoundError – If the bucket does not exist.

create_multipart_upload(path: S3Path, **params) → S3MultipartUpload[source]

Start a multipart upload to an object with CreateMultipartUpload.

Parameters:
  • path – The path of the object to write, without a version ID.

  • **params – Additional request parameters. The bucket and key of the path take precedence over parameters of the same name.

Returns:

The upload retaining the path’s bucket and key, including an access point alias or ARN, and the response’s upload ID and checksum configuration.

Raises:

ValueError – If the path has no key, or has a version ID, which a write cannot replace.

upload_part(upload: S3MultipartUpload, part_number: int, body: bytes, **params) → S3MultipartUploadPart[source]

Upload a part of a multipart upload with UploadPart.

Parameters:
  • upload – The multipart upload returned by creation. Its checksum algorithm is passed to the SDK to calculate the part checksum.

  • part_number – The number of the part, from 1.

  • body – The data of the part.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The uploaded part.

Raises:

ValueError – If the upload has no bucket, key, or upload ID.

upload_part_copy(upload: S3MultipartUpload, part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params) → S3MultipartUploadPart[source]

Copy a part of a multipart upload from an object with UploadPartCopy.

Parameters:
  • upload – The multipart upload returned by creation.

  • part_number – The number of the part, from 1.

  • source – The path of the object to copy, with the version ID to copy, if any.

  • range – The (start, end) byte range of the source to copy, with an exclusive end. None sends no range of its own, so the whole source is copied unless params or request_kwargs have CopySourceRange.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The copied part.

Raises:
  • ValueError – If the upload has no bucket, key, or upload ID, or the source has no key.

  • FileNotFoundError – If the upload, or the source object or version, does not exist.

  • OSError – With EINVAL for an invalid request, such as a source version that is a delete marker.

complete_multipart_upload(upload: S3MultipartUpload, parts: Sequence[S3MultipartUploadPart], **params) → S3CompleteMultipartUpload[source]

Complete a multipart upload with CompleteMultipartUpload.

Parameters:
  • upload – The multipart upload returned by creation. Its checksum algorithm selects the matching part checksum, and its checksum type is sent when present. Without an algorithm, only the ETag and part number are sent, even if the SDK added a part checksum.

  • parts – The uploaded parts, in part-number order.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The completed upload.

Raises:

ValueError – If the upload has no bucket, key, or upload ID.

abort_multipart_upload(upload: S3MultipartUpload, **params) → None[source]

Abort a multipart upload with AbortMultipartUpload.

An abort cleans up, so the stop scopes of the caller do not stop its retries (see unstoppable_retries()).

Parameters:
  • upload – The multipart upload returned by creation or listing.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Raises:
  • ValueError – If the upload has no bucket, key, or upload ID.

  • FileNotFoundError – If the upload does not exist, for example because it was completed or aborted.

part_ranges(size: int, block_size: int) → list[tuple[int, int]][source]

Split an object into the source ranges of the parts that copy it.

The object is split into ranges of block_size bytes, or of a larger size that splits it into at most MULTIPART_UPLOAD_MAX_PARTS ranges. A last range shorter than MULTIPART_UPLOAD_MIN_PART_SIZE is merged into the previous one, which is split in half if the result exceeds MULTIPART_UPLOAD_MAX_PART_SIZE. Every range is then within the S3 part size limits, including the last one unless the whole object is smaller than the minimum part size, so that more parts can follow the copied ones, as in an append.

Parameters:
  • size – The size of the source object in bytes.

  • block_size – The size in bytes to split the object by, between MULTIPART_UPLOAD_MIN_PART_SIZE and MULTIPART_UPLOAD_MAX_PART_SIZE. It is raised to size divided by MULTIPART_UPLOAD_MAX_PARTS, rounded up, if smaller. The range that a short last range is merged into can be longer, up to MULTIPART_UPLOAD_MAX_PART_SIZE.

Returns:

The (start, end) byte ranges, with an exclusive end, that cover the whole object in order; empty for an empty object.

check_multipart_upload_size(size: int, block_size: int) → None[source]

Check that data fits in a multipart upload before it is uploaded.

Sends no request.

Parameters:
  • size – The size of the data in bytes.

  • block_size – The size in bytes of the parts that upload the data.

Raises:

ValueError – If the data takes more than MULTIPART_UPLOAD_MAX_PARTS blocks. The message gives the minimum block size, which is at least MULTIPART_UPLOAD_MIN_PART_SIZE.

copy_object(source: S3Path, destination: S3Path, **params) → None[source]

Copy an object, or a version of it, with CopyObject.

Parameters:
  • source – The path of the object to copy, with the version ID to copy, if any.

  • destination – The path of the object to write, without a version ID.

  • **params – Additional request parameters, sent as given.

Raises:
  • ValueError – If the source or the destination has no key, or the destination has a version ID, which a write cannot replace.

  • FileNotFoundError – If a bucket, or the source object or version, does not exist.

  • OSError – With EINVAL for an invalid request, such as a source version that is a delete marker.

plan_multipart_copy(source: S3Path, destination: S3Path, block_size: int | None = None, **params) → S3MultipartCopyPlan[source]

Plan a copy with a multipart upload that copies as CopyObject does.

The source is read with HeadObject. Without a version in the path, a version ID other than null that it reports is the version to copy, so that the parts, the tags and the annotations come from the same object even if the source is replaced during the copy. A null version, which a write can replace, is not pinned. If the reported size fits in a single CopyObject request, nothing else is read and the plan says so.

No multipart request accepts the directives of CopyObject, so the plan applies them as CopyObject does. With the COPY metadata directive (the default), the content headers and the user-defined metadata of the source are used, and the values of params are ignored. With the COPY tagging directive (the default), the tags are read with GetObjectTagging, and the Tagging of params is ignored. REPLACE uses the values of params instead. With the COPY annotation directive (the default), the annotations are listed with ListObjectAnnotations, unless the source is encrypted with SSE-C or is in a directory bucket, which cannot have annotations; an object in a directory bucket has no tags either. The requests that read the source receive RequestPayer, and the source’s expected bucket owner and SSE-C parameters (ExpectedSourceBucketOwner and CopySourceSSECustomer*) under their names in those requests.

Parameters:
  • source – The path of the object to copy, with the version ID to copy, if any.

  • destination – The path of the object to write, without a version ID.

  • block_size – The size in bytes of the copied ranges, between MULTIPART_UPLOAD_MIN_PART_SIZE and MULTIPART_UPLOAD_MAX_PART_SIZE (the default); see part_ranges().

  • **params – The CopyObject parameters of the copy. Each request receives those that it accepts; CreateMultipartUpload also receives those that CopyObject does not accept either, so that botocore rejects them as it would for CopyObject.

Returns:

The plan.

Raises:

ValueError – If the source or the destination has no key, the destination has a version ID, block_size is out of the part size limits, a directive has a value that CopyObject does not accept, or HeadObject reports no size.

list_object_annotations(path: S3Path, **params) → list[str][source]

List the names of the annotations of an object with ListObjectAnnotations.

Sends one request per page.

Parameters:
  • path – The path of the object, with the version ID to list, if any.

  • **params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.

Returns:

The annotation names, across all pages.

Raises:

ValueError – If the path has no key.

copy_object_annotation(name: str, source: S3Path, destination: S3Path, version_id: str | None, etag: str | None, **params) → None[source]

Copy an annotation of an object onto the object that a copy wrote.

Reads the annotation with GetObjectAnnotation and writes it with PutObjectAnnotation. The annotation is written to the version that the copy created, if the bucket is versioned, and only if the destination still has the ETag of the copy, so that it is not attached to an object written over the copy.

The payload of the annotation is read whole and then closed, also when the read fails. As for get_object(), a failure to read it is neither translated nor retried: the botocore exception propagates.

Parameters:
  • name – The annotation name.

  • source – The path of the copied object, with the version ID that was copied, if any.

  • destination – The path of the object that the copy wrote, without a version ID.

  • version_id – The version ID that the copy created, if any.

  • etag – The ETag of the object that the copy wrote, if any.

  • **params – The CopyObject parameters of the copy. GetObjectAnnotation receives those of the source, mapped as for plan_multipart_copy(), and PutObjectAnnotation those that it accepts; the fields that the other arguments set take precedence.

Raises:

ValueError – If the source or the destination has no key.

get_object_tagging(path: S3Path, **params) → dict[str, str][source]

Get the tags of an object, or of a version of it, with GetObjectTagging.

Parameters:
  • path – The path of the object, with the version ID to read, if any, including null.

  • **params – Additional request parameters, sent as given.

Returns:

The tags, mapping each key to its value, in the order of the response.

Raises:
put_object_tagging(path: S3Path, tags: Mapping[str, str], **params) → None[source]

Replace the tags of an object, or of a version of it, with PutObjectTagging.

Parameters:
  • path – The path of the object, with the version ID to tag, if any, including null.

  • tags – The tags, mapping each key to its value. They replace all the existing tags.

  • **params – Additional request parameters, sent as given.

Raises:
put_object_acl(path: S3Path, acl: str, **params) → None[source]

Apply a canned ACL to an object, or to a version of it, with PutObjectAcl.

Parameters:
  • path – The path of the object, with the version ID to apply the ACL to, if any, including null.

  • acl – The canned ACL, one of OBJECT_ACLS.

  • **params – Additional request parameters, sent as given.

Raises:
  • ValueError – If the path has no key, or the ACL is not in OBJECT_ACLS.

  • FileNotFoundError – If the object or version does not exist.

put_bucket_acl(bucket: str, acl: str, **params) → None[source]

Apply a canned ACL to a bucket with PutBucketAcl.

Parameters:
  • bucket – The name of the bucket.

  • acl – The canned ACL, one of BUCKET_ACLS.

  • **params – Additional request parameters, sent as given.

Raises:
replace_object_metadata(path: S3Path, head: S3Metadata, metadata: Mapping[str, str], **params) → None[source]

Replace the user-defined metadata of an object by copying it onto itself.

S3 does not update the metadata of an object in place, so the object is copied onto itself with CopyObject and the REPLACE metadata directive, which writes a new object, or a new version in a versioned bucket. With that directive, S3 does not copy what the request omits, so the request also sends the content headers, Expires, WebsiteRedirectLocation and StorageClass of head, and, unless params set an encryption parameter, its ServerSideEncryption, SSEKMSKeyId and BucketKeyEnabled. Fields that are None in head are not sent; note that S3Metadata reports STANDARD when HeadObject omits the storage class. HeadObject does not return the KMS encryption context, so it is not retained.

Parameters:
  • path – The path of the object, without a version ID.

  • head – The HeadObject result of path (see head_object()), whose fields are retained.

  • metadata – The user-defined metadata of the copy, which replaces all the existing user-defined metadata.

  • **params – Additional CopyObject parameters. They take precedence over the retained fields; one that the copy itself sets, such as Metadata, MetadataDirective or Key, raises TypeError.

Raises:

ValueError – If the path has no key or has a version ID, which a write cannot replace.

generate_presigned_url(path: S3Path, client_method: str = 'get_object', expires_in: int = 3600, **params) → str[source]

Generate a presigned URL for a request on an object.

The URL is signed locally with the client’s credentials; no request is sent. request_kwargs are not added to the signed parameters.

Parameters:
  • path – The path of the object, with the version ID to sign for, if any. A path without a key is not rejected; botocore validates the parameters of the method.

  • client_method – The name of the client method to sign, such as get_object or put_object.

  • expires_in – The number of seconds for which the URL is valid.

  • **params – Parameters of the method to sign. They take precedence over the bucket, key and version ID of the path.

Returns:

The presigned URL.

static copy_source_params(params: Mapping[str, Any]) → dict[str, Any][source]

Map the parameters of a copy to those of the requests that read its source.

The source of a copy is looked up, read, or deleted (by a move) with the parameters that authorize the access to it, which CopyObject names apart from those of the destination.

Parameters:

params – The CopyObject parameters of the copy.

Returns:

RequestPayer, and the source’s expected bucket owner and SSE-C parameters under the names of the requests that read the source (ExpectedBucketOwner and SSECustomer*), where given.

static copy_from_source_params(params: Mapping[str, Any]) → dict[str, Any][source]

Map the parameters of the requests that read an object to those of a copy from it.

The inverse of copy_source_params(), for a copy whose source is read with params, such as the part copies of an append.

Parameters:

params – The parameters of the requests that read the object.

Returns:

RequestPayer, and the expected bucket owner and SSE-C parameters under the names of the copy source (ExpectedSourceBucketOwner and CopySourceSSECustomer*), where given.

list_objects_page(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) → S3ListObjectsPage[source]

List one page of objects with ListObjectsV2.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list.

  • delimiter – The delimiter to group keys by; None or "" lists recursively.

  • max_keys – The maximum number of keys of the page.

  • continuation_token – The token of the page to list.

  • **params – Additional request parameters, sent as given.

Returns:

The page.

list_objects(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) → Iterator[S3ListObjectsPage][source]

List the pages of objects with ListObjectsV2, one request each.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list.

  • delimiter – The delimiter to group keys by; None or "" lists recursively.

  • max_keys – The maximum number of keys per page.

  • continuation_token – The token of the first page to list.

  • **params – Additional request parameters, sent as given.

Yields:

The pages, until one has no continuation token.

Raises:

TypeError – If params has ContinuationToken, which the iterator advances; pass continuation_token instead.

list_object_versions_page(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) → S3ListObjectVersionsPage[source]

List one page of versions with ListObjectVersions.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list.

  • delimiter – The delimiter to group keys by; None or "" lists recursively.

  • key_marker – The key marker of the page to list.

  • version_id_marker – The version ID marker of the page to list. S3 accepts it only with a key marker.

  • **params – Additional request parameters, sent as given.

Returns:

The page.

list_object_versions(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) → Iterator[S3ListObjectVersionsPage][source]

List the pages of versions with ListObjectVersions, one request each.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list.

  • delimiter – The delimiter to group keys by; None or "" lists recursively.

  • key_marker – The key marker of the first page to list.

  • version_id_marker – The version ID marker of the first page to list, sent with key_marker.

  • **params – Additional request parameters, sent as given.

Yields:

The pages, until one is not truncated or has no key marker.

Raises:

TypeError – If params has KeyMarker or VersionIdMarker, which the iterator advances; pass key_marker and version_id_marker instead.

list_multipart_uploads_page(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) → S3ListMultipartUploadsPage[source]

List one page of in-progress multipart uploads with ListMultipartUploads.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.

  • key_marker – The key marker of the page to list.

  • upload_id_marker – The upload ID marker of the page to list. S3 accepts it only with a key marker.

  • **params – Additional request parameters, sent as given.

Returns:

The page.

Raises:

FileNotFoundError – If the bucket does not exist.

list_multipart_uploads(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) → Iterator[S3ListMultipartUploadsPage][source]

List the pages of in-progress multipart uploads, one request each.

Parameters:
  • bucket – The bucket to list.

  • prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.

  • key_marker – The key marker of the first page to list.

  • upload_id_marker – The upload ID marker of the first page to list, sent with key_marker.

  • **params – Additional request parameters, sent as given.

Yields:

The pages, until one is not truncated or lacks either next marker.

Raises:
  • TypeError – If params has KeyMarker or UploadIdMarker, which the iterator advances; pass key_marker and upload_id_marker instead.

  • FileNotFoundError – If the bucket does not exist.

list_buckets_page(continuation_token: str | None = None, **params) → S3ListBucketsPage[source]

List one page of the buckets of the caller with ListBuckets.

Parameters:
  • continuation_token – The token of the page to list.

  • **params – Additional request parameters, sent as given.

Returns:

The page.

list_buckets(continuation_token: str | None = None, **params) → Iterator[S3ListBucketsPage][source]

List the pages of the buckets of the caller, one request each.

Parameters:
  • continuation_token – The token of the first page to list.

  • **params – Additional request parameters, sent as given.

Yields:

The pages, until one has no continuation token.

Raises:

TypeError – If params has ContinuationToken, which the iterator advances; pass continuation_token instead.

class pyathena.filesystem.s3_core.S3ObjectSummary(bucket: str, key: str, size: int | None = None, etag: str | None = None, last_modified: datetime | None = None, storage_class: str | None = None)[source]

An object listed by ListObjectsV2 (a Contents entry).

bucket

The bucket of the object.

Type:

str

key

The key of the object.

Type:

str

size

The size of the object in bytes.

Type:

int | None

etag

The entity tag of the object.

Type:

str | None

last_modified

When the object was last modified.

Type:

datetime.datetime | None

storage_class

The storage class of the object.

Type:

str | None

bucket: str
key: str
size: int | None = None
etag: str | None = None
last_modified: datetime | None = None
storage_class: str | None = None
classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3ObjectSummary[source]

Build the object from a Contents entry of ListObjectsV2.

Parameters:
  • bucket – The bucket that was listed.

  • entry – The Contents entry.

Returns:

The listed object.

property path: S3Path

The path of the object.

__init__(bucket: str, key: str, size: int | None = None, etag: str | None = None, last_modified: datetime | None = None, storage_class: str | None = None) → None
class pyathena.filesystem.s3_core.S3CommonPrefix(bucket: str, prefix: str)[source]

A key prefix listed with a delimiter (a CommonPrefixes entry).

bucket

The bucket that was listed.

Type:

str

prefix

The key prefix, ending with the delimiter.

Type:

str

bucket: str
prefix: str
classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3CommonPrefix[source]

Build the prefix from a CommonPrefixes entry.

Parameters:
  • bucket – The bucket that was listed.

  • entry – The CommonPrefixes entry.

Returns:

The listed prefix.

__init__(bucket: str, prefix: str) → None
class pyathena.filesystem.s3_core.S3Bucket(name: str, creation_date: datetime | None = None, bucket_region: str | None = None)[source]

A bucket, as listed by ListBuckets or found by HeadBucket.

name

The name of the bucket.

Type:

str

creation_date

When the bucket was created, if returned.

Type:

datetime.datetime | None

bucket_region

The region of the bucket, if returned.

Type:

str | None

name: str
creation_date: datetime | None = None
bucket_region: str | None = None
classmethod from_response(entry: Mapping[str, Any]) → S3Bucket[source]

Build the bucket from a Buckets entry of ListBuckets.

Parameters:

entry – The Buckets entry.

Returns:

The listed bucket.

__init__(name: str, creation_date: datetime | None = None, bucket_region: str | None = None) → None
class pyathena.filesystem.s3_core.S3ListObjectsPage(bucket: str, objects: tuple[S3ObjectSummary, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), key_count: int | None = None, is_truncated: bool = False, next_continuation_token: str | None = None)[source]

One page of a ListObjectsV2 listing.

bucket

The bucket that was listed.

Type:

str

objects

The listed objects (Contents).

Type:

tuple[pyathena.filesystem.s3_core.S3ObjectSummary, …]

common_prefixes

The listed key prefixes (CommonPrefixes).

Type:

tuple[pyathena.filesystem.s3_core.S3CommonPrefix, …]

key_count

The number of objects and prefixes of the page (KeyCount), if returned.

Type:

int | None

is_truncated

Whether more pages follow.

Type:

bool

next_continuation_token

The token of the next page, if any.

Type:

str | None

bucket: str
objects: tuple[S3ObjectSummary, ...] = ()
common_prefixes: tuple[S3CommonPrefix, ...] = ()
key_count: int | None = None
is_truncated: bool = False
next_continuation_token: str | None = None
classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListObjectsPage[source]

Build the page from a ListObjectsV2 response.

Parameters:
  • bucket – The bucket that was listed.

  • response – The ListObjectsV2 response.

Returns:

The page.

__init__(bucket: str, objects: tuple[S3ObjectSummary, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), key_count: int | None = None, is_truncated: bool = False, next_continuation_token: str | None = None) → None
class pyathena.filesystem.s3_core.S3ListObjectVersionsPage(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None)[source]

One page of a ListObjectVersions listing.

bucket

The bucket that was listed.

Type:

str

versions

The listed versions (Versions).

Type:

tuple[pyathena.filesystem.s3_object.S3ObjectVersion, …]

delete_markers

The listed delete markers (DeleteMarkers).

Type:

tuple[pyathena.filesystem.s3_object.S3ObjectVersion, …]

common_prefixes

The listed key prefixes (CommonPrefixes).

Type:

tuple[pyathena.filesystem.s3_core.S3CommonPrefix, …]

is_truncated

Whether more pages follow.

Type:

bool

next_key_marker

The key marker of the next page, if any.

Type:

str | None

next_version_id_marker

The version ID marker of the next page, if any.

Type:

str | None

bucket: str
versions: tuple[S3ObjectVersion, ...] = ()
delete_markers: tuple[S3ObjectVersion, ...] = ()
common_prefixes: tuple[S3CommonPrefix, ...] = ()
is_truncated: bool = False
next_key_marker: str | None = None
next_version_id_marker: str | None = None
classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListObjectVersionsPage[source]

Build the page from a ListObjectVersions response.

Parameters:
  • bucket – The bucket that was listed.

  • response – The ListObjectVersions response.

Returns:

The page.

__init__(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None) → None
class pyathena.filesystem.s3_core.S3ListBucketsPage(buckets: tuple[S3Bucket, ...] = (), continuation_token: str | None = None)[source]

One page of a ListBuckets listing.

buckets

The listed buckets.

Type:

tuple[pyathena.filesystem.s3_core.S3Bucket, …]

continuation_token

The token of the next page, if any.

Type:

str | None

buckets: tuple[S3Bucket, ...] = ()
continuation_token: str | None = None
classmethod from_response(response: Mapping[str, Any]) → S3ListBucketsPage[source]

Build the page from a ListBuckets response.

Parameters:

response – The ListBuckets response.

Returns:

The page.

__init__(buckets: tuple[S3Bucket, ...] = (), continuation_token: str | None = None) → None
class pyathena.filesystem.s3_core.S3ListMultipartUploadsPage(bucket: str, uploads: tuple[S3MultipartUpload, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_upload_id_marker: str | None = None)[source]

One page of a ListMultipartUploads listing.

bucket

The bucket that was listed.

Type:

str

uploads

The listed in-progress multipart uploads (Uploads), with the listed bucket as their bucket.

Type:

tuple[pyathena.filesystem.s3_object.S3MultipartUpload, …]

is_truncated

Whether more pages follow.

Type:

bool

next_key_marker

The key marker of the next page, if any.

Type:

str | None

next_upload_id_marker

The upload ID marker of the next page, if any.

Type:

str | None

bucket: str
uploads: tuple[S3MultipartUpload, ...] = ()
is_truncated: bool = False
next_key_marker: str | None = None
next_upload_id_marker: str | None = None
classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3ListMultipartUploadsPage[source]

Build the page from a ListMultipartUploads response.

Parameters:
  • bucket – The bucket that was listed.

  • response – The ListMultipartUploads response.

Returns:

The page.

__init__(bucket: str, uploads: tuple[S3MultipartUpload, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_upload_id_marker: str | None = None) → None
class pyathena.filesystem.s3_core.S3DeleteBatch(bucket: str, objects: tuple[S3Path, ...], quiet: bool = True)[source]

The objects of one bucket that one DeleteObjects request deletes.

A batch is valid by construction: it has 1 to MAX_KEYS objects, each with a key in the bucket of the batch. A path with a version ID deletes that version.

bucket

The bucket of the objects.

Type:

str

objects

The paths of the objects to delete.

Type:

tuple[pyathena.filesystem.s3_path.S3Path, …]

quiet

Whether S3 omits the deleted objects from the response.

Type:

bool

MAX_KEYS: ClassVar[int] = 1000
bucket: str
objects: tuple[S3Path, ...]
quiet: bool = True
__post_init__() → None[source]

Validate the objects of the batch.

Raises:

ValueError – If the batch has no objects or more than MAX_KEYS, or an object has no key or another bucket.

classmethod from_paths(paths: Iterable[S3Path], quiet: bool = True) → list[S3DeleteBatch][source]

Group the paths into batches by bucket, in the order of the paths.

Parameters:
  • paths – The paths of the objects to delete.

  • quiet – Whether S3 omits the deleted objects from the responses.

Returns:

The batches of up to MAX_KEYS objects of one bucket each.

Raises:

ValueError – If a path has no key.

__init__(bucket: str, objects: tuple[S3Path, ...], quiet: bool = True) → None
class pyathena.filesystem.s3_core.S3DeleteResult(bucket: str, deleted: tuple[S3Path, ...] = (), errors: tuple[S3DeleteError, ...] = ())[source]

The result of a DeleteObjects request.

S3 answers a request with 200 also when it could not delete some of the objects, and lists them in errors.

bucket

The bucket of the request.

Type:

str

deleted

The deleted objects, with the version ID of the request, if any. Empty for a quiet batch.

Type:

tuple[pyathena.filesystem.s3_path.S3Path, …]

errors

The objects that S3 could not delete.

Type:

tuple[pyathena.filesystem.s3_core.S3DeleteError, …]

bucket: str
deleted: tuple[S3Path, ...] = ()
errors: tuple[S3DeleteError, ...] = ()
classmethod from_response(bucket: str, response: Mapping[str, Any]) → S3DeleteResult[source]

Build the result from a DeleteObjects response.

Parameters:
  • bucket – The bucket of the request.

  • response – The DeleteObjects response.

Returns:

The result.

__init__(bucket: str, deleted: tuple[S3Path, ...] = (), errors: tuple[S3DeleteError, ...] = ()) → None
class pyathena.filesystem.s3_core.S3DeleteError(path: S3Path, code: str | None = None, message: str | None = None)[source]

An object that DeleteObjects could not delete (an Errors entry).

str() gives "path (code: message)".

path

The path of the object, with the version ID of the request, if any.

Type:

pyathena.filesystem.s3_path.S3Path

code

The error code.

Type:

str | None

message

The error message.

Type:

str | None

path: S3Path
code: str | None = None
message: str | None = None
classmethod from_response(bucket: str, entry: Mapping[str, Any]) → S3DeleteError[source]

Build the error from an Errors entry of DeleteObjects.

Parameters:
  • bucket – The bucket of the request.

  • entry – The Errors entry.

Returns:

The error.

__init__(path: S3Path, code: str | None = None, message: str | None = None) → None
class pyathena.filesystem.s3_core.S3MultipartCopyPlan(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False)[source]

The requests of a copy with a multipart upload, as CopyObject copies.

S3Core.plan_multipart_copy() reads the source and builds the plan; the caller schedules the requests: CreateMultipartUpload with create_params, one UploadPartCopy per range with part_params, then CompleteMultipartUpload with complete_params, or AbortMultipartUpload with abort_params after a failure, and finally the copy of each annotation with S3Core.copy_object_annotation(). If fits_single_request is true, the source is copied with S3Core.copy_object() instead, and ranges, the parameters and annotations are empty.

source

The object to copy, with the version that HeadObject reported unless the path has one or the version is null.

Type:

pyathena.filesystem.s3_path.S3Path

destination

The object that the copy writes.

Type:

pyathena.filesystem.s3_path.S3Path

size

The size in bytes of the source, from HeadObject.

Type:

int

ranges

The (start, end) byte ranges of the source that the parts copy, with an exclusive end, in part-number order.

Type:

tuple[tuple[int, int], …]

create_params

The parameters of CreateMultipartUpload, with the metadata and the tags that CopyObject would write.

Type:

collections.abc.Mapping[str, Any]

part_params

The parameters of each UploadPartCopy.

Type:

collections.abc.Mapping[str, Any]

complete_params

The parameters of CompleteMultipartUpload.

Type:

collections.abc.Mapping[str, Any]

abort_params

The parameters of AbortMultipartUpload.

Type:

collections.abc.Mapping[str, Any]

annotations

The names of the annotations to copy; empty if the directive excludes them or the source cannot have any.

Type:

tuple[str, …]

fits_single_request

Whether the source fits in a single CopyObject request.

Type:

bool

source: S3Path
destination: S3Path
size: int
ranges: tuple[tuple[int, int], ...] = ()
create_params: Mapping[str, Any]
part_params: Mapping[str, Any]
complete_params: Mapping[str, Any]
abort_params: Mapping[str, Any]
annotations: tuple[str, ...] = ()
fits_single_request: bool = False
__init__(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False) → None

S3 Objects

class pyathena.filesystem.s3_object.S3Object(init: dict[str, Any], **kwargs)[source]

Represents an S3 object with metadata and filesystem-like properties.

This class provides a dictionary-like interface to S3 object metadata, making it easier to work with S3 objects in filesystem operations. It handles the mapping between S3 API field names and more pythonic property names.

The object supports both dictionary-style access and property-style access to metadata fields like content type, storage class, encryption settings, and object lock configurations. Dictionary-style access behaves like a dictionary, so a missing key raises KeyError. Property-style access returns None for a known field that the object does not have, and raises AttributeError for any other missing name.

Example

>>> s3_obj = S3Object({"ContentType": "text/csv", "ContentLength": 1024})
>>> print(s3_obj.content_type)  # "text/csv"
>>> print(s3_obj["content_length"])  # 1024
>>> s3_obj.storage_class = "STANDARD_IA"

Note

This class is primarily used internally by S3FileSystem for representing S3 objects in filesystem operations.

__init__(init: dict[str, Any], **kwargs) → None[source]

Initialize the object from an S3 API response.

Only the fields of init that have a property mapping are kept, under their property names (e.g., ContentType -> content_type). storage_class defaults to STANDARD when init has no StorageClass, and size is taken from Size or ContentLength, or is 0, also for an empty init. name is set to bucket/key, or to the bucket when there is no key.

Parameters:
  • init – An S3 API response or listing entry, such as a HeadObject response or a ListObjectsV2 Contents entry.

  • **kwargs – Additional fields stored as-is, such as type, bucket, key and version_id. S3 API field names are stored under their property names.

get(k[, d]) → D[k] if k in D, else d.  d defaults to None.[source]
__getattr__(item: str) → Any[source]

Return None for a known field that the object does not have.

Called only when normal attribute lookup fails, so fields that the object has are returned without reaching this method.

Parameters:

item – The attribute name.

Returns:

None, if item is a known S3 object field.

Raises:

AttributeError – If item is not a known S3 object field.

copy() → S3Object[source]

Return a shallow copy of the object.

Returns:

A new S3Object with the same fields.

to_dict() → dict[str, Any][source]

Convert S3Object to dictionary representation.

Returns:

Deep copy of the object’s attributes as a dictionary.

to_api_repr() → dict[str, Any][source]

Convert the object metadata to S3 API request parameters.

Returns:

Dictionary keyed by S3 API field names (e.g., ContentType) of the fields that are set, excluding ETag, ContentLength and LastModified.

classmethod from_summary(summary: S3ObjectSummary) → S3Object[source]

Build the file entry of an object listed by ListObjectsV2.

Parameters:

summary – The listed object.

Returns:

The file entry, named bucket/key.

classmethod from_version(version: S3ObjectVersion) → S3Object[source]

Build the file entry of a version listed by ListObjectVersions.

The entry is named bucket/key?versionId=<id> so that the version can be addressed, except for the null version, which a write to the key replaces and which is named bucket/key.

Parameters:

version – The listed version.

Returns:

The file entry.

classmethod from_metadata(metadata: S3Metadata, version_id: str | None = None) → S3Object[source]

Build the file entry of an object looked up with HeadObject.

The fields that the response does not have are left out, except Metadata, which botocore always returns.

Parameters:
  • metadata – The metadata of the object, with the looked up path.

  • version_id – The version ID of the entry, which may be pinned or omitted apart from the looked up path.

Returns:

The file entry, named bucket/key.

Raises:

ValueError – If the metadata has no object path.

classmethod from_bucket(bucket: S3Bucket) → S3Object[source]

Build the directory entry of a bucket.

Parameters:

bucket – The bucket.

Returns:

The directory entry, named after the bucket.

class pyathena.filesystem.s3_object.S3ObjectType[source]

Constants for S3 object types in filesystem operations.

These constants are used to distinguish between directories and files when working with S3 paths through the S3FileSystem interface.

S3_OBJECT_TYPE_DIRECTORY: str = 'directory'
S3_OBJECT_TYPE_FILE: str = 'file'
class pyathena.filesystem.s3_object.S3StorageClass[source]

Constants for Amazon S3 storage classes.

S3 storage classes determine the availability, durability, and cost characteristics of stored objects. Each class is optimized for different access patterns and use cases.

Storage classes:
  • STANDARD: Default storage for frequently accessed data

  • REDUCED_REDUNDANCY: Lower cost, reduced durability (deprecated)

  • STANDARD_IA: Infrequently accessed data with rapid retrieval

  • ONEZONE_IA: Lower cost IA storage in single availability zone

  • INTELLIGENT_TIERING: Automatic tiering between frequent/infrequent

  • GLACIER: Archive storage for long-term backup

  • DEEP_ARCHIVE: Lowest cost archive storage

  • GLACIER_IR: Archive with faster retrieval than standard Glacier

  • OUTPOSTS: Storage on AWS Outposts

  • BUCKET: Pseudo storage class PyAthena assigns to bucket entries

  • DIRECTORY: Pseudo storage class PyAthena assigns to directory entries

See also

AWS S3 storage classes documentation: https://docs.aws.amazon.com/s3/latest/userguide/storage-class-intro.html

S3_STORAGE_CLASS_STANDARD: str = 'STANDARD'
S3_STORAGE_CLASS_REDUCED_REDUNDANCY: str = 'REDUCED_REDUNDANCY'
S3_STORAGE_CLASS_STANDARD_IA: str = 'STANDARD_IA'
S3_STORAGE_CLASS_ONEZONE_IA: str = 'ONEZONE_IA'
S3_STORAGE_CLASS_INTELLIGENT_TIERING: str = 'INTELLIGENT_TIERING'
S3_STORAGE_CLASS_GLACIER: str = 'GLACIER'
S3_STORAGE_CLASS_DEEP_ARCHIVE: str = 'DEEP_ARCHIVE'
S3_STORAGE_CLASS_OUTPOSTS: str = 'OUTPOSTS'
S3_STORAGE_CLASS_GLACIER_IR: str = 'GLACIER_IR'
S3_STORAGE_CLASS_BUCKET: str = 'BUCKET'
S3_STORAGE_CLASS_DIRECTORY: str = 'DIRECTORY'
class pyathena.filesystem.s3_object.S3Metadata(response: dict[str, Any], path: S3Path | None = None)[source]

Represents the metadata of an S3 object as returned by HeadObject.

Behaves as a read-only mapping of the user-defined metadata (x-amz-meta-*, whose keys are arbitrary user-chosen strings), so it is a drop-in for implementations that return the user-defined metadata as a plain dictionary, and compares equal to such dictionaries. The system-defined metadata (content type, encryption settings, storage class, etc.) is exposed as typed properties.

Example

>>> metadata = fs.metadata("s3://bucket/key")
>>> metadata["attr1"]  # user-defined metadata
'value1'
>>> metadata.content_type  # system-defined metadata
'text/plain'

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html

__init__(response: dict[str, Any], path: S3Path | None = None) → None[source]

Initialize the metadata from a HeadObject response.

Parameters:
  • response – The HeadObject response.

  • path – The path that was looked up, if known.

property cache_control: str | None

The CacheControl header of the object.

property content_disposition: str | None

The ContentDisposition header of the object.

property content_encoding: str | None

The ContentEncoding header of the object.

property content_language: str | None

The ContentLanguage header of the object.

property content_length: int | None

The ContentLength of the object in bytes.

property content_type: str | None

The ContentType of the object.

property etag: str | None

The ETag (entity tag) of the object.

property expiration: str | None

The Expiration header of the object.

property expires: datetime | None

The Expires date of the object.

property last_modified: datetime | None

The LastModified time of the object.

property storage_class: str

The StorageClass of the object; STANDARD when S3 omits it.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the object.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the object.

property bucket_key_enabled: bool | None

Whether the object uses an S3 Bucket Key (BucketKeyEnabled).

property website_redirect_location: str | None

The WebsiteRedirectLocation of the object.

property version_id: str | None

The VersionId of the object.

property path: S3Path | None

The path that was looked up, or None if it is not known.

property object_lock_mode: str | None

The ObjectLockMode of the object.

property object_lock_retain_until_date: datetime | None

The ObjectLockRetainUntilDate of the object.

The ObjectLockLegalHoldStatus of the object.

property user_metadata: dict[str, str]

A copy of the user-defined metadata (x-amz-meta-*).

The keys are arbitrary user-chosen strings, returned as stored in S3 (S3 normalizes them to lowercase). The same key/value pairs are also accessible directly through the mapping interface of this class.

class pyathena.filesystem.s3_object.S3ObjectVersion(bucket: str, is_delete_marker: bool, response: dict[str, Any])[source]

Represents a version of an S3 object as returned by ListObjectVersions.

bucket

S3 bucket name.

key

Object key.

version_id

The version ID of the object.

is_latest

Whether the version is the latest version of the object.

is_delete_marker

Whether the version is a delete marker.

last_modified

Date and time when the version was last modified.

etag

Entity tag of the version. None for delete markers.

size

Size in bytes of the version. None for delete markers.

storage_class

Storage class of the version. None for delete markers.

owner

Owner of the version.

__init__(bucket: str, is_delete_marker: bool, response: dict[str, Any]) → None[source]

Initialize the version from a ListObjectVersions entry.

Parameters:
  • bucket – The name of the bucket that contains the version.

  • is_delete_marker – Whether the entry comes from DeleteMarkers rather than Versions.

  • response – A Versions or DeleteMarkers entry of the ListObjectVersions response.

property bucket: str

The name of the bucket that contains the version.

property key: str

The Key of the object.

property name: str

The path of the version in bucket/key form.

property version_id: str | None

The VersionId of the version.

property is_latest: bool

Whether the version is the latest version of the object (IsLatest).

property is_delete_marker: bool

Whether the version is a delete marker.

property last_modified: datetime | None

The LastModified time of the version.

property etag: str | None

The ETag of the version; None for delete markers.

property size: int | None

The Size of the version in bytes; None for delete markers.

property storage_class: str | None

The StorageClass of the version; None for delete markers.

property owner: S3Owner | None

The Owner of the version, or None if the response has none.

S3 Upload Operations

class pyathena.filesystem.s3_object.S3PutObject(response: dict[str, Any])[source]

Represents the response from an S3 PUT object operation.

This class encapsulates the metadata returned when uploading an object to S3, including encryption details, versioning information, and integrity checksums.

expiration

Object expiration time if lifecycle policy applies.

version_id

Version ID if bucket versioning is enabled.

etag

Entity tag for the uploaded object.

server_side_encryption

Server-side encryption method used.

Various checksum properties

For data integrity verification.

Note

This class is used internally by S3FileSystem operations and typically not instantiated directly by users.

__init__(response: dict[str, Any]) → None[source]

Initialize the result from a PutObject response.

Parameters:

response – The PutObject response.

property expiration: str | None

The Expiration header of the uploaded object.

property version_id: str | None

The VersionId of the uploaded object.

property etag: str | None

The ETag of the uploaded object.

property checksum_crc32: str | None

The ChecksumCRC32 of the uploaded object.

property checksum_crc32c: str | None

The ChecksumCRC32C of the uploaded object.

property checksum_sha1: str | None

The ChecksumSHA1 of the uploaded object.

property checksum_sha256: str | None

The ChecksumSHA256 of the uploaded object.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the uploaded object.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the uploaded object.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the uploaded object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the uploaded object.

property sse_kms_encryption_context: str | None

The SSEKMSEncryptionContext of the uploaded object.

property bucket_key_enabled: bool | None

Whether the uploaded object uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_dict() → dict[str, Any][source]

Convert the response to a dictionary.

Returns:

Deep copy of the instance attributes, keyed by their attribute names (e.g., _etag).

class pyathena.filesystem.s3_object.S3MultipartUpload(response: dict[str, Any])[source]

Represents an S3 multipart upload operation.

This class manages the metadata for multipart uploads, which allow uploading large files in chunks for better reliability and performance. It tracks upload identifiers, encryption settings, and lifecycle rules.

bucket

S3 bucket name for the upload.

key

Object key being uploaded.

upload_id

Unique identifier for the multipart upload.

server_side_encryption

Encryption method applied to the upload.

abort_date/abort_rule_id

Lifecycle rule information for upload cleanup.

initiated/storage_class/owner/initiator

Fields returned by the ListMultipartUploads API for in-progress uploads.

Note

Used internally by S3FileSystem for large file upload operations, and returned by S3FileSystem.list_multipart_uploads.

__init__(response: dict[str, Any]) → None[source]

Initialize the upload from an S3 API response.

Parameters:

response – A CreateMultipartUpload response or an Uploads entry of a ListMultipartUploads response.

property abort_date: datetime | None

The AbortDate of the upload set by a lifecycle rule.

property abort_rule_id: str | None

The AbortRuleId of the lifecycle rule that applies to the upload.

property bucket: str | None

The Bucket of the upload.

property key: str | None

The Key of the object being uploaded.

property upload_id: str | None

The UploadId of the multipart upload.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the upload.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the upload.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the upload.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the upload.

property sse_kms_encryption_context: str | None

The SSEKMSEncryptionContext of the upload.

property bucket_key_enabled: bool | None

Whether the upload uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

property checksum_algorithm: str | None

The ChecksumAlgorithm of the upload.

property checksum_type: str | None

COMPOSITE or FULL_OBJECT.

Type:

The ChecksumType of the upload

property initiated: datetime | None

The Initiated time of the upload, returned by ListMultipartUploads.

property storage_class: str | None

The StorageClass of the upload, returned by ListMultipartUploads.

property owner: S3Owner | None

The Owner of the upload, returned by ListMultipartUploads.

property initiator: S3Owner | None

The Initiator of the upload, returned by ListMultipartUploads.

class pyathena.filesystem.s3_object.S3MultipartUploadPart(part_number: int, response: dict[str, Any])[source]

Represents a single part in an S3 multipart upload operation.

Each part in a multipart upload has its own metadata including checksums, encryption details, and part identification. This class manages that metadata and provides methods to convert it to API-compatible formats.

part_number

The sequential part number (1-based).

etag

Entity tag for this specific part.

checksum_*

Various integrity checksums for the part data.

server_side_encryption

Encryption settings for this part.

Note

Parts must be at least 5MB except for the last part. Used internally by S3FileSystem for chunked upload operations.

__init__(part_number: int, response: dict[str, Any]) → None[source]

Initialize the part from an UploadPart or UploadPartCopy response.

For an UploadPartCopy response, the ETag, LastModified and checksums are read from its CopyPartResult.

Parameters:
  • part_number – The part number of the part.

  • response – The UploadPart or UploadPartCopy response.

property part_number: int

The part number of the part.

property copy_source_version_id: str | None

The CopySourceVersionId of the source object of a copied part.

property last_modified: datetime | None

The LastModified time from CopyPartResult; None for uploaded parts.

property etag: str | None

The ETag of the part.

property checksum_crc32: str | None

The ChecksumCRC32 of the part.

property checksum_crc32c: str | None

The ChecksumCRC32C of the part.

property checksum_sha1: str | None

The ChecksumSHA1 of the part.

property checksum_sha256: str | None

The ChecksumSHA256 of the part.

property checksum_crc64nvme: str | None

The ChecksumCRC64NVME of the part.

property checksum_sha512: str | None

The ChecksumSHA512 of the part.

property checksum_md5: str | None

The ChecksumMD5 of the part.

property checksum_xxhash64: str | None

The ChecksumXXHASH64 of the part.

property checksum_xxhash3: str | None

The ChecksumXXHASH3 of the part.

property checksum_xxhash128: str | None

The ChecksumXXHASH128 of the part.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the part.

property sse_customer_algorithm: str | None

The SSECustomerAlgorithm of the part.

property sse_customer_key_md5: str | None

The SSECustomerKeyMD5 of the customer-provided key for the part.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the part.

property bucket_key_enabled: bool | None

Whether the part uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_api_repr() → dict[str, Any][source]

Convert the part to a part entry of a CompleteMultipartUpload request.

Returns:

Dictionary with the ETag, checksum and PartNumber fields of the part, omitting fields whose value is None.

class pyathena.filesystem.s3_object.S3CompleteMultipartUpload(response: dict[str, Any])[source]

Represents the completion of an S3 multipart upload operation.

This class encapsulates the final response when a multipart upload is completed, including the final object location, versioning information, and consolidated metadata from all parts.

location

Final S3 URL of the completed object.

bucket

S3 bucket containing the object.

key

Final object key.

version_id

Version ID if bucket versioning is enabled.

etag

Final entity tag of the complete object.

server_side_encryption

Encryption applied to the final object.

Note

This represents the successful completion of a multipart upload. Used internally by S3FileSystem operations.

__init__(response: dict[str, Any]) → None[source]

Initialize the result from a CompleteMultipartUpload response.

Parameters:

response – The CompleteMultipartUpload response.

property location: str | None

The Location URI of the completed object.

property bucket: str | None

The Bucket of the completed object.

property key: str | None

The Key of the completed object.

property expiration: str | None

The Expiration header of the completed object.

property version_id: str | None

The VersionId of the completed object.

property etag: str | None

The ETag of the completed object.

property checksum_crc32: str | None

The ChecksumCRC32 of the completed object.

property checksum_crc32c: str | None

The ChecksumCRC32C of the completed object.

property checksum_sha1: str | None

The ChecksumSHA1 of the completed object.

property checksum_sha256: str | None

The ChecksumSHA256 of the completed object.

property server_side_encryption: str | None

The ServerSideEncryption algorithm of the completed object.

property sse_kms_key_id: str | None

The SSEKMSKeyId of the KMS key for the completed object.

property bucket_key_enabled: bool | None

Whether the completed object uses an S3 Bucket Key (BucketKeyEnabled).

property request_charged: str | None

The RequestCharged field of the response.

to_dict()[source]

Convert the response to a dictionary.

Returns:

Deep copy of the instance attributes, keyed by their attribute names (e.g., _etag).

Multipart Writer

class pyathena.filesystem.s3_writer.S3MultipartWriter(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None)[source]

Plan parts and send the requests of a multipart write.

The caller owns the buffer, schedules requests, collects part results in number order, and waits for running requests before completing or aborting. Initiation and finalization must be serialized; part requests may run in parallel after initiation. No request is sent by the constructor or planners.

upload

The identity returned by creation, retained after completion and after an abort failure. A successful abort clears it.

__init__(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None) → None[source]

Initialize a writer without starting its upload.

Parameters:
  • core – The synchronous S3 operations to use.

  • path – The destination, with a key and without a version ID.

  • block_size – The nominal part size, between the core’s minimum and maximum multipart part sizes, inclusive.

  • request_kwargs – Parameters inherited by each request that accepts them. A copy is kept; parameters given to a request override it.

Raises:

ValueError – If the destination or block size is invalid.

iter_parts(stream: BinaryIO, first_part_number: int = 1) → Iterator[tuple[int, bytes]][source]

Plan numbered parts from the stream’s current position to EOF.

A short trailing block is merged into the preceding block, splitting the result in half if it reaches the maximum part size. This keeps every part large enough for later writes, unless the entire stream is shorter than the minimum part size and must be the final part. Reads are bounded by the block size; the stream is not rewound.

Parameters:
  • stream – A blocking binary stream of buffered data.

  • first_part_number – The first number, including any earlier copied or uploaded parts in the caller’s count.

Yields:

The part number and bytes, in order, without sending requests.

Raises:

ValueError – If a generated part number is outside the S3 limit.

iter_copy_parts(size: int, first_part_number: int = 1) → Iterator[tuple[int, tuple[int, int]]][source]

Plan numbered copy ranges, with an exclusive end, without requests.

Parameters:
  • size – The nonnegative size of the existing object in bytes.

  • first_part_number – The first part number to use.

Yields:

The part number and source range, in order. The ranges use the core’s maximum part size and keep a short tail within S3 limits so that appended parts may follow, unless the whole source is smaller than the minimum part size and must be the final part.

Raises:

ValueError – If the size is negative or a part number exceeds the limit.

initiate(**params: Any) → S3MultipartUpload[source]

Create the upload, or return its already known identity.

Parameters:

**params – Parameters accepted by CreateMultipartUpload.

Returns:

The upload, stored before this method returns so that a caller can recover it after an interrupted wait for a scheduled request.

upload_part(part_number: int, body: bytes, **params: Any) → S3MultipartUploadPart[source]

Send an UploadPart request for the initialized upload.

Parameters:
  • part_number – The part number, between 1 and the core’s maximum.

  • body – The bytes to upload, with the size selected by the caller.

  • **params – Parameters accepted by UploadPart.

Returns:

The uploaded part, retaining its number and checksums.

Raises:
upload_part_copy(part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params: Any) → S3MultipartUploadPart[source]

Send an UploadPartCopy request for the initialized upload.

Parameters:
  • part_number – The part number, between 1 and the core’s maximum.

  • source – The source object, with an optional version ID.

  • range – The source range with an exclusive end, or None to send no range of its own, as in S3Core.upload_part_copy().

  • **params – Parameters accepted by UploadPartCopy.

Returns:

The copied part, retaining its number and checksums.

Raises:
complete(parts: Sequence[S3MultipartUploadPart], **params: Any) → S3CompleteMultipartUpload[source]

Complete the upload after the caller has collected all its parts.

Parameters:
  • parts – Successfully uploaded parts in part-number order.

  • **params – Parameters accepted by CompleteMultipartUpload.

Returns:

The completion result. The upload identity is retained.

Raises:

RuntimeError – If no upload is initialized. Request failures propagate; the caller must settle running work before aborting.

abort(**params: Any) → None[source]

Abort a known upload after the caller has settled running requests.

Does nothing if no upload is known. A successful abort clears the identity; a failed or interrupted abort retains it for a later retry.

Parameters:

**params – Parameters accepted by AbortMultipartUpload.

class pyathena.filesystem.s3_writer.S3MultipartStateError[source]

An operation that the state of a write no longer allows.

For example, committing a file whose upload was discarded, or whose write failed, or uploading a part after a part failed.

S3 Errors

class pyathena.filesystem.s3_errors.S3ClientError(error: ClientError)[source]

Represents an S3 error response with its translation to an OSError.

Wraps a botocore ClientError and exposes the error response fields as properties, along with os_error, the equivalent standard Python exception. The error is mapped by its S3 error code first, then by its HTTP status code; if neither is recognized, a generic OSError with the original error message is used. A PreconditionFailed error of an If-None-Match condition, from a write that must not replace an existing object, is mapped to FileExistsError. A 405 error with the x-amz-delete-marker: true header, from a request for a version that is a delete marker, is mapped to FileNotFoundError.

Example

>>> try:
...     client.head_object(Bucket="bucket", Key="key")
... except botocore.exceptions.ClientError as e:
...     raise S3ClientError(e).os_error from e
__init__(error: ClientError) → None[source]

Initialize the error from a botocore ClientError.

Parameters:

error – The ClientError raised by the S3 client.

classmethod code_of(error: BaseException) → str | None[source]

Return the S3 error code of an error raised from a ClientError.

Parameters:

error – An error, such as the OSError of a request.

Returns:

The code of the ClientError that the error was raised from, or None if it was not raised from one.

property code: str

The S3 error code that uniquely identifies the error condition.

property message: str

The error message returned by the server.

property http_status_code: int | None

The HTTP status code of the error response.

property os_error: OSError

The standard Python exception equivalent to this error response.

Resolved once at construction time. The exception is instantiated and ready to be raised; raise it with raise ... from error to preserve the original exception.

Note

OSError specializes itself into a subclass for some errno values (e.g., EPERM -> PermissionError), so the returned exception may be a subclass of OSError even for the errno-based mappings.