File System Integration¶
This section covers S3 filesystem integration and object management.
S3 FileSystem¶
- pyathena.filesystem.s3_filesystem(connection: Connection[Any], **kwargs: Any) S3FileSystem[source]¶
- pyathena.filesystem.s3_filesystem(connection: Connection[Any], filesystem_class: type[_FileSystemT], **kwargs: Any) _FileSystemT
Create an S3 filesystem with the S3 client and retry policy of a connection.
The filesystem borrows
connection.s3_client, which the connection shares with its result sets, and usesconnection.retry_config. A policy with the defaultexceptionsalso retries the transient S3 error codes (RETRY_ERROR_CODES). Closing the filesystem does not close the client. A filesystem created this way cannot be pickled; seeS3FileSystem.Example
>>> from pyathena import connect >>> from pyathena.filesystem import s3_filesystem >>> fs = s3_filesystem(connect(s3_staging_dir="s3://my-bucket/path/"))
- Parameters:
connection – The PyAthena connection.
filesystem_class –
S3FileSystem,AioS3FileSystem, or a subclass of either.**kwargs – The other arguments of the filesystem, except the arguments from which it builds a client.
- Returns:
The filesystem.
- class pyathena.filesystem.s3.S3FileSystem(*args, **kwargs)[source]¶
A filesystem interface for Amazon S3 that implements the fsspec protocol.
This class provides a file-system like interface to Amazon S3, allowing you to use familiar file operations (ls, open, cp, rm, etc.) with S3 objects. It’s designed to be compatible with s3fs while offering PyAthena-specific optimizations.
The filesystem supports standard S3 operations including:
Listing objects and directories
Reading and writing files
Copying and moving objects
Reading and writing object metadata, tags, and canned ACLs
Multipart uploads for large files, including management of incomplete uploads
Version-aware reads and object version listing (see
version_aware)Creating and removing buckets (disabled by default; see
allow_bucket_creation/allow_bucket_deletion)Various S3 storage classes and encryption options
Translating S3 error responses into standard Python exceptions (e.g.,
404->FileNotFoundError,403->PermissionError)
- allow_bucket_creation¶
Whether mkdir/makedirs may create buckets. Defaults to False.
- allow_bucket_deletion¶
Whether rmdir may delete buckets. Defaults to False.
- version_aware¶
Whether reads pin the object version observed at open time and ls may list all versions. Requires the s3:GetObjectVersion / s3:ListBucketVersions permissions. Defaults to False.
Example
>>> from pyathena.filesystem.s3 import S3FileSystem >>> fs = S3FileSystem() >>> >>> # List objects in a bucket >>> files = fs.ls('s3://my-bucket/data/') >>> >>> # Read a file >>> with fs.open('s3://my-bucket/data/file.csv', 'r') as f: ... content = f.read() >>> >>> # Write a file >>> with fs.open('s3://my-bucket/output/result.txt', 'w') as f: ... f.write('Hello, S3!') >>> >>> # Copy files >>> fs.cp('s3://source-bucket/file.txt', 's3://dest-bucket/file.txt')
Note
This filesystem is used internally by PyAthena for handling query results stored in S3, but can also be used independently for S3 file operations.
- __init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int | None = None, s3_additional_kwargs=None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, *args, batch_size: int | None = None, client: BaseClient | None = None, retry_config: RetryConfig | None = None, **kwargs) None[source]¶
Create a filesystem for Amazon S3.
The filesystem sends its requests with the boto3 S3 client given as
client, or with a client that it builds from s3fs-compatible arguments inkwargs.pyathena.filesystem.s3_filesystem()creates one with the S3 client and retry policy of a PyAthena connection.- Parameters:
connection – Deprecated; use
clientandretry_config, orpyathena.filesystem.s3_filesystem(). A PyAthena connection whose S3 client (Connection.s3_client) and retry policy the filesystem uses, with aDeprecationWarning.default_block_size – The block size for reads and writes; defaults to
DEFAULT_BLOCK_SIZE.default_cache_type – The fsspec cache type for reads, the name of a cache in
fsspec.caching.caches; defaults to"bytes". The available names depend on the installed fsspec version. Common ones are"none", which keeps nothing and requests the bytes of each read;"bytes", which reads one block beyond each read and keeps one contiguous range;"readahead", which reads one block beyond each read and keeps only the last range read;"blockcache", which requests whole blocks and keeps the most recently used ones; and"background", which is like"blockcache"and also reads the next block in a thread.max_workers – The number of threads of the request pool of the filesystem, which its files and parallel operations share, and the default number of parallel requests of a file. If None, the
max_pool_connectionsof the S3 client (10 unless configured). A client that the filesystem builds getsmax_pool_connectionsfrommax_workersunlessconfig_kwargssets it; the connection pool of a givenclientkeeps its size.s3_additional_kwargs – Extra arguments for the object requests of
open(),pipe_file()andput_file(). Their authorization parameters (RequestPayer,ExpectedBucketOwnerandSSECustomer*) are sent with every request, listings and copies included; a copy also sends them for its source (ExpectedSourceBucketOwnerandCopySourceSSECustomer*). Each request receives those that its operation accepts, and the parameters of a call take precedence.allow_bucket_creation – Whether
mkdir/makedirsmay create a bucket.allow_bucket_deletion – Whether
rmdirmay delete a bucket.version_aware – Whether reads pin the object version observed at open time.
*args – Passed to
fsspec.AbstractFileSystem.batch_size – The number of per-object calls of a batch operation (
cat,get,put,copy, and the like) that run at once, and of DeleteObjects requests ofrm, unless the call gives its own. None or -1 ismax_workers.client – The boto3 S3 client that the filesystem sends its requests with. The filesystem does not close it. A filesystem with a given client cannot be pickled.
retry_config – The retry policy of the requests; defaults to
RetryConfig(). A policy with the defaultexceptionsalso retries the transient S3 error codes (RETRY_ERROR_CODES).**kwargs – Passed to
fsspec.AbstractFileSystem; withoutclientorconnection, also s3fs-compatible client arguments.requester_pays=Truesends requester-pays requests with the operations that acceptRequestPayer.
- Raises:
ValueError – If
batch_sizeis neither positive, -1, nor None.TypeError – If
connectionis given withclientorretry_config, orclientorconnectionwith an argument from which the filesystem builds a client, such askey,profile,anon,endpoint_url,client_kwargs, orregion_name, other than one that fsspec’s configuration supplies.
- __reduce__() tuple[Any, ...][source]¶
Reduce the filesystem for pickling, as fsspec does.
- Returns:
The reduction of
fsspec.AbstractFileSystem, which creates the filesystem again from its arguments.- Raises:
TypeError – If the filesystem was built with
clientorconnection, whose boto3 client cannot be pickled.
- property request_kwargs: dict[str, Any]¶
The parameters sent with every operation that accepts them.
{"RequestPayer": "requester"}withrequester_pays=True, and the authorization parameters ofs3_additional_kwargs, also under the names of a copy source; seecore.request_kwargs.
- close() None[source]¶
Release the threads of the request pool, which stays usable, once they are idle.
See
S3ThreadPool.release(). The calls in progress, including the requests of open files, run to their end, and a call after the release creates new threads.
- __enter__() S3FileSystem[source]¶
Return the filesystem, whose threads the end of the block releases.
- ls(path: str | PathLike[str], detail: bool = False, refresh: bool = False, **kwargs) list[S3Object] | list[str][source]¶
List contents of an S3 path.
Lists buckets (when path is root) or objects within a bucket/prefix. Compatible with fsspec interface for filesystem operations.
- Parameters:
path – S3 path to list (e.g., “s3://bucket” or “s3://bucket/prefix”).
detail – If True, return S3Object instances; if False, return paths as strings.
refresh – If True, bypass cache and fetch fresh results from S3.
**kwargs –
Additional arguments including: versions: If True, list all versions of the objects, named
bucket/key?versionId=<id>except for thenullversion. Requires the filesystem to be constructed withversion_aware=True.
- Returns:
List of S3Object instances (if detail=True) or paths as strings (if detail=False).
- Raises:
FileNotFoundError – If the bucket does not exist, or if a key path is neither an object nor a directory, a path that some key starts with followed by
/, or if HeadObject reports the version that a key path with a version ID names missing.
Example
>>> fs = S3FileSystem() >>> fs.ls("s3://my-bucket") # List objects in bucket >>> fs.ls("s3://my-bucket/", detail=True) # Get detailed object info
- info(path: str | PathLike[str] | S3Path, **kwargs) S3Object[source]¶
Return information about an S3 path.
Uses the directory cache first: a cached entry for the path is returned, the entry of the path in a cached listing of its parent is returned, preferring an object to a key prefix of the same name as HeadObject does, and a cached listing of its parent without it means it does not exist. The cached bucket listing holds only the buckets that the caller owns, so a bucket missing from it is looked up with HeadBucket. Otherwise, a key path is looked up with HeadObject and, if no object exists, with a ListObjectsV2 request (
Delimiter="/",MaxKeys=1) that checks whether it is a key prefix; a bucket path is looked up with HeadBucket. If these requests find a listed object missing, or find a key prefix, the cached listing of the parent is removed. Withversion_aware, a cached file entry without a version ID is looked up again. With an explicit version, the cached entries of the path are skipped, the HeadObject result is cached under the name and the version ID, apart from the key and other versions, except for thenullversion, which an overwrite replaces, and a missing version is not looked up as a key prefix, since a version names an object. With request parameters on which the authorization of the requests depends (ExpectedBucketOwner,RequestPayer, and theSSECustomer*parameters of an object encrypted with a customer-provided key), each request receives those that its operation accepts, and only the cached HeadObject or HeadBucket results of lookups with the same values of these parameters are used.- Parameters:
path – S3 path (e.g., “s3://bucket” or “s3://bucket/key”), or an
S3Path, whose bucket, key and version are looked up as they are, such as a key that a listing returns.**kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. version_id: The version ID to look up when the path has none. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters described above. Other request parameters are ignored.
- Returns:
S3Object describing the bucket, directory, or file. The root path (
"","/"or"s3://") is a directory.- Raises:
FileNotFoundError – If the path does not exist.
- find(path: str | PathLike[str], maxdepth: int | None = None, withdirs: bool | None = None, detail: bool = False, **kwargs) dict[str, S3Object] | list[str][source]¶
Find all files below a given S3 path.
Recursively searches for files under the specified path, with optional depth limiting and directory inclusion. Uses efficient S3 list operations with delimiter handling for performance. As in fsspec, the result includes the path itself if withdirs is True and objects exist below it, unless it is a bucket, or if it is an object and nothing is listed below it. A key path with a version ID names that version of the object: the result is the version, named as
ls(versions=True)names it, or empty if HeadObject reports it missing.- Parameters:
path – S3 path to search under (e.g., “s3://bucket/prefix”).
maxdepth – Maximum number of levels to descend, at least 1 (None for unlimited). With 1, only the entries directly under the path are listed.
withdirs – Whether to include directories in results (None = default behavior).
detail – If True, return dict of {path: S3Object}; if False, return list of paths.
**kwargs –
Additional arguments including: prefix: Key prefix, relative to the path, to filter the listed keys
by. Each slash in the prefix counts as one level of maxdepth. With withdirs, the directories above the prefix, such as
subforsub/deep/, are not included.refresh: If True, bypass the cache and list from S3.
- Returns:
Dictionary mapping paths to S3Objects (if detail=True) or list of paths (if detail=False).
- Raises:
ValueError – If
maxdepthis less than 1 or the path is the root.
Example
>>> fs = S3FileSystem() >>> fs.find("s3://bucket/data/", maxdepth=2) # Limit depth >>> fs.find("s3://bucket/", withdirs=True) # Include directories
- walk(path, maxdepth=None, topdown=True, on_error='omit', **kwargs)[source]¶
Walk the directories below a path, as fsspec’s
walk()does.Unlike fsspec, a listed directory is listed by its key prefix rather than by its name (see
_directory_prefix()), so a directory whose name ends in a version ID query or in an empty segment is walked as it was listed. Its name in the listing of its parent is the part of its name after the parent’s name, such as""for the directorybucket/dir/of the keysdir//xinbucket/dir, and it is walked under its name as listed.- Parameters:
path – The path to walk.
maxdepth – Maximum number of levels to walk, at least 1 (None for unlimited).
topdown – Whether to yield a directory before the directories below it. When True, removing names from the yielded directory names skips their directories.
on_error – For the listing of the path,
"omit"to yield nothing if it fails,"raise"to raise its error, or a callable that receives the error. As in fsspec, a directory below the path whose listing fails is skipped.**kwargs – Additional arguments passed to
ls(), such asrefreshandversions;detail=Trueyields the entries by name instead of the names.
- Yields:
Tuples of a directory path, its directories, and its files.
- Raises:
ValueError – If
maxdepthis less than 1.
- du(path, total=True, maxdepth=None, withdirs=False, **kwargs)[source]¶
Return the sizes of the objects below a path, as fsspec’s
du()does.Unlike fsspec, each entry that
find()returns is looked up by its bucket and key (see_entry_path()), never by parsing its name.- Parameters:
path – The path to search under.
total – Whether to return the sum of the sizes instead of the sizes by name.
maxdepth – Maximum number of levels to descend, at least 1 (None for unlimited).
withdirs – Whether to include the directories, and the path itself if it is a directory.
**kwargs – Additional arguments passed to
find().
- Returns:
The total size, or the sizes by the names that
info()gives.
- glob(path, maxdepth=None, **kwargs)[source]¶
Find the paths that match a glob pattern.
As fsspec’s
glob(), except that only the keys that start with the part of the last path component before the first wildcard are listed, as with fsspec’s asynchronous_glob().- Parameters:
path – S3 glob pattern, as a string or an
os.PathLike.maxdepth – Maximum depth of the paths below the literal part of the pattern, at least 1.
**kwargs – Additional arguments passed to
find, such asdetail.
- Returns:
The matching paths, or a dictionary of them and their objects with
detail=True.
- tree(path: str | PathLike[str] = '/', recursion_limit: int = 2, max_display: int = 25, display_size: bool = False, prefix: str = '', is_last: bool = True, first: bool = True, indent_size: int = 4) str[source]¶
Return a tree of the paths below a path, as fsspec’s
tree()does.Unlike fsspec, a listed directory is listed by its key prefix rather than by its name, as in
walk().- Parameters:
path – The root path, as a string or an
os.PathLike.recursion_limit – Maximum depth of the tree.
max_display – Maximum number of entries shown per directory.
display_size – Whether to show the sizes of the files.
prefix – The prefix of each line.
is_last – Whether the path is the last entry of its directory.
first – Whether the path is the root of the tree.
indent_size – Number of spaces per level.
- Returns:
The tree as a string.
- get_mapper(root: str | PathLike[str] = '', check: bool = False, create: bool = False, missing_exceptions=None)[source]¶
Return a key/value store at a path, as fsspec’s
get_mapper()does.- Parameters:
root – The root path of the store, as a string or an
os.PathLike.check – Whether to check that the root is writable.
create – Whether to create the root if it does not exist.
missing_exceptions – The exceptions that mean a missing key.
- Returns:
The
fsspec.mapping.FSMapof the root.
- unstrip_protocol(name: str | PathLike[str]) str[source]¶
Add the protocol to a path, as fsspec’s
unstrip_protocol()does.- Parameters:
name – The path, as a string or an
os.PathLike.- Returns:
The path with the
s3://protocol.
- expand_path(path, recursive=False, maxdepth=None, **kwargs) list[str][source]¶
Expand glob patterns and directories into the paths they match.
As in fsspec, except that a path with a version ID names that version of an object: it is not a glob pattern, although its
?is one in fsspec, nor is anything expanded below it. Withrecursive, it is included only if the version exists, as fsspec includes a path that exists.- Parameters:
path – S3 path, glob pattern, or list of them.
recursive – Whether to include the paths below the directories.
maxdepth – Maximum depth of the expansion, at least 1.
**kwargs – Additional arguments passed to
globandfind.
- Returns:
The sorted matching paths.
- Raises:
ValueError – If
maxdepthis less than 1.FileNotFoundError – If nothing matches.
- exists(path: str | PathLike[str], **kwargs) bool[source]¶
Check if an S3 path exists.
Determines whether a bucket, object, or prefix exists in S3. Uses caching and efficient head operations to minimize API calls.
- Parameters:
path – S3 path to check (e.g., “s3://bucket” or “s3://bucket/key”).
**kwargs – Additional arguments including: refresh: If True, bypass the cache and query S3. ExpectedBucketOwner, RequestPayer, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5: The request parameters on which the authorization of the requests depends, as described in
info(). With them, cached listings are not used. Other request parameters are ignored.
- Returns:
True if the path exists, False otherwise. A bucket that HeadBucket denies access to (403) exists.
- Raises:
PermissionError – If the lookup of an object is denied (403). Without
s3:ListBucketon the bucket, S3 also answers 403 for a key that does not exist.
Example
>>> fs = S3FileSystem() >>> fs.exists("s3://my-bucket/file.txt") >>> fs.exists("s3://my-bucket/")
- rm_file(path: str | PathLike[str], **kwargs) None[source]¶
Delete an S3 object with DeleteObject.
Does nothing for a bucket path. If the path has a version ID, that version is deleted.
- Parameters:
path – S3 path (s3://bucket/key) of the object to delete.
**kwargs – Additional parameters passed to the DeleteObject API, such as
RequestPayer. Those that it does not accept are ignored, for fsspec compatibility.
- Raises:
TypeError – If kwargs has
bucket,keyorversion_id, orBucket,KeyorVersionId, which the path gives.
- rm(path, recursive=False, maxdepth=None, batch_size=None, **kwargs) None[source]¶
Delete objects with DeleteObjects requests.
Expands the paths with
expand_pathand deletes the matched objects in parallel requests of up toS3DeleteBatch.MAX_KEYSkeys each, one set of requests per bucket. A path with a version ID deletes that version without expansion.- Parameters:
path – S3 path (s3://bucket/key) or list of paths to delete.
recursive – Whether to delete all objects below the paths.
maxdepth – Maximum depth to expand when
recursiveis True.batch_size – Number of DeleteObjects requests to send at the same time. By default the
batch_sizeof the filesystem; None or -1 there ismax_workers.max_workersis a deprecated alias.**kwargs – Additional parameters passed to the DeleteObjects API.
Quiet(default True) sets the quiet mode of the requests.
- Raises:
ValueError – If a path is a bucket, or
batch_sizeis neither positive, -1, nor None.TypeError – If both
batch_sizeandmax_workersare given.OSError – If S3 could not delete some of the objects.
- mkdir(path: str | PathLike[str], create_parents: bool = True, **kwargs) None[source]¶
Create an S3 bucket.
S3 has no real directories below the bucket level; creating a key prefix requires no operation. This method creates the bucket when the path points at a bucket (or when
create_parentsis True and the bucket does not exist yet), and does nothing for key prefixes under an existing bucket.Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass
allow_bucket_creation=Trueto the filesystem constructor to enable bucket creation.- Parameters:
path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).
create_parents – If True, create the bucket when it does not exist, even if the path contains a key prefix.
**kwargs –
Additional arguments including: acl: Canned ACL to apply to the bucket. region_name: Region to create the bucket in. Defaults to the
client’s region.
Other parameters that the CreateBucket API accepts, such as
ObjectLockEnabledForBucket, are passed to it; the others are ignored, for fsspec compatibility.
- Raises:
FileExistsError – If the path is a bucket that already exists.
FileNotFoundError – If the bucket does not exist and
create_parentsis False.PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.
ValueError – If the ACL is invalid, the path is empty, or botocore rejects the parameters of the CreateBucket request.
- makedirs(path: str | PathLike[str], exist_ok: bool = False) None[source]¶
Recursively create a directory, creating the bucket if necessary.
Creating the bucket requires
allow_bucket_creation=Trueon the filesystem constructor; seemkdir().- Parameters:
path – S3 path (e.g., “s3://bucket” or “s3://bucket/prefix”).
exist_ok – If False, raise FileExistsError when the path is a bucket that already exists.
- Raises:
FileExistsError – If the path is a bucket that already exists and
exist_okis False.PermissionError – If the bucket would be created but bucket creation is not enabled on this filesystem instance.
- rmdir(path: str | PathLike[str]) None[source]¶
Remove an S3 bucket, which must be empty.
S3 has no real directories below the bucket level, so only bucket paths can be removed.
Bucket lifecycle operations are disabled by default because they are infrastructure-level changes; pass
allow_bucket_deletion=Trueto the filesystem constructor to enable bucket deletion.- Parameters:
path – S3 bucket path (e.g., “s3://bucket”).
- Raises:
FileExistsError – If the path contains a key that exists. The user may have meant
rm(path, recursive=True).FileNotFoundError – If the path contains a key that does not exist, or the bucket does not exist.
PermissionError – If bucket deletion is not enabled on this filesystem instance.
OSError – If the bucket is not empty.
- touch(path: str | PathLike[str], truncate: bool = True, **kwargs) dict[str, Any][source]¶
Create an empty object with PutObject.
- Parameters:
path – S3 path (s3://bucket/key) of the object.
truncate – If True, replace an existing object with an empty one; if False, raise if the object exists. A key that is only a prefix of other keys has no object.
**kwargs – Additional parameters passed to the PutObject API, except
Body: the object is empty.
- Returns:
The PutObject response as a dictionary (see
S3PutObject.to_dict()).- Raises:
TypeError – If kwargs has
Body.ValueError – If the path has a version ID, is a bucket, or exists while
truncateis False.
- mv(path1, path2, recursive=False, maxdepth=None, batch_size=None, **kwargs) None[source]¶
Move files from one S3 location to another.
Copies the files to the destinations that
copy()gives them, and then deletes only the source objects that were copied. fsspec’sAbstractFileSystem.mv()instead removespath1by expanding it again, which also deletes copies placed wherepath1matches them and files thatmaxdepthkept from being copied. A file whose destination is the file itself is left in place, and directories, which S3 does not store as objects, are not copied. Anullversion moved onto its key is copied and deleted only when bucket versioning is enabled. Without versioning, or with versioning suspended, that move is left in place because the copy would replace the version that the deletion removes.Comparing a
nullversion with its unversioned key calls GetBucketVersioning once per bucket during the move’s planning and requiress3:GetBucketVersioning. The result is not cached across moves. Directory buckets do not support versioning and need no lookup. A source without a version whose destination conflicts with another pair is looked up with HeadObject during the planning too.The files are copied in parallel as
copy()copies them, and are deleted once every copy has succeeded; nothing is deleted after a failed copy.- Parameters:
path1 – Source S3 path, glob pattern, or list of paths.
path2 – Destination S3 path, or list of paths when
path1is a list.recursive – Whether to move the directories with their contents.
maxdepth – Maximum depth of a recursive move.
batch_size – Number of copies to run at the same time, as in
copy().**kwargs – Additional S3 copy parameters passed to
cp_file(). The lookups and the deletion of the sources receiveRequestPayerand the source’s expected bucket owner (seeS3Core.copy_source_params()); the lookups also receive the source’s SSE-C parameters.
- Raises:
ValueError – If two sources have the same destination, or a destination is another source, including one left in place, which is checked before anything is copied. A source without an object at its key, such as a directory, writes no destination and does not conflict. Also if
batch_sizeis neither positive, -1, nor None.FileNotFoundError – If a source does not exist, once the running copies have ended; nothing is deleted.
OSError – If bucket versioning cannot be read.
- copy(path1, path2, recursive=False, maxdepth=None, on_error=None, batch_size=None, **kwargs) None[source]¶
Copy files within S3.
As fsspec’s
copy(), except that a source with a version ID copies that version of the object to a destination named after its key, ascopy_pairs()pairs them, and that the files are copied in parallel,batch_sizeat a time. Pairs that would depend on the order of the copies raise before anything is copied (seecheck_copy_pairs()).An error that
on_errordoes not ignore starts no further copy, stops the running ones before their next request where it can (a multipart copy sends no further part and is aborted), and is raised once they have ended. Copies that ended before it stay in place.- Parameters:
path1 – Source S3 path, glob pattern, or list of them.
path2 – Destination S3 path, or list of paths when
path1is a list.recursive – Whether to copy the directories with their contents.
maxdepth – Maximum depth of a recursive copy.
on_error –
"raise"or"ignore"for a missing source; by default"ignore"withrecursiveand"raise"otherwise.batch_size – Number of copies to run at the same time. By default the
batch_sizeof the filesystem; None or -1 there ismax_workers. Withncopies at once, the smaller ofbatch_sizeand the number of pairs, each multipart copy sends at mostmax(1, max_workers // n)parts at once.**kwargs – Additional S3 copy parameters passed to
cp_file().
- Raises:
ValueError – If two sources have the same destination, or a destination is another source, except for a source without an object at its key, such as a directory, which is not copied. Also if
batch_sizeis neither positive, -1, nor None.OSError – If bucket versioning cannot be read for the checks.
- get(rpath, lpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, batch_size=None, **kwargs) None[source]¶
Copy files from S3 to the local filesystem.
As fsspec’s
get(), except that a source with a version ID downloads that version of the object to a local path named after its key, ascopy_pairs()pairs them, and that the files are downloaded in parallel, as withcopy(). The destinations named after the sources are checked to lie underlpath. Withoutfile, the files are written to it one after another, in order.A failed download starts no further one, the running ones read and write no further block, and the error is raised once they have ended.
- Parameters:
rpath – Source S3 path, glob pattern, or list of them.
lpath – Local destination path, or list of paths or file-like objects when
rpathis a list.recursive – Whether to copy the directories with their contents.
callback – Progress callback. It and its branches for the files are used from the threads that download them, one at a time.
maxdepth – Maximum depth of a recursive copy.
batch_size – Number of downloads to run at the same time, as in
copy().**kwargs – Additional parameters passed to
get_file().
- Raises:
ValueError – If a destination named after a source lies outside
lpath, two downloads write the same local file, orbatch_sizeis neither positive, -1, nor None.
- cp_file(path1: str | PathLike[str] | S3Path, path2: str | PathLike[str] | S3Path, recursive=False, maxdepth=None, on_error=None, **kwargs)[source]¶
Copy an S3 object to another S3 location.
Performs server-side copy of S3 objects, which is more efficient than downloading and re-uploading. Automatically chooses between simple copy and multipart copy based on object size.
- Parameters:
path1 – Source S3 path (s3://bucket/key), or an
S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns.path2 – Destination S3 path (s3://bucket/key), or an
S3Path.recursive – Unused parameter for fsspec compatibility.
maxdepth – Unused parameter for fsspec compatibility.
on_error – Unused parameter for fsspec compatibility.
**kwargs – Additional S3 copy parameters (e.g., metadata, storage class). The
block_sizeandmax_workersparameters control a multipart copy and are not sent to S3. The lookup of the source receivesRequestPayer, and the source’s expected bucket owner and SSE-C parameters under the names of the requests that read it (seeS3Core.copy_source_params()).
- Raises:
ValueError – If trying to copy to a versioned file or copy buckets, or if a directive of a multipart copy has an invalid value.
Note
Uses multipart copy for objects larger than the maximum part size to optimize performance for large files. The copy operation is performed entirely on the S3 service without data transfer. The multipart copy applies the metadata, tagging and annotation directives as CopyObject does (default
COPY). A directorypath1, which recursivecopy()passes along with the files under it, is skipped.
- pipe(path, value=None, batch_size=None, **kwargs) None[source]¶
Write bytes into one or more paths.
As fsspec’s
pipe(), except that the values of a dict are written in parallel withpipe_file(), ascopy()copies its files.- Parameters:
path – S3 path to write
valueto, or a dict of the values to write by path.value – The bytes to write to a single path; ignored with a dict.
batch_size – Number of writes to run at the same time, as in
copy().**kwargs – Additional parameters passed to
pipe_file().
- Raises:
ValueError – If
pathis neither a string nor a dict, two paths of a dict name the same object, orbatch_sizeis neither positive, -1, nor None.
- pipe_file(path: str | PathLike[str], value: bytes | bytearray | memoryview, mode: str = 'overwrite', **kwargs) None[source]¶
Write bytes into the path.
Writes data up to the block size with a single PutObject request instead of the inherited
open()+write()path. Larger data and writes inside an fsspec transaction go through the buffered path, which uploads the data as a parallel multipart upload and keeps the deferred-commit semantics of transactions. A write that fails on that path leaves the existing object unchanged. Both paths write to the path without a trailing slash, asopen()does.- Parameters:
path – S3 path (s3://bucket/key) to write to.
value – The bytes to write.
mode – “overwrite” (default) or “create”. With “create”, raise FileExistsError when the object already exists, including one created during the write, which is not replaced.
**kwargs – Additional parameters passed to the PutObject API (e.g., ContentType, StorageClass) on the single-request path. The
block_size,max_workers, ands3_additional_kwargsparameters of theopen()path are also accepted, and so iscompression: the codec ofopen()to compress the value with before it is uploaded, or"infer"to take it from the extension of the path.
- Raises:
FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the write is committed.
ValueError – If the path does not contain a key or specifies a version, if the compression is not supported, or if the data takes more than
S3Core.MULTIPART_UPLOAD_MAX_PARTSblocks.
- cat(path, recursive=False, on_error='raise', batch_size=None, **kwargs)[source]¶
Read the contents of one or more paths.
As fsspec’s
cat(), except that the objects are read in parallel withcat_file(), ascopy()copies its files. Withon_error="raise", an error starts no further read and is raised once the running reads have ended.- Parameters:
path – S3 path, glob pattern, or list of them.
recursive – Whether to read the files below the directories.
on_error –
"raise"to raise an error of a read,"omit"to leave out the paths whose read failed, or"return"to return the error in place of their contents.batch_size – Number of reads to run at the same time, as in
copy().**kwargs – Additional parameters passed to
expand_path()andcat_file().
- Returns:
The contents of the path, or a dict of the contents by path if there are several paths or the path was expanded.
- Raises:
ValueError – If
batch_sizeis neither positive, -1, nor None.
- cat_ranges(paths, starts, ends, max_gap=None, on_error='return', batch_size=None, **kwargs) list[bytes | Exception][source]¶
Read byte ranges of one or more paths.
As fsspec’s
cat_ranges(), except that the ranges are read in parallel withcat_file(), ascopy()copies its files. Unlesson_erroris"return", an error starts no further read and is raised once the running reads have ended.- Parameters:
paths – The S3 paths, one per range.
starts – The start offset of each range, or one for all of them.
ends – The end offset of each range, or one for all of them.
max_gap – Not supported; must be None.
on_error –
"return"to return the error of a read in place of its bytes; otherwise the error is raised.batch_size – Number of reads to run at the same time, as in
copy().**kwargs – Additional parameters passed to
cat_file().
- Returns:
The bytes of each range, or the error of its read.
- Raises:
NotImplementedError – If
max_gapis given.TypeError – If
pathsis not a list.ValueError – If the numbers of paths, starts, and ends differ, or if
batch_sizeis neither positive, -1, nor None.
- cat_file(path: str | PathLike[str] | S3Path, start: int | None = None, end: int | None = None, **kwargs) bytes[source]¶
Read the contents of an S3 object with GetObject.
startandendselect bytes like a slice of the object: an empty range, or one that starts at or past the end of the object, returnsb"", and an end past the object reads up to its end. Non-negative offsets are sent to S3 as they are, and so is a negativestartwithout anend, as a suffix range of the last bytes. Other negative offsets are resolved against the size frominfo(), which also checks that the object exists for an empty range and receives the lookup parameters amongkwargs.- Parameters:
path – S3 path (s3://bucket/key) of the object, or an
S3Path, whose bucket, key and version are read as they are, such as a key that a listing returns.start – Byte offset to start reading at. A negative value counts from the end of the object.
end – Byte offset to stop reading at (exclusive). A negative value counts from the end of the object.
**kwargs – Additional parameters passed to the GetObject API, except
version_id: the version ID to read when the path has none.
- Returns:
The bytes read from the object.
- Raises:
FileNotFoundError – If the path has no key or the key does not exist.
- put(lpath, rpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, batch_size=None, **kwargs) None[source]¶
Copy files from the local filesystem to S3.
As fsspec’s
put(), except that the files are uploaded in parallel withput_file(), ascopy()copies its files. A failed upload starts no further one, the running ones send no further part and do not commit, and the error is raised once they have ended.- Parameters:
lpath – Local source path, glob pattern, or list of them.
rpath – S3 destination path, or list of paths when
lpathis a list.recursive – Whether to copy the directories with their contents.
callback – Progress callback. It and its branches for the files are used from the threads that upload them, one at a time.
maxdepth – Maximum depth of a recursive copy.
batch_size – Number of uploads to run at the same time, as in
copy().**kwargs – Additional parameters passed to
put_file().
- Raises:
ValueError – If two files have the same destination, or
batch_sizeis neither positive, -1, nor None.
- put_file(lpath: str | ~os.PathLike[str], rpath: str | ~os.PathLike[str], callback=<fsspec.callbacks.NoOpCallback object>, mode: str = 'overwrite', **kwargs)[source]¶
Upload a local file to S3.
Uploads a file from the local filesystem to an S3 location. Supports automatic content type detection based on file extension and provides progress callback functionality. An upload that fails before it is completed, including one of a local file that cannot be read, leaves the existing object unchanged.
- Parameters:
lpath – Local file path to upload.
rpath – S3 destination path (s3://bucket/key).
callback – Progress callback for tracking upload progress.
mode – “overwrite” (default) or “create”. With “create”, the file is written as with
open()inxbmode: raise FileExistsError when the object already exists, including one created during the upload, which is not replaced.**kwargs – Additional S3 parameters (e.g., ContentType, StorageClass). The
block_size,max_workers, ands3_additional_kwargsparameters ofopen()are also accepted.
- Raises:
FileExistsError – If the mode is “create” and the path already exists, or an object is created at it before the upload is committed.
ValueError – If the file takes more than
S3Core.MULTIPART_UPLOAD_MAX_PARTSblocks.
Note
Directories are not supported for upload. If lpath is a directory, the method returns without performing any operation. Bucket-only destinations (without key) are also not supported.
- get_file(rpath: str | ~os.PathLike[str] | ~pyathena.filesystem.s3_path.S3Path, lpath=None, callback=<fsspec.callbacks.NoOpCallback object>, outfile=None, **kwargs)[source]¶
Download an S3 file to local filesystem.
The object is read in waves of parallel range requests of the block size, at most
max_workersof them at once, but no more than the request pool of the filesystem runs, and the bodies of each wave are written in order before the next wave is read, so that the download holds at most one wave of bodies; a thread of the pool that has just read a range drops its body when its call returns (seeS3File._download()). The reads are conditional on the version and the ETag of the object looked up when the download starts, as withopen(), so that an object replaced during the download fails it instead of mixing two objects, unless the download reads a version other thannull, which a replacement does not change. A failed download leaves the part written so far in a local filelpath, as fsspec’sAbstractFileSystem.get_file()does.As with fsspec’s
AbstractFileSystem.get_file(), a directoryrpathcreates the local directorylpath, and the parent directories of a local filelpathare created as needed.- Parameters:
rpath – S3 source path (s3://bucket/key), or an
S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns.lpath – Local destination path, or a file-like object to write to. Not needed when
outfileis given.callback – Progress callback for tracking download progress, with the size of the object.
outfile – A file-like object to write to instead of
lpath.**kwargs – The arguments of
open(), such asblock_size,max_workers,version_id, ands3_additional_kwargs, and additional S3 parameters.cache_typeandcache_optionsare not used. Withcompression, the object is decompressed into the destination, read in order throughopen()with a block cache, whose block size, if not positive, is that of the filesystem, orDEFAULT_BLOCK_SIZEif that is not positive either, and the size of the callback is unknown.
- checksum(path: str | PathLike[str], **kwargs)[source]¶
Get checksum for S3 object or directory.
Computes a checksum for the specified S3 path. For individual objects, returns the ETag converted to an integer. For directories, returns a checksum based on the directory’s tokenized representation.
- Parameters:
path – S3 path (s3://bucket/key) to get checksum for.
**kwargs – Additional arguments including: refresh: If True, refresh cached info before computing checksum.
- Returns:
Integer checksum value derived from S3 ETag or directory token.
Note
For multipart uploads, ETag format is different and only the first part before the dash is used for checksum calculation.
- sign(path: str | PathLike[str], expiration: int = 3600, **kwargs)[source]¶
Generate a presigned URL for S3 object access.
Creates a presigned URL that allows temporary access to an S3 object without requiring AWS credentials. Useful for sharing files or providing time-limited access to resources.
- Parameters:
path – S3 path (s3://bucket/key) to generate URL for.
expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).
**kwargs –
Additional parameters including: client_method: S3 operation (‘get_object’, ‘put_object’, etc.).
Defaults to ‘get_object’.
Additional parameters passed to the S3 operation.
- Returns:
Presigned URL string that provides temporary access to the S3 object.
Example
>>> fs = S3FileSystem() >>> url = fs.sign("s3://my-bucket/file.txt", expiration=7200) >>> # URL valid for 2 hours >>> >>> # Generate upload URL >>> upload_url = fs.sign( ... "s3://my-bucket/upload.txt", ... client_method="put_object" ... )
- metadata(path: str | PathLike[str], **kwargs) S3Metadata[source]¶
Return the metadata of the path.
- Parameters:
path – S3 path (s3://bucket/key) to get metadata for.
**kwargs – Additional parameters passed to the HeadObject API.
- Returns:
S3Metadata, which behaves as a read-only mapping of the user-defined metadata (
x-amz-meta-*) and exposes the system-defined metadata (content type, encryption settings, etc.) as typed properties.
- getxattr(path: str | PathLike[str], attr_name: str, **kwargs) str | None[source]¶
Get an attribute from the user-defined metadata of the path.
- Parameters:
path – S3 path (s3://bucket/key) to get the attribute for.
attr_name – The name of the attribute.
**kwargs – Additional parameters passed to
metadata().
- Returns:
The value of the attribute, or None if the attribute is not set.
- setxattr(path: str | PathLike[str], copy_kwargs: dict[str, Any] | None = None, **kw_args) None[source]¶
Set the user-defined metadata of the path.
S3 does not allow updating the metadata of an existing object in place, so the object is copied onto itself with the REPLACE metadata directive. Note that this rewrites the object and updates its last-modified time. The system-defined metadata (e.g.,
ContentTypeandCacheControl), the storage class, and the server-side encryption algorithm and KMS key of the object are kept. HeadObject does not return the KMS encryption context, and anExpiresvalue that botocore cannot parse as a date is not kept.- Parameters:
path – S3 path (s3://bucket/key) to set metadata for. A path with a version ID is rejected, since copying a version onto the key would replace the current object with it.
copy_kwargs – Additional parameters to use for the underlying CopyObject API call. They take precedence over the kept system-defined metadata and storage class. Any encryption parameter replaces all kept encryption settings. The lookup of the object receives
RequestPayer, and the source’s expected bucket owner and SSE-C parameters (e.g.,CopySourceSSECustomerKey) under the names of HeadObject (seeS3Core.copy_source_params()).**kw_args – Key-value pairs to set, where the values must be strings. The keys are used as-is; names that are not valid Python identifiers (e.g., containing hyphens) can be passed by unpacking a dictionary. Does not alter existing fields, unless the field appears here - if the value is None, delete the field.
Example
>>> fs = S3FileSystem() >>> fs.setxattr("s3://bucket/key", attribute1="value1") >>> fs.setxattr("s3://bucket/key", **{"attribute-2": "value2"})
- Raises:
ValueError – If the path is a bucket or has a version ID.
- get_tags(path: str | PathLike[str]) dict[str, str][source]¶
Retrieve the tag key/values for the given path.
- Parameters:
path – S3 path (s3://bucket/key) to get tags for.
- Returns:
Dictionary mapping tag keys to tag values.
- put_tags(path: str | PathLike[str], tags: dict[str, str], mode: str = 'o') None[source]¶
Set the tags for the given existing key.
Tags are a str:str mapping that can be attached to any key, distinct from the user-defined metadata, which is usually set at key creation time. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html
- Parameters:
path – S3 path (s3://bucket/key) of the existing key to attach tags to.
tags – Tags to apply.
mode – One of ‘o’ or ‘m’. ‘o’ will over-write any existing tags. ‘m’ will merge in new tags with existing tags, which incurs two remote calls.
- chmod(path: str | PathLike[str], acl: str, recursive: bool = False, **kwargs) None[source]¶
Set the Access Control on a bucket/key.
See https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl
- Parameters:
path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.
acl – The value of the canned ACL to apply.
recursive – Whether to apply the ACL to all keys below the given path too, and to the object at the key, if any. The keys are updated in parallel; once one fails, the updates that have not started are cancelled, and the running ones make no further attempt. A path with a version ID updates that version only.
**kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.
- list_multipart_uploads(path: str | PathLike[str]) list[S3MultipartUpload][source]¶
List in-progress (incomplete) multipart uploads in a bucket.
Incomplete multipart uploads continue to accrue storage costs until they are completed or aborted. Use
clear_multipart_uploads()to abort all of them.- Parameters:
path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under
key/are listed, not those to sibling keys that merely start with the same characters (e.g.,prefix2/a).- Returns:
List of S3MultipartUpload instances describing the in-progress multipart uploads.
- object_version_info(path: str | PathLike[str], delete_markers: bool = False, **kwargs) list[S3ObjectVersion][source]¶
List the versions of the object or of the objects under the path.
A key path without a trailing slash selects that key if it has any versions or delete markers, and otherwise the keys under
key/. The choice does not depend ondelete_markers, so a key that has only delete markers yields no versions without them. A key path with a trailing slash selects the keys under it, and a bucket path selects all the keys in the bucket. Sibling keys that merely start with the same characters (e.g.,key.bak) are never included.- Parameters:
path – S3 path (s3://bucket/key or a key prefix) to list the versions for.
delete_markers – Whether to include delete markers in the result.
**kwargs – Additional parameters passed to the ListObjectVersions API.
- Returns:
List of S3ObjectVersion instances describing the versions.
- clear_multipart_uploads(path: str | PathLike[str]) None[source]¶
Abort any incomplete multipart uploads in the bucket.
Uploads completed or aborted after listing are already cleared. All abort results are checked before another abort error is raised.
- Parameters:
path – S3 bucket or key path (e.g., “bucket”, “s3://bucket” or “s3://bucket/prefix”). If the path contains a key, only the uploads to that key and to the keys under
key/are aborted, as listed bylist_multipart_uploads().
- created(path: str | PathLike[str]) datetime[source]¶
Return the creation time of the path.
Returns the same value as
modified().- Parameters:
path – S3 path (s3://bucket/key).
- Returns:
The last-modified time of the object.
- modified(path: str | PathLike[str]) datetime[source]¶
Return the last-modified time of the path.
- Parameters:
path – S3 path (s3://bucket/key).
- Returns:
The
last_modifiedfield frominfo(), which is None for buckets and directories.
- invalidate_cache(path: str | PathLike[str] | None = None) None[source]¶
Remove the cached entries of the path and its parent paths.
The cached bucket listing is removed only by the root path (
"","/"or"s3://"), not by the paths of buckets or keys. A version-qualified path invalidates the lookup of the version, and also the object path without the version, because deleting or copying a version can change the current version of the object. It also invalidates the keys that end in its query, in each spelling that names the version, such as the keysa/x?versionId=v1anda/x?version_id=v1ofbucket/a/x?versionId=v1, which a listing names as the paths. Only the path itself is parsed for a version: its parent paths are directories, named as they are listed, such asbucket/a/dir?versionId=v1for the keya/dir?versionId=v1/x. A path that ends in slashes is invalidated both with and without them, since lookups strip them while a key such asdir//is listed in the directorybucket/dir/.- Parameters:
path – The path to invalidate. If None, clear the whole cache.
- class pyathena.filesystem.s3.S3File(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]¶
A buffered file object for reading and writing an S3 object.
Instances are returned by
S3FileSystem.open().- fs: S3FileSystem¶
- __init__(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) None[source]¶
Initialize the file for the path and mode.
In read mode, the object is looked up with
info()and the reads are made conditional on its ETag (IfMatch). In append mode, an existing object smaller thanS3Core.MULTIPART_UPLOAD_MIN_PART_SIZEis read into the write buffer; a larger one is copied withUploadPartCopyas the first parts of a multipart upload, whatever the block size. In exclusive-create mode, the object must not exist when the file is opened, and the upload is committed withIfNoneMatch="*"so that it does not replace an object created in the meantime.- Parameters:
fs – The filesystem that the file belongs to.
path – S3 path (s3://bucket/key) of the file, or an
S3Path(seeS3FileTarget.resolve()).mode – The file mode:
rb,wb,ab, orxb.version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.
max_workers – The number of parallel requests of the file: range reads, part copies, and part uploads. A write keeps at most
max_workersparts, including the part copies of an append, in flight, waiting for one of them to finish before it submits another. If None, themax_workersof the filesystem.executor – The executor for parallel operations. If None, one on the request pool of the filesystem (
fs._create_executor()).block_size – The block size for reads and writes. Must be between
S3Core.MULTIPART_UPLOAD_MIN_PART_SIZEandS3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.cache_type – The fsspec cache type for reads; see
S3FileSystem.__init__().autocommit – Whether to commit the written data when the file is closed. If False,
commit()must be called.cache_options – Options for the fsspec cache.
size – The size of the object, if known. Passed to
fsspec.spec.AbstractBufferedFile.s3_additional_kwargs – Additional parameters for the S3 requests of the file, such as
ContentTypeorRequestPayer. Each request receives those that its operation accepts.**kwargs – Additional parameters for the S3 requests of the file, which take precedence over
s3_additional_kwargs.
- Raises:
FileExistsError – If an object exists at the path in exclusive-create mode.
FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.
ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between
S3Core.MULTIPART_UPLOAD_MIN_PART_SIZEandS3Core.MULTIPART_UPLOAD_MAX_PART_SIZEfor writing, ormax_workersis not positive.
- close() None[source]¶
Close the file, flushing any written data, and shut down its executor.
In a forked child process, a file that the parent opened is only marked closed: its data is neither flushed nor committed, and its upload is not aborted, since they belong to the parent. Called by another finalizer while the cyclic garbage collector runs in this thread, such as that of a
TextIOWrapper, the file is closed in a new thread once the collection has ended (see__del__()), and this returns before that. A close handed over to another thread is waited for, except during a collection or in a thread of anS3ThreadPool, where this returns without waiting.
- __del__() None[source]¶
Close the file when it is finalized without having been closed.
A file whose last reference is dropped in a user thread is closed here. A file that the cyclic garbage collector finalizes, or that is finalized in a thread of an
S3ThreadPool, is closed in a new thread instead, since a wait here could last forever (T1): the requests of the file may be queued behind this thread or include the one that runs the collection, and this thread may hold a lock that the close takes. During a collection, the thread starts once the collection has ended, after the other finalizers of the collection, such as that of aTextIOWrapper, have written to the file.
- read(length: int = -1) bytes[source]¶
Read bytes from the file.
- Parameters:
length – The number of bytes to read; -1 reads to the end.
- Returns:
The bytes read.
- Raises:
RuntimeError – In a forked child process of the process that opened the file.
ValueError – If the file is closed or not open for reading.
- flush(force: bool = False) None[source]¶
Upload the buffered data as a part once a block is full, or all of it with
force.- Parameters:
force – Whether to upload the remaining data and commit the file, after which it cannot be written.
- Raises:
RuntimeError – In a forked child process of the process that opened the file.
ValueError – If
forceis given again after a forced flush.
- write(data: bytes | bytearray | memoryview) int[source]¶
Write data to the buffer, uploading each full block as a part.
- Parameters:
data – The bytes to write.
- Returns:
The number of bytes written.
- Raises:
BaseException – The error of a request of the write that failed since the last write, after which the file is closed without committing it.
RuntimeError – In a forked child process of the process that opened the file.
S3MultipartStateError – If the write was discarded.
ValueError – If the file is not open for writing.
- property multipart_upload: S3MultipartUpload | None¶
The multipart upload of the write, from its creation until it is aborted.
- property completed_parts: tuple[S3MultipartUploadPart, ...]¶
The uploaded or copied parts in part-number order; empty after an abort.
- commit() None[source]¶
Commit the written data.
Creates an empty object if nothing was written, uploads the buffered data with PutObject if no multipart upload was started, and otherwise completes the multipart upload once its parts have been uploaded. A failed part is raised as soon as it fails, without waiting for the parts before it. If a part or the completion fails or is interrupted, the file is closed without committing it, and its upload is aborted; if the abort also fails, the upload is kept so that
discard()can abort it. Committing again after a successful commit sends no request. Invalidates the cache of the path afterwards. Does nothing for a file opened for reading. Waits first for a close of the file that was handed over to another thread (seeclose()).- Raises:
FileExistsError – If an object was created at the path after the file was opened in exclusive-create mode.
RuntimeError – In a forked child process of the process that opened the file, or if a close of the file that was handed over to another thread runs and this is called during a garbage collection or in a thread of an
S3ThreadPool.S3MultipartStateError – If the write was discarded, or the write or an earlier commit failed.
- discard() None[source]¶
Discard the written data, and abort the multipart upload, if any.
The part requests that have not started are cancelled, and the running ones make no further attempt and are waited for before the abort, whose own retries are not stopped. If the abort fails or is interrupted, the upload is kept, and calling this again retries the abort. Afterwards,
commit()raisesS3MultipartStateError. Does nothing for a committed file or a file opened for reading. Waits first for a close of the file that was handed over to another thread (seeclose()).- Raises:
RuntimeError – In a forked child process of the process that opened the file, or if a close of the file that was handed over to another thread runs and this is called during a garbage collection or in a thread of an
S3ThreadPool.
- url(expiration: int = 3600, **kwargs) str[source]¶
Generate a presigned HTTP URL to read this file (if it already exists).
The URL is signed with the lookup parameters of the file that the signed operation accepts, such as
RequestPayeror theSSECustomer*parameters, whose headers the request with the URL must then send.- Parameters:
expiration – URL expiration time in seconds. Defaults to 3600 (1 hour).
**kwargs – Additional parameters passed to
S3FileSystem.sign(), which take precedence over those of the file.
- Returns:
Presigned URL string that provides temporary access to the S3 object.
- metadata(**kwargs) S3Metadata[source]¶
Return the metadata of the file.
See
S3FileSystem.metadata(). The HeadObject request receives the lookup parameters of the file, such asRequestPayeror theSSECustomer*parameters.- Parameters:
**kwargs – Additional parameters passed to the HeadObject API, which take precedence over those of the file.
- Returns:
S3Metadata, which behaves as a read-only mapping of the user-defined metadata and exposes the system-defined metadata as typed properties.
- getxattr(xattr_name: str, **kwargs) str | None[source]¶
Get an attribute from the user-defined metadata of the file.
See
S3FileSystem.getxattr(), andmetadata()for the parameters of the request.- Parameters:
xattr_name – The name of the attribute.
**kwargs – Additional parameters passed to the HeadObject API, which take precedence over those of the file.
- Returns:
The value of the attribute, or None if the attribute is not set.
- setxattr(copy_kwargs: dict[str, Any] | None = None, **kwargs) None[source]¶
Set the user-defined metadata of the file.
See
S3FileSystem.setxattr(). The copy of the object onto itself receives the lookup parameters of the file both for its source and for its destination:RequestPayer, the expected bucket owner asExpectedSourceBucketOwnerandExpectedBucketOwner, and theSSECustomer*parameters also asCopySourceSSECustomer*, so that an object encrypted with a customer-provided key keeps it. AServerSideEncryptionincopy_kwargsreplaces theSSECustomer*parameters of the file for the destination; those in thes3_additional_kwargsof the filesystem are still sent.- Parameters:
copy_kwargs – Additional parameters to use for the underlying CopyObject API call, which take precedence over those of the file.
**kwargs – Key-value pairs of metadata to set.
- class pyathena.filesystem.s3.S3FileTarget(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None)[source]¶
The object that a file reads or writes, resolved when it is opened.
In read mode, the object is looked up, and the reads are made conditional on its ETag (
IfMatch). In append mode, an existing object is looked up; one smaller thanS3Core.MULTIPART_UPLOAD_MIN_PART_SIZEis also read, so that the file rewrites it from its buffer, and a larger one is copied by the file as the first parts of a multipart upload. In exclusive-create mode, the object must not exist, and the upload is committed withIfNoneMatch="*".- path¶
The path of the file. A version that is read is carried in the path as a
?versionId=suffix, so that a reopened file reads the same version.- Type:
- request_kwargs¶
The parameters of the S3 requests of the file, with the conditions above and, in append mode, the attributes of the existing object that the request parameters (those of the call and of the filesystem) do not give, which the rewritten object keeps: its system-defined and user-defined metadata, storage class, and encryption, unless they give any encryption parameter. Its object lock is not kept.
- details¶
The object that is read or appended, or an empty dictionary.
- Type:
- appended¶
The existing object in append mode, if any.
- Type:
- append_data¶
The data of an appended object that the file rewrites from its buffer, if any.
- Type:
bytes | None
- classmethod resolve(fs: S3FileSystem, path: str | S3Path, mode: str, *, block_size: int, version_id: str | None = None, size: int | None = None, request_kwargs: Mapping[str, Any] | None = None) S3FileTarget[source]¶
Validate the arguments of a file and look up its object.
Sends the lookup requests of the mode, which block.
- Parameters:
fs – The filesystem that the file belongs to.
path – S3 path (s3://bucket/key) of the file, or an
S3Path, whose bucket, key and version are used as they are, such as a key that a listing returns, and which is looked up by them. The path of the file is then the string of theS3Path.mode – The file mode:
rb,wb,ab, orxb.block_size – The block size of the file.
version_id – The version ID to read. Must match the version ID in the path if both are given. A version cannot be given, in either form, for writing or appending.
size – The size of the object to read, if known.
request_kwargs – The parameters of the S3 requests of the file. A copy is kept.
- Returns:
The resolved target.
- Raises:
FileExistsError – If an object exists at the path in exclusive-create mode.
FileNotFoundError – If no object exists at the path when reading, including when the path is a prefix.
ValueError – If the path has no key, the version IDs do not match, a version is given for writing, or the block size is not between
S3Core.MULTIPART_UPLOAD_MIN_PART_SIZEandS3Core.MULTIPART_UPLOAD_MAX_PART_SIZEfor writing.
- __init__(path: str, bucket: str, key: str, version_id: str | None, request_kwargs: dict[str, ~typing.Any], details: ~pyathena.filesystem.s3_object.S3Object | dict[str, ~typing.Any] = <factory>, size: int | None = None, appended: ~pyathena.filesystem.s3_object.S3Object | None = None, append_data: bytes | None = None) None¶
Async S3 FileSystem¶
- class pyathena.filesystem.s3_async.AioS3FileSystem(*args, **kwargs)[source]¶
An async filesystem interface for Amazon S3 using fsspec’s AsyncFileSystem.
This class wraps
S3FileSystemto provide native asyncio support. Its coroutines run the blocking boto3 calls in anS3ThreadPoolwithmax_workersthreads that the filesystem owns, instead of the default executor of the event loop, and run parallel per-object calls, such as the copies of_mv(), withasyncio.gather. The single requests that these calls run in parallel, such as the parts of a file or a copy, and the DeleteObjects batches of_rm(), run in the request pool of the wrappedS3FileSystem, which its files share, throughmap_async()or a write session.The implementation uses composition: an internal
S3FileSysteminstance handles all boto3 calls, while this class delegates to it in the threads of its pool. This avoids diamond inheritance issues and keeps all boto3 logic in one place.File handles created by
_openrun their parallel operations (range reads, multipart uploads) in that request pool, asS3Filedoes.open_async()returns anAioS3StreamedFile, whose reads and writes are coroutines.close(), or the end of awithorasync withblock, releases the threads of the filesystem, which stays usable.- _sync_fs¶
The internal synchronous S3FileSystem instance.
Example
>>> from pyathena.filesystem.s3_async import AioS3FileSystem >>> fs = AioS3FileSystem(asynchronous=True) >>> >>> # Use in async context >>> files = await fs._ls('s3://my-bucket/data/') >>> >>> # Sync wrappers (auto-generated by fsspec) need an instance created >>> # without asynchronous=True; they block the caller until done >>> files = AioS3FileSystem().ls('s3://my-bucket/data/')
- mirror_sync_methods = True¶
- async_impl = True¶
- __init__(connection: Connection[Any] | None = None, default_block_size: int | None = None, default_cache_type: str | None = None, max_workers: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, allow_bucket_creation: bool = False, allow_bucket_deletion: bool = False, version_aware: bool = False, asynchronous: bool = False, loop: Any | None = None, batch_size: int | None = None, *, client: BaseClient | None = None, retry_config: RetryConfig | None = None, **kwargs) None[source]¶
Initialize the filesystem and its internal
S3FileSystem.- Parameters:
connection – Deprecated; use
clientandretry_config, orpyathena.filesystem.s3_filesystem()withAioS3FileSystem. A PyAthena connection whose S3 client and retry policy the filesystem uses, with aDeprecationWarning.default_block_size – Passed to the internal
S3FileSystem.default_cache_type – Passed to the internal
S3FileSystem.max_workers – The number of threads that run the blocking calls of the filesystem, and, passed to the internal
S3FileSystem, the size of its request pool and the default number of parallel requests of its files and copies. If None, themax_pool_connectionsof the S3 client (10 unless configured), as forS3FileSystem.s3_additional_kwargs – Passed to the internal
S3FileSystem.allow_bucket_creation – Passed to the internal
S3FileSystem.allow_bucket_deletion – Passed to the internal
S3FileSystem.version_aware – Passed to the internal
S3FileSystem.asynchronous – Passed to
fsspec.asyn.AsyncFileSystem.loop – Passed to
fsspec.asyn.AsyncFileSystem.batch_size – Passed to
fsspec.asyn.AsyncFileSystem.client – Passed to the internal
S3FileSystem. A filesystem with a given client cannot be pickled.retry_config – Passed to the internal
S3FileSystem.**kwargs – Passed to both
fsspec.asyn.AsyncFileSystemand the internalS3FileSystem.
- Raises:
TypeError – If
connectionis given withclientorretry_config, or the internalS3FileSystemrejects the arguments.
- __reduce__() tuple[Any, ...][source]¶
Reduce the filesystem for pickling, as fsspec does.
- Returns:
The reduction of
fsspec.AbstractFileSystem, which creates the filesystem again from its arguments.- Raises:
TypeError – If the filesystem was built with
clientorconnection, whose boto3 client cannot be pickled.
- close() None[source]¶
Release the threads of the filesystem, which stays usable, once they are idle.
Releases the threads of the blocking calls and those of the request pool of the internal
S3FileSystem(seeS3FileSystem.close()andS3ThreadPool.release()). The calls in progress, including the requests of open files, run to their end.
- __enter__() AioS3FileSystem[source]¶
Return the filesystem, whose threads the end of the block releases.
- async __aenter__() AioS3FileSystem[source]¶
Return the filesystem, whose threads the end of the block releases.
- async __aexit__(exc_type: Any, exc_val: Any, exc_tb: Any) None[source]¶
Release the threads with
close().
- mv(path1, path2, recursive=False, maxdepth=None, **kwargs) None[source]¶
Move files from one S3 location to another.
Runs
_mv(); fsspec does not provide the synchronous wrapper of_mv.- Parameters:
path1 – Source S3 path, glob pattern, or list of paths.
path2 – Destination S3 path, or list of paths when
path1is a list.recursive – Whether to move the directories with their contents.
maxdepth – Maximum depth of a recursive move.
**kwargs – Additional S3 copy parameters.
- Raises:
ValueError – If the move has conflicting paths.
OSError – If bucket versioning cannot be read.
- walk(path, maxdepth=None, topdown=True, on_error='omit', **kwargs)[source]¶
Walk the directories below a path, as fsspec’s
walk()does.Runs the walk of
S3FileSystem.walk(), which lists a listed directory by its key prefix rather than by its name. As fsspec’swalk()of an asynchronous filesystem, which runs_walk(), it yields a directory whose listing fails with no entries.- Parameters:
path – The path to walk.
maxdepth – Maximum number of levels to walk, at least 1 (None for unlimited).
topdown – Whether to yield a directory before the directories below it.
on_error –
"omit","raise", or a callable that receives the error of the listing of the path, as inS3FileSystem.walk().**kwargs – Additional arguments passed to
ls().
- Yields:
Tuples of a directory path, its directories, and its files.
- tree(path: str | PathLike[str] = '/', recursion_limit: int = 2, max_display: int = 25, display_size: bool = False, prefix: str = '', is_last: bool = True, first: bool = True, indent_size: int = 4) str[source]¶
Return a tree of the paths below a path, as fsspec’s
tree()does.Runs
S3FileSystem.tree(), which lists a listed directory by its key prefix rather than by its name.- Parameters:
path – The root path, as a string or an
os.PathLike.recursion_limit – Maximum depth of the tree.
max_display – Maximum number of entries shown per directory.
display_size – Whether to show the sizes of the files.
prefix – The prefix of each line.
is_last – Whether the path is the last entry of its directory.
first – Whether the path is the root of the tree.
indent_size – Number of spaces per level.
- Returns:
The tree as a string.
- get_mapper(root: str | PathLike[str] = '', check: bool = False, create: bool = False, missing_exceptions=None)[source]¶
Return a key/value store at a path, as fsspec’s
get_mapper()does.- Parameters:
root – The root path of the store, as a string or an
os.PathLike.check – Whether to check that the root is writable.
create – Whether to create the root if it does not exist.
missing_exceptions – The exceptions that mean a missing key.
- Returns:
The
fsspec.mapping.FSMapof the root.
- unstrip_protocol(name: str | PathLike[str]) str[source]¶
Add the protocol to a path, as fsspec’s
unstrip_protocol()does.- Parameters:
name – The path, as a string or an
os.PathLike.- Returns:
The path with the
s3://protocol.
- async open_async(path: str | PathLike[str], mode: str = 'rb', block_size: int | None = None, max_workers: int | None = None, version_id: str | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) AioS3StreamedFile[source]¶
Open a file whose reads and writes are coroutines.
The lookups of the mode, as in
open(), run in the threads of the filesystem. If the caller is cancelled, a file opened after the cancellation is closed without committing anything (seecreate_in_thread()). SeeAioS3StreamedFilefor how the file reads, writes, and commits.- Parameters:
path – S3 path (s3://bucket/key) of the file.
mode – The file mode:
rb,wb,ab, orxb.block_size – The block size for reads and writes. Defaults to the default block size of the filesystem.
max_workers – The number of parallel requests of the file, in the request pool of the filesystem. Defaults to the
max_workersof the filesystem.version_id – The version ID to read.
s3_additional_kwargs – Additional parameters for the S3 requests of the file, which take precedence over those of the filesystem.
**kwargs – Additional parameters for the S3 requests of the file, which take precedence over
s3_additional_kwargs.
- Returns:
The opened file.
- Raises:
ValueError – If the mode is not binary, compression is requested, or the arguments are invalid, as in
open().NotImplementedError – If the mode is not supported, or the file is opened for writing in a transaction or with
autocommit=False, as the file commits when it is closed.FileExistsError – If an object exists at the path in exclusive-create mode.
FileNotFoundError – If no object exists at the path when reading.
- rmdir(path: str | PathLike[str]) None[source]¶
Remove an S3 bucket, which must be empty.
See
S3FileSystem.rmdir().- Parameters:
path – S3 bucket path (e.g., “s3://bucket”).
- sign(path: str | PathLike[str], expiration: int = 3600, **kwargs) str[source]¶
Generate a presigned URL for S3 object access.
See
S3FileSystem.sign().- Parameters:
path – S3 path (s3://bucket/key) to generate the URL for.
expiration – URL expiration time in seconds.
**kwargs – Additional parameters passed to
S3FileSystem.sign().
- Returns:
The presigned URL.
- metadata(path: str | PathLike[str], **kwargs) S3Metadata[source]¶
Return the metadata of the path.
See
S3FileSystem.metadata().- Parameters:
path – S3 path (s3://bucket/key) to get metadata for.
**kwargs – Additional parameters passed to the HeadObject API.
- Returns:
S3Metadata of the object.
- getxattr(path: str | PathLike[str], attr_name: str, **kwargs) str | None[source]¶
Get an attribute from the user-defined metadata of the path.
See
S3FileSystem.getxattr().- Parameters:
path – S3 path (s3://bucket/key) to get the attribute for.
attr_name – The name of the attribute.
**kwargs – Additional parameters passed to the HeadObject API.
- Returns:
The value of the attribute, or None if the attribute is not set.
- setxattr(path: str | PathLike[str], copy_kwargs: dict[str, Any] | None = None, **kwargs) None[source]¶
Set the user-defined metadata of the path.
See
S3FileSystem.setxattr().- Parameters:
path – S3 path (s3://bucket/key) to set metadata for.
copy_kwargs – Additional parameters to use for the underlying CopyObject API call.
**kwargs – Key-value pairs of metadata to set; a None value deletes the key.
- get_tags(path: str | PathLike[str]) dict[str, str][source]¶
Retrieve the tag key/values for the given path.
See
S3FileSystem.get_tags().- Parameters:
path – S3 path (s3://bucket/key) to get tags for.
- Returns:
Dictionary mapping tag keys to tag values.
- put_tags(path: str | PathLike[str], tags: dict[str, str], mode: str = 'o') None[source]¶
Set the tags for the given existing key.
See
S3FileSystem.put_tags().- Parameters:
path – S3 path (s3://bucket/key) of the existing key.
tags – Tags to apply.
mode –
oto overwrite the existing tags ormto merge with them.
- chmod(path: str | PathLike[str], acl: str, recursive: bool = False, **kwargs) None[source]¶
Set the canned ACL of a bucket or key.
See
S3FileSystem.chmod().- Parameters:
path – S3 path (s3://bucket or s3://bucket/key) to set the ACL on.
acl – The canned ACL to apply.
recursive – Whether to apply the ACL to all keys below the path too.
**kwargs – Additional parameters passed to the PutObjectAcl or PutBucketAcl API.
- object_version_info(path: str | PathLike[str], delete_markers: bool = False, **kwargs) list[S3ObjectVersion][source]¶
List the versions of the object or of the objects under the path.
See
S3FileSystem.object_version_info().- Parameters:
path – S3 path (s3://bucket/key or a key prefix) to list the versions for.
delete_markers – Whether to include delete markers in the result.
**kwargs – Additional parameters passed to the ListObjectVersions API.
- Returns:
List of S3ObjectVersion instances describing the versions.
- list_multipart_uploads(path: str | PathLike[str]) list[S3MultipartUpload][source]¶
List in-progress (incomplete) multipart uploads in a bucket.
See
S3FileSystem.list_multipart_uploads().- Parameters:
path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).
- Returns:
List of S3MultipartUpload instances describing the uploads.
- clear_multipart_uploads(path: str | PathLike[str]) None[source]¶
Abort any incomplete multipart uploads in the bucket.
See
S3FileSystem.clear_multipart_uploads().- Parameters:
path – S3 bucket or key path (e.g., “s3://bucket” or “s3://bucket/prefix”).
- checksum(path: str | PathLike[str], **kwargs) int[source]¶
Get the checksum of an S3 object or directory.
See
S3FileSystem.checksum().- Parameters:
path – S3 path (s3://bucket/key) to get the checksum for.
**kwargs – Additional arguments passed to
S3FileSystem.checksum().
- Returns:
Integer checksum derived from the ETag or the directory token.
- created(path: str | os.PathLike[str]) datetime[source]¶
Return the creation time of the path.
See
S3FileSystem.created().- Parameters:
path – S3 path (s3://bucket/key).
- Returns:
The last-modified time of the object.
- modified(path: str | os.PathLike[str]) datetime[source]¶
Return the last-modified time of the path.
See
S3FileSystem.modified().- Parameters:
path – S3 path (s3://bucket/key).
- Returns:
The last-modified time of the object.
- invalidate_cache(path: str | PathLike[str] | None = None) None[source]¶
Remove the cached entries of the path and its parent paths.
See
S3FileSystem.invalidate_cache().- Parameters:
path – The path to invalidate. If None, clear the whole cache.
- touch(path: str | PathLike[str], truncate: bool = True, **kwargs) dict[str, Any][source]¶
Create an empty object with PutObject.
See
S3FileSystem.touch().- Parameters:
path – S3 path (s3://bucket/key) of the object.
truncate – If True, replace an existing object with an empty one; if False, raise if the object exists.
**kwargs – Additional parameters passed to the PutObject API.
- Returns:
The PutObject response as a dictionary.
- class pyathena.filesystem.s3_async.AioS3File(fs: S3FileSystem, path: str | S3Path, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, executor: S3Executor | None = None, block_size: int = 5242880, cache_type: str = 'bytes', autocommit: bool = True, cache_options: dict[Any, Any] | None = None, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]¶
The file handle that
AioS3FileSystem.open()returns.Functionally identical to
S3File, whose parallel operations (range reads, multipart uploads) it runs in the request pool of the filesystem, at mostmax_workersof them at once; it exists as a distinct type forisinstancechecks. For reads and writes that are coroutines, seeAioS3StreamedFile.
- class pyathena.filesystem.s3_async.AioS3StreamedFile(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs)[source]¶
An S3 file whose reads and writes are coroutines.
Returned by
AioS3FileSystem.open_async(). The requests of the file run in the request pool of the filesystem, at mostmax_workersof them at once, and the coroutines wait for them without blocking the event loop:A read splits the requested range into ranges of the block size, read in parallel. Nothing is cached. As with
S3File, the reads are conditional on the ETag of the object, and on its version when the filesystem is version-aware.A write buffers the data, and uploads each full block as a part of a multipart upload, with at most
max_workersparts, including the part copies of an append, in flight. A write that would submit another part waits for one of them to finish first. A request found failed by the next write, or before a part is submitted, stops the write with its error. An append copies an existing object withUploadPartCopy, or rewrites one smaller than a part from the buffer, asS3Filedoes.Closing the file commits the written data, as
S3File.close()does: with PutObject if no part was uploaded, otherwise by completing the multipart upload, which is aborted if the completion fails.Leaving an
async withblock with an exception, a failed write, and a cancelled write close the file without committing it: the requests that have started are waited for and the multipart upload is aborted, so that the existing object is left unchanged.
The commit and the abort, which wait for the requests, run to their end in the threads of the filesystem, also if the task that waits for them is cancelled, including a commit that still waits for a thread; once the commit has been submitted, nothing else aborts the upload. In a forked child process, a file that the parent opened raises
RuntimeErrorfrom its reads, writes, and flushes, and closing it only marks it closed. A file opened for writing must be closed withawait close()or anasync withblock; garbage collection cannot await it, so the data of a file that is collected before it is closed or its commit has been submitted is never committed; its multipart upload, if any, is aborted, with aResourceWarning.- fs: AioS3FileSystem¶
- __init__(fs: AioS3FileSystem, path: str, mode: str = 'rb', version_id: str | None = None, max_workers: int | None = None, block_size: int = 5242880, size: int | None = None, s3_additional_kwargs: dict[str, Any] | None = None, **kwargs) None[source]¶
Initialize the file for the path and mode.
Sends the lookup requests of the mode (see
S3FileTarget), which block;AioS3FileSystem.open_async()runs this in a thread.- Parameters:
fs – The filesystem that the file belongs to.
path – S3 path (s3://bucket/key) of the file.
mode – The file mode:
rb,wb,ab, orxb.version_id – The version ID to read.
max_workers – The number of parallel requests of the file, in the request pool of the filesystem. If None, the
max_workersof the filesystem.block_size – The block size for reads and writes. Must be between
S3Core.MULTIPART_UPLOAD_MIN_PART_SIZEandS3Core.MULTIPART_UPLOAD_MAX_PART_SIZE, inclusive, unless reading.size – The size of the object to read, if known.
s3_additional_kwargs – Additional parameters for the S3 requests of the file. Each request receives those that its operation accepts.
**kwargs – Additional parameters for the S3 requests of the file, which take precedence over
s3_additional_kwargs.
- Raises:
FileExistsError – If an object exists at the path in exclusive-create mode.
FileNotFoundError – If no object exists at the path when reading.
NotImplementedError – If the mode is not supported.
ValueError – If
max_workersis not positive, or the arguments are invalid as forS3FileTarget.resolve().
- __enter__() AioS3StreamedFile[source]¶
Reject a synchronous
withblock, whose exit cannot close the file.- Raises:
TypeError – Always; use
async with.
- async read(length: int = -1) bytes[source]¶
Read bytes from the file.
- Parameters:
length – The number of bytes to read; -1 reads to the end.
- Returns:
The bytes read.
- Raises:
RuntimeError – In a forked child process of the process that opened the file.
ValueError – If the file is closed or not open for reading.
- async write(data: bytes) int[source]¶
Write data to the buffer, uploading each full block as a part.
- Parameters:
data – The bytes to write.
- Returns:
The number of bytes written.
- Raises:
RuntimeError – In a forked child process of the process that opened the file.
ValueError – If the file is not open for writing, is closed, or has been force-flushed.
BaseException – The error of a request of the write that failed since the last write, or before a part is submitted, after which the file is closed without committing it.
- async flush(force: bool = False) None[source]¶
Upload the buffered data if it fills a block, or all of it if forced.
- Parameters:
force – Whether this is the final flush of the file, after which the written data is committed.
- Raises:
RuntimeError – In a forked child process of the process that opened the file.
ValueError – If the file is closed, or force-flushed twice.
- async close() None[source]¶
Close the file, committing the written data.
Returns once the requests of the file and its scheduled commit have ended, also when the close or an earlier read or flush was cancelled; the event loop runs meanwhile. In a forked child process, a file that the parent opened is only marked closed, without committing or aborting anything.
Read-ahead Stream¶
- class pyathena.filesystem.reader.ReadAheadStream(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None)[source]¶
Sequential stream over a file that reads ranges ahead in parallel.
The file is read in parts of
part_sizebytes with a range-read function, and up tomax_partsparts are read in a thread pool ahead of the consumer, so the transfer is not limited to one request at a time. Parts are returned in order, and at mostmax_partsparts plus the one being consumed are held in memory. A read that returns fewer bytes than requested raisesOSError, and an error from the range-read function is raised by the read that reaches its part; every later read raises the same error.Closing the stream cancels the queued reads, waits for the running ones, and then calls
on_close.Example
>>> file = fs.open_input_file(path) >>> stream = ReadAheadStream( ... lambda offset, size: file.read_at(size, offset), file.size(), on_close=file.close ... ) >>> with pyarrow.PythonFile(stream, mode="r") as f: ... table = pyarrow.csv.read_csv(f)
- __init__(read_range: Callable[[int, int], bytes], size: int, part_size: int | None = None, max_parts: int | None = None, on_close: Callable[[], None] | None = None) None[source]¶
Initialize the stream without reading the file.
- Parameters:
read_range – A function that reads
sizebytes of the file fromoffset, called asread_range(offset, size)from several threads at once.size – The size of the file in bytes.
part_size – Bytes requested by one
read_range()call; if None,DEFAULT_PART_SIZE.max_parts – The most parts read ahead of the consumer at once; if None,
DEFAULT_MAX_PARTS.on_close – A function called once when the stream is closed, after the running reads have finished, such as the
close()of the file.
- Raises:
ValueError – If
part_sizeormax_partsis not positive.
- readable() bool[source]¶
Return whether object was opened for reading.
If False, read() will raise OSError.
- readinto(buffer: Any) int[source]¶
Fill
bufferfrom the parts read ahead.- Parameters:
buffer – A writable buffer.
- Returns:
The number of bytes read, less than the size of
bufferonly at the end of the file.- Raises:
ValueError – If the stream is closed, or its closing was interrupted.
S3 Executor¶
- class pyathena.filesystem.s3_executor.S3Executor[source]¶
Abstract executor for parallel S3 operations.
Defines the interface used by
S3FileandS3FileSystemfor submitting work to run in parallel and for shutting down the executor when done. Bothsubmitandshutdownmirror theconcurrent.futures.Executorinterface so thatas_completed()andFuture.cancel()work unchanged. Each submitted callable runs with a copy of the context variables of its submitter, so that the stop scopes ofstop_retries_on()reach the requests that it makes.- abstractmethod submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) Future[T][source]¶
Submit a callable for execution and return a Future.
An implementation runs the callable in a copy of the submitter’s context, such as with
contextvars.copy_context().run, so that the stop scopes of the submitter reach it.
- abstractmethod shutdown(wait: bool = True) None[source]¶
Shut down the executor, freeing any resources.
- async run(fn: Callable[[...], T], *args: Any, **kwargs: Any) T[source]¶
Run a blocking call in the executor and wait for it.
Cancelling the waiting task cancels a call that has not started. A running call makes no further attempt of its AWS API calls, as with
run_in_thread(), and a write in it sends no further part and does not commit (seeraise_if_stopped()); the task waits for the call to end before it re-raises the cancellation, also when the event loop shuts down withasyncio.run(). A repeated cancellation stops that wait, and the call then ends in the executor.- Parameters:
fn – The blocking callable.
*args – Positional arguments passed to the callable.
**kwargs – Keyword arguments passed to the callable.
- Returns:
The return value of the call.
- Raises:
RuntimeError – If the executor has been shut down.
- async run_shielded(fn: Callable[[...], T], *args: Any, **kwargs: Any) T[source]¶
Run a blocking call that a cancellation of the waiting task does not stop.
The call runs to its end, even if it has not started when the waiting task is cancelled, and its retries are not stopped; its outcome is then dropped. See
wait_shielded().- Parameters:
fn – The blocking callable.
*args – Positional arguments passed to the callable.
**kwargs – Keyword arguments passed to the callable.
- Returns:
The return value of the call.
- Raises:
RuntimeError – If the executor has been shut down.
- async wait_shielded(future: Future[T]) T[source]¶
Wait for a submitted call that a cancellation of the waiting task does not cancel.
Unlike awaiting
asyncio.wrap_future(future), a cancellation of the waiting task leaves the future to the caller, which can still wait for it or cancel it, and its outcome is retrieved, so that asyncio does not log a failure that ends after the cancellation as never retrieved.- Parameters:
future – The future of a call submitted to this executor.
- Returns:
The return value of the call.
- static waiter(future: Future[T]) Future[T][source]¶
Wrap a future of an executor for asyncio, retrieving its outcome.
The outcome belongs to the caller that awaits the wrapper; it is retrieved, so that asyncio does not log a failure as never retrieved when no one awaits it, such as after a cancellation.
- Parameters:
future – The future.
- Returns:
The asyncio future.
- submit_to(future: Future[T], fn: Callable[[...], T], *args: Any, **kwargs: Any) None[source]¶
Submit a callable whose outcome resolves a future of the caller.
The caller creates the future before calling this method, so that it keeps a reference to the future even if an interrupt stops the scheduling midway, before
submit()returns. Cancelling the future keeps a callable that has not started from running; a started callable sets the future’s result or exception. If the executor drops the callable before it starts, such as by cancelling it at its shutdown, the future is cancelled or gets the error that the executor reported.The arguments of a failed callable, such as the body of a part, are freed when it fails: its error keeps the frames that it was raised from, without their local variables. The errors that it was raised from keep theirs;
S3Core.call()clears those of theClientErrorof a request.- Parameters:
future – A pending future that no executor has started.
fn – The callable to run.
*args – Positional arguments passed to the callable.
**kwargs – Keyword arguments passed to the callable.
- map(fn: Callable[[ItemT], T], items: Iterable[ItemT], *, stop_on_failure: bool = True, counts: Callable[[BaseException], bool] | None = None) S3MapResult[T][source]¶
Run a callable for each item in the executor and wait for the calls.
Every item is submitted at once, and the executor applies its own limit, such as the
max_workersof a scope. Each call runs in a scope ofstop_retries_on()that the stop of the batch ends.With
stop_on_failure, the first failure that counts stops the batch: the calls that have not started are cancelled, the running ones make no further attempt, and the failure is raised once they have ended. An interrupt of the waiting thread, such asKeyboardInterrupt, does the same whateverstop_on_failureis, and is raised once the running calls have ended. A further interrupt stops that wait; the running calls then end in the executor.A thread of a scope’s pool that maps on that scope runs the calls itself. A thread of a bare
S3ThreadPoolthat maps on that pool waits for threads that it may hold, so a pool must not map on itself.- Parameters:
fn – The callable to run for each item.
items – The items.
stop_on_failure – Whether a failure that counts stops the batch.
counts – Whether a failure counts, by default every one. A failure that does not count neither stops the batch nor becomes
S3MapResult.failure.
- Returns:
The outcomes of the calls.
- Raises:
BaseException – The failure that stopped the batch.
RuntimeError – If the executor has been shut down.
- async map_async(fn: Callable[[ItemT], T], items: Iterable[ItemT], *, stop_on_failure: bool = True, counts: Callable[[BaseException], bool] | None = None) S3MapResult[T][source]¶
Run a callable for each item in the executor and await the calls.
As
map(), with a cancellation of the waiting task in place of an interrupt. The task waits for the running calls also when the event loop shuts down withasyncio.run(), which cancels it and runs the loop until it ends; a loop that is closed without cancelling it leaves the running calls to end in the executor.- Parameters:
fn – The callable to run for each item.
items – The items.
stop_on_failure – Whether a failure that counts stops the batch.
counts – Whether a failure counts, by default every one.
- Returns:
The outcomes of the calls.
- Raises:
BaseException – The failure that stopped the batch.
RuntimeError – If the executor has been shut down.
- class pyathena.filesystem.s3_executor.S3ThreadPool(max_workers: int | None = None)[source]¶
A thread pool for the requests of the S3 filesystems.
It runs at most
max_workerscalls at once in threads of its own, so that, unlikeasyncio.to_thread(), which runs a call in the default executor of the event loop, neither the size nor the load of that executor limits or blocks the calls of asyncio code. The calls are queued in the order they were submitted. The threads are created as the calls need them, and each call runs with a copy of the caller’s context variables, as withasyncio.to_thread(). In a forked child process, whose copy has no threads, the pool creates new ones, as it does afterrelease(). The pool can be shared by several threads and event loops.S3ThreadPoolExecutorandS3AioThreadPoolare aliases of this class.- Parameters:
max_workers – The maximum number of calls that run at once.
- Raises:
ValueError – If
max_workersis not positive.
- __init__(max_workers: int | None = None) None[source]¶
Initialize the pool without starting any thread.
- Parameters:
max_workers – The maximum number of calls that run at once. If None, the default of
ThreadPoolExecutor:min(32, CPUs + 4).- Raises:
ValueError – If
max_workersis not positive.
- submit(fn: Callable[[...], T], *args: Any, **kwargs: Any) Future[T][source]¶
Schedule a blocking call in a thread of the pool.
As with
ThreadPoolExecutor, the returned future can be cancelled only before the call starts. The calls are queued in the order they were submitted, so that the calls submitted before a call that waits for them are taken by threads before it, and it does not wait for a thread that it holds.- Parameters:
fn – The blocking callable.
*args – Positional arguments passed to the callable.
**kwargs – Keyword arguments passed to the callable.
- Returns:
The future of the call.
- Raises:
RuntimeError – If the pool has been shut down.
- shutdown(wait: bool = True) None[source]¶
Stop accepting calls and release the threads once they are idle.
The calls that were submitted before still run.
- Parameters:
wait – Whether to wait for the submitted calls to finish.
- release() None[source]¶
Release the threads once no call runs or waits, keeping the pool usable.
Until then, the calls, including those submitted after this method, run in the threads of the pool, at most
max_workersat once. A call submitted after the threads have been released creates new ones. This method does not block.
- pyathena.filesystem.s3_executor.S3ThreadPoolExecutor = <class 'pyathena.filesystem.s3_executor.S3ThreadPool'>¶
An alias of
S3ThreadPool, kept for compatibility.
- pyathena.filesystem.s3_executor.S3AioThreadPool = <class 'pyathena.filesystem.s3_executor.S3ThreadPool'>¶
An alias of
S3ThreadPool, kept for compatibility.
- class pyathena.filesystem.s3_executor.S3AioExecutor(loop: AbstractEventLoop | None = None, max_workers: int = 20)[source]¶
A thread pool kept for compatibility, an
S3ThreadPool.It used to dispatch the calls through an asyncio event loop. The calls now run directly in the pool, so they need no running loop, and they run to their end when the loop stops.
- Parameters:
loop – Ignored, with a
DeprecationWarningif given.max_workers – The maximum number of calls that run at once.
- Raises:
ValueError – If
max_workersis not positive.
- __init__(loop: AbstractEventLoop | None = None, max_workers: int = 20) None[source]¶
Initialize the pool without starting any thread.
- Parameters:
loop – Ignored, with a
DeprecationWarningif given.max_workers – The maximum number of calls that run at once.
- Raises:
ValueError – If
max_workersis not positive.
S3 Paths¶
- class pyathena.filesystem.s3_path.S3Path(bucket: str, key: str | None = None, version_id: str | None = None)[source]¶
An S3 path: a bucket, optionally a key, and optionally a version ID.
Paths are parsed from strings such as
s3://bucket/key,s3a://bucket/keyorbucket/key, optionally followed by a version ID query (?versionId=,?versionID=,?versionid=or?version_id=). Only a query at the end of the path is a version ID; any other?is part of the key. The root path, which names no bucket, is not anS3Path.- key¶
The key, or None for a bucket path. An empty key is stored as None. A trailing slash is kept, and a key of only slashes (
bucket//) names the bucket (seeis_bucket).- Type:
str | None
Example
>>> path = S3Path.parse("s3://bucket/dir/key?versionId=v1") >>> path.bucket, path.key, path.version_id ('bucket', 'dir/key', 'v1') >>> path.name 'bucket/dir/key' >>> str(path) # The name, with a version ID query if there is one. 'bucket/dir/key?versionId=v1' >>> path.uri 's3://bucket/dir/key?versionId=v1'
- VERSION_QUERY: ClassVar[Pattern[str]] = re.compile('\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z')¶
- PATTERN: ClassVar[Pattern[str]] = re.compile('(^s3://|^s3a://|^)(?P<bucket>[a-zA-Z0-9.\\-_]+)(/|/(?P<key>.+?))?(\\Z|\\?version(Id|ID|id|_id)=(?P<version_id>[^?]+)\\Z)', re.DOTALL)¶
- __post_init__() None[source]¶
Store an empty key as None, so that the path is the bucket path it prints as.
- classmethod parse(path: str) S3Path[source]¶
Parse a string into an S3 path.
- Parameters:
path – The S3 path (e.g., “s3://bucket/key?versionId=…”).
- Returns:
The path. The key is None for a bucket path (
bucketorbucket/), and keeps a trailing slash otherwise.- Raises:
ValueError – If the string is not a valid S3 path.
- classmethod of(path: str | S3Path) S3Path[source]¶
Return a path string parsed, or an S3 path as it is.
An
S3Path, such as a key that a listing returns, is never parsed again, so that a key that ends in what looks like a version ID query keeps naming that key.- Parameters:
path – The S3 path string, as for
parse(), or anS3Path.- Returns:
The path.
- Raises:
ValueError – If the string is not a valid S3 path.
- classmethod split_version_id(path: str) tuple[str, str | None][source]¶
Split the version ID query from the end of a path string.
Unlike
parse(), any string is accepted, such as the names that fsspec builds.- Parameters:
path – The path string.
- Returns:
Tuple of the string without the version ID query and the version ID, or of the string itself and None if it ends with no version ID query.
- classmethod has_version_id(path: str | PathLike[str]) bool[source]¶
Return whether a path string ends with a version ID query.
Unlike
parse(), any string is accepted, as withsplit_version_id().- Parameters:
path – The path.
- Returns:
Whether the path ends with a version ID query.
- property is_bucket: bool¶
it has no key, or a key of only slashes.
- Type:
Whether the path names the bucket
- property is_directory_bucket: bool¶
Whether the bucket is a directory bucket (S3 Express One Zone).
The names of directory buckets end with
--x-s3.
- property target: S3Path¶
The default move target, assuming bucket versioning is not enabled.
A
nullversion is taken to name the key itself: in a bucket without versioning, or with versioning suspended, a write to the key replaces itsnullversion. With versioning enabled, a write adds a new version instead, so callers with that state must keep thenullversion distinct from its key. Any other path is its own target.
- class pyathena.filesystem.s3_path_pairing.S3PathPairing(path1: str | list[str], path2: str | list[str], recursive: bool = False, maxdepth: int | None = None)[source]¶
The pairing of the paths of one
copy(),get()ormv().The sources are paired with their destinations as fsspec’s
copy()pairs them, except that a path with a version ID names that version of an object: it is not a glob pattern, and its destination is named after its key without the version. A source may also be anS3Path, such as a key that a listing returns, which names its bucket and key as they are, also when the key ends in what looks like a version ID query, and which is never parsed again (sees3_copy_pairs()). A move compares the paths by what they name (seetarget), keepingnullversions distinct from their keys in the caller’s versioning-enabled buckets.delete_paths()splits the paths of anrm().The pairing is a frozen dataclass of the paths as given, and holds no filesystem. The caller makes the lookups that it asks for (
expands,skips_directories,looks_up_destinationandconflict_candidates()) with its own requests and cache: it expands the sources, leaves out the directories whenskips_directoriessays so, and passes the results in. Asources,destination_is_dirormissingthat a rule needs and that is not passed raisesValueError.Example
>>> pairing = S3PathPairing("s3://bucket/dir/", "s3://bucket/copy/", recursive=True) >>> sources = fs.expand_path(pairing.path1, recursive=True) >>> pairing.copy_pairs(sources)
- property expands: bool¶
Whether the sources are expanded.
False if both paths are lists, which are paired as given, without expansion or lookups.
- property skips_directories: bool¶
Whether the directories among the expanded sources are left out.
True for a string source copied without
recursive, or with amaxdepth. A directory is a source that ends with a slash or that the filesystem reports as a directory.
- property looks_up_destination: bool¶
Whether the pairing depends on the destination being a directory.
True for a string source that is neither a glob pattern nor ends with a slash, copied to a string destination that does not end with a slash.
copy_pairs()then needsdestination_is_dir.
- copy_pairs(sources: Sequence[str | S3Path] | None = None, destination_is_dir: bool | None = None) list[tuple[str | S3Path, str]][source]¶
Pair the sources with their destinations as fsspec’s
copy()does.A source string with a version ID is named after its key without the version, and an
S3Pathsource after its bucket and key (itsname), whose suffix the destination keeps as it is.- Parameters:
sources – The expansion of
path1, without the directories thatskips_directoriesleaves out; needed whenexpands.destination_is_dir – Whether
path2is a directory; needed whenlooks_up_destinationis true and there are sources.
- Returns:
The sources and their destinations, which keep the form of
path2. When both paths are lists, they are paired as given, up to the end of the shorter list, as in fsspec’scopy(). Empty if there are no sources.- Raises:
ValueError – If
sourcesordestination_is_diris needed and None.
- s3_copy_pairs(sources: Sequence[str | S3Path] | None = None, destination_is_dir: bool | None = None) list[tuple[str | S3Path, str | S3Path]][source]¶
Pair the sources with their S3 destinations, as
copy_pairs()does.The destination of an
S3Pathsource, whenpath2is a string, is anS3Pathtoo:path2parsed, with the suffix thatcopy_pairs()appends to it added to its key as it is, so that a key such asa/x?versionId=nullis copied to the keyb/x?versionId=nullinstead of being parsed as a version. The other destinations are strings, ascopy_pairs()returns them.- Parameters:
sources – As for
copy_pairs().destination_is_dir – As for
copy_pairs().
- Returns:
The sources and their destinations.
- Raises:
ValueError – If
sourcesordestination_is_diris needed and None, or ifpath2names a version, which cannot be written, and a destination is anS3Path.
- conflict_candidates(pairs: Sequence[tuple[str | S3Path, str | S3Path]], *, versioning_enabled_buckets: Collection[str] = ()) list[str | S3Path][source]¶
Return the sources of a move or a copy whose conflicts depend on an object at their key.
A source without a version may be a directory, such as a prefix given in a list or one with other sources below it. If no object exists at its key, it is not copied, so it writes no destination and is left out of the conflict checks of
move_pairs()andcheck_copy_pairs(). A path with a version always names an object.- Parameters:
pairs – The sources and destinations of the move or the copy, as
copy_pairs()ors3_copy_pairs()of this pairing returns them. A string is parsed; anS3Pathis used as it is.versioning_enabled_buckets – Buckets whose versioning is enabled. Their
nullversions are distinct from their unversioned keys. Use the same collection formove_pairs()orcheck_copy_pairs().
- Returns:
The sources without a version whose destination conflicts, one per pair in the order of the pairs, as their
target: inbucket/keyform for a source string, and as anS3Pathfor anS3Pathsource. Empty if nothing needs to be looked up.- Raises:
TypeError – If
versioning_enabled_bucketsis a string.
- move_pairs(pairs: Sequence[tuple[str | S3Path, str | S3Path]], missing: Collection[str | S3Path] | None = None, *, versioning_enabled_buckets: Collection[str] = ()) list[tuple[str | S3Path, str | S3Path]][source]¶
Check the pairs of a move and leave out the sources that stay in place.
- Parameters:
pairs – The sources and destinations of the move, as
copy_pairs()ors3_copy_pairs()of this pairing returns them.missing – The
conflict_candidates()without an object at their key, as strings in any form that names them or asS3Pathobjects; needed when there are candidates.versioning_enabled_buckets – Buckets whose versioning is enabled. Their
nullversions are distinct from their unversioned keys. Use the same collection forconflict_candidates().
- Returns:
The pairs, except those whose destination is the source itself or, for a
nullversion in a bucket without enabled versioning, the key of the source.- Raises:
ValueError – If two sources have the same destination, or a destination is another source, including one left in place, except for a source without a version in
missing, which writes no destination. Also ifmissingis needed and None.TypeError – If
missingis a string instead of a collection. Also ifversioning_enabled_bucketsis a string.
- check_copy_pairs(pairs: Sequence[tuple[str | S3Path, str | S3Path]], missing: Collection[str | S3Path] | None = None, *, versioning_enabled_buckets: Collection[str] = ()) None[source]¶
Check the pairs of a copy whose objects are copied in parallel.
A copy has the conflicts of a move (see
move_pairs()), since a parallel copy has no order in which one pair would overwrite another or read a source after another pair has written it. A pair whose destination is its own source is no conflict: it rewrites the object in place, such as withMetadataDirective="REPLACE".- Parameters:
pairs – The sources and destinations of the copy, as
copy_pairs()ors3_copy_pairs()of this pairing returns them.missing – The
conflict_candidates()without an object at their key, as strings in any form that names them or asS3Pathobjects; needed when there are candidates.versioning_enabled_buckets – Buckets whose versioning is enabled. Use the same collection for
conflict_candidates().
- Raises:
ValueError – If two sources have the same destination, or a destination is another source, except for a source without a version in
missing, which writes no destination. Also ifmissingis needed and None.TypeError – If
missingis a string instead of a collection. Also ifversioning_enabled_bucketsis a string.
- classmethod delete_paths(path: str | list[str]) tuple[list[str], list[str]][source]¶
Split the paths that
rm()deletes into those deleted as given and those expanded.- Parameters:
path – S3 path or list of paths.
- Returns:
The paths with a version ID, which are deleted as given, and the other paths, which the filesystem expands.
- Raises:
ValueError – If a path is a bucket.
S3 Core¶
- class pyathena.filesystem.s3_core.S3Core(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None)[source]¶
Typed S3 operations on a boto3 S3 client.
Each operation sends one request, or one per page for the iterators, except those that say otherwise, such as
plan_multipart_copy(),copy_object_annotation()andgenerate_presigned_url(). The requests are sent with the retry policy, and S3 errors are translated intoOSErrorsubclasses (seeS3ClientError): a missing bucket or multipart upload, or a missing object or version that an operation reads, raisesFileNotFoundError, and a denied requestPermissionError.copy_object()andupload_part_copy()with a source version that is a delete marker raiseOSErrorwithEINVAL. As in S3, deleting a missing key is not an error. Nothing is cached.Example
>>> core = S3Core(boto3.client("s3")) >>> core.head_object(S3Path.parse("s3://bucket/key")).content_length >>> for page in core.list_objects("bucket", prefix="dir/", delimiter="/"): ... print([o.key for o in page.objects])
- OBJECT_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'aws-exec-read', 'bucket-owner-full-control', 'bucket-owner-read', 'private', 'public-read', 'public-read-write'})¶
- BUCKET_ACLS: ClassVar[frozenset[str]] = frozenset({'authenticated-read', 'private', 'public-read', 'public-read-write'})¶
- RETRY_ERROR_CODES: ClassVar[tuple[str, ...]] = ('SlowDown', 'ServiceUnavailable', 'InternalError', 'RequestTimeout')¶
The S3 error codes of transient failures (status 503, 500, and 400), which botocore’s retry modes also retry. A retry policy with the default codes retries them too.
- __init__(client: BaseClient, retry_config: RetryConfig | None = None, request_kwargs: Mapping[str, Any] | None = None) None[source]¶
Create the operations on a client.
- Parameters:
client – The boto3 S3 client to send the requests with.
retry_config – The retry policy of the requests; defaults to
RetryConfig(). The requests are sent with the policy thatresolve_retry_config()returns for it.request_kwargs – Parameters sent with every operation that accepts them, such as
{"RequestPayer": "requester"}.
- classmethod resolve_retry_config(retry_config: RetryConfig) RetryConfig[source]¶
Resolve the retry policy that S3 requests are sent with.
A policy whose
exceptionsare the default codes (THROTTLING_ERROR_CODES), in any order, also retriesRETRY_ERROR_CODES. A policy with otherexceptionsis used as given.- Parameters:
retry_config – The retry policy.
- Returns:
A copy of the policy that also retries
RETRY_ERROR_CODESwith the same attempts and waits, or the policy itself.
- property client: BaseClient¶
The boto3 S3 client.
- property retry_config: RetryConfig¶
The retry policy of the requests, with
RETRY_ERROR_CODESif they were added.
- property request_kwargs: dict[str, Any]¶
A copy of the parameters sent with every operation that accepts them.
- operation_params(method: str, params: Mapping[str, Any]) dict[str, Any][source]¶
Select the parameters that an S3 operation accepts.
Parameters inherited by several requests are filtered by the input shape of each operation, so that, e.g.,
ServerSideEncryptionfor writes is not sent with GetObject.- Parameters:
method – The name of the client method, such as
get_object.params – The parameters to select from.
- Returns:
The parameters that the operation accepts. Empty for a method that is not an S3 API operation, such as
generate_presigned_url.
- call(method: str | Callable[[...], Any], **request) dict[str, Any][source]¶
Send a request with the retry policy and translate its errors.
request_kwargsthat the operation accepts are added; a parameter of the request takes precedence. The operation is identified by the name of the method; afunctools.partialis identified by the method it wraps, and its bound keywords are parameters of the request. A callable whose__name__is not that of a client method is sent without them.A
Bodythat is a stream, given to the call or bound by afunctools.partial, is sent from the position where the call found it on each attempt of the policy. One whose position cannot be read withtell()and restored withseek()is sent once, without the retries of the policy, since a retry would send only the rest of the stream.- Parameters:
method – The name of the client method, or the method itself.
**request – The request parameters, sent as given.
- Returns:
The response.
- Raises:
ValueError – If botocore rejects the parameters of the request (
ParamValidationError), which is not sent.OSError – The translation of an S3 error response (see
S3ClientError).
Note
The frames of a raised error below this method, those of the retries and of botocore, have no local variables, so that they keep neither the request, such as the body of a part, nor the retry state, which keeps the error in a reference cycle.
- head_object(path: S3Path, **params) S3Metadata[source]¶
Look up an object, or a version of it, with HeadObject.
- Parameters:
path – The path of the object, with the version ID to look up, if any.
**params – Additional request parameters, sent as given.
- Returns:
The metadata of the object.
- Raises:
ValueError – If the path has no key.
FileNotFoundError – If the object or version does not exist.
- head_bucket(bucket: str, **params) S3Bucket[source]¶
Look up a bucket with HeadBucket.
- Parameters:
bucket – The name of the bucket.
**params – Additional request parameters, sent as given.
- Returns:
The bucket, with its region if returned.
- Raises:
FileNotFoundError – If the bucket does not exist.
- get_object(path: S3Path, range_: tuple[int, int | None] | None = None, **params) bytes[source]¶
Read an object, a version of it, or a byte range of it with GetObject.
The body of the response is read whole and then closed, also when the read fails. A failure to read the body is neither translated nor retried: the botocore exception, such as
ReadTimeoutErrororIncompleteReadError, propagates.- Parameters:
path – The path of the object, with the version ID to read, if any.
range – The
(start, end)byte range to read, with an exclusive end, or None as the end to read to the end of the object. A negative start without an end reads the last-startbytes, or the whole object if it is shorter. None sends no range of its own, so the whole object is read unlessparamsorrequest_kwargshaveRange.**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The bytes read.
- Raises:
ValueError – If the path has no key, or the range is empty or has both a negative start and an end, which S3 would ignore and return the whole object for.
FileNotFoundError – If the object or version does not exist.
OSError – If the range starts at or past the end of the object. Its
__cause__is theClientErrorwith theInvalidRangecode.
- put_object(path: S3Path, body: bytes | None = None, **params) S3PutObject[source]¶
Write an object with PutObject.
- Parameters:
path – The path of the object to write, without a version ID.
body – The data to write. None or empty bytes send no body of their own, so an empty object is written unless
paramsorrequest_kwargshaveBody.**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The result of the write.
- Raises:
ValueError – If the path has no key, or has a version ID, which a write cannot replace.
- create_bucket(bucket: str, acl: str | None = None, region_name: str | None = None, **params) None[source]¶
Create a bucket with CreateBucket.
A call creates the bucket; the
allow_bucket_creationoption of the filesystem applies only to the filesystem’s methods.- Parameters:
bucket – The name of the bucket.
acl – The canned ACL of the bucket, one of
BUCKET_ACLS. None or an empty string sends no ACL.region_name – The region to create the bucket in. None or an empty string uses the region of the client. A location constraint is sent for every region except
us-east-1, which does not accept one.**params – Additional request parameters, sent as given, except
CreateBucketConfiguration. Its fields are kept, and theLocationConstraintofregion_nameis added unless the region isus-east-1or the configuration has aLocationConstraint,LocationorBucket. A directory bucket is created withLocationandBucket, so noLocationConstraintis added to its configuration. An empty or None configuration is treated as not given. The given configuration is not modified.
- Raises:
ValueError – If the ACL is not a canned ACL of buckets.
- delete_bucket(bucket: str, **params) None[source]¶
Delete a bucket, which must be empty, with DeleteBucket.
A call deletes the bucket; the
allow_bucket_deletionoption of the filesystem applies only to the filesystem’s methods.- Parameters:
bucket – The name of the bucket.
**params – Additional request parameters, sent as given.
- Raises:
FileNotFoundError – If the bucket does not exist.
- get_bucket_versioning(bucket: str, **params) str | None[source]¶
Get the versioning state of a bucket with GetBucketVersioning.
- Parameters:
bucket – The name of the bucket.
**params – Additional request parameters, sent as given.
- Returns:
EnabledorSuspended, or None if versioning has never been enabled on the bucket.- Raises:
FileNotFoundError – If the bucket does not exist.
- delete_object(path: S3Path, **params) None[source]¶
Delete an object, or a version of it, with DeleteObject.
- Parameters:
path – The path of the object, with the version ID to delete, if any.
**params – Additional request parameters, sent as given.
- Raises:
ValueError – If the path has no key.
FileNotFoundError – If the bucket does not exist.
- delete_objects(batch: S3DeleteBatch, **params) S3DeleteResult[source]¶
Delete the objects of a batch with one DeleteObjects request.
- Parameters:
batch – The objects to delete.
**params – Additional request parameters, sent as given.
- Returns:
The result, with the objects that S3 could not delete in its
errors.- Raises:
FileNotFoundError – If the bucket does not exist.
- create_multipart_upload(path: S3Path, **params) S3MultipartUpload[source]¶
Start a multipart upload to an object with CreateMultipartUpload.
- Parameters:
path – The path of the object to write, without a version ID.
**params – Additional request parameters. The bucket and key of the path take precedence over parameters of the same name.
- Returns:
The upload retaining the path’s bucket and key, including an access point alias or ARN, and the response’s upload ID and checksum configuration.
- Raises:
ValueError – If the path has no key, or has a version ID, which a write cannot replace.
- upload_part(upload: S3MultipartUpload, part_number: int, body: bytes, **params) S3MultipartUploadPart[source]¶
Upload a part of a multipart upload with UploadPart.
- Parameters:
upload – The multipart upload returned by creation. Its checksum algorithm is passed to the SDK to calculate the part checksum.
part_number – The number of the part, from 1.
body – The data of the part.
**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The uploaded part.
- Raises:
ValueError – If the upload has no bucket, key, or upload ID.
- upload_part_copy(upload: S3MultipartUpload, part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params) S3MultipartUploadPart[source]¶
Copy a part of a multipart upload from an object with UploadPartCopy.
- Parameters:
upload – The multipart upload returned by creation.
part_number – The number of the part, from 1.
source – The path of the object to copy, with the version ID to copy, if any.
range – The
(start, end)byte range of the source to copy, with an exclusive end. None sends no range of its own, so the whole source is copied unlessparamsorrequest_kwargshaveCopySourceRange.**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The copied part.
- Raises:
ValueError – If the upload has no bucket, key, or upload ID, or the source has no key.
FileNotFoundError – If the upload, or the source object or version, does not exist.
OSError – With
EINVALfor an invalid request, such as a source version that is a delete marker.
- complete_multipart_upload(upload: S3MultipartUpload, parts: Sequence[S3MultipartUploadPart], **params) S3CompleteMultipartUpload[source]¶
Complete a multipart upload with CompleteMultipartUpload.
- Parameters:
upload – The multipart upload returned by creation. Its checksum algorithm selects the matching part checksum, and its checksum type is sent when present. Without an algorithm, only the ETag and part number are sent, even if the SDK added a part checksum.
parts – The uploaded parts, in part-number order.
**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The completed upload.
- Raises:
ValueError – If the upload has no bucket, key, or upload ID.
- abort_multipart_upload(upload: S3MultipartUpload, **params) None[source]¶
Abort a multipart upload with AbortMultipartUpload.
An abort cleans up, so the stop scopes of the caller do not stop its retries (see
unstoppable_retries()).- Parameters:
upload – The multipart upload returned by creation or listing.
**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Raises:
ValueError – If the upload has no bucket, key, or upload ID.
FileNotFoundError – If the upload does not exist, for example because it was completed or aborted.
- part_ranges(size: int, block_size: int) list[tuple[int, int]][source]¶
Split an object into the source ranges of the parts that copy it.
The object is split into ranges of
block_sizebytes, or of a larger size that splits it into at mostMULTIPART_UPLOAD_MAX_PARTSranges. A last range shorter thanMULTIPART_UPLOAD_MIN_PART_SIZEis merged into the previous one, which is split in half if the result exceedsMULTIPART_UPLOAD_MAX_PART_SIZE. Every range is then within the S3 part size limits, including the last one unless the whole object is smaller than the minimum part size, so that more parts can follow the copied ones, as in an append.- Parameters:
size – The size of the source object in bytes.
block_size – The size in bytes to split the object by, between
MULTIPART_UPLOAD_MIN_PART_SIZEandMULTIPART_UPLOAD_MAX_PART_SIZE. It is raised tosizedivided byMULTIPART_UPLOAD_MAX_PARTS, rounded up, if smaller. The range that a short last range is merged into can be longer, up toMULTIPART_UPLOAD_MAX_PART_SIZE.
- Returns:
The
(start, end)byte ranges, with an exclusive end, that cover the whole object in order; empty for an empty object.
- check_multipart_upload_size(size: int, block_size: int) None[source]¶
Check that data fits in a multipart upload before it is uploaded.
Sends no request.
- Parameters:
size – The size of the data in bytes.
block_size – The size in bytes of the parts that upload the data.
- Raises:
ValueError – If the data takes more than
MULTIPART_UPLOAD_MAX_PARTSblocks. The message gives the minimum block size, which is at leastMULTIPART_UPLOAD_MIN_PART_SIZE.
- copy_object(source: S3Path, destination: S3Path, **params) None[source]¶
Copy an object, or a version of it, with CopyObject.
- Parameters:
source – The path of the object to copy, with the version ID to copy, if any.
destination – The path of the object to write, without a version ID.
**params – Additional request parameters, sent as given.
- Raises:
ValueError – If the source or the destination has no key, or the destination has a version ID, which a write cannot replace.
FileNotFoundError – If a bucket, or the source object or version, does not exist.
OSError – With
EINVALfor an invalid request, such as a source version that is a delete marker.
- plan_multipart_copy(source: S3Path, destination: S3Path, block_size: int | None = None, **params) S3MultipartCopyPlan[source]¶
Plan a copy with a multipart upload that copies as CopyObject does.
The source is read with HeadObject. Without a version in the path, a version ID other than
nullthat it reports is the version to copy, so that the parts, the tags and the annotations come from the same object even if the source is replaced during the copy. Anullversion, which a write can replace, is not pinned. If the reported size fits in a single CopyObject request, nothing else is read and the plan says so.No multipart request accepts the directives of CopyObject, so the plan applies them as CopyObject does. With the COPY metadata directive (the default), the content headers and the user-defined metadata of the source are used, and the values of
paramsare ignored. With the COPY tagging directive (the default), the tags are read with GetObjectTagging, and theTaggingofparamsis ignored. REPLACE uses the values ofparamsinstead. With the COPY annotation directive (the default), the annotations are listed with ListObjectAnnotations, unless the source is encrypted with SSE-C or is in a directory bucket, which cannot have annotations; an object in a directory bucket has no tags either. The requests that read the source receiveRequestPayer, and the source’s expected bucket owner and SSE-C parameters (ExpectedSourceBucketOwnerandCopySourceSSECustomer*) under their names in those requests.- Parameters:
source – The path of the object to copy, with the version ID to copy, if any.
destination – The path of the object to write, without a version ID.
block_size – The size in bytes of the copied ranges, between
MULTIPART_UPLOAD_MIN_PART_SIZEandMULTIPART_UPLOAD_MAX_PART_SIZE(the default); seepart_ranges().**params – The CopyObject parameters of the copy. Each request receives those that it accepts; CreateMultipartUpload also receives those that CopyObject does not accept either, so that botocore rejects them as it would for CopyObject.
- Returns:
The plan.
- Raises:
ValueError – If the source or the destination has no key, the destination has a version ID,
block_sizeis out of the part size limits, a directive has a value that CopyObject does not accept, or HeadObject reports no size.
- list_object_annotations(path: S3Path, **params) list[str][source]¶
List the names of the annotations of an object with ListObjectAnnotations.
Sends one request per page.
- Parameters:
path – The path of the object, with the version ID to list, if any.
**params – Additional request parameters. The fields that the other arguments set take precedence over parameters of the same name.
- Returns:
The annotation names, across all pages.
- Raises:
ValueError – If the path has no key.
- copy_object_annotation(name: str, source: S3Path, destination: S3Path, version_id: str | None, etag: str | None, **params) None[source]¶
Copy an annotation of an object onto the object that a copy wrote.
Reads the annotation with GetObjectAnnotation and writes it with PutObjectAnnotation. The annotation is written to the version that the copy created, if the bucket is versioned, and only if the destination still has the ETag of the copy, so that it is not attached to an object written over the copy.
The payload of the annotation is read whole and then closed, also when the read fails. As for
get_object(), a failure to read it is neither translated nor retried: the botocore exception propagates.- Parameters:
name – The annotation name.
source – The path of the copied object, with the version ID that was copied, if any.
destination – The path of the object that the copy wrote, without a version ID.
version_id – The version ID that the copy created, if any.
etag – The ETag of the object that the copy wrote, if any.
**params – The CopyObject parameters of the copy. GetObjectAnnotation receives those of the source, mapped as for
plan_multipart_copy(), and PutObjectAnnotation those that it accepts; the fields that the other arguments set take precedence.
- Raises:
ValueError – If the source or the destination has no key.
- get_object_tagging(path: S3Path, **params) dict[str, str][source]¶
Get the tags of an object, or of a version of it, with GetObjectTagging.
- Parameters:
path – The path of the object, with the version ID to read, if any, including
null.**params – Additional request parameters, sent as given.
- Returns:
The tags, mapping each key to its value, in the order of the response.
- Raises:
ValueError – If the path has no key.
FileNotFoundError – If the object or version does not exist.
- put_object_tagging(path: S3Path, tags: Mapping[str, str], **params) None[source]¶
Replace the tags of an object, or of a version of it, with PutObjectTagging.
- Parameters:
path – The path of the object, with the version ID to tag, if any, including
null.tags – The tags, mapping each key to its value. They replace all the existing tags.
**params – Additional request parameters, sent as given.
- Raises:
ValueError – If the path has no key.
FileNotFoundError – If the object or version does not exist.
- put_object_acl(path: S3Path, acl: str, **params) None[source]¶
Apply a canned ACL to an object, or to a version of it, with PutObjectAcl.
- Parameters:
path – The path of the object, with the version ID to apply the ACL to, if any, including
null.acl – The canned ACL, one of
OBJECT_ACLS.**params – Additional request parameters, sent as given.
- Raises:
ValueError – If the path has no key, or the ACL is not in
OBJECT_ACLS.FileNotFoundError – If the object or version does not exist.
- put_bucket_acl(bucket: str, acl: str, **params) None[source]¶
Apply a canned ACL to a bucket with PutBucketAcl.
- Parameters:
bucket – The name of the bucket.
acl – The canned ACL, one of
BUCKET_ACLS.**params – Additional request parameters, sent as given.
- Raises:
ValueError – If the ACL is not in
BUCKET_ACLS.FileNotFoundError – If the bucket does not exist.
- replace_object_metadata(path: S3Path, head: S3Metadata, metadata: Mapping[str, str], **params) None[source]¶
Replace the user-defined metadata of an object by copying it onto itself.
S3 does not update the metadata of an object in place, so the object is copied onto itself with CopyObject and the REPLACE metadata directive, which writes a new object, or a new version in a versioned bucket. With that directive, S3 does not copy what the request omits, so the request also sends the content headers,
Expires,WebsiteRedirectLocationandStorageClassofhead, and, unlessparamsset an encryption parameter, itsServerSideEncryption,SSEKMSKeyIdandBucketKeyEnabled. Fields that are None inheadare not sent; note thatS3MetadatareportsSTANDARDwhen HeadObject omits the storage class. HeadObject does not return the KMS encryption context, so it is not retained.- Parameters:
path – The path of the object, without a version ID.
head – The HeadObject result of
path(seehead_object()), whose fields are retained.metadata – The user-defined metadata of the copy, which replaces all the existing user-defined metadata.
**params – Additional CopyObject parameters. They take precedence over the retained fields; one that the copy itself sets, such as
Metadata,MetadataDirectiveorKey, raisesTypeError.
- Raises:
ValueError – If the path has no key or has a version ID, which a write cannot replace.
- generate_presigned_url(path: S3Path, client_method: str = 'get_object', expires_in: int = 3600, **params) str[source]¶
Generate a presigned URL for a request on an object.
The URL is signed locally with the client’s credentials; no request is sent.
request_kwargsare not added to the signed parameters.- Parameters:
path – The path of the object, with the version ID to sign for, if any. A path without a key is not rejected; botocore validates the parameters of the method.
client_method – The name of the client method to sign, such as
get_objectorput_object.expires_in – The number of seconds for which the URL is valid.
**params – Parameters of the method to sign. They take precedence over the bucket, key and version ID of the path.
- Returns:
The presigned URL.
- static copy_source_params(params: Mapping[str, Any]) dict[str, Any][source]¶
Map the parameters of a copy to those of the requests that read its source.
The source of a copy is looked up, read, or deleted (by a move) with the parameters that authorize the access to it, which CopyObject names apart from those of the destination.
- Parameters:
params – The CopyObject parameters of the copy.
- Returns:
RequestPayer, and the source’s expected bucket owner and SSE-C parameters under the names of the requests that read the source (ExpectedBucketOwnerandSSECustomer*), where given.
- static copy_from_source_params(params: Mapping[str, Any]) dict[str, Any][source]¶
Map the parameters of the requests that read an object to those of a copy from it.
The inverse of
copy_source_params(), for a copy whose source is read withparams, such as the part copies of an append.- Parameters:
params – The parameters of the requests that read the object.
- Returns:
RequestPayer, and the expected bucket owner and SSE-C parameters under the names of the copy source (ExpectedSourceBucketOwnerandCopySourceSSECustomer*), where given.
- list_objects_page(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) S3ListObjectsPage[source]¶
List one page of objects with ListObjectsV2.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list.
delimiter – The delimiter to group keys by; None or
""lists recursively.max_keys – The maximum number of keys of the page.
continuation_token – The token of the page to list.
**params – Additional request parameters, sent as given.
- Returns:
The page.
- list_objects(bucket: str, prefix: str = '', delimiter: str | None = None, max_keys: int | None = None, continuation_token: str | None = None, **params) Iterator[S3ListObjectsPage][source]¶
List the pages of objects with ListObjectsV2, one request each.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list.
delimiter – The delimiter to group keys by; None or
""lists recursively.max_keys – The maximum number of keys per page.
continuation_token – The token of the first page to list.
**params – Additional request parameters, sent as given.
- Yields:
The pages, until one has no continuation token.
- Raises:
TypeError – If
paramshasContinuationToken, which the iterator advances; passcontinuation_tokeninstead.
- list_object_versions_page(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) S3ListObjectVersionsPage[source]¶
List one page of versions with ListObjectVersions.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list.
delimiter – The delimiter to group keys by; None or
""lists recursively.key_marker – The key marker of the page to list.
version_id_marker – The version ID marker of the page to list. S3 accepts it only with a key marker.
**params – Additional request parameters, sent as given.
- Returns:
The page.
- list_object_versions(bucket: str, prefix: str = '', delimiter: str | None = None, key_marker: str | None = None, version_id_marker: str | None = None, **params) Iterator[S3ListObjectVersionsPage][source]¶
List the pages of versions with ListObjectVersions, one request each.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list.
delimiter – The delimiter to group keys by; None or
""lists recursively.key_marker – The key marker of the first page to list.
version_id_marker – The version ID marker of the first page to list, sent with
key_marker.**params – Additional request parameters, sent as given.
- Yields:
The pages, until one is not truncated or has no key marker.
- Raises:
TypeError – If
paramshasKeyMarkerorVersionIdMarker, which the iterator advances; passkey_markerandversion_id_markerinstead.
- list_multipart_uploads_page(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) S3ListMultipartUploadsPage[source]¶
List one page of in-progress multipart uploads with ListMultipartUploads.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.
key_marker – The key marker of the page to list.
upload_id_marker – The upload ID marker of the page to list. S3 accepts it only with a key marker.
**params – Additional request parameters, sent as given.
- Returns:
The page.
- Raises:
FileNotFoundError – If the bucket does not exist.
- list_multipart_uploads(bucket: str, prefix: str | None = None, key_marker: str | None = None, upload_id_marker: str | None = None, **params) Iterator[S3ListMultipartUploadsPage][source]¶
List the pages of in-progress multipart uploads, one request each.
- Parameters:
bucket – The bucket to list.
prefix – The key prefix to list, which S3 matches as a plain string prefix of the keys. None sends no prefix.
key_marker – The key marker of the first page to list.
upload_id_marker – The upload ID marker of the first page to list, sent with
key_marker.**params – Additional request parameters, sent as given.
- Yields:
The pages, until one is not truncated or lacks either next marker.
- Raises:
TypeError – If
paramshasKeyMarkerorUploadIdMarker, which the iterator advances; passkey_markerandupload_id_markerinstead.FileNotFoundError – If the bucket does not exist.
- list_buckets_page(continuation_token: str | None = None, **params) S3ListBucketsPage[source]¶
List one page of the buckets of the caller with ListBuckets.
- Parameters:
continuation_token – The token of the page to list.
**params – Additional request parameters, sent as given.
- Returns:
The page.
- list_buckets(continuation_token: str | None = None, **params) Iterator[S3ListBucketsPage][source]¶
List the pages of the buckets of the caller, one request each.
- Parameters:
continuation_token – The token of the first page to list.
**params – Additional request parameters, sent as given.
- Yields:
The pages, until one has no continuation token.
- Raises:
TypeError – If
paramshasContinuationToken, which the iterator advances; passcontinuation_tokeninstead.
- class pyathena.filesystem.s3_core.S3ObjectSummary(bucket: str, key: str, size: int | None = None, etag: str | None = None, last_modified: datetime | None = None, storage_class: str | None = None)[source]¶
An object listed by ListObjectsV2 (a
Contentsentry).- last_modified¶
When the object was last modified.
- Type:
datetime.datetime | None
- class pyathena.filesystem.s3_core.S3CommonPrefix(bucket: str, prefix: str)[source]¶
A key prefix listed with a delimiter (a
CommonPrefixesentry).
- class pyathena.filesystem.s3_core.S3Bucket(name: str, creation_date: datetime | None = None, bucket_region: str | None = None)[source]¶
A bucket, as listed by ListBuckets or found by HeadBucket.
- creation_date¶
When the bucket was created, if returned.
- Type:
datetime.datetime | None
- class pyathena.filesystem.s3_core.S3ListObjectsPage(bucket: str, objects: tuple[S3ObjectSummary, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), key_count: int | None = None, is_truncated: bool = False, next_continuation_token: str | None = None)[source]¶
One page of a ListObjectsV2 listing.
- objects¶
The listed objects (
Contents).
- common_prefixes¶
The listed key prefixes (
CommonPrefixes).
- objects: tuple[S3ObjectSummary, ...] = ()¶
- common_prefixes: tuple[S3CommonPrefix, ...] = ()¶
- class pyathena.filesystem.s3_core.S3ListObjectVersionsPage(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None)[source]¶
One page of a ListObjectVersions listing.
- versions¶
The listed versions (
Versions).
- delete_markers¶
The listed delete markers (
DeleteMarkers).
- common_prefixes¶
The listed key prefixes (
CommonPrefixes).
- versions: tuple[S3ObjectVersion, ...] = ()¶
- delete_markers: tuple[S3ObjectVersion, ...] = ()¶
- common_prefixes: tuple[S3CommonPrefix, ...] = ()¶
- classmethod from_response(bucket: str, response: Mapping[str, Any]) S3ListObjectVersionsPage[source]¶
Build the page from a ListObjectVersions response.
- Parameters:
bucket – The bucket that was listed.
response – The ListObjectVersions response.
- Returns:
The page.
- __init__(bucket: str, versions: tuple[S3ObjectVersion, ...] = (), delete_markers: tuple[S3ObjectVersion, ...] = (), common_prefixes: tuple[S3CommonPrefix, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_version_id_marker: str | None = None) None¶
- class pyathena.filesystem.s3_core.S3ListBucketsPage(buckets: tuple[S3Bucket, ...] = (), continuation_token: str | None = None)[source]¶
One page of a ListBuckets listing.
- buckets¶
The listed buckets.
- Type:
- class pyathena.filesystem.s3_core.S3ListMultipartUploadsPage(bucket: str, uploads: tuple[S3MultipartUpload, ...] = (), is_truncated: bool = False, next_key_marker: str | None = None, next_upload_id_marker: str | None = None)[source]¶
One page of a ListMultipartUploads listing.
- uploads¶
The listed in-progress multipart uploads (
Uploads), with the listed bucket as their bucket.
- uploads: tuple[S3MultipartUpload, ...] = ()¶
- class pyathena.filesystem.s3_core.S3DeleteBatch(bucket: str, objects: tuple[S3Path, ...], quiet: bool = True)[source]¶
The objects of one bucket that one DeleteObjects request deletes.
A batch is valid by construction: it has 1 to
MAX_KEYSobjects, each with a key in the bucket of the batch. A path with a version ID deletes that version.- objects¶
The paths of the objects to delete.
- Type:
- __post_init__() None[source]¶
Validate the objects of the batch.
- Raises:
ValueError – If the batch has no objects or more than
MAX_KEYS, or an object has no key or another bucket.
- classmethod from_paths(paths: Iterable[S3Path], quiet: bool = True) list[S3DeleteBatch][source]¶
Group the paths into batches by bucket, in the order of the paths.
- Parameters:
paths – The paths of the objects to delete.
quiet – Whether S3 omits the deleted objects from the responses.
- Returns:
The batches of up to
MAX_KEYSobjects of one bucket each.- Raises:
ValueError – If a path has no key.
- class pyathena.filesystem.s3_core.S3DeleteResult(bucket: str, deleted: tuple[S3Path, ...] = (), errors: tuple[S3DeleteError, ...] = ())[source]¶
The result of a DeleteObjects request.
S3 answers a request with 200 also when it could not delete some of the objects, and lists them in
errors.- deleted¶
The deleted objects, with the version ID of the request, if any. Empty for a quiet batch.
- Type:
- errors¶
The objects that S3 could not delete.
- errors: tuple[S3DeleteError, ...] = ()¶
- class pyathena.filesystem.s3_core.S3DeleteError(path: S3Path, code: str | None = None, message: str | None = None)[source]¶
An object that DeleteObjects could not delete (an
Errorsentry).str()gives"path (code: message)".- path¶
The path of the object, with the version ID of the request, if any.
- class pyathena.filesystem.s3_core.S3MultipartCopyPlan(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False)[source]¶
The requests of a copy with a multipart upload, as CopyObject copies.
S3Core.plan_multipart_copy()reads the source and builds the plan; the caller schedules the requests: CreateMultipartUpload withcreate_params, one UploadPartCopy per range withpart_params, then CompleteMultipartUpload withcomplete_params, or AbortMultipartUpload withabort_paramsafter a failure, and finally the copy of each annotation withS3Core.copy_object_annotation(). Iffits_single_requestis true, the source is copied withS3Core.copy_object()instead, andranges, the parameters andannotationsare empty.- source¶
The object to copy, with the version that HeadObject reported unless the path has one or the version is
null.
- destination¶
The object that the copy writes.
- ranges¶
The
(start, end)byte ranges of the source that the parts copy, with an exclusive end, in part-number order.
- create_params¶
The parameters of CreateMultipartUpload, with the metadata and the tags that CopyObject would write.
- Type:
collections.abc.Mapping[str, Any]
- part_params¶
The parameters of each UploadPartCopy.
- Type:
collections.abc.Mapping[str, Any]
- complete_params¶
The parameters of CompleteMultipartUpload.
- Type:
collections.abc.Mapping[str, Any]
- abort_params¶
The parameters of AbortMultipartUpload.
- Type:
collections.abc.Mapping[str, Any]
- annotations¶
The names of the annotations to copy; empty if the directive excludes them or the source cannot have any.
- __init__(source: ~pyathena.filesystem.s3_path.S3Path, destination: ~pyathena.filesystem.s3_path.S3Path, size: int, ranges: tuple[tuple[int, int], ...] = (), create_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, part_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, complete_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, abort_params: ~collections.abc.Mapping[str, ~typing.Any] = <factory>, annotations: tuple[str, ...] = (), fits_single_request: bool = False) None¶
S3 Objects¶
- class pyathena.filesystem.s3_object.S3Object(init: dict[str, Any], **kwargs)[source]¶
Represents an S3 object with metadata and filesystem-like properties.
This class provides a dictionary-like interface to S3 object metadata, making it easier to work with S3 objects in filesystem operations. It handles the mapping between S3 API field names and more pythonic property names.
The object supports both dictionary-style access and property-style access to metadata fields like content type, storage class, encryption settings, and object lock configurations. Dictionary-style access behaves like a dictionary, so a missing key raises KeyError. Property-style access returns None for a known field that the object does not have, and raises AttributeError for any other missing name.
Example
>>> s3_obj = S3Object({"ContentType": "text/csv", "ContentLength": 1024}) >>> print(s3_obj.content_type) # "text/csv" >>> print(s3_obj["content_length"]) # 1024 >>> s3_obj.storage_class = "STANDARD_IA"
Note
This class is primarily used internally by S3FileSystem for representing S3 objects in filesystem operations.
- __init__(init: dict[str, Any], **kwargs) None[source]¶
Initialize the object from an S3 API response.
Only the fields of
initthat have a property mapping are kept, under their property names (e.g.,ContentType->content_type).storage_classdefaults toSTANDARDwheninithas noStorageClass, andsizeis taken fromSizeorContentLength, or is 0, also for an emptyinit.nameis set tobucket/key, or to the bucket when there is no key.- Parameters:
init – An S3 API response or listing entry, such as a HeadObject response or a ListObjectsV2
Contentsentry.**kwargs – Additional fields stored as-is, such as
type,bucket,keyandversion_id. S3 API field names are stored under their property names.
- __getattr__(item: str) Any[source]¶
Return None for a known field that the object does not have.
Called only when normal attribute lookup fails, so fields that the object has are returned without reaching this method.
- Parameters:
item – The attribute name.
- Returns:
None, if
itemis a known S3 object field.- Raises:
AttributeError – If
itemis not a known S3 object field.
- copy() S3Object[source]¶
Return a shallow copy of the object.
- Returns:
A new S3Object with the same fields.
- to_dict() dict[str, Any][source]¶
Convert S3Object to dictionary representation.
- Returns:
Deep copy of the object’s attributes as a dictionary.
- to_api_repr() dict[str, Any][source]¶
Convert the object metadata to S3 API request parameters.
- Returns:
Dictionary keyed by S3 API field names (e.g.,
ContentType) of the fields that are set, excludingETag,ContentLengthandLastModified.
- classmethod from_summary(summary: S3ObjectSummary) S3Object[source]¶
Build the file entry of an object listed by ListObjectsV2.
- Parameters:
summary – The listed object.
- Returns:
The file entry, named
bucket/key.
- classmethod from_version(version: S3ObjectVersion) S3Object[source]¶
Build the file entry of a version listed by ListObjectVersions.
The entry is named
bucket/key?versionId=<id>so that the version can be addressed, except for thenullversion, which a write to the key replaces and which is namedbucket/key.- Parameters:
version – The listed version.
- Returns:
The file entry.
- classmethod from_metadata(metadata: S3Metadata, version_id: str | None = None) S3Object[source]¶
Build the file entry of an object looked up with HeadObject.
The fields that the response does not have are left out, except
Metadata, which botocore always returns.- Parameters:
metadata – The metadata of the object, with the looked up path.
version_id – The version ID of the entry, which may be pinned or omitted apart from the looked up path.
- Returns:
The file entry, named
bucket/key.- Raises:
ValueError – If the metadata has no object path.
- class pyathena.filesystem.s3_object.S3ObjectType[source]¶
Constants for S3 object types in filesystem operations.
These constants are used to distinguish between directories and files when working with S3 paths through the S3FileSystem interface.
- class pyathena.filesystem.s3_object.S3StorageClass[source]¶
Constants for Amazon S3 storage classes.
S3 storage classes determine the availability, durability, and cost characteristics of stored objects. Each class is optimized for different access patterns and use cases.
- Storage classes:
STANDARD: Default storage for frequently accessed data
REDUCED_REDUNDANCY: Lower cost, reduced durability (deprecated)
STANDARD_IA: Infrequently accessed data with rapid retrieval
ONEZONE_IA: Lower cost IA storage in single availability zone
INTELLIGENT_TIERING: Automatic tiering between frequent/infrequent
GLACIER: Archive storage for long-term backup
DEEP_ARCHIVE: Lowest cost archive storage
GLACIER_IR: Archive with faster retrieval than standard Glacier
OUTPOSTS: Storage on AWS Outposts
BUCKET: Pseudo storage class PyAthena assigns to bucket entries
DIRECTORY: Pseudo storage class PyAthena assigns to directory entries
See also
AWS S3 storage classes documentation: https://docs.aws.amazon.com/s3/latest/userguide/storage-class-intro.html
- class pyathena.filesystem.s3_object.S3Metadata(response: dict[str, Any], path: S3Path | None = None)[source]¶
Represents the metadata of an S3 object as returned by HeadObject.
Behaves as a read-only mapping of the user-defined metadata (
x-amz-meta-*, whose keys are arbitrary user-chosen strings), so it is a drop-in for implementations that return the user-defined metadata as a plain dictionary, and compares equal to such dictionaries. The system-defined metadata (content type, encryption settings, storage class, etc.) is exposed as typed properties.Example
>>> metadata = fs.metadata("s3://bucket/key") >>> metadata["attr1"] # user-defined metadata 'value1' >>> metadata.content_type # system-defined metadata 'text/plain'
See https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
- __init__(response: dict[str, Any], path: S3Path | None = None) None[source]¶
Initialize the metadata from a HeadObject response.
- Parameters:
response – The HeadObject response.
path – The path that was looked up, if known.
- property bucket_key_enabled: bool | None¶
Whether the object uses an S3 Bucket Key (
BucketKeyEnabled).
- class pyathena.filesystem.s3_object.S3ObjectVersion(bucket: str, is_delete_marker: bool, response: dict[str, Any])[source]¶
Represents a version of an S3 object as returned by ListObjectVersions.
- bucket¶
S3 bucket name.
- key¶
Object key.
- version_id¶
The version ID of the object.
- is_latest¶
Whether the version is the latest version of the object.
- is_delete_marker¶
Whether the version is a delete marker.
- last_modified¶
Date and time when the version was last modified.
- etag¶
Entity tag of the version. None for delete markers.
- size¶
Size in bytes of the version. None for delete markers.
- storage_class¶
Storage class of the version. None for delete markers.
- owner¶
Owner of the version.
- __init__(bucket: str, is_delete_marker: bool, response: dict[str, Any]) None[source]¶
Initialize the version from a ListObjectVersions entry.
- Parameters:
bucket – The name of the bucket that contains the version.
is_delete_marker – Whether the entry comes from
DeleteMarkersrather thanVersions.response – A
VersionsorDeleteMarkersentry of the ListObjectVersions response.
S3 Upload Operations¶
- class pyathena.filesystem.s3_object.S3PutObject(response: dict[str, Any])[source]¶
Represents the response from an S3 PUT object operation.
This class encapsulates the metadata returned when uploading an object to S3, including encryption details, versioning information, and integrity checksums.
- expiration¶
Object expiration time if lifecycle policy applies.
- version_id¶
Version ID if bucket versioning is enabled.
- etag¶
Entity tag for the uploaded object.
- server_side_encryption¶
Server-side encryption method used.
- Various checksum properties
For data integrity verification.
Note
This class is used internally by S3FileSystem operations and typically not instantiated directly by users.
- __init__(response: dict[str, Any]) None[source]¶
Initialize the result from a PutObject response.
- Parameters:
response – The PutObject response.
- property server_side_encryption: str | None¶
The
ServerSideEncryptionalgorithm of the uploaded object.
- property sse_customer_key_md5: str | None¶
The
SSECustomerKeyMD5of the customer-provided key for the uploaded object.
- property sse_kms_encryption_context: str | None¶
The
SSEKMSEncryptionContextof the uploaded object.
- class pyathena.filesystem.s3_object.S3MultipartUpload(response: dict[str, Any])[source]¶
Represents an S3 multipart upload operation.
This class manages the metadata for multipart uploads, which allow uploading large files in chunks for better reliability and performance. It tracks upload identifiers, encryption settings, and lifecycle rules.
- bucket¶
S3 bucket name for the upload.
- key¶
Object key being uploaded.
- upload_id¶
Unique identifier for the multipart upload.
- server_side_encryption¶
Encryption method applied to the upload.
- abort_date/abort_rule_id
Lifecycle rule information for upload cleanup.
- initiated/storage_class/owner/initiator
Fields returned by the ListMultipartUploads API for in-progress uploads.
Note
Used internally by S3FileSystem for large file upload operations, and returned by
S3FileSystem.list_multipart_uploads.- __init__(response: dict[str, Any]) None[source]¶
Initialize the upload from an S3 API response.
- Parameters:
response – A CreateMultipartUpload response or an
Uploadsentry of a ListMultipartUploads response.
- property abort_rule_id: str | None¶
The
AbortRuleIdof the lifecycle rule that applies to the upload.
- property sse_customer_key_md5: str | None¶
The
SSECustomerKeyMD5of the customer-provided key for the upload.
- property bucket_key_enabled: bool | None¶
Whether the upload uses an S3 Bucket Key (
BucketKeyEnabled).
- property initiated: datetime | None¶
The
Initiatedtime of the upload, returned by ListMultipartUploads.
- class pyathena.filesystem.s3_object.S3MultipartUploadPart(part_number: int, response: dict[str, Any])[source]¶
Represents a single part in an S3 multipart upload operation.
Each part in a multipart upload has its own metadata including checksums, encryption details, and part identification. This class manages that metadata and provides methods to convert it to API-compatible formats.
- part_number¶
The sequential part number (1-based).
- etag¶
Entity tag for this specific part.
- checksum_*
Various integrity checksums for the part data.
- server_side_encryption¶
Encryption settings for this part.
Note
Parts must be at least 5MB except for the last part. Used internally by S3FileSystem for chunked upload operations.
- __init__(part_number: int, response: dict[str, Any]) None[source]¶
Initialize the part from an UploadPart or UploadPartCopy response.
For an UploadPartCopy response, the
ETag,LastModifiedand checksums are read from itsCopyPartResult.- Parameters:
part_number – The part number of the part.
response – The UploadPart or UploadPartCopy response.
- property copy_source_version_id: str | None¶
The
CopySourceVersionIdof the source object of a copied part.
- property last_modified: datetime | None¶
The
LastModifiedtime fromCopyPartResult; None for uploaded parts.
- property sse_customer_key_md5: str | None¶
The
SSECustomerKeyMD5of the customer-provided key for the part.
- class pyathena.filesystem.s3_object.S3CompleteMultipartUpload(response: dict[str, Any])[source]¶
Represents the completion of an S3 multipart upload operation.
This class encapsulates the final response when a multipart upload is completed, including the final object location, versioning information, and consolidated metadata from all parts.
- location¶
Final S3 URL of the completed object.
- bucket¶
S3 bucket containing the object.
- key¶
Final object key.
- version_id¶
Version ID if bucket versioning is enabled.
- etag¶
Final entity tag of the complete object.
- server_side_encryption¶
Encryption applied to the final object.
Note
This represents the successful completion of a multipart upload. Used internally by S3FileSystem operations.
- __init__(response: dict[str, Any]) None[source]¶
Initialize the result from a CompleteMultipartUpload response.
- Parameters:
response – The CompleteMultipartUpload response.
- property server_side_encryption: str | None¶
The
ServerSideEncryptionalgorithm of the completed object.
Multipart Writer¶
- class pyathena.filesystem.s3_writer.S3MultipartWriter(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None)[source]¶
Plan parts and send the requests of a multipart write.
The caller owns the buffer, schedules requests, collects part results in number order, and waits for running requests before completing or aborting. Initiation and finalization must be serialized; part requests may run in parallel after initiation. No request is sent by the constructor or planners.
- upload¶
The identity returned by creation, retained after completion and after an abort failure. A successful abort clears it.
- __init__(core: S3Core, path: S3Path, *, block_size: int, request_kwargs: Mapping[str, Any] | None = None) None[source]¶
Initialize a writer without starting its upload.
- Parameters:
core – The synchronous S3 operations to use.
path – The destination, with a key and without a version ID.
block_size – The nominal part size, between the core’s minimum and maximum multipart part sizes, inclusive.
request_kwargs – Parameters inherited by each request that accepts them. A copy is kept; parameters given to a request override it.
- Raises:
ValueError – If the destination or block size is invalid.
- iter_parts(stream: BinaryIO, first_part_number: int = 1) Iterator[tuple[int, bytes]][source]¶
Plan numbered parts from the stream’s current position to EOF.
A short trailing block is merged into the preceding block, splitting the result in half if it reaches the maximum part size. This keeps every part large enough for later writes, unless the entire stream is shorter than the minimum part size and must be the final part. Reads are bounded by the block size; the stream is not rewound.
- Parameters:
stream – A blocking binary stream of buffered data.
first_part_number – The first number, including any earlier copied or uploaded parts in the caller’s count.
- Yields:
The part number and bytes, in order, without sending requests.
- Raises:
ValueError – If a generated part number is outside the S3 limit.
- iter_copy_parts(size: int, first_part_number: int = 1) Iterator[tuple[int, tuple[int, int]]][source]¶
Plan numbered copy ranges, with an exclusive end, without requests.
- Parameters:
size – The nonnegative size of the existing object in bytes.
first_part_number – The first part number to use.
- Yields:
The part number and source range, in order. The ranges use the core’s maximum part size and keep a short tail within S3 limits so that appended parts may follow, unless the whole source is smaller than the minimum part size and must be the final part.
- Raises:
ValueError – If the size is negative or a part number exceeds the limit.
- initiate(**params: Any) S3MultipartUpload[source]¶
Create the upload, or return its already known identity.
- Parameters:
**params – Parameters accepted by CreateMultipartUpload.
- Returns:
The upload, stored before this method returns so that a caller can recover it after an interrupted wait for a scheduled request.
- upload_part(part_number: int, body: bytes, **params: Any) S3MultipartUploadPart[source]¶
Send an UploadPart request for the initialized upload.
- Parameters:
part_number – The part number, between 1 and the core’s maximum.
body – The bytes to upload, with the size selected by the caller.
**params – Parameters accepted by UploadPart.
- Returns:
The uploaded part, retaining its number and checksums.
- Raises:
RuntimeError – If no upload is initialized.
ValueError – If the part number is outside the S3 limit.
- upload_part_copy(part_number: int, source: S3Path, range_: tuple[int, int] | None = None, **params: Any) S3MultipartUploadPart[source]¶
Send an UploadPartCopy request for the initialized upload.
- Parameters:
part_number – The part number, between 1 and the core’s maximum.
source – The source object, with an optional version ID.
range – The source range with an exclusive end, or None to send no range of its own, as in
S3Core.upload_part_copy().**params – Parameters accepted by UploadPartCopy.
- Returns:
The copied part, retaining its number and checksums.
- Raises:
RuntimeError – If no upload is initialized.
ValueError – If the part number is outside the S3 limit.
- complete(parts: Sequence[S3MultipartUploadPart], **params: Any) S3CompleteMultipartUpload[source]¶
Complete the upload after the caller has collected all its parts.
- Parameters:
parts – Successfully uploaded parts in part-number order.
**params – Parameters accepted by CompleteMultipartUpload.
- Returns:
The completion result. The upload identity is retained.
- Raises:
RuntimeError – If no upload is initialized. Request failures propagate; the caller must settle running work before aborting.
- abort(**params: Any) None[source]¶
Abort a known upload after the caller has settled running requests.
Does nothing if no upload is known. A successful abort clears the identity; a failed or interrupted abort retains it for a later retry.
- Parameters:
**params – Parameters accepted by AbortMultipartUpload.
S3 Errors¶
- class pyathena.filesystem.s3_errors.S3ClientError(error: ClientError)[source]¶
Represents an S3 error response with its translation to an OSError.
Wraps a botocore
ClientErrorand exposes the error response fields as properties, along withos_error, the equivalent standard Python exception. The error is mapped by its S3 error code first, then by its HTTP status code; if neither is recognized, a genericOSErrorwith the original error message is used. APreconditionFailederror of anIf-None-Matchcondition, from a write that must not replace an existing object, is mapped toFileExistsError. A405error with thex-amz-delete-marker: trueheader, from a request for a version that is a delete marker, is mapped toFileNotFoundError.Example
>>> try: ... client.head_object(Bucket="bucket", Key="key") ... except botocore.exceptions.ClientError as e: ... raise S3ClientError(e).os_error from e
- __init__(error: ClientError) None[source]¶
Initialize the error from a botocore
ClientError.- Parameters:
error – The
ClientErrorraised by the S3 client.
- classmethod code_of(error: BaseException) str | None[source]¶
Return the S3 error code of an error raised from a
ClientError.- Parameters:
error – An error, such as the
OSErrorof a request.- Returns:
The code of the
ClientErrorthat the error was raised from, or None if it was not raised from one.
- property os_error: OSError¶
The standard Python exception equivalent to this error response.
Resolved once at construction time. The exception is instantiated and ready to be raised; raise it with
raise ... from errorto preserve the original exception.Note
OSErrorspecializes itself into a subclass for some errno values (e.g.,EPERM->PermissionError), so the returned exception may be a subclass ofOSErroreven for the errno-based mappings.