S3 filesystem¶
PyAthena ships its own fsspec-compatible
filesystem implementation for Amazon S3 (S3FileSystem), built on boto3, with an API
surface compatible with s3fs for users migrating from it.
The filesystem is used internally by the pandas, Polars, and S3FS result sets to read query results from S3, and can also be used independently for S3 file operations.
fsspec registration¶
Importing pyathena.pandas or pyathena.polars registers S3FileSystem as the fsspec
s3 / s3a protocols via pyathena.filesystem.register_s3_filesystem. This replaces
fsspec’s default lazy mapping of the s3 protocol to s3fs, which means
fsspec.filesystem("s3") returns PyAthena’s implementation and s3fs-specific settings
(such as the S3FS_LOGGING_LEVEL environment variable) have no effect.
The registered class is the synchronous S3FileSystem, whose async_impl is False.
Like fsspec’s other synchronous filesystems, it accepts and ignores asynchronous=True,
so fsspec.filesystem("s3", asynchronous=True) returns a synchronous filesystem; libraries
such as zarr check async_impl and wrap it. For a native asynchronous filesystem, use
AioS3FileSystem.
A filesystem class that has already been registered explicitly is also overwritten, with a warning log, so that the replacement is diagnosable. To restore another implementation, re-register it afterwards:
import fsspec
import s3fs
import pyathena.pandas # Registers PyAthena's S3FileSystem.
fsspec.register_implementation("s3", s3fs.S3FileSystem, clobber=True)
Basic usage¶
The filesystem can be created from a PyAthena connection with s3_filesystem(), which
gives it the S3 client and the retry policy of the connection (see “S3 client” in
Usage), with a boto3 S3 client, or with s3fs-compatible credential
arguments:
import boto3
from pyathena import connect
from pyathena.filesystem import s3_filesystem
from pyathena.filesystem.s3 import S3FileSystem
fs = s3_filesystem(connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
region_name="us-west-2"))
# Or with a boto3 S3 client, and optionally a pyathena.util.RetryConfig.
fs = S3FileSystem(client=boto3.client("s3", region_name="us-west-2"))
# Or with direct credentials (s3fs-compatible arguments).
fs = S3FileSystem(key="YOUR_ACCESS_KEY", secret="YOUR_SECRET_KEY")
# Or with a named profile.
fs = S3FileSystem(profile="YOUR_PROFILE")
# Or anonymously for public buckets.
fs = S3FileSystem(anon=True)
The filesystem uses a given client as it is and does not close it. Combining client
with an argument from which the filesystem builds a client, such as key, secret,
token, profile, anon, endpoint_url, client_kwargs, config_kwargs, or
region_name, raises TypeError; such settings from fsspec’s configuration, such as
the FSSPEC_S3_ENDPOINT_URL environment variable, are ignored. requester_pays, max_workers, the cache options,
and the arguments of fsspec can be combined with it. s3_filesystem() takes the other
arguments of the filesystem, and AioS3FileSystem as its second argument.
The connection argument of S3FileSystem and AioS3FileSystem is deprecated and
emits a DeprecationWarning; it will be removed in the next major version. It is
subject to the same restrictions as client.
The S3 client computes checksums of uploads and validates those of downloads by
botocore’s request_checksum_calculation and response_checksum_validation options,
set with s3_config of the connection, with config_kwargs, or in the configuration of
a given client; see “Checksums” under “S3 client” in Usage for their
settings and their effect on data integrity.
Standard fsspec operations work as expected:
fs.ls("s3://YOUR_S3_BUCKET/path/to/")
fs.find("s3://YOUR_S3_BUCKET/path/to/")
fs.exists("s3://YOUR_S3_BUCKET/path/to/object")
fs.info("s3://YOUR_S3_BUCKET/path/to/object")
with fs.open("s3://YOUR_S3_BUCKET/path/to/object", "rb") as f:
data = f.read()
fs.pipe("s3://YOUR_S3_BUCKET/path/to/object", b"data")
fs.cat("s3://YOUR_S3_BUCKET/path/to/object")
fs.cp("s3://YOUR_S3_BUCKET/src", "s3://YOUR_S3_BUCKET/dst")
fs.rm("s3://YOUR_S3_BUCKET/path/to/", recursive=True)
cat, cat_ranges, get, put, pipe with a dict of values, cp, and mv transfer
their objects in parallel, batch_size objects at a time:
fs.get("s3://YOUR_S3_BUCKET/path/to/", "local/dir/", recursive=True, batch_size=16)
batch_size is an argument of these methods, of rm, and of S3FileSystem. By default,
and with -1, it is the filesystem’s max_workers. The transfers of a batch run in
threads of their own. With several transfers at once, each object sends at most
max(1, max_workers // n) range reads, parts, or part copies at once, n being the
smaller of batch_size and the number of objects, so that a batch keeps about
max_workers requests in the request pool and as many parts in memory; a smaller
max_workers given to the method is kept. A single object keeps the filesystem’s
max_workers. cat, cat_ranges, and a small pipe send their single request from the
transfer’s thread, so a batch_size larger than max_workers sends that many requests
at once.
When a transfer fails, no further transfer starts, the retries of the running ones stop,
and the error is raised once they have ended. A running multipart upload or copy sends
no further part and is aborted instead of being completed, and a running download reads
and writes no further block. A request already sent, such as the PutObject of a small
object, runs to its end; a multipart copy that has been completed still copies its
annotations. With cache_type="background", a block that fsspec’s cache prefetches in
its own thread may still be read after the stop. The transfers that ended before the failure stay done, and their number is
not bounded by batch_size. mv deletes the sources only once every copy has
succeeded. on_error of cat, cat_ranges, and cp keeps its fsspec meaning; with
on_error="omit" or "return" of cat, "return" of cat_ranges, or "ignore" of
cp for a missing source, those errors do not stop the batch.
Pairs whose result would depend on the order of the transfers raise ValueError before
anything is transferred: two sources of get, put, cp, or pipe with the same
destination, including two paths of one existing local file or one file-like object,
and a cp destination that is another source. Local paths that do not exist yet are
compared as strings, after os.path.normcase, so on a case-insensitive filesystem other
than Windows, such as the default one of macOS, give get destinations that differ by
more than case. A cp of an object onto itself, such as with
MetadataDirective="REPLACE", is no conflict. get with outfile writes the objects
into it one after another, in order.
rm deletes up to 1,000 objects per DeleteObjects request, batch_size requests at a
time. Its max_workers argument is a deprecated alias of batch_size.
Writes with pipe/pipe_file issue a single PutObject request for data up to the
block size (5 MiB by default); larger data is uploaded as a parallel multipart upload
through the buffered file path. Inside an
fsspec transaction,
writes are deferred until the transaction commits and are discarded on rollback.
With the compression argument (a codec of fsspec, or "infer" from the extension of
the path), pipe/pipe_file compress the data before uploading it, and the sizes in
this section apply to the compressed data.
A multipart write of a file returned by open keeps at most max_workers parts in
flight, counting the parts of the written data and the parts copied from the existing
object in an append, and waits for one of them to finish before it submits another. The
parts take about max_workers × the block size of memory, in addition to the buffered
data of the file. If a part has failed before the write submits another part, the write
raises its error instead of uploading the rest of the data, and aborts the multipart
upload after the parts being uploaded or copied finish. A part that fails after the last
write raises its error at the next write, when the file is closed, or when the
transaction commits, and the multipart upload is aborted.
max_workers is an argument of S3FileSystem, AioS3FileSystem, open, put, and
pipe.
The requests that the files and operations of a filesystem run in parallel (range
reads, parts, part copies, the ACLs of a recursive chmod, the DeleteObjects batches of
rm, and the aborts of clear_multipart_uploads) share one request pool of the
filesystem, with max_workers threads. Each file or operation runs at most its own
max_workers requests at once in that pool. The max_workers of the filesystem
defaults to the max_pool_connections of its S3 client: 10 unless configured, or that
of the given client, such as the client of the connection. A client that the
filesystem builds takes its max_pool_connections from max_workers unless
config_kwargs sets it; max_workers does not change the connection pool of a given
client. For more parallel requests, raise max_workers, and with a given client, also
its max_pool_connections, such as with the s3_config of the connection. close(), or the end of a with block, releases the threads of the pool once they
are idle, and the filesystem stays usable.
Pickling and processes¶
A filesystem created with s3fs-compatible arguments can be pickled, as for Dask, Ray, or
multiprocessing workers: the unpickled filesystem builds its own client from the same
arguments. A filesystem created with client, connection, or s3_filesystem(), and
the files that it opens for reading, raise TypeError when they are pickled, as a boto3
client cannot be pickled; as in fsspec, a file opened for writing cannot be pickled. To use a filesystem in such workers, create it with s3fs-compatible
arguments, such as profile or region_name, or create a filesystem in each worker.
In a forked child process, create a new filesystem. A file that the parent opened raises
RuntimeError there from its reads, writes, and flushes, commit(), and discard(),
and closing it only marks it closed, without uploading or aborting anything.
An interrupt, such as Ctrl+C, or a failure while a file reads ranges in parallel
cancels the range reads that have not started, and the running ones make no further
attempt. The interrupt or the error is raised once they have ended, and their data is
released. A second interrupt stops that wait, and the reads still finish. rm and
clear_multipart_uploads send every request also after one fails; an interrupt cancels
those that have not started and is raised once the running ones have ended. An interrupt of
a batch operation, such as get or cp, does the same as a failed transfer. Closing
the file waits for its running requests; an interrupt of that wait stops it, and the
requests still finish. An interrupt while a
request is being submitted can let one more request than max_workers run at once
until the request pool is idle. These guarantees hold for one interrupt at a time. If another interrupt arrives while the cleanup of an interrupt or
an error runs, the call still does not wait forever, but the cleanup may stop early:
the data of finished requests can stay referenced until the garbage collector frees it,
and a multipart upload can be left incomplete. clear_multipart_uploads removes such
uploads.
A file returned by open whose write or commit has failed, or which has been
discarded, cannot be committed: its commit() raises S3MultipartStateError. Inside a
transaction, a write error that the caller catches therefore makes the transaction fail
when it commits, and fsspec discards the files of the transaction that are not
committed yet.
The block size for writing, given by the block_size argument of open or by the
filesystem’s default_block_size, must be between 5 MiB and 5 GiB, inclusive, the part
size limits of a multipart upload. Otherwise, open raises ValueError.
A multipart upload consists of at most 10,000 parts. put and pipe upload one part
per block, so with the default block size they can upload up to about 48.8 GiB
(10,000 × 5 MiB). To upload a larger object, use a block size of at least its size
divided by 10,000, either with the block_size argument of put, pipe, and open
or with the default_block_size argument of S3FileSystem. put and pipe check
the size before uploading anything and raise ValueError with the minimum block size
if the data needs more parts. A file written with open can take more parts, because
each write that fills the buffer uploads the data beyond its last full block as a
separate part when that data is at least 5 MiB. In an append, the parts copied from
the existing object also count toward the limit. A write with open that reaches the
limit raises ValueError and aborts its multipart upload. Multipart copies with cp
use parts large enough to stay within the limit.
cp copies an object larger than 5 GiB with a multipart upload instead of a single
CopyObject request, with the same result as CopyObject. If HeadObject reports a size
of at most 5 GiB when the copy starts, for example because the size that cp found
came from a cached listing, the object is copied with CopyObject instead. CopyObject parameters given as
keyword arguments are sent to the multipart requests that accept them, such as
CopySourceIfMatch to each part copy. With the default COPY value of
MetadataDirective, TaggingDirective, and AnnotationDirective, the content headers
(such as ContentType) and user-defined metadata, the tags, and the annotations of the
source are copied, and the values given for them are ignored, as CopyObject does. A
REPLACE directive uses the given values instead, and AnnotationDirective="EXCLUDE"
skips the annotations. Copying the tags needs s3:GetObjectTagging on the source, and
copying the annotations needs s3:ListObjectAnnotations and s3:GetObjectAnnotation on
the source and s3:PutObjectAnnotation on the destination. A source without a
?versionId= suffix whose HeadObject reports a version ID other than null is copied
from that version, which needs s3:GetObjectVersion and, to copy the tags,
s3:GetObjectVersionTagging on the source. A null version is not pinned. The annotations are listed before anything is written and copied after the
upload completes, so the destination exists without them until the last one is
written. If an annotation fails to copy, the error is raised and the destination is
kept. A failed part copy aborts the multipart upload. So does an interrupt, or the
cancellation of an AioS3FileSystem copy, including one by the shutdown of the event
loop at the end of asyncio.run(), unless the upload has already completed. A
CreateMultipartUpload request and the part copies in flight finish first, without a
further attempt. The CompleteMultipartUpload request of an AioS3FileSystem copy
also finishes first, with its retries; the requests of an AioS3FileSystem copy that
have not started are cancelled. A further interrupt
while the CreateMultipartUpload request is waited for is ignored. One while the part
copies are waited for or the upload is aborted stops the cleanup, and the upload ID is
logged.
Every method of the filesystems also accepts paths as os.PathLike objects, such as
pathlib.PurePosixPath("YOUR_S3_BUCKET/dir/key"), and uses them as their strings.
pathlib removes a trailing slash and collapses //, so a path with the s3://
protocol or a trailing slash needs to be a string. On Windows, pathlib.Path joins
its parts with backslashes, so S3 paths need PurePosixPath.
Paths are normalized as in fsspec, which drops a trailing slash, so info, isfile,
and open treat s3://YOUR_S3_BUCKET/dir/ as s3://YOUR_S3_BUCKET/dir: the object
dir if it exists, and otherwise the directory dir. An object whose key ends in a
slash, such as a folder marker, is therefore not a file for these methods. Opening
dir/ for reading reads the object dir or raises FileNotFoundError. Opening it for
writing, pipe, pipe_file, and put_file write the object dir. A path with a
?versionId= suffix keeps the slash and refers to the object. find, and ls of the directory, list the object as a file
entry. cat_file uses the key as written. Without a ?versionId= suffix, it reads
such an object without a range, with a non-empty range of non-negative offsets, or with
a negative start and no end, and raises FileNotFoundError for other ranges.
A directory is named after its key prefix without the last slash. A key with an empty
segment, such as dir//x, therefore lies in the directory dir/, which ls of dir
lists and find descends into. Since ls drops a trailing slash, ls of dir/ lists
dir, and walk does not descend into dir/.
S3 request parameters, such as ContentType, ServerSideEncryption, or RequestPayer,
can be given to open, pipe, and put as keyword arguments or in
s3_additional_kwargs, and to all of them through the s3_additional_kwargs argument
of S3FileSystem. The parameters of a call take precedence over those of the
filesystem. A file sends each of its requests only the parameters that the S3 operation
accepts, so, for example, ServerSideEncryption for writes is not sent with reads. A
pipe of data up to the block size sends its parameters with a single PutObject
request as given. put sets ContentType from the file extension unless the call or
the filesystem gives one.
The parameters in the s3_additional_kwargs argument of S3FileSystem that authorize
the access to the objects, RequestPayer, ExpectedBucketOwner, and the SSECustomer*
parameters, are sent with every request of the filesystem whose operation accepts them,
listings and copies included. A copy also sends them for its source, as
ExpectedSourceBucketOwner and the CopySourceSSECustomer* parameters. The parameters
of a call take precedence. With an SSECustomerKey there, every object that the
filesystem reads or copies must be encrypted with that key, and every object that it
writes is: a call that asks for another encryption with ServerSideEncryption sends both,
which S3 rejects. sign signs only the parameters given to it.
fs = S3FileSystem(s3_additional_kwargs={"RequestPayer": "requester", **sse_c})
data = fs.cat_file("s3://YOUR_S3_BUCKET/path/to/encrypted.csv")
fs = S3FileSystem(s3_additional_kwargs={"ServerSideEncryption": "AES256"})
with fs.open("s3://YOUR_S3_BUCKET/path/to/data.csv", "wb", ContentType="text/csv") as f:
f.write(b"col1\n1\n")
info and exists accept ExpectedBucketOwner, RequestPayer, and the
SSECustomer* parameters of an object encrypted with a customer-provided key (SSE-C)
as keyword arguments, and ignore other request parameters. The lookups of the object
by open, and the existence check of pipe with mode="create", send these
parameters of the file or the write. A lookup with them uses only the cached results of
lookups with the same values, not cached listings.
sse_c = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": YOUR_32_BYTE_KEY}
with fs.open("s3://YOUR_S3_BUCKET/path/to/encrypted.csv", "rb", **sse_c) as f:
data = f.read()
The lookups of the sources of cp_file, copy, and mv, and of the object of
setxattr, send the source’s parameters of the copy: RequestPayer,
ExpectedSourceBucketOwner as ExpectedBucketOwner, and the CopySourceSSECustomer*
parameters as SSECustomer*. mv deletes the copied sources with RequestPayer and
that expected bucket owner. The requests that copy and mv send to expand and pair
their paths, such as the listings and the directory checks of the sources and of the
destination, do not send these parameters of the call, but those of the filesystem
described above. copy and mv with lists of both sources and destinations pair the
paths as given, without these requests.
fs.cp_file(
"s3://YOUR_S3_BUCKET/path/to/encrypted.csv",
"s3://YOUR_S3_BUCKET/path/to/copy.csv",
CopySourceSSECustomerAlgorithm="AES256",
CopySourceSSECustomerKey=YOUR_32_BYTE_KEY,
**sse_c,
)
A file opened for reading sends its reads on the condition of the ETag looked up when it
was opened (IfMatch), so reading an object replaced since raises OSError; an
IfMatch given to open is sent instead. A file that fsspec reopens, such as an
unpickled file, keeps the request parameters of the file, including the
SSECustomerKey of an SSE-C object, and this condition. metadata(), getxattr(),
url(), and setxattr() of a file send its lookup parameters; setxattr() sends them
also under the names of the copy source. In an append, the existing object is read when
the file is opened, or copied as parts when its upload starts, on the condition of the
ETag looked up when the file was opened. If the object was replaced before it is read or
copied, the append raises OSError, and its multipart upload, if any, is aborted. The
write of the appended object is not conditional, so it replaces an object written after
the existing object was read or copied. The appended object keeps the attributes of the
existing object that neither the call nor the s3_additional_kwargs of the filesystem
give: its system-defined metadata, such as ContentType, its user-defined metadata, its
storage class, and its encryption, unless they give any encryption parameter, such as
ServerSideEncryption. It does not keep the object lock of the existing object. A key that is only a prefix of other keys has no object: reading it raises
FileNotFoundError, and appending, exclusive creation ("xb"), and pipe/put with
mode="create" write a new object.
Read caching¶
A file opened for reading fetches its data through an fsspec cache, chosen by the
cache_type argument of open or by the default_cache_type argument of
S3FileSystem, and "bytes" by default. The value is the name of a cache registered in
fsspec.caching.caches, so the available names depend on the installed fsspec version.
The common ones are:
"none": keeps nothing; each read requests the bytes it reads."bytes": reads one block beyond each read and keeps one contiguous range, which later reads extend."readahead": reads one block beyond each read and keeps only the last range read, which suits many small sequential reads."blockcache": requests whole blocks and keeps the most recently used ones, up tomaxblocks(32 by default)."background": like"blockcache", and also reads the next block in a thread while the current one is consumed.
The block size is the block_size argument of open or the filesystem’s
default_block_size (5 MiB by default). Options of the cache, such as maxblocks, are
given in the cache_options argument of open. See
Read Buffering
in the fsspec documentation for the other cache types.
fs = S3FileSystem(default_cache_type="readahead")
with fs.open(
"s3://YOUR_S3_BUCKET/path/to/data.csv",
"rb",
block_size=32 * 2**20,
cache_type="background",
cache_options={"maxblocks": 2},
) as f:
data = f.read()
The cache_type option of PandasCursor and PolarsCursor sets the default_cache_type
of the filesystem that reads their results; see S3 read options and
S3 read options. For the cache that S3FSCursor uses, see
S3 read options.
Error translation¶
S3 error responses are translated into standard Python exceptions, so filesystem
operations raise natural errors instead of botocore’s ClientError:
S3 error |
Python exception |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Others |
|
S3ClientError.code_of() returns the S3 error code of a translated error, or None for
an error that is not translated from an S3 error response:
from pyathena.filesystem.s3_errors import S3ClientError
try:
fs.pipe_file("s3://YOUR_S3_BUCKET/path/to/object", b"data")
except OSError as e:
if S3ClientError.code_of(e) == "SlowDown":
...
Request parameters that botocore rejects before sending the request
(ParamValidationError), such as a misspelled parameter name, raise ValueError.
exists() returns True for a bucket that HeadBucket answers with 403. For an object,
a 403 raises PermissionError: without the s3:ListBucket permission on the bucket,
S3 also answers 403 for a key that does not exist.
Multipart upload management¶
Incomplete multipart uploads continue to accrue storage costs until they are completed or aborted. The filesystem can discover and abort them:
uploads = fs.list_multipart_uploads("s3://YOUR_S3_BUCKET")
for upload in uploads:
print(upload.key, upload.upload_id, upload.initiated)
# Abort all incomplete uploads to a key and the keys under it.
fs.clear_multipart_uploads("s3://YOUR_S3_BUCKET/path/to/")
Versioning¶
With version_aware=True, reads pin the object version observed at open time, so a
file handle keeps returning consistent data even if the object is overwritten while
reading. The file path carries the pinned version as a ?versionId= suffix, which
metadata(), getxattr() and url() of the file also use. Explicit versions can
always be read with the ?versionId= suffix or the version_id argument. Only a
?versionId= (or ?versionID=, ?versionid=, ?version_id=) query at the end of a
path is a version; any other ? is part of the key. A path with a version is not a
glob pattern: copy(), mv() and get() copy that version to a destination named
after its key. ls() and find() of a key path with a version return only that
version, named as ls(versions=True) names it, without listing the keys under the
path. They look the version up as info() does: if HeadObject reports it missing,
ls() raises FileNotFoundError and find() returns nothing, and other errors, such
as the 400 Bad Request of a version ID that the bucket rejects, are raised.
fs = S3FileSystem(
connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2"),
version_aware=True,
)
with fs.open("s3://YOUR_S3_BUCKET/path/to/object", "rb") as f:
data = f.read() # Pinned to the version observed at open time.
# List all versions of the objects under a prefix. Each version is named
# "bucket/key?versionId=<id>", except the "null" version, which is named by its key.
fs.ls("s3://YOUR_S3_BUCKET/path/to/", versions=True)
# Typed version information, including delete markers if requested.
versions = fs.object_version_info("s3://YOUR_S3_BUCKET/path/to/object")
for version in versions:
print(version.version_id, version.is_latest, version.last_modified)
Version-aware operations require the s3:GetObjectVersion and
s3:ListBucketVersions permissions.
Moving a ?versionId=null source onto its own key copies that version and
deletes the source version when bucket versioning is enabled. If versioning has
never been enabled or is suspended, the move leaves the source in place: copying
onto the key would replace the null version that the subsequent deletion removes.
The same distinction applies to move conflict checks.
A move compares a null version with its unversioned key using GetBucketVersioning,
which requires s3:GetBucketVersioning. Each relevant bucket is looked up once
during planning; other moves do not make this request. The result is not cached
between moves. A failed lookup stops the move before any copy or deletion.
Directory buckets do not support versioning, so their null versions are treated
as the key itself without a versioning lookup.
The opt-in S3 versioning integration tests create dedicated temporary buckets and check unversioned, enabled, and suspended states. See Testing for the command and required test permissions.
Bucket lifecycle¶
Bucket creation and deletion are infrastructure-level changes and are disabled by
default: mkdir/makedirs and rmdir raise PermissionError when they would
create or delete a bucket. Pass the opt-in flags to enable them:
fs = S3FileSystem(
connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2"),
allow_bucket_creation=True,
allow_bucket_deletion=True,
)
fs.mkdir("s3://YOUR_NEW_BUCKET")
fs.rmdir("s3://YOUR_NEW_BUCKET") # The bucket must be empty.
Creating a key prefix under an existing bucket requires no operation (S3 has no real directories below the bucket level) and is always a no-op.
Typed S3 operations¶
Parse an S3 path¶
Use S3Path.parse() to obtain the bucket, key and version ID of a path:
from pyathena.filesystem.s3_path import S3Path
path = S3Path.parse("s3://bucket/key?versionId=v1")
print(path.bucket, path.key, path.version_id)
In 4.0, S3FileSystem.PATTERN_PATH and the parse_path() methods of
S3FileSystem and AioS3FileSystem are removed.
S3Path.parse() returns an S3Path object; access its attributes instead of
unpacking the tuple returned by parse_path().
For direct access to the regular expression, use S3Path.PATTERN instead of
S3FileSystem.PATTERN_PATH.
Send S3 requests¶
S3FileSystem.core is an S3Core, the typed operations that the filesystem sends
its listing, lookup, read, write, delete, multipart upload, copy, tagging, ACL and
metadata replacement requests with. It can also be built on a boto3 S3 client. Each
operation sends one request (one per page for the iterators and
list_object_annotations()); plan_multipart_copy() and copy_object_annotation(),
described below, send several, and generate_presigned_url() signs a URL locally
without a request. The requests are sent with the retry policy. An operation raises
FileNotFoundError for a missing bucket or multipart upload, or for a missing object or
version that it reads, and caches nothing. copy_object() and upload_part_copy() with
a source version that is a delete marker raise OSError with EINVAL, while the
filesystem’s cp_file() raises FileNotFoundError for it, as for a missing file.
Requests sent through fs.core do not invalidate the filesystem’s cache: call
fs.invalidate_cache() after a change, or make it through the filesystem.
import boto3
from pyathena.filesystem.s3_core import S3Core
from pyathena.filesystem.s3_path import S3Path
core = fs.core # or S3Core(boto3.client("s3"))
metadata = core.head_object(S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object"))
print(metadata.content_length, metadata.version_id)
for page in core.list_objects("YOUR_S3_BUCKET", prefix="path/to/", delimiter="/"):
print([o.key for o in page.objects], [p.prefix for p in page.common_prefixes])
list_object_versions(), list_multipart_uploads() and list_buckets() return pages in
the same way. The core sends the prefix of list_multipart_uploads() as given, and S3
matches it as a plain string prefix, so it also lists the uploads to sibling keys that
start with it, which fs.list_multipart_uploads() excludes.
get_object() reads an object, a version of it, or a byte range and returns the bytes.
A range has an exclusive end; None as the end reads to the end of the object, and a
negative start without an end reads the last bytes. The response body is closed after
the read; a failure to read it raises the botocore exception and is not retried.
put_object() writes bytes and returns an S3PutObject.
data = core.get_object(S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object"), (0, 1024))
result = core.put_object(S3Path("YOUR_S3_BUCKET", "path/to/copy"), data)
print(result.etag, result.version_id)
delete_objects() deletes an S3DeleteBatch, the objects of one bucket that one
DeleteObjects request accepts. S3DeleteBatch.from_paths() groups paths into batches
of up to 1,000 objects per bucket. The objects that S3 could not delete are in the
errors of the returned S3DeleteResult, not raised; as in S3, deleting a key that
does not exist is not an error.
from pyathena.filesystem.s3_core import S3DeleteBatch
paths = [
S3Path.parse("s3://YOUR_S3_BUCKET/path/to/a"),
S3Path.parse("s3://YOUR_S3_BUCKET/path/to/b"),
]
for batch in S3DeleteBatch.from_paths(paths):
result = core.delete_objects(batch)
for error in result.errors:
print(error) # path (code: message)
get_object_tagging() returns the tags of an object as a dictionary, and
put_object_tagging() replaces all of its tags with the given ones. Both act on the
version ID of the path, if any, including null. put_object_acl() and
put_bucket_acl() apply a canned ACL, which must be in S3Core.OBJECT_ACLS or
S3Core.BUCKET_ACLS, respectively; another value raises ValueError, and an object
ACL also applies to the version ID of the path.
replace_object_metadata() replaces the user-defined metadata of an object, given the
head_object() result of the same path, by copying the object onto itself. The copy
rewrites the object, or creates a new version in a versioned bucket, so the path must
not have a version ID. The copy retains the content headers (CacheControl,
ContentDisposition, ContentEncoding, ContentLanguage, ContentType), Expires,
WebsiteRedirectLocation, StorageClass, and the ServerSideEncryption,
SSEKMSKeyId and BucketKeyEnabled of the object. Additional CopyObject parameters
take precedence over the retained fields, and any encryption parameter replaces all of
the retained encryption fields.
path = S3Path.parse("s3://YOUR_S3_BUCKET/path/to/object")
core.put_object_tagging(path, {**core.get_object_tagging(path), "tag2": "value2"})
head = core.head_object(path)
core.replace_object_metadata(path, head, {**head, "attr1": "value1"})
generate_presigned_url() signs a URL locally, for get_object unless another client
method is given, and sends no request. Its parameters take precedence over the
Bucket, Key and VersionId of the path. The request_kwargs of the core, such as
RequestPayer, are not included in the signed parameters; pass them as parameters
of generate_presigned_url() to sign them.
url = core.generate_presigned_url(path, expires_in=600)
upload_url = core.generate_presigned_url(path, "put_object", ContentType="text/csv")
create_bucket() and delete_bucket() create and delete a bucket when called. The
allow_bucket_creation and allow_bucket_deletion options apply only to the
filesystem’s mkdir/makedirs and rmdir, not to calls through the core.
get_bucket_versioning() returns the versioning state of a bucket, Enabled or
Suspended, or None if versioning has never been enabled on it.
S3Path.is_directory_bucket tells whether the bucket of a path is a directory bucket.
create_bucket() sends a LocationConstraint for the region_name argument, or for
the client’s region by default, except in us-east-1. The fields of a
CreateBucketConfiguration passed as a parameter are kept, and the LocationConstraint
is added to it only if it has none of LocationConstraint, Location and Bucket; an
empty or None configuration is treated as not passed. To
create a directory bucket, pass its Location and Bucket in
CreateBucketConfiguration; no LocationConstraint is added.
Multipart writer¶
S3MultipartWriter provides synchronous multipart requests and part planning without fsspec.
It is the writer used by S3File; applications can also construct it on an S3Core.
The constructor validates the destination and block size without sending a request.
The caller owns the buffer and executor, collects part results in number order, and waits for running requests before completing or aborting.
Serialize initiation and finalization; part requests can run in parallel after initiation.
from io import BytesIO
from pyathena.filesystem.s3_writer import S3MultipartWriter
writer = S3MultipartWriter(
core,
S3Path("YOUR_S3_BUCKET", "path/to/object"),
block_size=5 * 2**20,
request_kwargs={"ChecksumAlgorithm": "SHA256"},
)
writer.initiate()
try:
parts = [
writer.upload_part(number, body)
for number, body in writer.iter_parts(BytesIO(b"your data"))
]
result = writer.complete(parts)
except BaseException:
writer.abort()
raise
iter_parts() consumes a blocking binary stream from its current position and produces numbered part bodies without sending requests.
It merges a short tail into the preceding block and splits the result if it reaches the maximum part size.
A stream shorter than the minimum part size must be the final part.
iter_copy_parts(size) produces numbered source ranges for copying an existing object before appending data.
A source smaller than the minimum part size must be the final part; S3File reads a small existing object into its buffer when appending.
Both planners accept first_part_number to include previously copied or uploaded parts in the 10,000-part limit.
upload_part_copy() sends a planned range with an exclusive end; None sends no range of its own, as with the core primitive.
The writer filters inherited and per-request parameters by operation; per-request parameters take precedence.
It retains the upload identity after completion and after an abort failure, and clears it only after a successful abort.
The caller can therefore retry a failed abort.
The writer does not invalidate the filesystem cache; call fs.invalidate_cache() after a direct write.
When an S3File wait for multipart creation is interrupted, it waits for any creation that has already started and recovers the successful response before aborting.
It drops the buffer and closes the file before waiting, so a later close cannot upload the interrupted write, and a deferred commit raises S3MultipartStateError.
It suppresses Ctrl-C during the blocking recovery wait and re-raises the original error when cleanup finishes.
An additional interruption outside that wait can stop cleanup; once the upload identity is known, discard() can retry the abort.
An abort failure retains the upload identity for discard() to retry.
A failed multipart commit() also closes the file without its buffered data and aborts the upload; a later commit() raises S3MultipartStateError.
Cancelling an async buffered write through asyncio.to_thread still allows its writer thread to finish.
Multipart requests¶
create_multipart_upload(), upload_part(), upload_part_copy(),
complete_multipart_upload() and abort_multipart_upload() send the requests of a
multipart upload. part_ranges() sends no request: it splits an object into the byte
ranges of the parts that copy it, by the part limits MULTIPART_UPLOAD_MIN_PART_SIZE (5 MiB),
MULTIPART_UPLOAD_MAX_PART_SIZE (5 GiB) and MULTIPART_UPLOAD_MAX_PARTS (10,000) of
S3Core. check_multipart_upload_size() sends no request either: it raises
ValueError with the minimum block size if data of the given size takes more than
MULTIPART_UPLOAD_MAX_PARTS parts of the block size, the check that put and pipe
make before uploading.
copy_object() copies an object with one CopyObject request, which accepts objects
up to MULTIPART_UPLOAD_MAX_PART_SIZE. For a larger object, plan_multipart_copy()
reads the source and returns an S3MultipartCopyPlan: the version to copy, the byte
ranges of the parts, the parameters of each multipart upload request, and the
annotations to copy, so that the multipart upload writes the metadata, tags and
annotations that CopyObject would. It sends HeadObject, then GetObjectTagging and
ListObjectAnnotations unless the directives or the source exclude them, and writes
nothing. If HeadObject reports a size that fits in one CopyObject request, nothing else
is read, and the plan’s fits_single_request says to copy with copy_object()
instead. copy_object_annotation() copies one annotation onto the destination after
the upload completes, with GetObjectAnnotation and PutObjectAnnotation. As with
get_object(), the payload is closed after the read, and a failure to read it raises
the botocore exception and is not retried. The filesystems’ cp_file(), copy() and
mv() run these plans.
The multipart primitives take the S3MultipartUpload returned by creation,
keeping its bucket, key, upload ID and checksum configuration together.
The core retains no upload state.
upload_part() passes the upload’s checksum algorithm to the SDK;
completion selects the matching part checksum and sends the upload’s
ChecksumType when present, including FULL_OBJECT.
Without a creation algorithm, completion sends the ETag and part number without
checksums that the SDK may add to part uploads.
abort_multipart_upload() also accepts uploads returned by
list_multipart_uploads().
upload = core.create_multipart_upload(
S3Path("YOUR_S3_BUCKET", "path/to/object"), ChecksumAlgorithm="SHA256"
)
try:
part = core.upload_part(upload, 1, b"data")
result = core.complete_multipart_upload(upload, [part])
except Exception:
core.abort_multipart_upload(upload)
raise
Path pairing¶
S3PathPairing pairs the paths of one copy(), get() or mv(): mv() always, and
copy() and get() when a source has a version ID and the destination is one path
(fsspec pairs the others). Its delete_paths() splits the paths of an rm() into
those deleted as given and those expanded. The pairing is fsspec’s, except that a
path with a version ID names that version, and its destination is named after its key.
A pairing is a frozen dataclass of path1, path2, recursive and maxdepth, and
holds no filesystem. The filesystems read from it what to look up (expands,
skips_directories, looks_up_destination and conflict_candidates()), expand the
sources, leave out the directories when skips_directories is true, look up the
paths, and pass the results to copy_pairs() and move_pairs(). A sources,
destination_is_dir or missing that a rule needs and that is not passed raises
ValueError.
For moves that compare a null version with its unversioned key, the caller also
looks up bucket versioning and passes the names of the versioning-enabled buckets
as versioning_enabled_buckets to both conflict_candidates() and move_pairs().
Pass a collection of bucket names, such as a set, rather than a single string.
The default empty collection treats null versions as their keys, as in unversioned
or suspended buckets. The model makes no AWS requests.
from pyathena.filesystem.s3_path_pairing import S3PathPairing
pairing = S3PathPairing("s3://YOUR_S3_BUCKET/src/", "s3://YOUR_S3_BUCKET/dst/", recursive=True)
sources = fs.expand_path(pairing.path1, recursive=pairing.recursive)
for source, destination in pairing.copy_pairs(sources):
print(source, "->", destination)
Async filesystem¶
AioS3FileSystem provides the same functionality on top of fsspec’s
AsyncFileSystem, running its blocking calls in a thread pool sized by max_workers,
and its parallel requests in the request pool of its internal S3FileSystem.
Its open_async() returns a file whose reads and writes are coroutines.
See AioS3FileSystem for details.