Connection and Cursors¶
This section covers the core connection functionality and basic cursor operations.
Connection¶
DB API 2.0 interface to Amazon Athena: connect(), aio_connect(), and type objects.
- class pyathena.DBAPITypeObject[source]¶
A DB API type object that compares equal to each of its Athena type names.
https://www.python.org/dev/peps/pep-0249/#type-objects-and-constructors
- pyathena.connect(*, cursor_class: None = ..., **kwargs) Connection[Cursor][source]¶
- pyathena.connect(*, cursor_class: type[ConnectionCursor], **kwargs) Connection[ConnectionCursor]
Create a new database connection to Amazon Athena.
This function provides the main entry point for establishing connections to Amazon Athena. It follows the DB API 2.0 specification and returns a Connection object that can be used to create cursors for executing SQL queries. Connection settings must be passed as keyword arguments.
- Parameters:
s3_staging_dir – S3 location to store query results. Required if not using workgroups or if the workgroup doesn’t have a result location. Pass an empty string to explicitly disable S3 staging and skip the
AWS_ATHENA_S3_STAGING_DIRenvironment variable fallback (required for workgroups with managed query result storage).region_name – AWS region name. If not specified, uses the default region from your AWS configuration.
schema_name – Athena database/schema name. Defaults to “default”.
catalog_name – Athena data catalog name. Defaults to “awsdatacatalog”.
work_group – Athena workgroup name. Can be used instead of s3_staging_dir if the workgroup has a result location configured. Pass an empty string to skip the
AWS_ATHENA_WORK_GROUPenvironment variable fallback.poll_interval – Time in seconds between polling for query completion. Defaults to 1.0.
encryption_option – S3 encryption option for query results. Can be “SSE_S3”, “SSE_KMS”, or “CSE_KMS”.
kms_key – KMS key ID for encryption when using SSE_KMS or CSE_KMS.
profile_name – AWS profile name to use for authentication.
role_arn – ARN of IAM role to assume for authentication; its credentials are requested again before they expire.
role_session_name – Session name when assuming a role.
cursor_class – Custom cursor class to use. If not specified, uses the default Cursor class.
kill_on_interrupt – Whether to cancel running queries when interrupted. Defaults to True.
**kwargs – Additional keyword arguments passed to the Connection constructor.
- Returns:
A Connection object that can be used to create cursors and execute queries.
- Raises:
ProgrammingError – If neither s3_staging_dir nor work_group is provided.
Example
>>> import pyathena >>> conn = pyathena.connect( ... s3_staging_dir='s3://my-bucket/staging/', ... region_name='us-east-1', ... schema_name='mydatabase' ... ) >>> cursor = conn.cursor() >>> cursor.execute("SELECT * FROM mytable LIMIT 10") >>> results = cursor.fetchall()
- async pyathena.aio_connect(**kwargs) AioConnection[source]¶
Create a new async database connection to Amazon Athena.
This is the async counterpart of
connect(). It returns anAioConnectionwhose cursors use nativeasynciofor polling and API calls, keeping the event loop free.- Parameters:
**kwargs – Arguments forwarded to
AioConnection.create(). Seeconnect()for the full list of supported arguments.- Returns:
An
AioConnectionthat producesAioCursorinstances by default.
Example
>>> import pyathena >>> conn = await pyathena.aio_connect( ... s3_staging_dir='s3://my-bucket/staging/', ... region_name='us-east-1', ... ) >>> async with conn.cursor() as cursor: ... await cursor.execute("SELECT 1") ... print(await cursor.fetchone())
- class pyathena.connection.Connection(*, s3_staging_dir: str | None = ..., region_name: str | None = ..., schema_name: str | None = ..., catalog_name: str | None = ..., work_group: str | None = ..., poll_interval: float = ..., encryption_option: str | None = ..., kms_key: str | None = ..., profile_name: str | None = ..., role_arn: str | None = ..., role_session_name: str = ..., external_id: str | None = ..., serial_number: str | None = ..., duration_seconds: int = ..., converter: Converter | None = ..., formatter: Formatter | None = ..., retry_config: RetryConfig | None = ..., cursor_class: None = ..., cursor_kwargs: dict[str, Any] | None = ..., kill_on_interrupt: bool = ..., session: Session | None = ..., config: Config | None = ..., result_reuse_enable: bool = ..., result_reuse_minutes: int = ..., on_start_query_execution: Callable[[str], None] | None = ..., on_poll: Callable[[AthenaQueryExecution | AthenaCalculationExecutionStatus], None] | None = ..., glue_metadata_fallback: bool = ..., s3_config: Config | None = ..., **kwargs)[source]¶
- class pyathena.connection.Connection(*, s3_staging_dir: str | None = ..., region_name: str | None = ..., schema_name: str | None = ..., catalog_name: str | None = ..., work_group: str | None = ..., poll_interval: float = ..., encryption_option: str | None = ..., kms_key: str | None = ..., profile_name: str | None = ..., role_arn: str | None = ..., role_session_name: str = ..., external_id: str | None = ..., serial_number: str | None = ..., duration_seconds: int = ..., converter: Converter | None = ..., formatter: Formatter | None = ..., retry_config: RetryConfig | None = ..., cursor_class: type[ConnectionCursor] = ..., cursor_kwargs: dict[str, Any] | None = ..., kill_on_interrupt: bool = ..., session: Session | None = ..., config: Config | None = ..., result_reuse_enable: bool = ..., result_reuse_minutes: int = ..., on_start_query_execution: Callable[[str], None] | None = ..., on_poll: Callable[[AthenaQueryExecution | AthenaCalculationExecutionStatus], None] | None = ..., glue_metadata_fallback: bool = ..., s3_config: Config | None = ..., **kwargs)
A DB API 2.0 compliant connection to Amazon Athena.
The Connection class represents a database session and provides methods to create cursors for executing SQL queries against Amazon Athena. It handles authentication, session management, and query result storage in S3.
This class follows the Python Database API Specification v2.0 (PEP 249) and provides a familiar interface for database operations.
- s3_staging_dir¶
S3 location where query results are stored.
- region_name¶
AWS region name.
- schema_name¶
Default database/schema name for queries.
- catalog_name¶
Data catalog name (typically “awsdatacatalog”).
- work_group¶
Athena workgroup name.
- poll_interval¶
Interval in seconds for polling query status.
- encryption_option¶
S3 encryption option for query results.
- kms_key¶
KMS key for encryption when applicable.
- kill_on_interrupt¶
Whether to cancel queries on interrupt signals.
- result_reuse_enable¶
Whether to enable Athena’s result reuse feature.
- result_reuse_minutes¶
Minutes to reuse cached results.
Example
>>> conn = Connection( ... s3_staging_dir='s3://my-bucket/staging/', ... region_name='us-east-1', ... schema_name='mydatabase' ... ) >>> with conn: ... cursor = conn.cursor() ... cursor.execute("SELECT COUNT(*) FROM mytable") ... result = cursor.fetchone()
Note
Either s3_staging_dir or work_group must be specified. If using a workgroup, it must have a result location configured unless s3_staging_dir is also provided. For workgroups with managed query result storage, pass
s3_staging_dir=""to skip the environment variable fallback.- __init__(*, s3_staging_dir: str | None = None, region_name: str | None = None, schema_name: str | None = 'default', catalog_name: str | None = 'awsdatacatalog', work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, profile_name: str | None = None, role_arn: str | None = None, role_session_name: str = 'PyAthena-session-1795267768', external_id: str | None = None, serial_number: str | None = None, duration_seconds: int = 3600, converter: Converter | None = None, formatter: Formatter | None = None, retry_config: RetryConfig | None = None, cursor_class: None = None, cursor_kwargs: dict[str, Any] | None = None, kill_on_interrupt: bool = True, session: Session | None = None, config: Config | None = None, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, on_start_query_execution: Callable[[str], None] | None = None, on_poll: Callable[[AthenaQueryExecution | AthenaCalculationExecutionStatus], None] | None = None, glue_metadata_fallback: bool = True, s3_config: Config | None = None, **kwargs) None[source]¶
- __init__(*, s3_staging_dir: str | None = None, region_name: str | None = None, schema_name: str | None = 'default', catalog_name: str | None = 'awsdatacatalog', work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, profile_name: str | None = None, role_arn: str | None = None, role_session_name: str = 'PyAthena-session-1795267768', external_id: str | None = None, serial_number: str | None = None, duration_seconds: int = 3600, converter: Converter | None = None, formatter: Formatter | None = None, retry_config: RetryConfig | None = None, cursor_class: type[ConnectionCursor] = None, cursor_kwargs: dict[str, Any] | None = None, kill_on_interrupt: bool = True, session: Session | None = None, config: Config | None = None, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, on_start_query_execution: Callable[[str], None] | None = None, on_poll: Callable[[AthenaQueryExecution | AthenaCalculationExecutionStatus], None] | None = None, glue_metadata_fallback: bool = True, s3_config: Config | None = None, **kwargs) None
Initialize a new Athena database connection.
- Parameters:
s3_staging_dir – S3 location to store query results. Required if not using workgroups or if workgroup doesn’t have result location. None uses the
AWS_ATHENA_S3_STAGING_DIRenvironment variable. Pass an empty string to explicitly disable S3 staging and skip theAWS_ATHENA_S3_STAGING_DIRenvironment variable fallback. This is required when connecting to a workgroup with managed query result storage enabled.region_name – AWS region name. Uses default region if not specified.
schema_name – Default database/schema name. Defaults to “default”.
catalog_name – Data catalog name. Defaults to “awsdatacatalog”.
work_group – Athena workgroup name. Can substitute for s3_staging_dir if workgroup has result location configured. None uses the
AWS_ATHENA_WORK_GROUPenvironment variable. Pass an empty string to skip that environment variable fallback and leave the workgroup unset.poll_interval – Seconds between query status polls. Defaults to 1.0.
encryption_option – S3 encryption for results (“SSE_S3”, “SSE_KMS”, “CSE_KMS”).
kms_key – KMS key ID when using SSE_KMS or CSE_KMS encryption.
profile_name – AWS profile name for authentication.
role_arn – IAM role ARN to assume for authentication. The session and clients of the connection use the role’s credentials, which are requested again before they expire; the profile, credential and
botocore_sessionarguments sign only the AssumeRole requests.role_session_name – Session name when assuming IAM role.
external_id – External ID for role assumption (if required by role).
serial_number – MFA device serial number, for role assumption with
role_arnor for GetSessionToken without it. Each request for credentials, at connection and once they are about to expire, prompts for an MFA code withinput(); the other threads that need them wait.duration_seconds – The duration in seconds of the credentials of
role_arnorserial_number. Defaults to 3600.converter – Custom type converter. Uses DefaultTypeConverter if None.
formatter – Custom parameter formatter. Uses DefaultParameterFormatter if None.
retry_config – Retry configuration for API calls. Uses default if None.
cursor_class – Default cursor class for this connection.
cursor_kwargs – Default keyword arguments for cursor creation. Arguments passed to
cursor()override the same keys.kill_on_interrupt – Cancel running queries on interrupt. Defaults to True.
session – Pre-configured boto3 Session. Creates new session if None.
config – Boto3 Config object for client configuration.
result_reuse_enable – Enable Athena query result reuse. Defaults to False.
result_reuse_minutes – Minutes to reuse cached results.
on_start_query_execution – Callback invoked with each query ID before the cursor waits for the query, as for the
execute()argument of the same name.on_poll – Callback invoked once per poll iteration with the current execution object (
AthenaQueryExecution, orAthenaCalculationExecutionStatusfor Spark). Useful for monitoring live query progress. Defaults to None.glue_metadata_fallback – In
AwsDataCatalogand S3 Tables catalogs, answer a throttled table-metadata, table-listing or database-listing request from the AWS Glue Data Catalog before retrying it. Defaults to True.s3_config – Botocore Config options for the S3 client only, such as
max_pool_connections. They are merged overconfig.**kwargs – Additional arguments passed to boto3 Session and client.
- Raises:
ProgrammingError – If neither s3_staging_dir nor work_group is provided.
Note
Either s3_staging_dir or work_group must be specified. Environment variables AWS_ATHENA_S3_STAGING_DIR and AWS_ATHENA_WORK_GROUP are checked only when the corresponding argument is omitted or None. An explicit empty string disables that fallback and becomes None.
When using a workgroup with managed query result storage, pass
s3_staging_dir=""to prevent the environment variable fallback from sending aResultConfigurationthat conflicts withManagedQueryResultsConfiguration.
- property session: Session¶
Get the boto3 session used for AWS API calls.
- Returns:
The configured boto3 Session object.
- property client: BaseClient¶
Get the boto3 Athena client used for query operations.
- Returns:
The configured boto3 Athena client.
- property s3_client: BaseClient¶
The S3 client shared by the connection’s result sets, filesystems and Spark cursors.
It is built on first use from the connection’s session, region,
s3_configand client arguments, except Athena’sendpoint_urlandapi_version.
- property retry_config: RetryConfig¶
Get the retry configuration for AWS API calls.
- Returns:
The RetryConfig object that controls retry behavior for failed requests.
- __enter__()[source]¶
Enter the runtime context for the connection.
- Returns:
Self for use in context manager protocol.
- __exit__(exc_type, exc_val, exc_tb)[source]¶
Exit the runtime context and close the connection.
- Parameters:
exc_type – Exception type if an exception occurred.
exc_val – Exception value if an exception occurred.
exc_tb – Exception traceback if an exception occurred.
- cursor(cursor: None = None, **kwargs) ConnectionCursor[source]¶
- cursor(cursor: type[FunctionalCursor], **kwargs) FunctionalCursor
Create a new cursor object for executing queries.
Creates and returns a cursor object that can be used to execute SQL queries against Amazon Athena. The cursor inherits connection settings but can be customized with additional parameters.
- Parameters:
cursor – Custom cursor class to use. If not provided, uses the connection’s default cursor class.
**kwargs – Additional keyword arguments to pass to the cursor constructor. These override the connection’s
cursor_kwargsand its defaults.
- Returns:
A cursor object that can execute SQL queries.
Example
>>> cursor = connection.cursor() >>> cursor.execute("SELECT * FROM my_table LIMIT 10") >>> results = cursor.fetchall()
# Using a custom cursor type >>> from pyathena.pandas.cursor import PandasCursor >>> pandas_cursor = connection.cursor(PandasCursor) >>> df = pandas_cursor.execute(“SELECT * FROM my_table”).fetchall()
- close() None[source]¶
Close the connection.
Closes the network connections of the Athena client, of the STS client of
role_arnorserial_number, and of the Glue and S3 clients if they were built. A client used after this opens new network connections.Note
This method is called automatically when using the connection as a context manager (with statement).
- commit() None[source]¶
Commit any pending transaction.
This method is provided for DB API 2.0 compatibility. Since Athena does not support transactions, this method does nothing.
Note
Athena queries are auto-committed and cannot be rolled back.
- rollback() None[source]¶
Rollback any pending transaction.
This method is required by DB API 2.0 but is not supported by Athena since Athena does not support transactions.
- Raises:
NotSupportedError – Always raised since transactions are not supported.
Execution Options¶
- class pyathena.options.ExecuteOptions(work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int = 0, cache_expiration_time: int = 0, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, on_start_query_execution: Callable[[str], None] | None = None, result_set_type_hints: dict[str | int, str] | None = None)[source]¶
Shared options for
Cursor.execute()across all cursor implementations.This dataclass is the single source of truth for the query-execution arguments shared by every SQL cursor type (sync/async/aio and pandas/arrow/polars/s3fs variants). It can be passed to
execute()via theoptionskeyword argument as an alternative to individual :keyword >>> from pyathena.options import ExecuteOptions: :keyword >>> options = ExecuteOptions: :kwtype >>> options = ExecuteOptions: work_group=”primary”, cache_size=100 :keyword >>> cursor.execute: :kwtype >>> cursor.execute: “SELECT * FROM my_table”, options=optionsWhen both
optionsand individual keyword arguments are provided, the individual keyword arguments take precedence. This allows building a baseExecuteOptionsonce and tweaking it per call:>>> cursor.execute("SELECT ...", options=options, work_group="adhoc")
Passing None for an individual keyword argument is treated as “not provided” and leaves the corresponding
optionsfield unchanged.merge()also ignores None, so to clear a field, usedataclasses.replace()or construct a new instance.- work_group¶
Athena workgroup to use for this query. Overrides the connection-level workgroup.
- Type:
str | None
- s3_staging_dir¶
S3 location for query results. Overrides the connection-level staging directory.
- Type:
str | None
- cache_size¶
Number of recent queries to scan for client-side result caching. 0 (default) disables the cache lookup, unless
cache_expiration_timeis set to a positive value, in which case all queries within the expiration window are scanned. Aqmarkquery with parameters is never looked up.- Type:
- cache_expiration_time¶
Maximum age in seconds of a cached query result to consider for reuse. 0 (default) means no age limit.
- Type:
- result_reuse_enable¶
Enable Athena server-side result reuse for this query. None (default) falls back to the connection-level setting.
- Type:
bool | None
- result_reuse_minutes¶
Maximum age in minutes of a previous query result that Athena should consider for reuse. None (default) falls back to the connection-level setting.
- Type:
int | None
- paramstyle¶
Parameter style for this query (‘qmark’ or ‘pyformat’). None (default) uses the module-level
pyathena.paramstyle.- Type:
str | None
- on_start_query_execution¶
Callback invoked with the query ID before
execute()waits for the query: after the StartQueryExecution API call, or after a reusable query ID is found throughcache_size. Invoked by synchronous and aio cursors;AsyncCursor-based cursors return the query ID directly through their execution model and do not invoke it.- Type:
collections.abc.Callable[[str], None] | None
- result_set_type_hints¶
Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index. For example:
{"tags": "array(varchar)", "metadata": "map(varchar, integer)"}
- classmethod resolve(options: ExecuteOptions | None, **overrides: Any) ExecuteOptions[source]¶
Return
options(or a default instance) withoverridesapplied.This is the canonical way for
execute()implementations to combine theoptionsargument with the individual keyword arguments.- Parameters:
options – Base options, or None to start from the defaults.
**overrides – Field values to apply on top of
options. None values are ignored.
- Returns:
The effective
ExecuteOptionsfor the call.
Note
Without base options, construct the base class directly from the overrides. Subclass resolvers retain their
merge()dispatch.
- merge(**overrides: Any) ExecuteOptions[source]¶
Return a new instance with non-None
overridesapplied.- Parameters:
**overrides – Field values to apply on top of this instance. None values are ignored, so an omitted
execute()keyword argument never clobbers a value set onoptions.- Returns:
A new
ExecuteOptionswith the overrides applied.- Raises:
TypeError – If an override name is not a field of this class.
- __init__(work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int = 0, cache_expiration_time: int = 0, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, on_start_query_execution: Callable[[str], None] | None = None, result_set_type_hints: dict[str | int, str] | None = None) None¶
Standard Cursors¶
- class pyathena.cursor.Cursor(*, s3_staging_dir: str | None = None, schema_name: str | None = None, catalog_name: str | None = None, work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, kill_on_interrupt: bool = True, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, **kwargs)[source]¶
A DB API 2.0 compliant cursor for executing SQL queries on Amazon Athena.
The Cursor class provides methods for executing SQL queries against Amazon Athena and retrieving results. It follows the Python Database API Specification v2.0 (PEP 249) and provides familiar database cursor operations.
This cursor returns results as tuples by default. For other data formats, consider using specialized cursor classes like PandasCursor or ArrowCursor.
- description¶
Sequence of column descriptions for the last query.
- rowcount¶
Number of rows affected by the last query (-1 for SELECT queries).
- arraysize¶
Default number of rows to fetch with fetchmany().
Example
>>> cursor = connection.cursor() >>> cursor.execute("SELECT name, age FROM users WHERE age > %(age)s", {"age": 18}) >>> while True: ... row = cursor.fetchone() ... if not row: ... break ... print(f"Name: {row[0]}, Age: {row[1]}")
>>> cursor.execute("CREATE TABLE test AS SELECT 1 as id, 'test' as name") >>> print(f"Created table, rows affected: {cursor.rowcount}")
- __init__(*, s3_staging_dir: str | None = None, schema_name: str | None = None, catalog_name: str | None = None, work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, kill_on_interrupt: bool = True, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, **kwargs) None[source]¶
Initialize a Cursor.
- Parameters:
s3_staging_dir – S3 location for query results.
schema_name – Default schema name.
catalog_name – Default catalog name.
work_group – Athena workgroup name.
poll_interval – Query status polling interval in seconds.
encryption_option – S3 encryption option (SSE_S3, SSE_KMS, CSE_KMS).
kms_key – KMS key for encryption.
kill_on_interrupt – Cancel the query when a
KeyboardInterruptinterruptsexecute()while it starts the query, runs theon_start_query_executioncallbacks, or waits for the query.result_reuse_enable – Enable Athena query result reuse.
result_reuse_minutes – Maximum age in minutes of a reused result.
**kwargs – Arguments forwarded to
WithResultSet.__init__andBaseCursor.__init__, such asarraysize,connection,converter,formatter, andretry_config.
- property arraysize: int¶
The default number of rows per
fetchmany()call.execute()passes it to the new result set, so a change applies to the result sets of later executions. Setting it to zero or a negative value raisesProgrammingError.- Returns:
The default number of rows per
fetchmany()call.
- execute(operation: str, parameters: dict[str, Any] | list[str] | None = None, *, work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int | None = None, cache_expiration_time: int | None = None, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, on_start_query_execution: Callable[[str], None] | None = None, result_set_type_hints: dict[str | int, str] | None = None, options: ExecuteOptions | None = None, **kwargs) Cursor[source]¶
Execute a SQL query.
- Parameters:
operation – SQL query string to execute.
parameters – Query parameters (optional).
work_group – Athena workgroup to use for this query.
s3_staging_dir – S3 location for query results.
cache_size – Number of queries to check for result caching.
cache_expiration_time – Cache expiration time in seconds.
result_reuse_enable – Enable Athena result reuse for this query.
result_reuse_minutes – Minutes to reuse cached results.
paramstyle – Parameter style (‘qmark’ or ‘pyformat’).
on_start_query_execution – Callback invoked with the query ID before
execute()waits for the query: after theStartQueryExecutioncall, or after a reusable query ID is found throughcache_size. Function signature: (query_id: str) -> None This allows early access to query_id for monitoring/cancellation.result_set_type_hints – Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index. For example:
{"tags": "array(varchar)", "metadata": "map(varchar, integer)"}options – Shared execution options as an
ExecuteOptionsinstance. Individual keyword arguments take precedence overoptionsfields.**kwargs – Unknown keyword arguments raise TypeError.
- Returns:
Self reference for method chaining.
- Raises:
ProgrammingError – If the cursor is closed.
Example
>>> cursor.execute( ... "SELECT * FROM table_with_complex_types", ... result_set_type_hints={ ... "tags": "array(varchar)", ... "metadata": "map(varchar, integer)", ... } ... )
- DEFAULT_RESULT_REUSE_MINUTES = 60¶
- LIST_DATABASES_MAX_RESULTS = 50¶
- LIST_QUERY_EXECUTIONS_MAX_RESULTS = 50¶
- LIST_TABLE_METADATA_MAX_RESULTS = 50¶
- cancel() None¶
Cancel the currently executing query.
- Raises:
ProgrammingError – If the cursor is closed or no query is currently executing.
- close() None¶
Close the cursor and its result set.
Later operations on the cursor raise
ProgrammingError, andquery_idkeeps the ID of the last query. Closing a closed cursor does nothing.
- property connection: Connection[Any]¶
The connection that created this cursor.
- property data_manifest_location: str | None¶
The S3 location of the data manifest that lists the files the query wrote.
- property description: list[tuple[str, str, None, None, int, int, str]] | None¶
The DB API 2.0 column descriptions of the result set, or None without one.
- property encryption_option: str | None¶
The
EncryptionOptionof the query results, such asSSE_S3orSSE_KMS.
- property engine_execution_time_in_millis: int | None¶
The time in milliseconds that the query engine took to run the query.
- property error_category: int | None¶
1 for system, 2 for user, 3 for other.
- Type:
The
ErrorCategoryof the failure
- executemany(operation: str, seq_of_parameters: list[dict[str, Any] | list[str] | None], **kwargs) None¶
Execute a SQL query multiple times with different parameters.
On success,
rowcountis the sum of the affected row counts, or -1 if any execution has an unknown count. An empty parameter list sets it to 0. On failure, it is -1; earlier executions are not rolled back. Result sets are discarded.On failure,
query_idretains the current query ID when available. If parameter iteration fails, this can identify the last successful execution.- Parameters:
operation – SQL query string to execute.
seq_of_parameters – Sequence of parameter sets, one per execution.
**kwargs – Additional keyword arguments passed to each
execute(). An empty parameter list makes noexecute()call, so it does not report unsupported keyword names.
- Raises:
ProgrammingError – If the cursor is closed.
- property expected_bucket_owner: str | None¶
The AWS account ID expected to own the S3 bucket of the query results.
- fetchall() list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch all remaining rows from the result set.
- Returns:
The remaining rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- fetchmany(size: int | None = None) list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch multiple rows from the result set.
- Parameters:
size – Maximum number of rows to fetch. If None or not positive,
arraysizeis used.- Returns:
The fetched rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- fetchone() tuple[Any | None, ...] | dict[Any, Any | None] | None¶
Fetch the next row of the result set.
- Returns:
The next row (a tuple, or a dict for dict cursors), or None if no more rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- static get_default_converter(unload: bool = False) DefaultTypeConverter | Any¶
Get the default type converter for this cursor class.
- Parameters:
unload – Whether the converter is for UNLOAD operations. Some cursor types may return different converters for UNLOAD operations.
- Returns:
The default type converter instance for this cursor type.
- get_table_metadata(table_name: str, catalog_name: str | None = None, schema_name: str | None = None, logging_: bool = True) AthenaTableMetadata¶
Get one table’s metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
table_name – The table name.
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
logging – Whether to log a failed request.
- Returns:
The table’s metadata.
- Raises:
OperationalError – If the request fails, including when the table does not exist.
ProgrammingError – If the cursor is closed.
- list_databases(catalog_name: str | None, max_results: int | None = None) list[AthenaDatabase]¶
List the catalog’s databases.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
max_results – The page size of each request.
- Returns:
The catalog’s databases.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- list_table_metadata(catalog_name: str | None = None, schema_name: str | None = None, expression: str | None = None, max_results: int | None = None, logging_: bool = True) list[AthenaTableMetadata]¶
List a database’s table metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
expression – A table name pattern.
max_results – The page size of each request.
logging – Whether to log a failed request.
- Returns:
The metadata of the database’s tables.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- property query_id: str | None¶
The query execution ID of the last execution.
With
cache_sizeorcache_expiration_time, this can be the ID of a previous execution whose result is reused.- Returns:
The query execution ID, or None if there is none since the last reset.
- property query_planning_time_in_millis: int | None¶
The time in milliseconds that Athena took to plan the query.
- property query_queue_time_in_millis: int | None¶
The time in milliseconds that the query waited in the queue.
- property result_reuse_enabled: bool | None¶
Whether reuse of previous query results by age is enabled for the query.
- property result_reuse_minutes: int | None¶
The maximum age in minutes of a previous query result that Athena can reuse.
- property result_set: AthenaResultSet | None¶
The result set of the last executed query.
- Returns:
The result set, or None before a query succeeds or after a reset.
- property reused_previous_result: bool | None¶
Whether Athena reused a previous query result instead of running the query.
- property rowcount: int¶
Get the number of rows affected by the last operation.
For SELECT statements, this returns -1 as per DB API 2.0 specification. For DML operations (INSERT, UPDATE, DELETE) and CTAS, this returns the number of affected rows. After a successful
executemany(), this is the sum across executions, or -1 if any count is unknown.- Returns:
The number of rows, or -1 if not applicable or unknown.
- property rownumber: int | None¶
The zero-based index of the next row in the result set.
- Returns:
The row index, or None if there is no result set or the index is unknown.
- property s3_acl_option: str | None¶
The
S3AclOptionof the query results, such asBUCKET_OWNER_FULL_CONTROL.
- property service_processing_time_in_millis: int | None¶
The time in milliseconds that Athena took to publish the query results.
- setinputsizes(sizes)¶
Accept input sizes as DB API 2.0 requires, and ignore them.
- Parameters:
sizes – Sequence of parameter types or sizes.
- Raises:
ProgrammingError – If the cursor is closed.
- setoutputsize(size, column=None)¶
Accept a column buffer size as DB API 2.0 requires, and ignore it.
- Parameters:
size – Buffer size for large columns.
column – Index of the column the size applies to, or None for all large columns.
- Raises:
ProgrammingError – If the cursor is closed.
- property state_change_reason: str | None¶
The
StateChangeReasonthat gives further detail about the state.
- class pyathena.cursor.DictCursor(*, dict_type: type[Any] | None = None, **kwargs)[source]¶
A cursor that returns query results as dictionaries instead of tuples.
DictCursor provides the same functionality as the standard Cursor but returns rows as dictionaries where column names are keys. This makes it easier to access column values by name rather than position.
Example
>>> cursor = connection.cursor(DictCursor) >>> cursor.execute("SELECT id, name, email FROM users LIMIT 1") >>> row = cursor.fetchone() >>> print(f"User: {row['name']} ({row['email']})")
>>> cursor.execute("SELECT * FROM products") >>> for row in cursor.fetchall(): ... print(f"Product {row['id']}: {row['name']} - ${row['price']}")
- __init__(*, dict_type: type[Any] | None = None, **kwargs) None[source]¶
Initialize a DictCursor.
- Parameters:
dict_type – The type used to build each row of this cursor’s result sets. If None, the result set class’s
dict_typeis used.**kwargs – Arguments forwarded to
Cursor.__init__.
- DEFAULT_RESULT_REUSE_MINUTES = 60¶
- LIST_DATABASES_MAX_RESULTS = 50¶
- LIST_QUERY_EXECUTIONS_MAX_RESULTS = 50¶
- LIST_TABLE_METADATA_MAX_RESULTS = 50¶
- property arraysize: int¶
The default number of rows per
fetchmany()call.execute()passes it to the new result set, so a change applies to the result sets of later executions. Setting it to zero or a negative value raisesProgrammingError.- Returns:
The default number of rows per
fetchmany()call.
- cancel() None¶
Cancel the currently executing query.
- Raises:
ProgrammingError – If the cursor is closed or no query is currently executing.
- close() None¶
Close the cursor and its result set.
Later operations on the cursor raise
ProgrammingError, andquery_idkeeps the ID of the last query. Closing a closed cursor does nothing.
- property connection: Connection[Any]¶
The connection that created this cursor.
- property data_manifest_location: str | None¶
The S3 location of the data manifest that lists the files the query wrote.
- property description: list[tuple[str, str, None, None, int, int, str]] | None¶
The DB API 2.0 column descriptions of the result set, or None without one.
- property encryption_option: str | None¶
The
EncryptionOptionof the query results, such asSSE_S3orSSE_KMS.
- property engine_execution_time_in_millis: int | None¶
The time in milliseconds that the query engine took to run the query.
- property error_category: int | None¶
1 for system, 2 for user, 3 for other.
- Type:
The
ErrorCategoryof the failure
- execute(operation: str, parameters: dict[str, Any] | list[str] | None = None, *, work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int | None = None, cache_expiration_time: int | None = None, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, on_start_query_execution: Callable[[str], None] | None = None, result_set_type_hints: dict[str | int, str] | None = None, options: ExecuteOptions | None = None, **kwargs) Cursor¶
Execute a SQL query.
- Parameters:
operation – SQL query string to execute.
parameters – Query parameters (optional).
work_group – Athena workgroup to use for this query.
s3_staging_dir – S3 location for query results.
cache_size – Number of queries to check for result caching.
cache_expiration_time – Cache expiration time in seconds.
result_reuse_enable – Enable Athena result reuse for this query.
result_reuse_minutes – Minutes to reuse cached results.
paramstyle – Parameter style (‘qmark’ or ‘pyformat’).
on_start_query_execution – Callback invoked with the query ID before
execute()waits for the query: after theStartQueryExecutioncall, or after a reusable query ID is found throughcache_size. Function signature: (query_id: str) -> None This allows early access to query_id for monitoring/cancellation.result_set_type_hints – Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index. For example:
{"tags": "array(varchar)", "metadata": "map(varchar, integer)"}options – Shared execution options as an
ExecuteOptionsinstance. Individual keyword arguments take precedence overoptionsfields.**kwargs – Unknown keyword arguments raise TypeError.
- Returns:
Self reference for method chaining.
- Raises:
ProgrammingError – If the cursor is closed.
Example
>>> cursor.execute( ... "SELECT * FROM table_with_complex_types", ... result_set_type_hints={ ... "tags": "array(varchar)", ... "metadata": "map(varchar, integer)", ... } ... )
- executemany(operation: str, seq_of_parameters: list[dict[str, Any] | list[str] | None], **kwargs) None¶
Execute a SQL query multiple times with different parameters.
On success,
rowcountis the sum of the affected row counts, or -1 if any execution has an unknown count. An empty parameter list sets it to 0. On failure, it is -1; earlier executions are not rolled back. Result sets are discarded.On failure,
query_idretains the current query ID when available. If parameter iteration fails, this can identify the last successful execution.- Parameters:
operation – SQL query string to execute.
seq_of_parameters – Sequence of parameter sets, one per execution.
**kwargs – Additional keyword arguments passed to each
execute(). An empty parameter list makes noexecute()call, so it does not report unsupported keyword names.
- Raises:
ProgrammingError – If the cursor is closed.
- property expected_bucket_owner: str | None¶
The AWS account ID expected to own the S3 bucket of the query results.
- fetchall() list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch all remaining rows from the result set.
- Returns:
The remaining rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- fetchmany(size: int | None = None) list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch multiple rows from the result set.
- Parameters:
size – Maximum number of rows to fetch. If None or not positive,
arraysizeis used.- Returns:
The fetched rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- fetchone() tuple[Any | None, ...] | dict[Any, Any | None] | None¶
Fetch the next row of the result set.
- Returns:
The next row (a tuple, or a dict for dict cursors), or None if no more rows.
- Raises:
ProgrammingError – If the cursor is closed or has no result set.
- static get_default_converter(unload: bool = False) DefaultTypeConverter | Any¶
Get the default type converter for this cursor class.
- Parameters:
unload – Whether the converter is for UNLOAD operations. Some cursor types may return different converters for UNLOAD operations.
- Returns:
The default type converter instance for this cursor type.
- get_table_metadata(table_name: str, catalog_name: str | None = None, schema_name: str | None = None, logging_: bool = True) AthenaTableMetadata¶
Get one table’s metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
table_name – The table name.
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
logging – Whether to log a failed request.
- Returns:
The table’s metadata.
- Raises:
OperationalError – If the request fails, including when the table does not exist.
ProgrammingError – If the cursor is closed.
- list_databases(catalog_name: str | None, max_results: int | None = None) list[AthenaDatabase]¶
List the catalog’s databases.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
max_results – The page size of each request.
- Returns:
The catalog’s databases.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- list_table_metadata(catalog_name: str | None = None, schema_name: str | None = None, expression: str | None = None, max_results: int | None = None, logging_: bool = True) list[AthenaTableMetadata]¶
List a database’s table metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
expression – A table name pattern.
max_results – The page size of each request.
logging – Whether to log a failed request.
- Returns:
The metadata of the database’s tables.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- property query_id: str | None¶
The query execution ID of the last execution.
With
cache_sizeorcache_expiration_time, this can be the ID of a previous execution whose result is reused.- Returns:
The query execution ID, or None if there is none since the last reset.
- property query_planning_time_in_millis: int | None¶
The time in milliseconds that Athena took to plan the query.
- property query_queue_time_in_millis: int | None¶
The time in milliseconds that the query waited in the queue.
- property result_reuse_enabled: bool | None¶
Whether reuse of previous query results by age is enabled for the query.
- property result_reuse_minutes: int | None¶
The maximum age in minutes of a previous query result that Athena can reuse.
- property result_set: AthenaResultSet | None¶
The result set of the last executed query.
- Returns:
The result set, or None before a query succeeds or after a reset.
- property reused_previous_result: bool | None¶
Whether Athena reused a previous query result instead of running the query.
- property rowcount: int¶
Get the number of rows affected by the last operation.
For SELECT statements, this returns -1 as per DB API 2.0 specification. For DML operations (INSERT, UPDATE, DELETE) and CTAS, this returns the number of affected rows. After a successful
executemany(), this is the sum across executions, or -1 if any count is unknown.- Returns:
The number of rows, or -1 if not applicable or unknown.
- property rownumber: int | None¶
The zero-based index of the next row in the result set.
- Returns:
The row index, or None if there is no result set or the index is unknown.
- property s3_acl_option: str | None¶
The
S3AclOptionof the query results, such asBUCKET_OWNER_FULL_CONTROL.
- property service_processing_time_in_millis: int | None¶
The time in milliseconds that Athena took to publish the query results.
- setinputsizes(sizes)¶
Accept input sizes as DB API 2.0 requires, and ignore them.
- Parameters:
sizes – Sequence of parameter types or sizes.
- Raises:
ProgrammingError – If the cursor is closed.
- setoutputsize(size, column=None)¶
Accept a column buffer size as DB API 2.0 requires, and ignore it.
- Parameters:
size – Buffer size for large columns.
column – Index of the column the size applies to, or None for all large columns.
- Raises:
ProgrammingError – If the cursor is closed.
- property state_change_reason: str | None¶
The
StateChangeReasonthat gives further detail about the state.
Asynchronous Cursors¶
- class pyathena.async_cursor.AsyncCursor(*, s3_staging_dir: str | None = None, schema_name: str | None = None, catalog_name: str | None = None, work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, kill_on_interrupt: bool = True, max_workers: int = 20, arraysize: int = 1000, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, **kwargs)[source]¶
Asynchronous cursor for non-blocking Athena query execution.
This cursor allows multiple queries to be executed concurrently without blocking the main thread. It’s useful for applications that need to execute multiple queries in parallel or perform other work while queries are running.
The cursor maintains a thread pool for executing queries asynchronously and provides methods to check query status and retrieve results when ready.
- arraysize¶
Default number of rows that fetchmany() returns on the result sets this cursor creates.
Example
>>> cursor = connection.cursor(AsyncCursor) >>> >>> # Execute multiple queries concurrently >>> query_id1, future1 = cursor.execute("SELECT COUNT(*) FROM table1") >>> query_id2, future2 = cursor.execute("SELECT COUNT(*) FROM table2") >>> query_id3, future3 = cursor.execute("SELECT COUNT(*) FROM table3") >>> >>> # Check if queries are done and get results >>> if future1.done(): ... result1 = future1.result().fetchall() >>> >>> # Wait for all to complete >>> results = [f.result().fetchall() for f in [future1, future2, future3]]
Note
Each execute() call returns a
(query_id, future)tuple. The future resolves to the result set, which provides the column descriptions and the fetch methods.description(query_id)also returns the column descriptions as a Future. A result set whose future resolved before the cursor closed belongs to the caller. A future that still waits for its query when the cursor closes raisesProgrammingError, also if its last status check finds the query finished, and one that is building its result set resolves with a closed result set (seeclose()).- __init__(*, s3_staging_dir: str | None = None, schema_name: str | None = None, catalog_name: str | None = None, work_group: str | None = None, poll_interval: float = 1, encryption_option: str | None = None, kms_key: str | None = None, kill_on_interrupt: bool = True, max_workers: int = 20, arraysize: int = 1000, result_reuse_enable: bool = False, result_reuse_minutes: int = 60, **kwargs) None[source]¶
Initialize an AsyncCursor.
- Parameters:
s3_staging_dir – S3 location for query results.
schema_name – Default schema name.
catalog_name – Default catalog name.
work_group – Athena workgroup name.
poll_interval – Query status polling interval in seconds.
encryption_option – S3 encryption option (SSE_S3, SSE_KMS, CSE_KMS).
kms_key – KMS key for encryption.
kill_on_interrupt – Cancel a query whose start in
execute()is interrupted byKeyboardInterrupt. Waiting runs on worker threads, which do not receive the interrupt.max_workers – Size of the cursor thread pool for waiting and collecting results.
arraysize – Default number of rows per
fetchmany()call of the result sets the cursor creates.result_reuse_enable – Enable Athena query result reuse.
result_reuse_minutes – Maximum age in minutes of a reused result.
**kwargs – Arguments forwarded to
BaseCursor.__init__, such asconnection,converter,formatter, andretry_config.
- Raises:
ProgrammingError – If
arraysizeis not between 1 andCursorIterator.DEFAULT_FETCH_SIZE.
- close(wait: bool = False) None[source]¶
Close the cursor and shut down its executor.
Later operations on the cursor raise
ProgrammingError. The futures ofexecute(),description(), andpoll()that still wait for their query raiseProgrammingErrorat their next status check, also when that check finds the query finished; the query itself keeps running. A future ofexecute()that is building its result set resolves with a closed result set, and one that is just finishing can resolve with either an open or a closed one. A result set whose future resolved before the close is left to the caller. The futures ofcancel()andquery_execution()still run.- Parameters:
wait – Whether to wait for submitted futures to finish before returning.
- description(query_id: str) Future[list[tuple[str, str, None, None, int, int, str]] | None][source]¶
Get the column descriptions of a query’s result set asynchronously.
The future waits for the query to finish, then reads only the result set metadata with the metadata reader of the cursor: a
GetQueryResultspage of one row, or for an UNLOAD result of the pandas, Arrow, and Polars cursors, the schema of its Parquet files. The descriptions are those of the result set ofexecute()with the options of the cursor, not those given toexecute(). If the cursor closes first, the future raisesProgrammingError.- Parameters:
query_id – The Athena query execution ID.
- Returns:
Future object containing the DB API 2.0 column descriptions, or None.
- Raises:
ProgrammingError – If the cursor is closed.
- query_execution(query_id: str) Future[AthenaQueryExecution][source]¶
Get query execution details asynchronously.
Retrieves the current execution status and metadata for a query. This is useful for monitoring query progress without blocking.
- Parameters:
query_id – The Athena query execution ID.
- Returns:
Future object containing AthenaQueryExecution with query details.
- Raises:
ProgrammingError – If the cursor is closed.
- poll(query_id: str) Future[AthenaQueryExecution][source]¶
Poll for query completion asynchronously.
Waits for the query to complete (succeed, fail, or be cancelled) and returns the final execution status. This method blocks until completion but runs the polling in a background thread.
- Parameters:
query_id – The Athena query execution ID to poll.
- Returns:
Future object containing the final AthenaQueryExecution status.
Note
This method performs polling internally, so it will take time proportional to your query execution duration. If the cursor closes first, the future raises
ProgrammingError.- Raises:
ProgrammingError – If the cursor is closed.
- execute(operation: str, parameters: dict[str, Any] | list[str] | None = None, *, work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int | None = None, cache_expiration_time: int | None = None, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, result_set_type_hints: dict[str | int, str] | None = None, options: ExecuteOptions | None = None, **kwargs) tuple[str, Future[AthenaResultSet | Any]][source]¶
Execute a SQL query asynchronously.
Starts query execution on Amazon Athena and returns immediately without waiting for completion. The query runs in the background while your application can continue with other work.
- Parameters:
operation – SQL query string to execute.
parameters – Query parameters (optional).
work_group – Athena workgroup to use (optional).
s3_staging_dir – S3 location for query results (optional).
cache_size – Number of queries to check for result caching (optional).
cache_expiration_time – Cache expiration time in seconds (optional).
result_reuse_enable – Enable result reuse for identical queries (optional).
result_reuse_minutes – Result reuse duration in minutes (optional).
paramstyle – Parameter style to use (optional).
result_set_type_hints – Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index.
options – Shared execution options as an
ExecuteOptionsinstance. Individual keyword arguments take precedence overoptionsfields.**kwargs – Unknown keyword arguments raise TypeError.
- Returns:
query_id: Athena query execution ID for tracking
future: Future object for result retrieval, which raises
ProgrammingErrorif the cursor closes before the query finishes (seeclose())
- Return type:
Tuple of (query_id, future) where
- Raises:
ProgrammingError – If the cursor is closed.
Example
>>> query_id, future = cursor.execute("SELECT * FROM large_table") >>> print(f"Query started: {query_id}") >>> # Do other work while query runs... >>> result_set = future.result() # Wait for completion
- executemany(operation: str, seq_of_parameters: list[dict[str, Any] | list[str] | None], **kwargs) None[source]¶
Execute multiple queries asynchronously (not supported).
This method is not supported for asynchronous cursors because managing multiple concurrent queries would be complex and resource-intensive.
- Parameters:
operation – SQL query string.
seq_of_parameters – Sequence of parameter sets.
**kwargs – Additional arguments.
- Raises:
NotSupportedError – Always raised as this operation is not supported.
ProgrammingError – If the cursor is closed.
Note
For bulk operations, consider using execute() with parameterized queries or batch processing patterns instead.
- cancel(query_id: str) Future[None][source]¶
Cancel a running query asynchronously.
Submits a cancellation request for the specified query. The cancellation itself runs asynchronously in the background.
- Parameters:
query_id – The Athena query execution ID to cancel.
- Returns:
Future object that completes when the cancellation request finishes.
Example
>>> query_id, future = cursor.execute("SELECT * FROM huge_table") >>> # Later, cancel the query >>> cancel_future = cursor.cancel(query_id) >>> cancel_future.result() # Wait for cancellation to complete
- Raises:
ProgrammingError – If the cursor is closed.
- LIST_DATABASES_MAX_RESULTS = 50¶
- LIST_QUERY_EXECUTIONS_MAX_RESULTS = 50¶
- LIST_TABLE_METADATA_MAX_RESULTS = 50¶
- property connection: Connection[Any]¶
The connection that created this cursor.
- static get_default_converter(unload: bool = False) DefaultTypeConverter | Any¶
Get the default type converter for this cursor class.
- Parameters:
unload – Whether the converter is for UNLOAD operations. Some cursor types may return different converters for UNLOAD operations.
- Returns:
The default type converter instance for this cursor type.
- get_table_metadata(table_name: str, catalog_name: str | None = None, schema_name: str | None = None, logging_: bool = True) AthenaTableMetadata¶
Get one table’s metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
table_name – The table name.
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
logging – Whether to log a failed request.
- Returns:
The table’s metadata.
- Raises:
OperationalError – If the request fails, including when the table does not exist.
ProgrammingError – If the cursor is closed.
- list_databases(catalog_name: str | None, max_results: int | None = None) list[AthenaDatabase]¶
List the catalog’s databases.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
max_results – The page size of each request.
- Returns:
The catalog’s databases.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- list_table_metadata(catalog_name: str | None = None, schema_name: str | None = None, expression: str | None = None, max_results: int | None = None, logging_: bool = True) list[AthenaTableMetadata]¶
List a database’s table metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
expression – A table name pattern.
max_results – The page size of each request.
logging – Whether to log a failed request.
- Returns:
The metadata of the database’s tables.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- setinputsizes(sizes)¶
Accept input sizes as DB API 2.0 requires, and ignore them.
- Parameters:
sizes – Sequence of parameter types or sizes.
- Raises:
ProgrammingError – If the cursor is closed.
- setoutputsize(size, column=None)¶
Accept a column buffer size as DB API 2.0 requires, and ignore it.
- Parameters:
size – Buffer size for large columns.
column – Index of the column the size applies to, or None for all large columns.
- Raises:
ProgrammingError – If the cursor is closed.
- class pyathena.async_cursor.AsyncDictCursor(*, dict_type: type[Any] | None = None, **kwargs)[source]¶
Asynchronous cursor that returns query results as dictionaries.
Combines the asynchronous execution capabilities of AsyncCursor with the dictionary-based result format of DictCursor. Results are returned as dictionaries where column names are keys, making it easier to access column values by name rather than position.
Example
>>> cursor = connection.cursor(AsyncDictCursor) >>> query_id, future = cursor.execute("SELECT id, name, email FROM users") >>> result_set = future.result() >>> row = result_set.fetchone() >>> print(f"User: {row['name']} ({row['email']})")
- __init__(*, dict_type: type[Any] | None = None, **kwargs) None[source]¶
Initialize an AsyncDictCursor.
- Parameters:
dict_type – The type used to build each row of this cursor’s result sets. If None, the result set class’s
dict_typeis used.**kwargs – Arguments forwarded to
AsyncCursor.__init__.
- LIST_DATABASES_MAX_RESULTS = 50¶
- LIST_QUERY_EXECUTIONS_MAX_RESULTS = 50¶
- LIST_TABLE_METADATA_MAX_RESULTS = 50¶
- cancel(query_id: str) Future[None]¶
Cancel a running query asynchronously.
Submits a cancellation request for the specified query. The cancellation itself runs asynchronously in the background.
- Parameters:
query_id – The Athena query execution ID to cancel.
- Returns:
Future object that completes when the cancellation request finishes.
Example
>>> query_id, future = cursor.execute("SELECT * FROM huge_table") >>> # Later, cancel the query >>> cancel_future = cursor.cancel(query_id) >>> cancel_future.result() # Wait for cancellation to complete
- Raises:
ProgrammingError – If the cursor is closed.
- close(wait: bool = False) None¶
Close the cursor and shut down its executor.
Later operations on the cursor raise
ProgrammingError. The futures ofexecute(),description(), andpoll()that still wait for their query raiseProgrammingErrorat their next status check, also when that check finds the query finished; the query itself keeps running. A future ofexecute()that is building its result set resolves with a closed result set, and one that is just finishing can resolve with either an open or a closed one. A result set whose future resolved before the close is left to the caller. The futures ofcancel()andquery_execution()still run.- Parameters:
wait – Whether to wait for submitted futures to finish before returning.
- property connection: Connection[Any]¶
The connection that created this cursor.
- description(query_id: str) Future[list[tuple[str, str, None, None, int, int, str]] | None]¶
Get the column descriptions of a query’s result set asynchronously.
The future waits for the query to finish, then reads only the result set metadata with the metadata reader of the cursor: a
GetQueryResultspage of one row, or for an UNLOAD result of the pandas, Arrow, and Polars cursors, the schema of its Parquet files. The descriptions are those of the result set ofexecute()with the options of the cursor, not those given toexecute(). If the cursor closes first, the future raisesProgrammingError.- Parameters:
query_id – The Athena query execution ID.
- Returns:
Future object containing the DB API 2.0 column descriptions, or None.
- Raises:
ProgrammingError – If the cursor is closed.
- execute(operation: str, parameters: dict[str, Any] | list[str] | None = None, *, work_group: str | None = None, s3_staging_dir: str | None = None, cache_size: int | None = None, cache_expiration_time: int | None = None, result_reuse_enable: bool | None = None, result_reuse_minutes: int | None = None, paramstyle: str | None = None, result_set_type_hints: dict[str | int, str] | None = None, options: ExecuteOptions | None = None, **kwargs) tuple[str, Future[AthenaResultSet | Any]]¶
Execute a SQL query asynchronously.
Starts query execution on Amazon Athena and returns immediately without waiting for completion. The query runs in the background while your application can continue with other work.
- Parameters:
operation – SQL query string to execute.
parameters – Query parameters (optional).
work_group – Athena workgroup to use (optional).
s3_staging_dir – S3 location for query results (optional).
cache_size – Number of queries to check for result caching (optional).
cache_expiration_time – Cache expiration time in seconds (optional).
result_reuse_enable – Enable result reuse for identical queries (optional).
result_reuse_minutes – Result reuse duration in minutes (optional).
paramstyle – Parameter style to use (optional).
result_set_type_hints – Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index.
options – Shared execution options as an
ExecuteOptionsinstance. Individual keyword arguments take precedence overoptionsfields.**kwargs – Unknown keyword arguments raise TypeError.
- Returns:
query_id: Athena query execution ID for tracking
future: Future object for result retrieval, which raises
ProgrammingErrorif the cursor closes before the query finishes (seeclose())
- Return type:
Tuple of (query_id, future) where
- Raises:
ProgrammingError – If the cursor is closed.
Example
>>> query_id, future = cursor.execute("SELECT * FROM large_table") >>> print(f"Query started: {query_id}") >>> # Do other work while query runs... >>> result_set = future.result() # Wait for completion
- executemany(operation: str, seq_of_parameters: list[dict[str, Any] | list[str] | None], **kwargs) None¶
Execute multiple queries asynchronously (not supported).
This method is not supported for asynchronous cursors because managing multiple concurrent queries would be complex and resource-intensive.
- Parameters:
operation – SQL query string.
seq_of_parameters – Sequence of parameter sets.
**kwargs – Additional arguments.
- Raises:
NotSupportedError – Always raised as this operation is not supported.
ProgrammingError – If the cursor is closed.
Note
For bulk operations, consider using execute() with parameterized queries or batch processing patterns instead.
- static get_default_converter(unload: bool = False) DefaultTypeConverter | Any¶
Get the default type converter for this cursor class.
- Parameters:
unload – Whether the converter is for UNLOAD operations. Some cursor types may return different converters for UNLOAD operations.
- Returns:
The default type converter instance for this cursor type.
- get_table_metadata(table_name: str, catalog_name: str | None = None, schema_name: str | None = None, logging_: bool = True) AthenaTableMetadata¶
Get one table’s metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
table_name – The table name.
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
logging – Whether to log a failed request.
- Returns:
The table’s metadata.
- Raises:
OperationalError – If the request fails, including when the table does not exist.
ProgrammingError – If the cursor is closed.
- list_databases(catalog_name: str | None, max_results: int | None = None) list[AthenaDatabase]¶
List the catalog’s databases.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
max_results – The page size of each request.
- Returns:
The catalog’s databases.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- list_table_metadata(catalog_name: str | None = None, schema_name: str | None = None, expression: str | None = None, max_results: int | None = None, logging_: bool = True) list[AthenaTableMetadata]¶
List a database’s table metadata.
In
AwsDataCatalogand S3 Tables catalogs, a throttled request is answered from the AWS Glue Data Catalog; seeglue_metadata_fallback.- Parameters:
catalog_name – The catalog, or None for the cursor’s catalog.
schema_name – The database, or None for the cursor’s schema.
expression – A table name pattern.
max_results – The page size of each request.
logging – Whether to log a failed request.
- Returns:
The metadata of the database’s tables.
- Raises:
OperationalError – If the request fails.
ProgrammingError – If the cursor is closed.
- poll(query_id: str) Future[AthenaQueryExecution]¶
Poll for query completion asynchronously.
Waits for the query to complete (succeed, fail, or be cancelled) and returns the final execution status. This method blocks until completion but runs the polling in a background thread.
- Parameters:
query_id – The Athena query execution ID to poll.
- Returns:
Future object containing the final AthenaQueryExecution status.
Note
This method performs polling internally, so it will take time proportional to your query execution duration. If the cursor closes first, the future raises
ProgrammingError.- Raises:
ProgrammingError – If the cursor is closed.
- query_execution(query_id: str) Future[AthenaQueryExecution]¶
Get query execution details asynchronously.
Retrieves the current execution status and metadata for a query. This is useful for monitoring query progress without blocking.
- Parameters:
query_id – The Athena query execution ID.
- Returns:
Future object containing AthenaQueryExecution with query details.
- Raises:
ProgrammingError – If the cursor is closed.
- setinputsizes(sizes)¶
Accept input sizes as DB API 2.0 requires, and ignore them.
- Parameters:
sizes – Sequence of parameter types or sizes.
- Raises:
ProgrammingError – If the cursor is closed.
- setoutputsize(size, column=None)¶
Accept a column buffer size as DB API 2.0 requires, and ignore it.
- Parameters:
size – Buffer size for large columns.
column – Index of the column the size applies to, or None for all large columns.
- Raises:
ProgrammingError – If the cursor is closed.
Result Sets¶
- class pyathena.result_set.AthenaResultSet(connection: Connection[Any], converter: Converter, query_execution: AthenaQueryExecution, arraysize: int, retry_config: RetryConfig, _pre_fetch: bool = True, result_set_type_hints: dict[str | int, str] | None = None)[source]¶
Result set for Athena query execution using the GetQueryResults API.
This class provides a DB API 2.0 compliant result set implementation that fetches query results from Amazon Athena. It uses the GetQueryResults API to retrieve data in paginated chunks, converting each value according to its Athena data type.
The result set exposes query execution metadata (timing, data scanned, state, etc.) through read-only properties, allowing inspection of query performance and status.
This is the base result set implementation used by the standard Cursor. Specialized implementations exist for different output formats:
AthenaArrowResultSet: Apache Arrow formatAthenaPandasResultSet: Pandas DataFrameAthenaS3FSResultSet: S3 file-based access
Example
>>> cursor.execute("SELECT * FROM my_table") >>> result_set = cursor.result_set >>> print(f"Query ID: {result_set.query_id}") >>> print(f"Data scanned: {result_set.data_scanned_in_bytes} bytes") >>> for row in result_set: ... print(row)
See also
AWS Athena GetQueryResults API: https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html
- __init__(connection: Connection[Any], converter: Converter, query_execution: AthenaQueryExecution, arraysize: int, retry_config: RetryConfig, _pre_fetch: bool = True, result_set_type_hints: dict[str | int, str] | None = None) None[source]¶
Initialize the result set and fetch the first page if the query succeeded.
- Parameters:
connection – The connection that ran the query.
converter – The converter for result values.
query_execution – The query execution whose results to read.
arraysize – The number of rows per
GetQueryResultspage and the defaultfetchmany()size.retry_config – The retry configuration for API calls.
_pre_fetch – Whether to fetch the first page here when the query succeeded. The async result set passes False and fetches it itself, and so do the result sets that read their result with their own metadata reader.
result_set_type_hints – Athena type signatures for complex-type columns, keyed by column name (case-insensitive) or zero-based column index.
- Raises:
ProgrammingError – If
query_executionis not given.OperationalError – If fetching the first page fails.
- property state_change_reason: str | None¶
The
StateChangeReasonthat gives further detail about the state.
- property error_category: int | None¶
1 for system, 2 for user, 3 for other.
- Type:
The
ErrorCategoryof the failure
- property engine_execution_time_in_millis: int | None¶
The time in milliseconds that the query engine took to run the query.
- property query_queue_time_in_millis: int | None¶
The time in milliseconds that the query waited in the queue.
- property total_execution_time_in_millis: int | None¶
The total time in milliseconds that Athena took to run the query.
- property query_planning_time_in_millis: int | None¶
The time in milliseconds that Athena took to plan the query.
- property service_processing_time_in_millis: int | None¶
The time in milliseconds that Athena took to publish the query results.
- property data_manifest_location: str | None¶
The S3 location of the data manifest that lists the files the query wrote.
- property reused_previous_result: bool | None¶
Whether Athena reused a previous query result instead of running the query.
- property is_unload: bool¶
Whether the query is an UNLOAD statement that the cursor runs for its result.
Its result is read from the Parquet files of the UNLOAD output when the query has a result location; without one, as with managed query result storage, it is read from
GetQueryResults.- Returns:
True if the metadata reader of the result set finds an UNLOAD result.
- property encryption_option: str | None¶
The
EncryptionOptionof the query results, such asSSE_S3orSSE_KMS.
- property expected_bucket_owner: str | None¶
The AWS account ID expected to own the S3 bucket of the query results.
- property s3_acl_option: str | None¶
The
S3AclOptionof the query results, such asBUCKET_OWNER_FULL_CONTROL.
- property result_reuse_enabled: bool | None¶
Whether reuse of previous query results by age is enabled for the query.
- property result_reuse_minutes: int | None¶
The maximum age in minutes of a previous query result that Athena can reuse.
- property description: list[tuple[str, str, None, None, int, int, str]] | None¶
The DB API 2.0 column descriptions.
None without result metadata, or for
INSERT,UPDATE,DELETE, andMERGE. The metadata reader of the result set formats them (seeAthenaResultSetMetadataReader.description()). A subclass that changes the descriptions does so in its reader class (_metadata_reader_class), whichAsyncCursor.description()also reads with.
- property connection: Connection[Any]¶
The connection of the result set; raises
ProgrammingErrorif closed.
- fetchone() tuple[Any | None, ...] | dict[Any, Any | None] | None[source]¶
Fetch the next row of the result.
- fetchmany(size: int | None = None) list[tuple[Any | None, ...] | dict[Any, Any | None]][source]¶
Fetch the next set of rows of the query result.
- Parameters:
size – Maximum number of rows to fetch. If None or not positive,
arraysizeis used.- Returns:
The rows, fewer than
sizewhen the result is exhausted.- Raises:
TypeError – If
sizeis not an integer and not replaced witharraysize.ProgrammingError – If the result set is closed or an earlier fetch failed.
- fetchall() list[tuple[Any | None, ...] | dict[Any, Any | None]][source]¶
Fetch all remaining rows of the query result.
- Returns:
The remaining rows.
- Raises:
ProgrammingError – If the result set is closed or an earlier fetch failed.
- close() None[source]¶
Close the result set and discard its query execution, metadata, and rows.
Later fetches raise
ProgrammingError. Closing a closed result set does nothing.
- DEFAULT_RESULT_REUSE_MINUTES = 60¶
- class pyathena.result_set.AthenaDictResultSet(*args: Any, dict_type: type[Any] | None = None, **kwargs: Any)[source]¶
A result set that returns each row as a dictionary keyed by column name.
- __init__(*args: Any, dict_type: type[Any] | None = None, **kwargs: Any) None[source]¶
Initialize the result set with an optional row type for this instance.
- Parameters:
*args – Positional arguments passed to the next
__init__in the MRO.dict_type – The type used to build each row of this result set. If None, the class attribute
dict_typeis used.**kwargs – Keyword arguments passed to the next
__init__in the MRO.
- DEFAULT_RESULT_REUSE_MINUTES = 60¶
- close() None¶
Close the result set and discard its query execution, metadata, and rows.
Later fetches raise
ProgrammingError. Closing a closed result set does nothing.
- property connection: Connection[Any]¶
The connection of the result set; raises
ProgrammingErrorif closed.
- property data_manifest_location: str | None¶
The S3 location of the data manifest that lists the files the query wrote.
- property description: list[tuple[str, str, None, None, int, int, str]] | None¶
The DB API 2.0 column descriptions.
None without result metadata, or for
INSERT,UPDATE,DELETE, andMERGE. The metadata reader of the result set formats them (seeAthenaResultSetMetadataReader.description()). A subclass that changes the descriptions does so in its reader class (_metadata_reader_class), whichAsyncCursor.description()also reads with.
- property encryption_option: str | None¶
The
EncryptionOptionof the query results, such asSSE_S3orSSE_KMS.
- property engine_execution_time_in_millis: int | None¶
The time in milliseconds that the query engine took to run the query.
- property error_category: int | None¶
1 for system, 2 for user, 3 for other.
- Type:
The
ErrorCategoryof the failure
- property expected_bucket_owner: str | None¶
The AWS account ID expected to own the S3 bucket of the query results.
- fetchall() list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch all remaining rows of the query result.
- Returns:
The remaining rows.
- Raises:
ProgrammingError – If the result set is closed or an earlier fetch failed.
- fetchmany(size: int | None = None) list[tuple[Any | None, ...] | dict[Any, Any | None]]¶
Fetch the next set of rows of the query result.
- Parameters:
size – Maximum number of rows to fetch. If None or not positive,
arraysizeis used.- Returns:
The rows, fewer than
sizewhen the result is exhausted.- Raises:
TypeError – If
sizeis not an integer and not replaced witharraysize.ProgrammingError – If the result set is closed or an earlier fetch failed.
- property is_unload: bool¶
Whether the query is an UNLOAD statement that the cursor runs for its result.
Its result is read from the Parquet files of the UNLOAD output when the query has a result location; without one, as with managed query result storage, it is read from
GetQueryResults.- Returns:
True if the metadata reader of the result set finds an UNLOAD result.
- property query_planning_time_in_millis: int | None¶
The time in milliseconds that Athena took to plan the query.
- property query_queue_time_in_millis: int | None¶
The time in milliseconds that the query waited in the queue.
- property result_reuse_enabled: bool | None¶
Whether reuse of previous query results by age is enabled for the query.
- property result_reuse_minutes: int | None¶
The maximum age in minutes of a previous query result that Athena can reuse.
- property reused_previous_result: bool | None¶
Whether Athena reused a previous query result instead of running the query.
- property s3_acl_option: str | None¶
The
S3AclOptionof the query results, such asBUCKET_OWNER_FULL_CONTROL.
- property service_processing_time_in_millis: int | None¶
The time in milliseconds that Athena took to publish the query results.
- property state_change_reason: str | None¶
The
StateChangeReasonthat gives further detail about the state.
- class pyathena.result_set.AthenaResultSetMetadataReader(connection: Connection[Any], query_execution: AthenaQueryExecution, retry_config: RetryConfig)[source]¶
Reads the metadata of the result set of a query execution from the Athena API.
It builds and sends the
GetQueryResultsrequests of the result set, parses theirResultSetMetadata, and formats the DB API 2.0 column descriptions.read()reads the metadata and the update count from a page of one row;describe()returns the descriptions without the rows. The result sets page through the same requests, so that the descriptions of a result set and of its reader match. It reads no rows and holds no resources to close.- __init__(connection: Connection[Any], query_execution: AthenaQueryExecution, retry_config: RetryConfig) None[source]¶
Initialize the reader; it sends no request.
- Parameters:
connection – The connection that ran the query.
query_execution – The query execution whose result set to read.
retry_config – The retry configuration for API calls.
- property query_execution: AthenaQueryExecution¶
The query execution whose result set the reader reads.
- property is_unload: bool¶
Whether the query is an UNLOAD statement that the cursor runs for its result.
- property reads_unload_files: bool¶
Whether the result set is read from the Parquet files of the UNLOAD output.
- property metadata: tuple[dict[str, Any], ...] | None¶
The column metadata that
read()read; None before it.
- build_get_query_results_request(max_results: int, next_token: str | None = None) dict[str, Any][source]¶
Build a
GetQueryResultsrequest for the query.- Parameters:
max_results – The maximum number of rows in the page.
next_token – The token of the page to get; the first page if None.
- Returns:
The request parameters.
- Raises:
ProgrammingError – If the query ID is missing or the query has not succeeded.
- get_query_results(max_results: int, next_token: str | None = None) dict[str, Any][source]¶
Get a page of query results with
GetQueryResults.- Parameters:
max_results – The maximum number of rows in the page.
next_token – The token of the page to get; the first page if None.
- Returns:
The
GetQueryResultsresponse.- Raises:
ProgrammingError – If the query ID is missing or the query has not succeeded.
OperationalError – If the request fails.
- parse_metadata(response: dict[str, Any]) tuple[dict[str, Any], ...][source]¶
Parse the column metadata of a
GetQueryResultsresponse.- Parameters:
response – The
GetQueryResultsresponse.- Returns:
The
ColumnInfoof the response.- Raises:
DataError – If the response has no result metadata.
- read() None[source]¶
Read the metadata and the update count from a
GetQueryResultspage of one row.The rows of the page are not read.
- Raises:
ProgrammingError – If the query ID is missing or the query has not succeeded.
OperationalError – If the request fails.
DataError – If the response has no result metadata.
- description(metadata: tuple[dict[str, Any], ...] | None) list[tuple[str, str, None, None, int, int, str]] | None[source]¶
Format the DB API 2.0 column descriptions of column metadata.
- Parameters:
metadata – The column metadata.
- Returns:
The descriptions; None without metadata, or for
INSERT,UPDATE,DELETE, andMERGE.
- describe() list[tuple[str, str, None, None, int, int, str]] | None[source]¶
Read the column descriptions of the result set without its rows.
- Returns:
The descriptions, as
description()formats them; None if the query has not succeeded, without a request.- Raises:
OperationalError – If reading the metadata fails.
ProgrammingError – If the query execution of an UNLOAD result has no data manifest location.
- class pyathena.result_set.AthenaUnloadResultSetMetadataReader(connection: Connection[Any], query_execution: AthenaQueryExecution, retry_config: RetryConfig, unload: bool = False, unload_location: str | None = None)[source]¶
Reads the metadata of a result set that can be read from the output files of UNLOAD.
The pandas, Arrow, and Polars result sets read the Parquet files of an UNLOAD query instead of its CSV result. This reader decides whether the result set does, reads the data manifest of the query, and derives the unload location. A result without data files has no columns; otherwise a subclass reads the metadata from the Parquet files, as its result set reads them (see
_read_parquet_metadata()). Other results are read as the base class reads them.- __init__(connection: Connection[Any], query_execution: AthenaQueryExecution, retry_config: RetryConfig, unload: bool = False, unload_location: str | None = None) None[source]¶
Initialize the reader; it sends no request.
- Parameters:
connection – The connection that ran the query.
query_execution – The query execution whose result set to read.
retry_config – The retry configuration for API calls.
unload – Whether the cursor runs its queries as UNLOAD to Parquet.
unload_location – The S3 location of the UNLOAD output. If None, it is derived from the first file in the data manifest.
- property is_unload: bool¶
Whether the query is an UNLOAD statement that the cursor runs for its result.
- property reads_unload_files: bool¶
Whether the result set is read from the Parquet files of the UNLOAD output.
A query without a result location, as with managed query result storage, is read from
GetQueryResults.
- property unload_location: str | None¶
The S3 location of the UNLOAD output, given or derived by
read().
- property data_manifest: list[str]¶
The S3 paths of the Parquet files that
read()read from the data manifest.
- read() None[source]¶
Read the metadata, from the Parquet files of the UNLOAD output.
A result read from the files has no update count, and no
GetQueryResultsrequest is sent for it.- Raises:
OperationalError – If reading the data manifest or the metadata fails.
ProgrammingError – If the query execution has no data manifest location.
- class pyathena.result_set.WithResultSet(*, arraysize: int | None = None, **kwargs)[source]¶
Mixin that keeps a cursor’s query ID and result set.
Provides the query ID, the result set and its properties,
arraysize,rownumber,rowcount, andclose.WithFetchandWithAsyncFetchlist it beforeBaseCursor/AioBaseCursorandCursorIterator, so that these members take precedence over theirs.- __init__(*, arraysize: int | None = None, **kwargs) None[source]¶
Initialize the cursor with no query ID and no result set.
- Parameters:
arraysize – Default number of rows per
fetchmany()call, validated by thearraysizesetter. If None,DEFAULT_FETCH_SIZEis used.**kwargs – Arguments passed to the next
__init__in the MRO.
- Raises:
ProgrammingError – If
arraysizeis outside the range the cursor’sarraysizesetter accepts.
- property result_set: AthenaResultSet | None¶
The result set of the last executed query.
- Returns:
The result set, or None before a query succeeds or after a reset.
- property description: list[tuple[str, str, None, None, int, int, str]] | None¶
The DB API 2.0 column descriptions of the result set, or None without one.
- property query_id: str | None¶
The query execution ID of the last execution.
With
cache_sizeorcache_expiration_time, this can be the ID of a previous execution whose result is reused.- Returns:
The query execution ID, or None if there is none since the last reset.
- property state_change_reason: str | None¶
The
StateChangeReasonthat gives further detail about the state.
- property error_category: int | None¶
1 for system, 2 for user, 3 for other.
- Type:
The
ErrorCategoryof the failure
- property engine_execution_time_in_millis: int | None¶
The time in milliseconds that the query engine took to run the query.
- property query_queue_time_in_millis: int | None¶
The time in milliseconds that the query waited in the queue.
- property total_execution_time_in_millis: int | None¶
The total time in milliseconds that Athena took to run the query.
- property query_planning_time_in_millis: int | None¶
The time in milliseconds that Athena took to plan the query.
- property service_processing_time_in_millis: int | None¶
The time in milliseconds that Athena took to publish the query results.
- property data_manifest_location: str | None¶
The S3 location of the data manifest that lists the files the query wrote.
- property reused_previous_result: bool | None¶
Whether Athena reused a previous query result instead of running the query.
- property encryption_option: str | None¶
The
EncryptionOptionof the query results, such asSSE_S3orSSE_KMS.
- property expected_bucket_owner: str | None¶
The AWS account ID expected to own the S3 bucket of the query results.
- property s3_acl_option: str | None¶
The
S3AclOptionof the query results, such asBUCKET_OWNER_FULL_CONTROL.
- property result_reuse_enabled: bool | None¶
Whether reuse of previous query results by age is enabled for the query.
- property result_reuse_minutes: int | None¶
The maximum age in minutes of a previous query result that Athena can reuse.
- property rowcount: int¶
Get the number of rows affected by the last operation.
For SELECT statements, this returns -1 as per DB API 2.0 specification. For DML operations (INSERT, UPDATE, DELETE) and CTAS, this returns the number of affected rows. After a successful
executemany(), this is the sum across executions, or -1 if any count is unknown.- Returns:
The number of rows, or -1 if not applicable or unknown.
- property arraysize: int¶
The default number of rows per
fetchmany()call.execute()passes it to the new result set, so a change applies to the result sets of later executions. Setting it to zero or a negative value raisesProgrammingError.- Returns:
The default number of rows per
fetchmany()call.