export_table#

DynamoDB export to S3 tool box.

Reference:

DynamoDB data export to Amazon S3: how it works: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html

class aws_dynamodb_io.export_table.ManifestSummary(version: str, export_arn: str, table_id: str, table_arn: str, s3_bucket: str, s3_prefix: str, s3_sse_algorithm: str, s3_sse_kms_key_id: str, item_count: int, output_format: str, output_view: Optional[str], export_from_time_str: Optional[str], export_to_time_str: Optional[str], start_time_str: str, end_time_str: str, export_time_str: Optional[str], manifest_files_s3_key: str, billed_size_bytes: int, export_type: str)[source]#

The manifest-summary.json file data model.

to_dict() → Dict[str, Any][source]#: Convert the object to a dictionary.

class aws_dynamodb_io.export_table.DataFile(item_count: int, md5: str, etag: str, s3_bucket: str, s3_key: str, export_arn: str, export_format: str)[source]#

The s3://.../AWSDynamoDB/${timestamp}-${random_str]/data/${random_str}.json.gz data file data model.

Parameters:

item_count – number of item in this data file.
md5 – md5 hash of this data file.
etag – AWS S3 etag.
s3_bucket – S3 bucket name.
s3_key – S3 key.

property s3_uri: str#: The S3 URI of the data file.

read_dynamodb_json(s3_client: S3Client) → List[Dict[str, Dict[str, Any]]][source]#

Read items from the DynamoDB JSON data file.

Ref: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html

Example item:

{
    'key1': {'S': '...'},
    'attr1': {'S': '...'},
    'attr2': {'N': '...'},
    ...
},

read_amazon_ion(s3_client: S3Client, ion_loads_kwargs: Dict[str, Any] = None) → List[IonPyDict][source]#

Read items from the Amazon ION data file.

Parameters:: s3_client – S3 client for reading data.

Ref: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.Output.html

Example item:

{
    'key1': ...,
    'attr1': ...,
    'attr2': ...,
    ...
},

class aws_dynamodb_io.export_table.ExportStatusEnum(value)[source]#: An enumeration.

class aws_dynamodb_io.export_table.ExportFormatEnum(value)[source]#: An enumeration.

class aws_dynamodb_io.export_table.ExportTypeEnum(value)[source]#: An enumeration.

class aws_dynamodb_io.export_table.ExportJob(arn: str, status: str, start_time: Optional[datetime] = None, end_time: Optional[datetime] = None, export_time: Optional[datetime] = None, table_arn: Optional[str] = None, table_id: Optional[str] = None, client_token: Optional[str] = None, s3_bucket: Optional[str] = None, s3_prefix: Optional[str] = None, s3_sse_algorithm: Optional[str] = None, s3_sse_kms_key_id: Optional[str] = None, billed_size_bytes: Optional[int] = None, item_count: Optional[int] = None, export_format: Optional[str] = None, failure_code: Optional[str] = None, failure_message: Optional[str] = None, export_manifest: Optional[str] = None, export_type: Optional[str] = None, incremental_export_specification: Optional[dict] = None)[source]#

The DynamoDB export table job data model.

Ref:

export_table_to_point_in_time: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/export_table_to_point_in_time.html
describe_export: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/describe_export.html
list_exports: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/list_exports.html
How it works: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html

classmethod list_exports(dynamodb_client: DynamoDBClient, table_arn: str, page_size: int = 25, max_results: int = 1000, get_details: bool = False) → List[ExportJob][source]#: Ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/list_exports.html

classmethod from_export_description(desc: dict)[source]#

Parameters:: desc – The export description dictionary from export_table_to_point_in_time, describe_export or list_exports.

classmethod describe_export(dynamodb_client: DynamoDBClient, export_arn: str) → Optional[ExportJob][source]#: Ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/describe_export.html

property export_short_id: str#: The short ID of the export, which is a compound of the export timestamp and random string. Example: 1672531200000-a1b2c3d4. 1672531200000 is the timestamp of 2023-01-01 00:00:00

property s3uri_export: str#

The S3 folder you specified when you call the dynamodb_client.export_table_to_point_in_time(...) API.

Example: s3://bucket/prefix/

property s3uri_export_data: str#

Where the export data files are stored.

Example: s3://bucket/prefix/AWSDynamoDB/1672531200000-a1b2c3d4/data/

property s3uri_export_manifest_files: str#

The S3 location of the manifest files.

Example: s3://bucket/prefix/AWSDynamoDB/1672531200000-a1b2c3d4/manifest-files.json

property s3uri_export_manifest_summary: str#

The S3 location of the manifest summary file.

Example: s3://bucket/prefix/AWSDynamoDB/1672531200000-a1b2c3d4/manifest-summary.json

get_details(dynamodb_client: DynamoDBClient)[source]#: Get the details of the DynamoDB export, refresh it’s attributes values.

get_manifest_summary(dynamodb_client: DynamoDBClient, s3_client: S3Client) → ManifestSummary[source]#: Get the manifest summary of the DynamoDB export.

get_data_files(dynamodb_client: DynamoDBClient, s3_client: S3Client) → List[DataFile][source]#: Get the list of data files of the DynamoDB export.

read_dynamodb_json(dynamodb_client: DynamoDBClient, s3_client: S3Client) → Iterable[Dict[str, Dict[str, Any]]][source]#: Read the items of the DynamoDB export. This is a generator function.

read_amazon_ion(dynamodb_client: DynamoDBClient, s3_client: S3Client) → Iterable[IonPyDict][source]#: Read the items of the DynamoDB export. This is a generator function.

classmethod export_table_to_point_in_time(dynamodb_client: DynamoDBClient, table_arn: str, s3_bucket: str, s3_prefix: Optional[str] = None, export_time: Optional[datetime] = None, s3_bucket_owner: Optional[datetime] = None, s3_sse_algorithm: Optional[datetime] = None, s3_sse_kms_key_id: Optional[datetime] = None, export_format: str = 'DYNAMODB_JSON', export_type: str = 'FULL_EXPORT', client_token: Optional[str] = None, incremental_export_specification: Optional[dict] = None)[source]#

Export DynamoDB to point-in-time, and return the export object.

Ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/export_table_to_point_in_time.html

classmethod wait_until_complete(dynamodb_client: DynamoDBClient, export_arn: str, delays: int = 10, timeout: int = 900, verbose: bool = True) → ExportJob[source]#: Wait until the DynamoDB export is completed.

classmethod from_s3_dir(s3_client: S3Client, bucket: str, prefix: str)[source]#

Unlike describe_export reading from DynamoDB API, it directly reads the export metadata from the S3 folder of a completed export job. The DynamoDB export is only available for 35 days after the export is completed. After that, you can use this method to read the export from S3 directly.

Parameters:

s3_client – The boto3 S3 client
bucket – The S3 bucket
prefix – It should have a manifest-summary.json file in it. Example: “my-dynamodb-export/AWSDynamoDB/01725162280092-940349cc/