airflow.providers.amazon.aws.operators.dms¶

Classes¶

`DmsCreateTaskOperator`	Creates AWS DMS replication task.
`DmsModifyTaskOperator`	Modifies an existing AWS DMS replication task.
`DmsDeleteTaskOperator`	Deletes AWS DMS replication task.
`DmsDescribeTasksOperator`	Describes AWS DMS replication tasks.
`DmsStartTaskOperator`	Starts AWS DMS replication task.
`DmsStopTaskOperator`	Stops AWS DMS replication task.
`DmsDescribeReplicationConfigsOperator`	Describes AWS DMS Serverless replication configurations.
`DmsCreateReplicationConfigOperator`	Creates an AWS DMS Serverless replication configuration.
`DmsDeleteReplicationConfigOperator`	Deletes an AWS DMS Serverless replication configuration.
`DmsDescribeReplicationsOperator`	Describes AWS DMS Serverless replications.
`DmsStartReplicationOperator`	Starts an AWS DMS Serverless replication.
`DmsStopReplicationOperator`	Stops an AWS DMS Serverless replication.

Module Contents¶

class airflow.providers.amazon.aws.operators.dms.DmsCreateTaskOperator(*, replication_task_id, source_endpoint_arn, target_endpoint_arn, replication_instance_arn, table_mappings, migration_type='full-load', create_task_kwargs=None, **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Creates AWS DMS replication task.

See also

For more information on how to use this operator, take a look at the guide: Create a replication task

Parameters:

replication_task_id (str) – Replication task id
source_endpoint_arn (str) – Source endpoint ARN
target_endpoint_arn (str) – Target endpoint ARN
replication_instance_arn (str) – Replication instance ARN
table_mappings (dict) – Table mappings
migration_type (str) – Migration type (‘full-load’|’cdc’|’full-load-and-cdc’), full-load by default.
create_task_kwargs (dict | None) – Extra arguments for DMS replication task creation.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers: ClassVar[dict][source]¶

replication_task_id[source]¶

source_endpoint_arn[source]¶

target_endpoint_arn[source]¶

replication_instance_arn[source]¶

migration_type = 'full-load'[source]¶

table_mappings[source]¶

create_task_kwargs[source]¶

execute(context)[source]¶

Create AWS DMS replication task from Airflow.

Returns:: replication task arn

class airflow.providers.amazon.aws.operators.dms.DmsModifyTaskOperator(*, replication_task_arn, table_mappings=None, migration_type=None, replication_task_settings=None, cdc_start_time=None, cdc_start_position=None, cdc_stop_position=None, wait_for_completion=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Modifies an existing AWS DMS replication task.

The task must already be in a modifiable state before modification. Use DmsStopTaskOperator upstream in the Dag to stop it, and DmsStartTaskOperator downstream to restart it afterwards if needed.

Valid modifiable states are stopped, ready, and failed.

See also

For more information on how to use this operator, take a look at the guide: Modify a replication task

Parameters:

replication_task_arn (str) – Replication task ARN
table_mappings (dict | None) – New table mappings. If not provided, existing mappings are kept.
migration_type (str | None) – Migration type (‘full-load’|’cdc’|’full-load-and-cdc’). If not provided, existing type is kept.
replication_task_settings (dict | None) – Task settings dict. If not provided, existing settings are kept.
cdc_start_time (datetime.datetime | None) – Start time for CDC.
cdc_start_position (str | None) – Indicates when to start CDC (checkpoint or LSN/SCN format). Mutually exclusive with cdc_start_time.
cdc_stop_position (str | None) – Indicates when to stop CDC.
wait_for_completion (bool) – If True, wait for the modification to finish before returning. In deferrable mode the operator defers rather than blocking. Defaults to True.
deferrable (bool) – Run the operator in deferrable mode. Defaults to False.
waiter_delay (int) – Seconds between waiter polls (default: 30).
waiter_max_attempts (int) – Maximum waiter poll attempts (default: 60).
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

MODIFIABLE_STATES[source]¶

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers: ClassVar[dict][source]¶

replication_task_arn[source]¶

table_mappings = None[source]¶

migration_type = None[source]¶

replication_task_settings = None[source]¶

cdc_start_time = None[source]¶

cdc_start_position = None[source]¶

cdc_stop_position = None[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None, result=None)[source]¶

class airflow.providers.amazon.aws.operators.dms.DmsDeleteTaskOperator(*, replication_task_arn=None, **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Deletes AWS DMS replication task.

See also

For more information on how to use this operator, take a look at the guide: Delete a replication task

Parameters:

replication_task_arn (str | None) – Replication task ARN
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

replication_task_arn = None[source]¶

execute(context)[source]¶

Delete AWS DMS replication task from Airflow.

Returns:: replication task arn

class airflow.providers.amazon.aws.operators.dms.DmsDescribeTasksOperator(*, describe_tasks_kwargs=None, **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Describes AWS DMS replication tasks.

See also

For more information on how to use this operator, take a look at the guide: Get details of replication tasks

Parameters:

describe_tasks_kwargs (dict | None) – Describe tasks command arguments
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers: ClassVar[dict[str, str]][source]¶

describe_tasks_kwargs[source]¶

execute(context)[source]¶

Describe AWS DMS replication tasks from Airflow.

Returns:: Marker and list of replication tasks
Return type:: tuple[str | None, list]

class airflow.providers.amazon.aws.operators.dms.DmsStartTaskOperator(*, replication_task_arn, start_replication_task_type='start-replication', start_task_kwargs=None, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Starts AWS DMS replication task.

See also

For more information on how to use this operator, take a look at the guide: Start a replication task

Parameters:

replication_task_arn (str) – Replication task ARN
start_replication_task_type (str) – Replication task start type (default=’start-replication’) (‘start-replication’|’resume-processing’|’reload-target’)
start_task_kwargs (dict | None) – Extra start replication task arguments
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

replication_task_arn[source]¶

start_replication_task_type = 'start-replication'[source]¶

start_task_kwargs[source]¶

aws_conn_id = 'aws_default'[source]¶

execute(context)[source]¶

Start AWS DMS replication task from Airflow.

class airflow.providers.amazon.aws.operators.dms.DmsStopTaskOperator(*, replication_task_arn=None, **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Stops AWS DMS replication task.

See also

For more information on how to use this operator, take a look at the guide: Stop a replication task

Parameters:

replication_task_arn (str | None) – Replication task ARN
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

replication_task_arn = None[source]¶

execute(context)[source]¶

Stop AWS DMS replication task from Airflow.

class airflow.providers.amazon.aws.operators.dms.DmsDescribeReplicationConfigsOperator(*, filter=None, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Describes AWS DMS Serverless replication configurations.

See also

For more information on how to use this operator, take a look at the guide: Describe a serverless replication config

Parameters:

describe_config_filter – Filters block for filtering results.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

filter = None[source]¶

execute(context)[source]¶

Describe AWS DMS replication configurations.

Returns:: List of replication configurations
Return type:: list

class airflow.providers.amazon.aws.operators.dms.DmsCreateReplicationConfigOperator(*, replication_config_id, source_endpoint_arn, target_endpoint_arn, compute_config, replication_type, table_mappings, additional_config_kwargs=None, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Creates an AWS DMS Serverless replication configuration.

See also

For more information on how to use this operator, take a look at the guide: Create a serverless replication config

Parameters:

replication_config_id (str) – Unique identifier used to create a ReplicationConfigArn.
source_endpoint_arn (str) – ARN of the source endpoint
target_endpoint_arn (str) – ARN of the target endpoint
compute_config (dict[str, Any]) – Parameters for provisioning an DMS Serverless replication.
replication_type (str) – type of DMS Serverless replication
table_mappings (str) – JSON table mappings
tags – Key-value tag pairs
additional_config_kwargs (dict | None) – Additional configuration parameters for DMS Serverless replication. Passed directly to the API
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

replication_config_id[source]¶

source_endpoint_arn[source]¶

target_endpoint_arn[source]¶

compute_config[source]¶

replication_type[source]¶

table_mappings[source]¶

additional_config_kwargs[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.amazon.aws.operators.dms.DmsDeleteReplicationConfigOperator(*, replication_config_arn, wait_for_completion=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), waiter_delay=60, waiter_max_attempts=60, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Deletes an AWS DMS Serverless replication configuration.

See also

For more information on how to use this operator, take a look at the guide: Delete a serverless replication configuration

Parameters:

replication_config_arn (str) – ARN of the replication config
wait_for_completion (bool) – If True, waits for the replication config to be deleted before returning. If False, the operator will return immediately after the request is made.
deferrable (bool) – Run the operator in deferrable mode.
waiter_delay (int) – The number of seconds to wait between retries (default: 60).
waiter_max_attempts (int) – The maximum number of attempts to be made (default: 60).
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

VALID_STATES = ['failed', 'stopped', 'created'][source]¶

DELETING_STATES = ['deleting'][source]¶

TERMINAL_PROVISION_STATES = ['deprovisioned', ''][source]¶

replication_config_arn[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 60[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

handle_delete_wait()[source]¶

execute_complete(context, event=None)[source]¶

retry_execution(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.dms.DmsDescribeReplicationsOperator(*, filter=None, aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Describes AWS DMS Serverless replications.

See also

For more information on how to use this operator, take a look at the guide: Get the status of a serverless replication

Parameters:

filter (list[dict[str, Any]] | None) – Filters block for filtering results.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderer[source]¶

filter = None[source]¶

execute(context)[source]¶

Describe AWS DMS replications.

Returns:: Replications
Return type:: list[dict[str, Any]]

class airflow.providers.amazon.aws.operators.dms.DmsStartReplicationOperator(*, replication_config_arn, replication_start_type, cdc_start_time=None, cdc_start_pos=None, cdc_stop_pos=None, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Starts an AWS DMS Serverless replication.

See also

For more information on how to use this operator, take a look at the guide: Start a serverless replication

Parameters:

replication_config_arn (str) – ARN of the replication config
replication_start_type (str) – Type of replication.
cdc_start_time (datetime.datetime | str | None) – Start time of CDC
cdc_start_pos (str | None) – Indicates when to start CDC.
cdc_stop_pos (str | None) – Indicates when to stop CDC.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

RUNNING_STATES = ['running'][source]¶

STARTABLE_STATES = ['stopped', 'failed', 'created'][source]¶

TERMINAL_STATES = ['failed', 'stopped', 'created'][source]¶

TERMINAL_PROVISION_STATES = ['deprovisioned', ''][source]¶

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

replication_config_arn[source]¶

replication_start_type[source]¶

cdc_start_time = None[source]¶

cdc_start_pos = None[source]¶

cdc_stop_pos = None[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

wait_for_completion = True[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

retry_execution(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.dms.DmsStopReplicationOperator(*, replication_config_arn, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.dms.DmsHook]

Stops an AWS DMS Serverless replication.

See also

For more information on how to use this operator, take a look at the guide: Stop a serverless replication

Parameters:

replication_config_arn (str) – ARN of the replication config
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be

STOPPED_STATES = ['stopped'][source]¶

NON_STOPPABLE_STATES = ['stopped'][source]¶

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

replication_config_arn[source]¶

wait_for_completion = True[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

deferrable[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶