Google Compute Engine SSH Operators¶

Prerequisite Tasks¶

To use these operators, you must do a few things:

Select or create a Cloud Platform project using the Cloud Console.

Enable billing for your project, as described in the Google Cloud documentation.

Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'
Detailed information is available for Installation.
Setup a Google Cloud Connection.

ComputeEngineRemoteInstanceSSHOperator¶

Use the SSHOperator together with ComputeEngineSSHHook to execute a command on a remote instance.

This operator uses either the Cloud OS Login or instance metadata to manage SSH keys. To use Cloud OS Login, the service account must have compute.osAdminLogin IAM roles and the instance metadata must have Cloud OS Login enabled. This can be done by setting the instance metadata - enable-oslogin=TRUE

To use instance metadata, make sure to set the Cloud OS Login argument to False in the hook.

Please note that the target instance must allow tcp traffic on port 22.

Trusted Partner Cloud (TPC) guidance¶

TPC stands for Trusted Partner Cloud. In these environments, the practical question for Compute Engine SSH is which hook parameters to set.

Use these settings for direct SSH in TPC:

ComputeEngineSSHHook(
    ...,
    use_oslogin=False,
    use_iap_tunnel=False,
)

Use these settings for SSH over IAP in TPC:

ComputeEngineSSHHook(
    ...,
    use_oslogin=False,
    use_iap_tunnel=True,
)

For the IAP case, the caller must also have the IAM permissions required to open an IAP tunnel. In Google Cloud deployments, this is typically the IAP-secured Tunnel User role (roles/iap.tunnelResourceAccessor).

Avoid this setting in the tested TPC environment:

ComputeEngineSSHHook(
    ...,
    use_oslogin=True,
)

In the tested TPC environment, the OS Login SSH flow was not available for this hook. For users, the practical guidance is to use metadata-managed SSH keys and set use_oslogin=False.

When you use metadata-managed SSH keys in TPC:

set use_oslogin=False,
do not enable instance metadata enable-oslogin=TRUE for that SSH path,
set use_iap_tunnel=True only when the required IAP IAM permissions are present.

Below is the code to create the operator:

tests/system/google/cloud/compute/example_compute_ssh.py[source]

metadata_without_iap_tunnel1 = SSHOperator(
    task_id="metadata_without_iap_tunnel1",
    ssh_hook=ComputeEngineSSHHook(
        user="username",
        instance_name=GCE_INSTANCE_NAME,
        zone=LOCATION,
        project_id=PROJECT_ID,
        use_oslogin=False,
        use_iap_tunnel=False,
        cmd_timeout=1,
    ),
    command="echo metadata_without_iap_tunnel1",
)

You can also create the hook without project id - project id will be retrieved from the Google credentials used:

tests/system/google/cloud/compute/example_compute_ssh.py[source]

metadata_without_iap_tunnel2 = SSHOperator(
    task_id="metadata_without_iap_tunnel2",
    ssh_hook=ComputeEngineSSHHook(
        user="username",
        instance_name=GCE_INSTANCE_NAME,
        zone=LOCATION,
        use_oslogin=False,
        use_iap_tunnel=False,
        cmd_timeout=100,
    ),
    command="echo metadata_without_iap_tunnel2",
)

More information¶

See Google Compute Engine API documentation and Cloud OS Login API documentation