Configuration Reference¶

This page contains the list of all available Airflow configurations for the apache-airflow-providers-cncf-kubernetes provider that can be set in the airflow.cfg file or using environment variables.

Note

For more information see Setting Configuration Options.

[kubernetes_executor]¶

api_client_retry_configuration¶

Kwargs to override the default urllib3 Retry used in the kubernetes API client

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION
Example:: { "total": 3, "backoff_factor": 0.5 }

async_pod_creation¶

Added in version 10.20.0.

Create worker pods concurrently within each scheduler loop instead of sequentially. When enabled, the worker_pods_creation_batch_size pods dequeued per loop are submitted to the Kubernetes API concurrently (bounded by pod_creation_max_concurrency) using the asynchronous Kubernetes client. This reduces the time the scheduler loop spends blocked on pod creation when per-call latency is high (network round-trip, admission webhooks). Pod templates are still built synchronously; only the create API calls are parallelized.

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__ASYNC_POD_CREATION

cluster_context¶

When running with in_cluster=False change the default cluster_context or config_file options to Kubernetes client. Leave blank these to use default behaviour like kubectl has.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT

config_file¶

Path to the kubernetes configfile to be used when in_cluster is set to False

Type:: string
Default:: None
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE

delete_option_kwargs¶

Optional keyword arguments to pass to the delete_namespaced_pod kubernetes client core_v1_api method when using the Kubernetes Executor. This should be an object and can contain any of the options listed in the v1DeleteOptions class defined here: https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_OPTION_KWARGS
Example:: {"grace_period_seconds": 10}

delete_worker_pods¶

If True, all worker pods will be deleted upon termination

Type:: string
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS

delete_worker_pods_on_failure¶

If False (and delete_worker_pods is True), failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed.

Type:: string
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE

enable_tcp_keepalive¶

Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely when idle connection is time-outed on services like cloud load balancers or firewalls.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__ENABLE_TCP_KEEPALIVE

in_cluster¶

Use the service account kubernetes gives to pods to connect to kubernetes cluster. It’s intended for clients that expect to be running inside a pod running on kubernetes. It will raise an exception if called from a process not running in a kubernetes environment.

Type:: string
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER

kube_client_request_args¶

Keyword parameters to pass while calling a kubernetes client core_v1_api methods from Kubernetes Executor provided as a single line formatted JSON dictionary string. List of supported params are similar for all core_v1_apis, hence a single config variable for all apis. See: https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__KUBE_CLIENT_REQUEST_ARGS

logs_task_metadata¶

Flag to control the information added to kubernetes executor logs for better traceability

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__LOGS_TASK_METADATA

multi_namespace_mode¶

Allows users to launch pods in multiple namespaces. Will require creating a cluster-role for the scheduler, or use multi_namespace_mode_namespace_list configuration.

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE

multi_namespace_mode_namespace_list¶

If multi_namespace_mode is True while scheduler does not have a cluster-role, give the list of namespaces where the scheduler will schedule jobs Scheduler needs to have the necessary permissions in these namespaces.

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE_NAMESPACE_LIST

namespace¶

The Kubernetes namespace where airflow workers should be created. Defaults to default

Type:: string
Default:: default
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE

pod_creation_max_concurrency¶

Added in version 10.20.0.

Maximum number of concurrent pod-creation API calls when async_pod_creation is enabled. Bounds the burst of simultaneous requests to the Kubernetes API server (and admission webhooks) to avoid tripping API priority-and-fairness or rate limits. Has no effect when async_pod_creation is False. When set to 0 the limit falls back to worker_pods_creation_batch_size.

Type:: integer
Default:: 0
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__POD_CREATION_MAX_CONCURRENCY
Example:: 16

pod_launch_failure_excluded_container_reasons¶

Added in version 10.20.0.

Comma-separated list of container termination reasons that are excluded from the pod_launch_failure_retries requeue path even when the task instance is still in queued state. Pods that fail with an excluded reason consume a normal task retry instead of being transparently requeued. The default Error covers the case where the container started executing but the worker process exited before writing running to the database, which is most likely an Airflow-specific startup error rather than a transient infrastructure event. Set to an empty value to requeue these cases as well.

Type:: string
Default:: Error
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__POD_LAUNCH_FAILURE_EXCLUDED_CONTAINER_REASONS

pod_launch_failure_retries¶

Added in version 10.20.0.

The number of times the executor will transparently requeue a task whose worker pod failed before the task process started running (for example a node drain, autoscaler scale-down, node boot race, or transient image pull failure). The task instance is still in queued state in these cases, meaning no task code ran, so requeuing does not consume a task-level retry.

This changes the previous default behavior: such tasks are now requeued once before failing instead of failing on the first pod failure. Set this to 0 to restore the previous behavior (fail immediately, no requeue).

Use -1 for unlimited requeues, but with caution: a pod that fails on every launch (for example a misconfigured image that can never be pulled) will be requeued forever, and with the default delete_worker_pods_on_failure = False the failed pods are not cleaned up, so they accumulate.

Type:: integer
Default:: 1
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__POD_LAUNCH_FAILURE_RETRIES

pod_template_file¶

Path to the YAML pod file that forms the basis for KubernetesExecutor workers.

running_pod_log_lines¶

Added in version 10.20.0.

Number of lines read from the end of a running task’s pod log when the task log is served through the kube API, e.g. when viewing logs of a running task in the UI. The value must be greater than 0.

Type:: integer
Default:: 100
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__RUNNING_POD_LOG_LINES

ssl_ca_cert¶

Path to a CA certificate to be used by the Kubernetes client to verify the server’s SSL certificate.

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__SSL_CA_CERT

task_publish_max_retries¶

The Maximum number of retries for queuing the task to the kubernetes scheduler when failing due to Kube API exceeded quota errors before giving up and marking task as failed. -1 for unlimited times.

Type:: integer
Default:: 0
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES

tcp_keep_cnt¶

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe tcp_keep_cnt number of times before a connection is considered to be broken.

Type:: integer
Default:: 6
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_CNT

tcp_keep_idle¶

When the enable_tcp_keepalive option is enabled, TCP probes a connection that has been idle for tcp_keep_idle seconds.

Type:: integer
Default:: 120
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_IDLE

tcp_keep_intvl¶

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe after tcp_keep_intvl seconds.

Type:: integer
Default:: 30
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_INTVL

verify_ssl¶

Set this to false to skip verifying SSL certificate of Kubernetes python client.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__VERIFY_SSL

worker_container_repository¶

The repository of the Kubernetes Image for the Worker to Run

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY

worker_container_tag¶

The tag of the Kubernetes Image for the Worker to Run

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG

worker_pod_pending_fatal_container_state_reasons¶

Added in version 8.1.0.

If the worker pods are in a pending state due to a fatal container state reasons, then fail the task and delete the worker pod if delete_worker_pods is True and delete_worker_pods_on_failure is True.

Type:: string
Default:: CreateContainerConfigError,ErrImagePull,CreateContainerError,ImageInspectError,InvalidImageName
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS

worker_pods_creation_batch_size¶

Number of Kubernetes Worker Pod creation calls per scheduler loop. Note that the current default of “1” will only launch a single pod per-heartbeat. It is HIGHLY recommended that users increase this number to match the tolerance of their kubernetes cluster for better performance.

Type:: string
Default:: 1
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE

[local_kubernetes_executor]¶

This section only applies if you are using the LocalKubernetesExecutor in [core] section above

kubernetes_queue¶

Define when to send a task to KubernetesExecutor when using LocalKubernetesExecutor. When the queue of a task is the value of kubernetes_queue (default kubernetes), the task is executed via KubernetesExecutor, otherwise via LocalExecutor

Type:: string
Default:: kubernetes
Environment Variable:: AIRFLOW__LOCAL_KUBERNETES_EXECUTOR__KUBERNETES_QUEUE