Apache Kafka

apache-airflow-providers-apache-kafka

Apache Kafka

Works with Airflow 2.11+
7.2K
Monthly Downloads
13
Total Modules
View Docs Source
Install:
pip install apache-airflow-providers-apache-kafka==2.1.0

Airflow

2.11+

Python

>=3.11

Dependencies (6)

Show all Hide apache-airflow>=2.11.0 apache-airflow-providers-common-compat>=1.12.0 asgiref>=2.3.0; python_version < '3.14' asgiref>=3.11.1; python_version >= '3.14' confluent-kafka>=2.6.0; python_version < '3.14' confluent-kafka>=2.13.2; python_version >= '3.14'

Connections (1)

URI Schemes (1)

Schemes this release registers in provider.yaml, grouped by the Airflow feature that routes them to the provider.

Asset URIs
kafka://

Modules

O

ConsumeFromTopicOperator

An operator that consumes from Kafka a topic(s) and processing the messages.

airflow.providers.apache.kafka.operators.consume.ConsumeFromTopicOperator
View Docs Source
O

ProduceToTopicOperator

An operator that produces messages to a Kafka topic.

airflow.providers.apache.kafka.operators.produce.ProduceToTopicOperator
View Docs Source
H

KafkaBaseHook

A base hook for interacting with Apache Kafka.

airflow.providers.apache.kafka.hooks.base.KafkaBaseHook
View Docs Source
H

KafkaAdminClientHook

A hook for interacting with the Kafka Cluster.

airflow.providers.apache.kafka.hooks.client.KafkaAdminClientHook
View Docs Source
H

KafkaConsumerHook

A hook for creating a Kafka Consumer.

airflow.providers.apache.kafka.hooks.consume.KafkaConsumerHook
View Docs Source
H

KafkaProducerHook

A hook for creating a Kafka Producer.

airflow.providers.apache.kafka.hooks.produce.KafkaProducerHook
View Docs Source
S

AwaitMessageSensor Deferrable

An Airflow sensor that defers until a specific message is published to Kafka.

airflow.providers.apache.kafka.sensors.kafka.AwaitMessageSensor
View Docs Source
S

AwaitMessageTriggerFunctionSensor Deferrable

Defer until a specific message is published to Kafka, trigger a registered function, then resume waiting.

airflow.providers.apache.kafka.sensors.kafka.AwaitMessageTriggerFunctionSensor
View Docs Source
T

AwaitMessageTrigger

A trigger that waits for a message matching specific criteria to arrive in Kafka.

airflow.providers.apache.kafka.triggers.await_message.AwaitMessageTrigger
View Docs Source
T

KafkaMessageQueueTrigger

A dedicated trigger for Kafka message queues that extends ``MessageQueueTrigger``.

airflow.providers.apache.kafka.triggers.msg_queue.KafkaMessageQueueTrigger
View Docs Source
T

KafkaSharedStreamTrigger

Event-driven trigger that watches Kafka topics through a shared consumer.

airflow.providers.apache.kafka.triggers.shared_stream.KafkaSharedStreamTrigger
View Docs Source
Q

KafkaMessageQueueProvider

Configuration for Apache Kafka integration with common-messaging.

airflow.providers.apache.kafka.queues.kafka.KafkaMessageQueueProvider
View Docs Source
P

KafkaEventProducerPlugin

Publishes Airflow DagRun and TaskInstance event messages to a defined Kafka topic.

airflow.providers.apache.kafka.plugins.event_producer.KafkaEventProducerPlugin
View Docs Source