rocketkind
Infrastructure

Kafka REST

Kafka REST connects a workflow to an Apache Kafka cluster through Confluent's REST API v3 — the API Confluent Cloud serves for every cluster, and Confluent Platform serves through Confluent Server or the Kafka REST Proxy.

Authentication

  • API key and secret

Actions

Generic API request. Call any Kafka REST API v3 endpoint on this connection - broker and topic configs, partitions or reassignments - for the case where no shipped action covers it yet. The connection supplies the base URL, the authentication and the egress route, so only the path, query and body are yours to fill.

  • Create ACLDestructiveGrant a principal an operation on a resource - how an approved access request for a topic or consumer group is fulfilled in Kafka.
  • Create topicCreate a topic in the connection's cluster, with its partition count, replication factor and topic configs such as retention.ms or cleanup.policy.
  • Delete ACLsDestructiveRemove every ACL matching a filter - how a revoked grant or a leaver's access is taken back.
  • Delete topicDestructiveDelete a topic and every record in it.
  • Get consumer group lagRead how far a consumer group is behind: its total lag across partitions and the partition with the largest lag - to check that downstream consumers have caught up with the events a workflow published.
  • Get topicRead one topic's partition count, replication factor and internal flag.
  • List ACLsList the cluster's ACLs - who may read, write or administer which topics and groups - for an access review, or to find every grant a leaver's principal still holds.
  • List clustersList the Kafka clusters this REST endpoint serves, with each one's clusterid - on Confluent Platform, how to confirm the cluster id a connection uses.
  • List consumer groupsList the consumer groups in the connection's cluster, each with its state and partition assignor - to find which applications consume a topic.
  • List topicsList the topics in the connection's cluster, each with its partition count, replication factor and whether it is internal - to check that the topics a pipeline writes to exist.
  • Produce recordWrite one record to a topic - publish an identity event, such as a joiner or a revoked grant, for downstream systems to consume.
  • Produce recordsDestructiveWrite a batch of records to one topic in a single request.

Product names, logos and brands are the property of their respective owners and are used here for identification only. Their use does not imply endorsement by, or affiliation with, those owners.