Newsletter – Week 29, 2017

News:

Articles:

Videos:

 

Useful Apache Kafka Console Commands

Parameters used:

Parameter Comment
zookeeper_host the list of Zookeeper nodes in cluster
topic_name the topic name
broker_list the list of Kafka brokers in cluster
replication_factor the replication factor for topic
partition_count the number of partitions for topic

Topic management:

  • create a topic:

bin/kafka-topics.sh –create –zookeeper [zookeeper_list] –replication-factor [replication_factor] –partitions [partition_count] –topic [topic_name]

Example:

>bin/kafka-topics.sh –create –zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 –replication-factor 1  –partitions 3 –topic TestKafkaTopicName

  • view list of topics

bin/kafka-topics.sh –list –zookeeper [zookeeper_list]

Example:

>bin/kafka-topics.sh –list –zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181

  • describe a topic

bin/kafka-topics.sh –describe –zookeeper [zookeeper_list] –topic [topic_name]

Example:

>bin/kafka-topics.sh –describe –zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 –topic TestKafkaTopicName

Messaging:

  • produce a message to Kafka

bin/kafka-console-producer.sh –broker-list [broker_list] –topic [topic_name]

Example:

>bin/kafka-console-producer.sh  –broker-list kafka_nd1:6667,kafka_nd2:6667,kafka_nd3:6667 –topic TestKafkaTopicName

Note: you have to be extremely accurate when sending a message to Kafka from console because the message might be in incorrect format that would break the consumer logic.

  • consume all messages from Kafka topic (old ones and new arriving)

bin/kafka-console-consumer.sh –zookeeper [zookeeper_list] –topic [topic_name] –from-beginning

Example:

>bin/kafka-console-consumer.sh –zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 –topic TestKafkaTopicName

  • consume  the messages arriving in Kafka topic

bin/kafka-console-consumer.sh –zookeeper [zookeeper_list] –topic [topic_name] –from-beginning

Example:

>bin/kafka-console-consumer.sh –zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 –topic TestKafkaTopicName