Newsletter – Week 24, 2018

News:

Articles:

Videos:

Newsletter – Week 15, 2018

News:

Articles:

Videos:

Newsletter – Week 12, 2018

News:

Articles:

Videos:

Newsletter – Week 29, 2017

News:

Articles:

Videos:

 

Useful Apache Kafka Console Commands

Parameters used:

Parameter Comment
zookeeper_host the list of Zookeeper nodes in cluster
topic_name the topic name
broker_list the list of Kafka brokers in cluster
replication_factor the replication factor for topic
partition_count the number of partitions for topic

Topic management:

  • create a topic:
bin/kafka-topics.sh --create --zookeeper [zookeeper_list] --replication-factor [replication_factor] --partitions [partition_count] --topic [topic_name]

Example:

>bin/kafka-topics.sh --create --zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 --replication-factor 1  --partitions 3 --topic TestKafkaTopicName
  • view list of topics
bin/kafka-topics.sh --list --zookeeper [zookeeper_list]

Example:

>bin/kafka-topics.sh --list --zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181
  • describe a topic
bin/kafka-topics.sh --describe --zookeeper [zookeeper_list] --topic [topic_name]

Example:

>bin/kafka-topics.sh --describe --zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 --topic TestKafkaTopicName

Messaging:

  • produce a message to Kafka
bin/kafka-console-producer.sh --broker-list [broker_list] --topic [topic_name]

Example:

>bin/kafka-console-producer.sh  --broker-list kafka_nd1:6667,kafka_nd2:6667,kafka_nd3:6667 --topic TestKafkaTopicName

Note: you have to be extremely accurate when sending a message to Kafka from console because the message might be in incorrect format that would break the consumer logic.
  • consume all messages from Kafka topic (old ones and new arriving)
bin/kafka-console-consumer.sh --zookeeper [zookeeper_list] --topic [topic_name] --from-beginning

Example:

>bin/kafka-console-consumer.sh --zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 --topic TestKafkaTopicName
  • consume  the messages arriving in Kafka topic
bin/kafka-console-consumer.sh --zookeeper [zookeeper_list] --topic [topic_name] --from-beginning

Example:

>bin/kafka-console-consumer.sh --zookeeper zookeeeper_nd1:2181,zookeeper_nd2:2181,zookeeper_nd3:2181 --topic TestKafkaTopicName