How to Check Whether Kafka Server Is Running

How to check Kafka server status or details?

If you are looking for the Kafka cluster broker status, you can use zookeeper cli to find the details for each broker as given below:

ls /brokers/ids returns the list of active brokers IDs on the cluster.

get /brokers/ids/<id> returns the details of the broker with the given ID.

Example :

kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[0]

kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/0
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:9092"],"jmx_port":-1,"host":"localhost","timestamp":"1558428038778","port":9092,"version":4}
cZxid = 0x116
ctime = Tue May 21 08:40:38 UTC 2019
mZxid = 0x116
mtime = Tue May 21 08:40:38 UTC 2019
pZxid = 0x116
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16ad9830f16000b
dataLength = 188
numChildren = 0

You can put these steps in some shell script to get the details for all brokers.

How to programmatically check if Kafka Broker is up and running in Python

I am afraid there is no direct approach for testing whether Kafka Brokers are up and running. Also note that if your consumer has already consumed the messages it doesn't mean that this is a bad behaviour and obviously it does not indicate that the Kafka broker is down.

---

A possible workaround would be to perform some sort of quick operation and see if the broker responds. An example would be listing the topics:

Using confluent-kafka-python and AdminClient

# Example using confuent_kafka
from confluent_kafka.admin import AdminClient

kafka_broker = {'bootstrap.servers': 'localhost:9092'}
admin_client = AdminClient(kafka_broker)
topics = admin_client.list_topics().topics

if not topics: 
    raise RuntimeError()

Using kafka-python and KafkaConsumer

# example using kafka-python
import kafka


consumer = kafka.KafkaConsumer(group_id='test', bootstrap_servers=['localhost:9092'])
topics = consumer.topics()

if not topics: 
    raise RuntimeError()

How to ensure a kafka cluster is fully up?

Once all brokers have been started we can use following cmds, to check whether they have formed a cluster or not.

• From kafka-1 run the following command against the rest of the brokers, i.e. i = 2, 3, 4 and 5:

  • nc -vz kafka-i 9092 [It should return connection succeeded]

• tail the server.log in each broker node. It should give the info about the cluster.

• From Kafka bin directory, You can periodically run ./zookeeper-shell.sh zk_host:zk_port and execute ls /brokers/ids. It should gives you five entries, e.g. [0, 1, 2, 3, 4] if all 5 brokers have registered to the zookeeper.

One dirty (less involved) hack might be to create a test topic with 5 partitions, and wait until each broker gets 1 partition to itself.

---

Related Topics