kcat (kafkacat) on AmazonLinux2 / CentOS

Photo by Bing Han

When I prepared some demos about Amazon Managed Streaming for Kafka (Amazon MSK) I realised that one of the CLI-tools that I like to use was not available on my EC2 instance: the useful open-source kcat tool. Note that the kafkacat has been renamed to kcat now (this text was written for kafkacat but applies to kcat too).

There seems to be no binary available for Amazon Linux via yum install, but you can quickly build it yourself. Doing so requires UNIX make, as well as a C and C++ compiler (so the whole task mentally throws you back to the time before the “write once, run anywhere” became a thing).

kafkacat Installation on Amazon Linux2

To build kafkacat yourself, just run the following commands on the EC2 instance that you want to use as a client (where you want to install kafkacat):

sudo yum update -y
sudo yum install git -y
# make sure you have the necessary dev tools avail, 
# or run the ff command:

sudo yum -y groupinstall base "Development tools" -- \ setopt=group_package_types=mandatory,default,optional
sudo yum install openssl-devel
sudo yum install libcurl-devel
# Download, build and install cmake *unless cmake exists*
wget https://cmake.org/files/v3.18/cmake-3.18.0.tar.gz
tar -xvzf cmake-3.18.0.tar.gz
cd cmake-3.18.0
./bootstrap
make
sudo make install
git clone https://github.com/edenhill/kafkacat.git
cd kafkacat/
./bootstrap.sh
./kcat -h

You might get away with less then the “Development Tools” group install, let me know if you find out.

Examples

Then try out your newly installed kafkacat tool, e.g. by using one of the following commands:

./kafkacat -h
./kafkacat -L -b YOUR.kafkabroker.com:9092

You can use the AWS MSK console to retrieve the bootstrap servers easily:

For an MSK cluster set up with plain text, you can query for the brokers as shown below:

./kafkacat -L -b b-1.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092,b-3.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092,b-6.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092

In this example I have 6 broker running across 3 availability zones:

More examples

There are many other examples that you might find useful when running Apache Kafka especially on AWS. I will continue this posting with some more examples… stay tuned!

This article about calling MSK via REST might be interesting.

Please clap for this article if you enjoyed reading it as much as I enjoyed writing it. I spend way too much time on Twitter, so feel free to connect: @frankmunz.