Kafka Cluster Setup

from my own experience I find that while setting up kafka cluster on AWS we face some issues so just want to highlight them.

a. First setup zookeeper cluster. Let’s say 3 node cluster. Modify each node zoo.conf to publish ip address as internal IP address.

server.id=<internal aws ip1>:2888:3888

server.id=<internal aws ip2>:2888:3888

server.id=<internal aws ip3>:2888:3888

b. Go to kafka server.properties and change broker.id

node1 — broker.id 0

node2 — broker.id 1

node 3 — broker.id2

Change bind address to <internal ip> so that it is not accessible from outside

Change advertised.host.name to <internal ip>

List all zookeeper nodes within your own cluster under setting zookeeper.connect

Start all kafka nodes and you they should be able to create a cluster

Create a topic, publish a message using kafka-console-producer and see if there are not errors.

Published by Marut Singh

Welcome to my blog. I am software architect, mentor, corporate trainer. Developing software for last 15 years in various domains..I work in different stacks and software architecture is my area of speciality..worked in c++, java, c#, scala, play vert.x, spring, nosql blah blah blah. And offcourse cloud technologies. Software Engineer cant imagine life without cloud :-) Always exploring new languages and tools to make sure that I do not loose touch, to ensure that I delivery high quality software in reasonable cost at all times. Love TDD, BDD, Agile and more than anything simplicity.. Normally I am very helpful so if you need some help do get in touch.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: