Installing Kafka and Hadoop

Setting up the two pillars of big data infrastructure

Posted by Syed Zain Raza on May 19th, 2026

Kafka and Hadoop serve different but complementary roles in a big data stack. Hadoop is a batch processing framework for storing and analyzing large datasets at rest. Kafka is a distributed streaming platform for data in motion — ingesting, routing, and processing high-throughput event streams in real time. This guide walks through installing both on Ubuntu.

Prerequisites

Both Kafka and Hadoop require Java. Install OpenJDK first:

sudo apt update
sudo apt install openjdk-11-jdk -y
java -version

Hadoop also requires SSH for its node communication. Set up passwordless SSH for localhost:

sudo apt install ssh -y
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost

Installing Hadoop

Download and extract Hadoop. Check the Apache Hadoop site for the latest stable version:

wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xzf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 /usr/local/hadoop

Set the environment variables. Add these to your ~/.bashrc:

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Reload the shell:

source ~/.bashrc

Set the Java path in Hadoop's environment file:

nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
# Add: export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

Format the HDFS namenode and start the cluster:

hdfs namenode -format
start-dfs.sh
start-yarn.sh

Verify all services are running:

jps
# Should show: NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager

The Hadoop web UI is available at http://localhost:9870.

Installing Kafka

Download and extract Kafka:

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 /usr/local/kafka

Add Kafka to your PATH in ~/.bashrc:

export KAFKA_HOME=/usr/local/kafka
export PATH=$PATH:$KAFKA_HOME/bin

Kafka depends on ZooKeeper for cluster coordination. Start ZooKeeper first, then Kafka:

zookeeper-server-start.sh -daemon $KAFKA_HOME/config/zookeeper.properties
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

Testing Kafka

Create a topic, produce a message, and consume it to verify the installation:

# Create a topic
kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# Start a producer (type messages, press Enter after each)
kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

# In a second terminal, start a consumer
kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

Messages you type in the producer terminal should appear in the consumer terminal.

How they work together

A common architecture is to use Kafka as the ingestion layer — applications stream events into Kafka topics — and Hadoop as the storage and batch processing layer. A Kafka consumer reads from the topics and writes the data to HDFS, where MapReduce or Spark jobs can process it. This gives you both real-time streaming capability and the ability to run large-scale historical analysis on the same data.