Installing Kafka and Hadoop

Setting up the two pillars of big data infrastructure

Posted by Syed Zain Raza

Kafka and Hadoop serve different but complementary roles in a big data stack. Hadoop is a batch processing framework for storing and analyzing large datasets at rest. Kafka is a distributed streaming platform for data in motion — ingesting, routing, and processing high-throughput event streams in real time. This guide walks through installing both on Ubuntu.

Prerequisites

Both Kafka and Hadoop require Java. Install OpenJDK first:

sudo apt update
sudo apt install openjdk-11-jdk -y
java -version

Hadoop also requires SSH for its node communication. Set up passwordless SSH for localhost:

sudo apt install ssh -y
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost

Installing Hadoop

Download and extract Hadoop. Check the Apache Hadoop site for the latest stable version:

wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xzf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 /usr/local/hadoop

Set the environment variables. Add these to your ~/.bashrc:

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Reload the shell:

source ~/.bashrc

Set the Java path in Hadoop's environment file:

nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
# Add: export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

Format the HDFS namenode and start the cluster:

hdfs namenode -format
start-dfs.sh
start-yarn.sh

Verify all services are running:

jps
# Should show: NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager

The Hadoop web UI is available at http://localhost:9870.

Installing Kafka

Download and extract Kafka:

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 /usr/local/kafka

Add Kafka to your PATH in ~/.bashrc:

export KAFKA_HOME=/usr/local/kafka
export PATH=$PATH:$KAFKA_HOME/bin

Kafka depends on ZooKeeper for cluster coordination. Start ZooKeeper first, then Kafka:

zookeeper-server-start.sh -daemon $KAFKA_HOME/config/zookeeper.properties
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

Testing Kafka

Create a topic, produce a message, and consume it to verify the installation:

# Create a topic
kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# Start a producer (type messages, press Enter after each)
kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

# In a second terminal, start a consumer
kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

Messages you type in the producer terminal should appear in the consumer terminal.

How they work together

A common architecture is to use Kafka as the ingestion layer — applications stream events into Kafka topics — and Hadoop as the storage and batch processing layer. A Kafka consumer reads from the topics and writes the data to HDFS, where MapReduce or Spark jobs can process it. This gives you both real-time streaming capability and the ability to run large-scale historical analysis on the same data.