简介

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

部署

CentOS部署

准备基础环境

1
2
3
# 安装源文件
sudo yum install epel-release -y
sudo yum update -y

安装Java环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 安装JDK
sudo yum install -y java-1.8.0-openjdk

# 查看Java版本
java -version

# 查看Java环境变量
echo $JAVA_HOME

# 配置Java环境变量
echo "JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")" | sudo tee -a /etc/profile

# 配置生效
source /etc/profile

下载Kafka

1
2
3
4
5
6
7
8
# 下载安装包：https://kafka.apache.org/downloads
sudo wget https://downloads.apache.org/kafka/3.5.1/kafka_2.12-3.5.1.tgz

# 解压
sudo tar -zvxf kafka_2.12-3.5.1.tgz -C  /opt/

# 创建软链接
sudo ln -s /opt/kafka_2.12-3.5.1 /opt/kafka

启动Zookeeper

手动形式启动：

1
sudo ./zookeeper-server-start.sh ../config/zookeeper.properties

服务形式启动：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# 创建服务
cat > /etc/systemd/system/zookeeper.service << EOF

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
User=root
Group=root

[Install]
WantedBy=multi-user.target
EOF

# 启动服务
sudo systemctl start zookeeper
sudo systemctl status zookeeper
sudo systemctl start zookeeper

启动Kafka

手动形式启动：

1
sudo ./kafka-server-start.sh ../config/server.properties

服务形式启动：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 创建服务
cat > /etc/systemd/system/kafka.service << EOF

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal


[Install]
WantedBy=multi-user.target
EOF

# 启动服务
sudo systemctl start kafka
sudo systemctl status kafka
sudo systemctl stop kafka

使用

命令行使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 创建Topic
/opt/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

# 查看Topic列表
/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

# 查看Topic详情
/opt/kafka/bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092

# 接受Topic消息
/opt/kafka/bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

# 发送Topic消息
/opt/kafka/bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092

参考文档