04 develop中间件mqkafka

kafka环境搭建

启动单节点kafka

kafka通过zookeeper管理集群配置，选举leader，以及在Consumer Group发生变化时进行rebalance

1
2
3
4
5
# 启动zookeeper（也可以使用独立的zookeeper）
zookeeper-server-start.sh config/zookeeper.properties

# 启动 kafka
kafka-server-start.sh config/server.properties

启动kafka集群（同台服务器）

集群中broker.id不能有重复

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#复制配置文件
cp config/server.properties config/server-1.properties

config/server-1.properties:
    broker.id=1
listeners=PLAINTEXT://192.168.85.143:9092
advertised.listeners=PLAINTEXT://192.168.85.143:9092
default.replication.factor=2

 
config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9092
    log.dirs=/tmp/kafka-logs-2

config/server-3.properties:
    broker.id=3
    listeners=PLAINTEXT://:9093
    log.dirs=/tmp/kafka-logs-23

启动多个结点

kafka-server-start.sh config/server-1.properties
kafka-server-start.sh config/server-2.properties
kafka-server-start.sh config/server-3.properties

模拟过程

创建主题

1
2
3
4
5
# 创建主题
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

# 查询所有主题
kafka-topics.sh --list --bootstrap-server localhost:9092

启动生产者

1
2
# 启动后，单行输出做为一个消息发出
kafka-console-producer.sh --broker-list localhost:9092 --topic test

启动消费者

1
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

演示leader节点炸，其他节点接手

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 创建多复制节点的主题
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 1 --topic my-replicated-topic

# 查看当前主题的leader节点
kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-replicated-topic

#返回结果：
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: my-replicated-topic	Partition: 0	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

# 干掉leader broker
ps aux | grep server-1.properties
kill -9 pid

# 再次查看主题信息
kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-replicated-topic

#返回结果：
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: my-replicated-topic	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1,2

[2020-03-03 21:19:35,058] WARN [Consumer clientId=consumer-console-consumer-84017-1, groupId=console-consumer-84017] Error while fetching metadata with correlation id 90 : {dahui=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解决方案 添加配置

1
2
listeners=PLAINTEXT://192.168.85.143:9092
advertised.listeners=PLAINTEXT://192.168.85.143:9092