未分類
24 4 月 2020

Golang 通过 sarama 连接到 kafka 的疑惑

Golang 通过 sarama 连接到 kafka 的疑惑

資深大佬 : hijoker 50

// 问题:

Golang 通过 sarama 连接到 kafka,没有指定 group,连接的 topic 只有一个 partition,通过 2 台机器上一样的程序连上 kafka 上, 这 2 个程序都能消费这一个 partition 的数据.

那么这个 2 台机器上的应用程序是属于同一个 group 的,而且是同一个 consumer? 要知道一个 partition 只能被每个 group 的一个 consumer 消费的!!!

// kafka 版本:

[email protected]:/usr/local/kafka_2.11-2.0.0/bin# ./kafka-topics.sh –version 2.0.0 (Commit:3402a8361b734732)

// topic 只有一个 partition

[email protected]:/usr/local/kafka_2.11-2.0.0/bin# kafka-topics.sh –describe –zookeeper 127.0.0.1:2181 –topic topic.XXX Topic:topic.skygram PartitionCount:1 ReplicationFactor:1 Configs: Topic: topic.XXX Partition: 0 Leader: 0 Replicas: 0 Isr: 0

// 只有一个 group,看不到其他的 group 和 consumer 的消息

[email protected]:/usr/local/kafka_2.11-2.0.0/bin# ./kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list

KMOffsetCache-ubuntu18

// 确实是两台机器连上的

[email protected]:/usr/local/kafka_2.11-2.0.0/bin# netstat -antop|grep 9092|grep ESTABLISHED tcp 0 0 192.168.198.145:50396 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (1.00/0/0) tcp 0 0 192.168.198.145:50394 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (4.07/0/0) tcp 0 0 192.168.198.145:50392 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (4.07/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50394 ESTABLISHED 4367/java keepalive (3630.28/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54181 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43006 ESTABLISHED 4367/java keepalive (4265.90/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54183 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:43012 192.168.198.145:9092 ESTABLISHED 4699/java keepalive (4266.62/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54182 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50396 ESTABLISHED 4367/java keepalive (3742.75/0/0) tcp6 0 0 192.168.198.145:43006 192.168.198.145:9092 ESTABLISHED 4699/java keepalive (4265.90/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50392 ESTABLISHED 4367/java keepalive (3630.28/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43012 ESTABLISHED 4367/java keepalive (4266.62/0/0

大佬有話說 (6)

  • 資深大佬 : znood

    没有指定 group，那 group 可能是随机的，去 zookeeper 里可以看到有哪些 comsumer group

  • 主 資深大佬 : hijoker

    @znood zk 里面是空的,很是令我困惑
    [zk: localhost:2181(CONNECTED) 2] ls /
    [cluster, controller_epoch, controller, brokers, zookeeper, kafka-manager, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]
    [zk: localhost:2181(CONNECTED) 3] ls /consumers
    []

  • 主 資深大佬 : hijoker

    我也偏向认为这是 2 个 group,但是怎么从 kafka 的后台来证明这是 2 个 group 呢
    还有,好像新版本的 kafka 的 consumer 的信息已经不在 zk 里了
    是用这个命令来看

    ./kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list

  • 資深大佬 : znood

    那 group 有可能是空的，KMOffsetCache-ubuntu18 可能是命令行测试生成的

  • 主 資深大佬 : hijoker

    @znood https://www.google.com/search?biw=1920&bih=937&sxsrf=ACYBGNRoNLV3NleeP-K5T0mwRDkF8D-A6g%3A1579151812697&ei=xPEfXvyKKreu0PEPsda2wAQ&q=KMOffsetCache&oq=KMOffsetCache&gs_l=psy-ab.12..35i39i19j35i39l2j0i10.4516664.4516664..4517922…3.0..2.1015.3690.4-1j1j2j1……0….2j1..gws-wiz…….33i160.1BsHGQK4MGc&ved=0ahUKEwj8_N32rofnAhU3FzQIHTGrDUgQ4dUDCAs
    不是命令行,这是由于这种消费者的信息是存放在 __consumer_offsets 中

  • 資深大佬 : znood

    不管 KMOffsetCache-ubuntu18 怎么来的，我的意思是你的两个程序可能都使用的空 groupID，我搜了下，groupID 只是一个 label 标识，默认为空，也就是可以为空
    Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

    https://stackoverflow.com/questions/43118671/kafka-consumer-default-group-id

    https://cwiki.apache.org/confluence/display/KAFKA/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer